Ethics-based Automated Negotiation in the Decision-making of Autonomous Systems

The presence of autonomous decision-making systems is growing rapidly and affecting many aspects of our daily lives. Designed to learn and act independently, they are capable of autonomous decision-making without human assistance. Although these systems promise many advantages, their increased degree of freedom raises concerns about their ethical behavior during decision-making. The demand for ethically aware AI is also driven by laws and regulations, such as the AI Act, which emphasizes responsible and transparent AI development, reinforcing the need to align autonomous systems with ethical values. As of today, however, human values and ethics are mostly not considered by autonomous systems that make decisions for them. This presents the challenge of incorporating user ethical preferences into the decision-making of autonomous systems, ensuring that they take into account the user's morals and beliefs, which may vary across contexts and are unique to each individual in society. Indeed, considering ethics in the decision-making process raises another challenge of how systems may interact should their ethical preferences differ. The absence of universal ethics implies that they need to reach an ethical agreement to interact with each other. Negotiation is a possible way to reach such an agreement and automated negotiation is the process through which multiple autonomous systems communicate by automatically exchanging bids, dialogues, and offers for that purpose. However, when considering ethics in negotiation, the challenge lies in quantifying user ethical preferences and formalizing negotiation rules, as systems require measurable parameters to assess the ethical implications of their decisions and ensure alignment with the ethical values of the users they represent. This raises another challenge of systematically capturing user ethical preferences and generating ethical profiles that encapsulate users' preferences for effective integration into decision-making. To address these challenges, we propose an ethics-based automated negotiation approach in which autonomous systems utilize users' ethical profiles together with contextual factors to control their autonomy while collaboratively negotiating with each other to reach an ethical agreement that satisfies the ethical beliefs of all parties involved. It is not realistic to expect that the agreement is reached once and for all; rather, it is situational in that it relates to or depends on (i) environmental factors (e.g., the location and surroundings of a place, social circumstances, weather conditions) and (ii) user status, which is a set of physical, social, or emotional conditions that (possibly) apply to the user in a given context at a particular time (e.g., specific health conditions or state of affairs such as elderly, injured, and crowd anxiety). Accordingly, the contributions of this thesis are as follows: i) providing the research community with a comprehensive overview of the current literature on automated negotiation through a systematic mapping study; (ii) proposing a novel approach that provides a domain-agnostic template solution for architecting autonomous systems that leverages the ethical profiles of end users and their contextual status to regulate their autonomous behavior and support decision-making through ethics-based negotiation; (iii) implementing the proposed approach to validate it and demonstrate the feasibility and effectiveness of the system in realizing ethics-based negotiation in real-life scenarios; and (iv) proposing an approach to develop a tool that facilitates the automated generation of ethical profile-building questionnaires to further generate user ethical profiles through their responses.