Conflict resolution via emerging technologies?

Journal of Physics: Conference Series PAPER • OPEN ACCESS Conflict resolution via emerging technologies? To cite this article: Chika Yinka-Banjo et al 2019 J. Phys.: Conf. Ser. 1235 012022 View the article online for updates and enhancements. This content was downloaded from IP address 207.90.39.16 on 24/07/2019 at 13:54 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 Conflict resolution via emerging technologies? Chika Yinka-Banjo1 , Ogban-Asuquo Ugot1 Sanjay Misra2,3, Adewole Adewumi2, Robertas Damasevicius4 and Rytis Maskeliunas4 1 University of Lagos, Lagos, Nigeria Covenant University Nigeria, 3Atilim University, Turkey 4 Kaunas University of Technology, Kanus, Lithuania 2 Abstract. This paper presents a review of the current techniques and approaches adopted in conflict resolution in Multi-Agent Systems (MAS). The review highlights the strength and weaknesses, and thus, their success in fostering cooperation and collaboration in multi-agent systems. We survey alternative approaches to conflict resolution that rely on emerging technologies such as deep learning. From the survey, we discuss the benefits of using these emerging technologies in the conflict resolution process. 1. Introduction An agent is a computational entity with the ability to perceive and act upon its environment, through sensors and effectors. Intelligent (rational) agents strive to achieve a goal by choosing the optimal path to a solution, one that maximizes a performance measure. The intelligent agent maximizes its performance measure by making rational decisions [20]. In a single-agent environment, trade-offs between decisions are made without considering how these decisions affect any other agent [20]. This is not the case when solving problems in a multiagent system. Conflict may occur when the optimality of an individual agent’s decision is limited by decisions of other agents within the same MAS. Conflict resolution therefore, is the essential process of managing conflict within multiagent environments in order to restore cooperation and collaboration. Current approaches to conflict resolution in MAS include negotiation and arbitration. Negotiation has been the focus of central interest in multiagent systems, as it has been in the social sciences [14]. This is because negotiation provides a powerful approach for dealing with inter-agent dependencies at run-time [5]. Negotiation can be used to resolve a wide range of MAS conflicts. When agents in conflict cannot achieve an agreement arbitration strategy are adopted. In arbitration, an arbitrage agent is entrusted with the responsibility of coordinating conflict [5]. We shall consider these approaches and others in detail. Section 2 of the paper presents a literature review of multiagent systems and section 3 discusses the current approaches to conflict resolution in multiagent systems. In section 4, a review of relevant modern approaches to conflict resolution in multiagent systems is presented, followed by a discussion of the relevance of these emerging techniques in section 5. 2. Literature Review 2.1. Multiagent Systems (MAS) Multiagent systems can be viewed as a subset of Distributed Artificial Intelligence (DAI) [2]. In MAS, the main focus is how agents organize their knowledge about their environment and activities and analyze logically, the process of coordination [6]. Agents are generally classified into passive and active agents; the former has no goals while the latter has clearly defined goals and it sets out to achieve those goals [2]. • • • • • The agents in MAS have several important characteristics [2]; Agents are intelligent and autonomous Centralized control Decentralized data Asynchronous processing and computation Applications of multiagent systems can be found in industrial and commercial environments [5, 7]. Some applications include; Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. Published under licence by IOP Publishing Ltd 1 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 • • • Modelling and optimization of transportation systems. Ecommerce and their markets, where agents act as buyers and sellers, and can thus purchase and sell goods on behalf of their users. Real-time monitoring and management of telecommunication networks, where agents are responsible, e.g., for call forwarding and multiplexing and transmission. 2.2. MAS Architecture and Environment The environment of an agent might be open or closed, deterministic or stochastic and it might be single or multiagent [16, 15]. Although situations exist where an agent may operate by itself in a single agent environment, the increase in the Internet of Things is making such situations rare, therefore agents usually interact with at least one other agent [17]. The focus of this subsection is on systems with multiple agents. Characteristics of a Multiagent environment include [2, 18]; • Multiagent environments, specify interaction protocols and communication protocols. • Environment is usually open and decentralized. • Agents in a multiagent environment are usually autonomous and distributed, and may be adversarial or cooperative. Fig 1: MAS environment 2.3. Deep Learning Deep Learning is a subset of Machine Learning which focuses on an area of algorithms which was inspired by our understanding of how the brain works in order to obtain knowledge. It’s also referred to as Deep Structured Learning or Hierarchical Learning. Deep Learning builds upon the idea of Artificial Neural Networks and scales it up to be able to consume large amounts of data by deepening (adding more layers) the networks [3]. By having a large number of layers, a deep learning model has the capability of extracting features from raw data and "learn" about those features little-by-little in each layer, building up to the higher-level knowledge of the data. This technique is called Hierarchical Feature Learning, and it allows such systems to automatically learn complex features through multiple levels of abstraction with minimal human intervention [3]. Implementing deep learning can be outlined in the following steps; • • • • raw data → s is fed to input layer Data is transformed by hidden layer Output layer returns target scalars Q (s; →) Using back-propagation, train network on labelled data 2 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 Fig 5: Deep Learning Neural Network with 3 hidden layers One of the most distinct characteristics of Deep Learning – and one that made it quite popular and practical – is that it scales well, that is, the more data given to it, the better it performs [3]. Unlike many older machine learning algorithms which has a higher bound to the amount of data they can ingest – often called a "plateau in performance" – Deep Learning models has no such limitations (theoretically) and they may be able to go beyond human comprehension, which is evident with the modern deep learning-based image processing systems being able to outperform humans [3]. 3. Conflict Resolution strategies in MAS- State of the art One can approach the research of conflict resolution from three perspectives: system autonomy [10, 4], adversarial domains [10] and cooperative multi-agent systems [6]. Conflict resolution of cooperative multi-agent systems can be classified into: distributed systems [8, 9], model description [19], and applications [12, 13]. From the literature, it is obvious that the most commonly used techniques for conflict resolution in non-cooperative MAS are Negotiation and Arbitration [3]. 3.1. Other Conflict Resolution Strategies In [3], the authors propose the ConfRSSM (Conflict Resolution Strategy Method) in the domain of Learning Management System (LMS). The researchers demonstrate enhancing agents’ interactions and cooperation can greatly benefit from classifying conflicts. Researchers in [7] presented a hybrid approach as a solution for issues related to resolving conflicts in MAS. The approach is well suited to avoidance, prevention and detection of conflicts. Majorly, the conflicts arise due to fuzzy tasks, indistinguishable sub goals and unknown associated constraints [1]. 3.2. Problems with Current Conflict Resolution Strategies The current research on resolving conflict fails to address some concerns such as; 1. 2. 3. 4. Attention is not given to the conflicting agents and their confidence level Attention is not given to the number of groups of conflicting agents and the number of issues the conflict is centered around Some conflicts are less important and can be ignored, there is no means of identify the importance of a conflict Cost and time efficiency are important factors when choosing a resolution strategy, there are no rules to selecting amongst resolution strategies [6]. 4. MAS Learning within the Context of Conflict Resolution This section discusses the role of learning in conflict resolution and it provides an entry point for discussing the potential of emerging deep learning techniques in conflict resolution. 4.1 Robot Path Planning and Collision Avoidance Agents with shared resources within a multiagent system, can benefit from prioritization [2]. Researchers in [3] explore the use of genetic algorithms to assign priority dynamically. The performance of a team of agents needs to be improved without affecting the individual agent’s performance. The study implements a decoupled heuristic approach where a high-level planner agent is trained to reduce conflict by assigning priority. This planner agent is introduced only after individual agents have learned to optimize their performance. Specially designed for 3 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 Partially Observable Markov Decision Process (POMDP) environments, the approach is demonstrated on a problem in 3D aircraft path planning. 4.2. Network Management Systems There is a reasonable amount of effort that has been dedicated to developing real-time traffic management systems [4]. By training the system on high resolution simulation models, agents are able to achieve state estimation and short-term prediction. State estimation is useful where real time predictions have to be made and short-term predictions provide fast predictions [5]. One will still need to augment online adjustment modules together with several consistency checking techniques could be integrated with the simulation model and periodically activated in order to maintain the model consistency. The study in [5] presents a multi-agent learning methodology for consistency checking and online calibration of real-time traffic network simulation models. The approach strives to adjust the agent’s performance enabling them to learn based on their own performance and percept sequence. The performance of the methodology is examined using real-world data. The results show that the methodology is promising as an efficient mechanism for maintaining model estimation consistency. 4.3. Reinforcement Learning for Resolving Demand Capacity imbalance The work in [9] proposes and investigates the use of collaborative reinforcement learning methods for resolving demand-capacity imbalances during pre-tactical Air Traffic Management. The work recognizes the importance of online real-time data and leverage on this by building data-driven techniques for predicting correlated aircraft trajectories and, as such, respond to a need to handle airplane collision, a problem identified in contemporary research and practice in air-traffic management. The simulations, designed based on real-world data, confirm the electiveness of their [9] methods in resolving the demand-capacity problem, even in extremely hard scenarios. 4.4. Reinforcement Learning for Autonomous Vehicles In [7], the authors investigate the application of Reinforcement Learning to two well-known decision dilemmas, namely Newcomb’s Problem and Prisoner’s Dilemma. These problems are exemplary for dilemmas that autonomous agents are faced with when interacting with humans [6]. Furthermore, that argue that a Newcomb-like formulation is more adequate in the human-machine interaction case and demonstrate empirically that the unmodified Reinforcement Learning algorithms end up with the well-known maximum expected utility solution. 4.5. Deep Reinforcement Learning in Conflict Resolution Recent advances in combining deep learning and Reinforcement Learning have shown a promising path for designing new control agents that can learn optimal policies for challenging control tasks [9, 13]. These new methods address the main limitations of conventional Reinforcement Learning methods such as customized feature engineering and small action/state space dimension requirements. In [6], the authors leverage one of the state-ofthe-art Reinforcement Learning methods, known as Trust Region Policy Optimization, to tackle intersection management for autonomous vehicles. They [6] show that using this method, they can perform fine-grained acceleration control of autonomous vehicles in a grid street plan to achieve a global design objective and avoid unnecessary conflict. 4.6. Deep Q-learning in Conflict Resolution Cooperation and competition are rising behaviors seen in versatile multiagent systems. Within the work in [9] the researchers think about how participation and competition are new in independent agents that utilize reinforcement learning prepared utilizing camera sensor information for state representation. The deep Q-Learning system is actualized in multiagent situations and the interaction between two learning agents is explored within the wellknown video game Pong. By manipulating the remunerate rules of Pong, the authors are able to illustrate how competitive and collaborative behavior develop. The conflict resolution procedures recommended focusses on ways to extend the motivation to collaboration and it guides the movement from competitive to collaborative conduct. Finally, by playing against another versatile agent, rather than against a hard-wired agent, the authors accomplish more vigorous techniques. The work [11] appears that Deep Q-Networks can serve as a valuable device for consideration when dealing with multiagent systems. 5. Discussion Multiagent deep reinforcement learning has a considerably large literature on conflict resolution in the multiagent environment [9, 11]. However, it is important to note that most of the research and experiments have been conducted in either simple grid worlds or with agents already equipped with abstract and high-level sensory perception. Despite this one can still observe the importance of learning in MAS. 4 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 Deep Learning and Deep reinforcement learning in particular, serve more as a prevention of conflict rather than a conflict resolution strategy [12, 10]. At first glance this might seem contradictory to the goal of the study which is conflict resolution via emerging technology. From our study of Deep Learning applied in MAS, one can see that when an agent possesses a learning element, the instances of conflict with other agents within the MAS reduces [6, 5]. Therefore, our proposition, based on the study is that conflict resolution strategies are required less if the agents learn about the multiagent environment using a deep reinforcement learning model. In any case, if conflict occurs, agents with a model about the MAS, obtained from learning, find it easier to resolve conflict [12, 13]. Future work on conflict resolution in MAS should adopt learning strategies to enhance the performance of the agents within the multiagent system. Such enhancements in turn can increase the efficiency of current conflict resolution strategies. Acknowledgment The authors gratefully acknowledge the financial support of African Institute for mathematical sciences (AIMS) Alumni small research grant (AASRG), the Organization for Women in Science for the Developing World (OWSD), and L’oreal-Unesco for Women in Science. References [1] Aaron R, Enrique M, Saul P and Enrique S 2012 Conflict resolution in multiagent systems: Balancing optimality and learning speed, Eleventh Mexican International Conference on Artifitia l Intelligence. [2] Albrecht S and Stone P Multiagent Learning: Foundations and Recent Trends, viewed on 26 November 2017, from http://www.cs.utexas.edu/~larg/ijcai17_tutorial/multiagent_learning.pdf [3] Albrecht S, Crandall J and Ramamoorthy S 2016 Belief and truth in hypothesised behaviours. Artificial Intelligence, 235:63–94. [4] Alicia Y, and Ghusoon S 2017 A Conflict Resolution Strategy Selection Method (ConfRSSM) in MultiAgent Systems, International Journal of Advanced Computer Science and Applications (IJACSA), Vol 8, Issue 5, pp. 398-404. [5] Banerjee B and Peng J 2004 Performance bounded reinforcement learning in strategic interactions. In Proceedings of the 19th AAAI Conference on Artificial Intelligence, pages 2–7. [6] Banerjee D and Sen S 2007 Reaching pareto-optimality in prisoner’s dilemma using conditional joint action learning. Autonomous Agents and Multi-Agent Systems, 15(1):91–108. [7] Barber K, Goel A and Martin C 2000 “Dynamic Adaptive Autonomy in Multi-agent Systems,” Jour. of Experimental and Theoretical AI, 12(2): 129-147. [8] Barber K, Han D and Liu T 2000 Coordinating Distributed Design Making Using Reusable Interaction Specifications [9] Barber K, Liu T and Ramaswamy S 2001 “Conflict Detection During Plan Integration for Multi-Agent Systems,” IEEE Trans. on Systems, Man, and Cybernetics, 31(4): 616-628. [10] Barber K, Liu T, Goel A and Martin C 1999 “Conflict Representation and Classification in a Domain-Independent Conflict Management Framework,” Proc. of the Third Inter.Conference on Autonomous Agents 1999, pp. 346-347. [11] Barrett S and Stone P 2015 Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, pages 2010–2016. [12] Bond A and Gasser L 2002 An analysis of problems and research in DAI. In A.H. Bond and L. Gasser, editors, Readings in Distributed Artificial Intelligence, pages 3-35. Morgan Kaufmann, San Mateo, CA. [13] Bowling M. and Veloso M.. Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2):215–250. [14] Brandolese A, Brun A and Portioli-Staudacher A A multi-agent approach for the capacity allocation problem, viewed on 18th November 2017 from https://doi.org/10.1016/S0925-5273(00)00004-9 [15] Busoniu L, Babuska R and Bart D 2008 A comprehensive survey of multiagent reinforce-ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2(38):156– 172. [16] Caruana R and Niculescu-Mizil A 2006 An Empirical Comparison of Supervised Learning Algorithms, Proc. 23rd Int'l Conf. Machine Learning (ICML '06), pp. 161-168. [17] Chakraborty D and Stone P 2014 Multiagent learning in the presence of memory-bounded agents. Autonomous Agents and Multi-Agent Systems, 28(2):182–213. 5 The 3rd International Conference on Computing and Applied Informatics 2018 IOP Publishing IOP Conf. Series: Journal of Physics: Conf. Series 1235 (2019) 012022 doi:10.1088/1742-6596/1235/1/012022 [18] Chen K, Lindsay A, Robinson P and Abbass H 2009 A hierarchical conflict resolution method for multiagent path planning. IEEE - Evolutionary Computation, pages 1169–1176. [19] Christoforos M, Valts B and Ross K 2017 Socially Competent Navigation Planning by Deep Learning of Multi-Agent Path Topologies., viewed on 29 November 2017 from http://bit.ly/2BFQM4e. [20] Huang C, Ceroni A and Nof S 2000 Agility of Networked Enterprises - Parallelism, Error Recovery and Conflict Resolution Computers in Industry, 42(2-3): 275-287. 6