Multi Agent Robotics
Multi Agent Robotics
Multi Agent Robotics
T. Laengle, H. Woern
University of Karlsruhe Department of Computer Science, Institute for Process Control and Robotics,
Kaiserstr. 12, Geb. 40.28, D-76128 Karlsruhe, Germany
Abstract. In this paper an new intelligent robot control scheme is presented which enables a cooperative work of humans and robots
through direct contact interaction in a partially known environment. Because of the high flexibility and adaptability, the human-robot
cooperation is expected to have a wide range of applications in uncertain environments, not only in future construction and manufacturing
industries but also in service branches. A multi-agent control architecture gives an appropriate frame for the flexibility of the human-
robot-team. Robots are considered as intelligent autonomous assistants of humans which can mutually interact on a symbolic level and a
physical level. This interaction is achieved through the exchange of information between humans and robots, the interpretation of the
transmitted information, the coordination of the activities and the cooperation between independent system components. Equipped with
sensing modalities for the perception of the environment, the robot system KAMRO (Karlsruhe Autonomous Mobile Robot) is introduced
to demonstrate the principles of the cooperation among humans and robot agents. Experiments were conducted to prove the effectiveness
of our concept.
1 INTRODUCTION
In the past, the main goal of automation was the construction of systems excluding the involvement of humans. Man has
been considered as the weak point in the operation of automized systems: he is subject to errors, suffers from fatigue and
often has an unreliable memory. In the best cases humans have been regarded as obstacles in an automatized manufacturing
rather than as active factors. In fact, to guarantee the safety of humans within the working area of a robot, additional sensors
and equipment are required, which would increase the cost of an automatic system. Therefore many efforts have been made
to separate the working area of robot systems from man.
But today the demands for high flexible robotic systems, which must have the capabilities of adapting themselves to an
uncertain environment, are rapidly increasing. This is the case not only in manufacturing, where customer requirements
have to be met, new technologies should be integrated, products and propositions of each customer must be identified and
individualized, but also in the services, where robots have to move in unstructured and changing environments.
Because of these requirements it is unrealistic to consider a fully automated factory or service facility. Furthermore the
trend consists of integrating the man in the production process. In this context, the human-robot-team has an important role
to play in this context. The main idea is to combine the complementary capabilities of robots and humans. The robot is
quicker and more reliable but relatively inflexible, while the man is flexible but often unreliable. The human-robot-
cooperation can be considered as one of the solutions for meeting the future requirements in the industry and services.
Robots are facing new application areas in service sector. There is a need of robots for medical surgery [8], hotel trade,
cleaning of rooms or household [25]. Contrary to industrial robots, which are working in a well known environment and
used by skill operators, it is not the case for service robots. Service robots have to work in a ”turbulent” environment, where
objects are not necessarily known or have not ”a priori” known locations nor trajectories.
In spite of promising researches in the field of artificial intelligence, autonomous robots have still difficulties to execute
complex tasks in ”turbulent” environment. The more autonomous the system is, the more structured the world model should
be, and the more specific its tasks are. A small deviation between the world model and the real world causes the system
break down. As a result autonomous robots suffer from typical domain restriction (Fig. 1.a). Up to now it is still not known
how to achieve complete autonomous execution of complex tasks in unstructured environment [Brooks].
A possible compromise to avoid the domain restriction problem is to find the right balance between robot autonomy and
human-robot-interaction. We do not try to eliminate the uncertainties in the environment nor increase the intelligence of the
robot. The main idea is to compensate the uncertainties in order to avoid the robot to become stuck or trapped by an
appropriate mixture of robot intelligence, knowledge about the environment and human-robot interaction (Fig. 1.b).
Through appropriate interaction it is possible to combine the complementary capabilities of human and robot in order to
execute a complex task [9].
Usually the human-robot-”interaction” is restricted to a human-robot communication through keyboard and joystick [13].
But this kind of interface is not intuitive for human, does not exploit the complementary capabilities of human and robot
(especially the high temporal and spatial perception capabilities of humans) and offers communication possibilities only
from the human to the robot. Another common user-robot interface is an haptic interface like a 6D Mouse with bilateral
feedback. Thus the operator receives an information feed back about the forces applied from the environment to the
manipulator. But in general it is not an easy task for an operator to control a manipulator with a lot of degrees of freedom
with such devices. These methods are not appropriated for people who have no knowledge about robotic systems.
In recent years new human-robot interfaces based on direct interaction through contact has been tackled by many
researchers. An important issue in this area is the arm-manipulator coordination for the load sharing problem. For the
cooperative motion control of a robot and a human the human characteristics are approximated by an impedance model in
[10]. In [2][ 3] a variable compliance control as a coordination mechanism for the arm and the manipulator is proposed. In
[30][19] a control scheme was proposed to accomplish the coordinated task execution of a human and a mobile manipulator.
The majority of these works on arm-manipulator cooperation however focuses only on the dynamics of the am-manipulator
chain. Hence there is no a priori planning used for the robot, it plays a passive role and it does not recognize or interpret the
action of the human according to the current situation or task context.
An other important issue in this area is the shared guidance of an object by the operator and the robot. This problem is
addressed especially by the so called ”Synergistic devices”. Synergistic devices are intended for cooperative physical
interaction operator by producing forces. They allow the operator to have control of motion within a particular plane, while
the device dictates motion perpendicular to that plane. Detailed description of the synergistic device COBOT for
manufacturing applications is given in [20]. An overview of such devices in medicine is given in [27].
2.3 Limitation and requirements
From the communication point of view the interaction between human and robot have to fulfill the following two
requirements:
1. Task definition. The operator should be able to define on-line and in a simple way the task which should be executed.
The definition of a task can be divided into task specification and task description:
• Task specification. The operator provides the parameters, which are necessary for the task execution. In our case
parameters are the objects to be handled or their locations.
• Task description. By describing the task the operator gives instruction to the robot, about how to solve a specific
task. In our case the task description occurs by selecting appropriated elementary operations and ordering them in the
time space.
2. Dynamic function allocation. Function allocation is one classic question of human-machine interaction [4]. One major
characteristic of this process is non-stationarity. Therefore it is vital to consider the dynamics of such a process at the
same time as we consider the nature of the mutual sharing [7].
Unfortunately these requirements are neglected in the works described in paragraph 2.2. In the next section we present a
new concept which allows the operator to specify on-line a pick and place task by direct contact interaction with the
manipulator. The nature of the interaction between the operator and the robot changes according to the evolution of the task.
The performance of any complex system depends on the interaction between its components. To make robots capable of
working together with men in different domains like transportation, assembly, maintenance and other services, it is
necessary to understand the nature of interactions between the human and machine components. Robots are considered as
intelligent autonomous assistant of humans, which can mutually interact on a symbolic (exchange of digital messages on a
network) and a physical level (visual, tactile, acoustic). Interaction can be achieved with three concepts [29]:
3.1 Communication
The communication between robots and humans comprises the transmission of the information at the physical level and
the interpretation of the information.
Computer
Human model
Robot
Decoding Coding
Human
Br ain
3.2 Coordination
Several agents can have overlapping working spaces. In order to avoid mutual disturbance and conflict between two
working agents and prevent the system from dead-lock, it is necessary to plan, to order or to coordinate the activities of
these agents during an action.
3.3 Cooperation
Many problems cannot be solved by single agents working in isolation, because they do not have all the necessary
expertise, resources or information. On the other hand, it may be impractical to assign permanently all necessary tasks to a
single agent, because of historical and physical constraints [11]. This explains the necessity of temporary alliances through
cooperation where each partner contributes with his specific capabilities to the accomplishment of the final goal.
The question of human safety becomes increasingly important as future robots will be required to share their working area
with human operator. The following issues have to be considered: What are the requirements on the hardware and on the
software? What are the possible dangers for the human? How can the robot recognize the danger? How should the robot
react in dangerous situations?
Standard benchmark problems and algorithms to evaluate the cooperation between the human and the robot have to be
defined. Formal metrics to evaluate the progress made have to be developed.
In our Institute several experiments have been made with the mobile two-arm robot system KAMRO (Karlsuhe
Autonomous Mobile Robot). The robot consists of a mobile platform with an omnidirectional drive system using
MECANUM wheels and two PUMA manipulators with 6 degrees of freedom mounted in a hanging configuration. It is also
equipped with a multitude of sensors: one over-head camera, two hand-cameras, force/torque sensors on each manipulator
and ultrasonic sensors on the mobile platform. (Fig. 4).
Based on these works, further studies must be done to demonstrate the human-robot-cooperation within the multi-agent
system:
• Extending the communication model to the physical level and extending the perception methods through the fusion
of various sensors, for example by combining the force/torque sensors data with the vision sensors data.
• Analysing and modelling of the human behaviour to create a knowledge base for the agents. This knowledge base
will be used for implicit communication, so that the human can control the robots without any additional
information about the system and can thus concentrate himself only on the important aspects of the executing task.
• Interpreting the intention of sensor data with the help of the created knowledge base. To guarantee the reliability of
the interpretation, it is important for the agents to have abilities for error detecting and self-readjusting.
• Developing a human agent and integrating it into the multi-agent control architecture KAMARA by constituting the
human-robot-team.
• Analysing the safety requirements and presenting corresponding methods to meet them.
Cell controller
Task
KAMARA
Channel
KAMRO
Our approach is based on the integration of human-robot cooperation in a task plan. However this plan tolerates some
degrees of uncertainty and gives some degrees of liberty to the human, who is able to specify on-line some task parameters
and recover system errors.
A typical task for a manipulator is a pick and place task and can be achieved directly by one operation cycle (Fig. 6). An
operation cycle means the following elementary operation (EO) sequence which appears in robot programs frequently and is
described in Tab. 1 [6].
The correct execution of this operation cycle requires sensing and actuating capabilities as well as a knowledge database.
These capabilities are useful in order to sense the state of the dynamic environment, avoid collision with the environment,
find the position of the workpiece, grasp the workpiece correctly, compensate the uncertainties about the environment.
According to our previous experience with the autonomous robot KAMRO (Karlsruhe Autonomous Mobile Robot) we
have to admit that in spite of all the efforts made to develop an highly fault-tolerant control architecture it still remains some
errors, which cannot be easily overcome. A Classification of possible errors, distinguishing between internal robot errors
(mechanics, control, algorithm, model...) and external errors (sensor) and cognitive errors (database) is given in [24]. A
possible solution to overcome these errors is to integrate the human in the ”control loop” and give him enough ”degrees of
freedom” to solve the conflict between and the internal representation of the world and the real state of the world.
5.3 Description
In order to integrate the human in the ”control loop” of the manipulator we define for each situation of the pick and place
task symmetrically to the EOs, which are executed automatically, the semi-automatic elementary operations SEOs. When
the robot is not able to accomplish the required task autonomously, it switches into the semi-automatic mode and gives then
the operator the possibility to help him. The integration of these semi-automatic SEO into the operation cycle is shown on
Fig. 7. We distinguish two levels for the task execution according to the degree of autonomy of the robot: the level A
represents the autonomous level S whereas the level represents the semi-autonomous level. The events which activate the
transition between the two levels are represented by the white arrows. The nature of these events will be discussed in the
next section.
6 IMPLEMENTATION
In the preceding section we presented our concept for the dynamic shared control of a manipulator between the operator
and the intelligent control system. In this section we show how to realize it. We first focus on the controller then on the
transition recognition mechanism and also briefly on the safety factors.
For the distributed control of the 6 degrees of freedom of the manipulator among the human and the robot we use a
hybrid position-force controller represented on Fig. 9 [21].
Fig. 9: The hybrid position-force controller for a 6DOF
[P,O] represents the internal states of the manipulator: the position (Px, Py, Pz) and the orientation (Ox, Oy, Oz) of the
end-effector. [F,T] represents the external state of the manipulator: The forces (Fx, Fy, Fz) and the torques (Tx, Ty, Tz)
applied on the force/torque sensor attached on the end-effector. I6 represents the 6 dimensional identity matrix whereas S
represents the diagonal selection matrix. If a component of S is ‘0’, the corresponding degree of freedom is controlled by
the ICS (Position control law), whereas if the component is ‘1’, the degree of freedom is controlled by the human (force
control law or compliant motion).
The cooperative work between human and robot requires not only the shared control of the different DOFs according to
the situation, but also the ability to switch to the autonomous mode (Level A) or to the semi-autonomous mode (Level S) at
the right time. This mode transition could be initiated whether by the ICS (automatic transition) or by the human (manual
transition).
Automatic transition. The transition between the level A and S occurs automatically when an error has been detected
during the execution of an EO or when the robot recognizes its incapacity to execute the EO because parameters are
missing.
Manual transition. The transition between the level A and S can also be initiated manually when the operator applies the
corresponding forces on the end-effector. All EOs are guarded motion, so that if the applied forces cross a given transition
threshold, the transition is initiated.
The transition recognition makes use of rules and facts which tell the system how to react to a given situation. In our case
the situation is determined by four different types of parameters:
1. The operation cycle status: it characterizes the elementary operation (EO or SEO) which is currently executed.
2. The operation status. It specifies if an error has been detected.
3. The task specification status. It specifies which parameters are missing for the autonomous execution of the task.
4. The sensor data. The sensors are force/torque sensors as well as position sensors (overhead camera or hand camera).
As an example we give the rule which activates the transition from the EO 6 to SEO 6.
if((operation_cycle_status = EO6)
and ( (operation_status = error)
or (operation_status = incomplete)
or (Fx > Transition_Threshold_x)
or (Fy > Transition_Threshold_y)))
then (transition = 1)
At the moment the safety of the operator cooperating with the robot is guaranteed by the following measures:
1. Speed: The execution speed of the manipulator is limited
2. Guarded Motion: If the forces applied on the end-effector cross a security threshold the task execution is interrupted.
3. Hardware solutions like bumpers or buttons in order to stop the robot immediately in an emergency situation
Our concept was implemented and tested on the robot system KAMRO (Karlsruhe Autonomous Mobile Robot). The
robot consists of a mobile platform with an omnidirectional drive system using MECANUM wheels and two PUMA
manipulators with 6 degrees of freedom mounted in a hanging configuration. Besides the robot is equipped with a multitude
of sensors: 1 over-head camera, 2 hand-cameras, force/torque sensors on each manipulator and ultrasonic sensors on the
mobile platform.
We present the results obtained for SEO 5 – 7 with our robot KAMRO. The evolution of the forces applied by the human
on the end-effector as well as its position are presented in Fig. 11. During the first phase (SEO 5) Px and Py are controlled
by the ICS and are quite regular, whereas Pz is controlled by the human. The transition to SEO 6 is initiated manually. The
transition threshold is crossed over at t=4,5s. The manual transition allows the human to specify the transfer height which is
controlled by the ICS during the next SEO (SEO 6).
100
-100
0 500 1000 1500 2000 2500
242
240
238
0 500 1000 1500 2000 2500
Force Fz (N)
Time (12ms)
Fig. 11: Position and Forces applied on a manipulator’s end-effector as a time function
The advantages of our approach are the following:
• The error recovery occurs on-line.
• The operator interacts directly with the robot. This is a very natural communication way which renders the task easier
for the human.
• Through direct interaction the human keeps a direct force and visual feed back from the working environment and can
better exploit his very good sensing and actuating capabilities.
• The control of the DOF of the manipulator is shared dynamically between the robot and the human. The right mixture of
robot autonomy and human interaction is adapted to the state of the task execution.
• The human activates the transitions between two different operations executed by the robot. So the robot has to adapt
himself to the working rhythm of the human.
8 CONCLUSION
In this paper we outline the problems of autonomous robots for executing a complex task in unstructured environments. A
possible compromise to overcome these limitations is to find the right balance between robot autonomy and human-robot-
interaction. We present a method which integrates the human-robot- interaction in the task plan and allows a dynamic
mixture of human and robot capabilities according to the state of the task execution. The combination of the capabilities
occurs through an hybrid position-force controller which supports physical interaction between human and robot. This kind
of communication presents the advantage that it is intuitive and allows to exploit optimally the very good perception
capabilities of the operator.
9 ACKNOWLEDGMENTS
The research work was performed at the Institute for Process Control and Robotics (IPR), Prof. Dr.-Ing H. Wörn, Faculty
for Computer Science, University of Karlsruhe. The research was partially funded by the Nato Contract CRG 951002, with
support of the German Research Foundation (Graduiertenkolleg "Controllability of Complex Systems", DFG). Many thanks
also to Thomas Hoeniger for his contribution to this work.
10 REFERENCES
1. Al-Jarrah, O.M.; Zheng, Y.F. (1996): Arm-manipulator coordination for load sharing using compliant control. Int. Conf. on Robotics and
Automation, vol.2, pp. 100-1005, 1996
2. Al-Jarrah, O. M.; Zheng, Y. F. (1997): Arm - Manipulator Coordination for Load Sharing Using Variable Compliance Control. IEEE Int. Conf. on
Robotics and Automation, pp. 895-900, Albuquerque, New Mexico, April 1997.
3. Al-Jarrah, O. M.; Zheng, Y. F. (1997): Arm - Manipulator Coordination for Load Sharing Using Reflexive Motion Control. IEEE Int. Conf. on
Robotics and Automation, pp. 2326-2331, Albuquerque, New Mexico, April 1997.
4. Birmingham, H. P.; Taylor, F. V. (1954). A design philosophy for man-machine control systems. Proceedings of the IRE, 42, 1748-1758.
5. Brooks, R. A. (1986): A robust layered control system for a mobile robot“. In IEEE Jour. of Robotics and Automation, vol. 2, no. 1, pp. 14-23, 1986.
6. Frommherz, B.; Hoerman, K. (1997): A Concept for a Robot Action Planning system. In Rembold, U; Hoermann, K. (Eds):Languages for Sensor-
Based Control in Robotics. NATO ASI Series, Vol. 29.
7. Hancock, P. A. (1992): On the Future Of Hybrid Human-Machine Systems. In ‘Verification and Validation of complex Systems: Human Factors
Issues’, Wise et al. (Eds), NATO ASI Series, Spinger Verlag.
8. Ho, S. C.; Hibberd, R. D.; Davies, B. L. (1995): Robot Assisted Knee Surgery. IEEE Engineering in Medicine and Biology, (1995). Reprinted in
Groen, F.C.A.; Hirose, S.; Thorpe, C.E. (Ed.), Proceedings of the International Conference IAS-3, IOS Press, Washington, pp. 449-458.
9. Hoeniger, Th.; Laengle, Th.; Zhu, L. (1997): Physical Interactions between Robots and Humans for Highly Flexible Manufacturing Systems. In Proc.
of the 13th ISPE/IEE Int. Conf. CAD/CAM, Robotics & Factories of the Future (CARS&FOF’97), pp. 899-904., Pereira Colombia, Dec. 15-17, 1997.
10. Ikeura, R.; Monden, H.; Inooka, H. (1994): Cooperative Motion Control of a Robot and a Human: Proc. of 3rd IEEE Int. Workshop on Robot and
Human Communication (RO-MAN’94), Nagoya, July 18-20, 1994
11. Jennings, N. (1993): Cooperation in industrial multi-agent system. World Scientific Sries in Computer Science, Vol. 43, 1993
12. Kajikawa, S.; Okino, T.; Ohba, K.; Inooka, H. (1995): Motion Planning for Hand-Over between Human and Robot. Int. Conf. on Intelligent Robots
and Systems, vol. 1, pp. 193-199, 1995
13. Kawamura, K.; Pack, R.T.; Bishay, M.; Iskarous, M. (1996): Design philosophy for service robots. Int. jour on Robotics and Autonomous Systems, N
1-2, pp. 109-116, 1996
14. Kazuhiro, K.; Fujisawa, Y.; Fukuda, T. (1993): Mechanical System Control with Man-Machine-Environment Interactions. Int. Conf. on Robotics and
Automation, Vol. 1, pp. 239-244, Atlanta, Georgia, May 2-6, 1993
15. Kuniyoshi, Y; Inaba, M; Inoue, H. (1992). Seeing, Understanding and Doing Human Task. Int. Conf. on Robotics and Auotmation, Vol. 1, pp. 2-9,
Nice, France, May 12-14, 1992
16. Laengle, T.; Rembold, U. (1996): A Distributed Control Architecture for intelligent Systems. SPIE's Int. Symp. on intelligent Systems and advanced
Manufacturing, Boston, USA, Nov. 18-22, 1996. In: SPIE Proc. of the Int. Conf. on Advanced Sensor and Control-System Interface, Vol. 2911, pp.
52-61
17. Mizoguchi, H.; Sato, T.; Ishikawa, T. (1996): Robotic-Office to Suport Office Work by Human Behavior Understanding Functin with Networked
Machines. IEEE Int. Conf. on Robotics and Automation, pp. 2968-2975, Minneapolis, Minnesota, Apr. 22-28, 1996
18. Nassal, U. (1996): Fuzzy Control for Mobile Manipulation. IEEE Int. Conf. on Robotics and Automation, pp. 2264-2269, 1996
19. Nassal, U. (1996): Bewegungskoordination und reaktive Steuerung autonomer mobiler Mehrmanipulatorsysteme.VDI Verlag, Reihe 8, No. 593,
Düsseldorf, 1996.
20. Peshkin, M.; Colgate, J. E. (1996): "Cobots" Work with People. In IEEE Robotics and Automation Magazine, Vol. 3, No. 4, pp.8-9, Dec. 1996.
21. Raibert, M. H.; Craig, J. J.: Hybrid Position/Force Control of Manipulators. ASME Jour. of Dynamics Systems, Measurement, and Control 102, pp.
126-133, June 1981.
22. Rembold, U.; Lueth, T.; Ogasawara, T. From Autonomous Assembly Robot to Service Robots for Factories. IEEE Int. Conf. on Intelligent Robots and
Systems (IROS’94), Munich, Germany, Sept. 12-16, 1994
23. Sato, T.; Nishida, Y.; Ichikawa, J.; Hatamura, Y.; Mizoguchi, H. (1994): Active Understanding of Human Intention by Robot through Monitoring of
Human Behavior. Int. Conf. on Intelligent Robots and Systems, vol. 1, pp. 405-414, Germany, 1994
24. Schloen, J. (1994): Wissensbasierte Bewegungsausführung für die Montageautomatisierung von Industrierobotern. PhD, IPR, University of Karlsruhe,
Dissertationen zur Künstlichen Intelliogenz, INFIX.
25. Schraft, R. D. (1994): Mechatronics and Robotics for Service Applications. IEEE Robotics and Automation Magezine, pp. 31-35.
26. Suzuki, T.; Yokota, K.; Asama, H.; Kaetsu, H.; Endo, I. (1995): Cooperation between the Human Operator and the Multi-Agent Robotic System:
Evaluation of Agent Monitoring Methods for the Human Interface System. Int. Conf. on Intelligent Robots and Systems, vol. 1, pp. 206-211, 1995
27. Troccaz, J.; Peshkin, M.; Davies, B. (1997): The Use of Localizers, Robots and Synergistic Devices in CAS. In Proc. First Joint Conference Computer
Vision, Virtual Reality and Robotics in Medicine and Medical Robtics and Computer-Assisted Surgery (CVRMed-MRCAS’97), Grenoble, March
1997, Troccaz, J.; Mösges, R. (Eds.), Lecture Notes in Computer Science 1205.
28. Vischer, D. (1992): Cooperating Robot with Visual and Tactile Skills. Int. Conf. on Robotics and Automation, Vol. 3, pp. 2018-2025, Nice, France,
May 12-14, 1992
29. Wettstein, H. (1993): Systemsrchitecktur. Hanser-Verlag, München, Germany
30. Yamamoto, Y.; Eda, H.; Yun, X. (1996): Coordinated Task Execution of a Human and a Mobile Manipulator. Int. Conf. on Robotics and Automation,
Vol.2, pp. 1006-1011, 1996.