CONFIDENTIAL. Limited circulation. For review only.

Contact Anticipation for Physical Human–Robot Interaction with

Robotic Manipulators using Onboard Proximity Sensors
Caleb Escobedo∗ , Matthew Strong, Mary West, Ander Aramburu, Alessandro Roncone

Abstract— In this paper, we present a framework that unites

obstacle avoidance and deliberate physical interaction for
robotic manipulators. As humans and robots begin to coexist
in work and household environments, pure collision avoidance
is insufficient, as human–robot contact is inevitable and, in
some situations, desired. Our work enables manipulators to
anticipate, detect, and act on contact. To achieve this, we
allow limited deviation from the robot’s original trajectory
through velocity reduction and motion restrictions. Then, if
contact occurs, a robot can detect it and maneuver based
on a novel dynamic contact thresholding algorithm. The core
contribution of this work is dynamic contact thresholding,
which allows a manipulator with onboard proximity sensors to
track nearby objects and reduce contact forces in anticipation
of a collision. Our framework elicits natural behavior during
physical human–robot interaction. We evaluate our system on a
variety of scenarios using the Franka Emika Panda robot arm;
collectively, our results demonstrate that our contribution is not
only able to avoid and react on contact, but also anticipate it. Fig. 1: Four sensor units shown in color placed throughout
the robot body equipped with proximity sensors to detect
I. I NTRODUCTION external objects, such as a human hand. The red portion of
Robots have begun to transition from assembly lines, the robot shows where contact is expected to be made. The
where they are separated from humans, to environments bottom right image shows a visualization of sensed objects
where human–robot interaction is inevitable [1]. With this (green spheres) in rviz based on the real sensor values closest
shift, research in physical human–robot interaction (HRI) to the human hand.
has grown to allow robots to work with and around humans
on complex tasks. Safe HRI requires robots to both avoid computationally expensive, low-bandwidth (they typically
harmful collisions and continue to work toward their main operate at a maximum of 30 Hz and are thus unsuitable
task, whenever possible. Furthermore, robots must reliably for highly dynamic environments), and prone to occlusions.
sense their surrounding environments and parse pertinent in- While [3] proposed solving the occlusion problem with
formation in real time to avoid potentially harmful collisions. multiple cameras, their approach still struggled to provide
However, as HRI scenarios become commonplace, resort- real-time avoidance. Interestingly, there exists an inverse
ing to pure collision avoidance is not sufficient—contact relationship between frequency of occlusions and proximity
is inevitable, and, at times, desirable. For example, when to the interaction—that is, the closer a human and robot
humans work in close proximity, if contact needs to be made, operate, the more frequently occlusions occur. In all, these
a single nudge or tap can cause a person to move away and limitations make it challenging for a robot to guarantee safety
create more space for movement. We take a similar approach in unstructured, dynamic environments. To solve this issue, in
in this work: when contact is made, the robot moves slightly this work, we equip a robot with multiple sensors distributed
away to allow free movement near the contact area. Previous along its body, as seen in Fig. 1, allowing perception of its
surveys ([1], [2]) in both collision avoidance and contact surroundings through onboard, real-time, and high-frequency
detection did not address the transition between the two. proximity sensors. Whole-body sensing enables a robot to
Similar to these methods, the first step to anticipating observe potentially occluded objects closer to the interaction
potential contact is to accurately perceive objects in the because such sensors are physically attached to the robot
space nearby the robot. One common solution consists of itself. We utilize custom-built, onboard printed circuit boards
depth sensing cameras, which are externally mounted and equipped with proximity sensors to anticipate collisions and
can perceive both humans and objects in order to allow increase sensitivity to contact.
the robot to avoid unwanted collisions. These systems are We posit that this environmental information enables new
robot behaviors that go beyond pure avoidance. With this in
All authors are with the Department of Computer Science, Univer- mind, a framework is introduced, outlined in Fig. 2, that al-
sity of Colorado Boulder, 1111 Engineering Drive, Boulder, CO USA
[email protected] lows a robot manipulator to use whole-body sensing to antici-
* Corresponding author. pate contact with objects in its immediate surroundings. With

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

contact anticipation, the force required to trigger a reactive On the other hand, work on collision detection and re-
behavior is reduced and in turn leads to softer interactions action has generally not integrated prior-to-contact sensing.
between robots and humans. To enable this behavior, we first Several recent works ([17]–[22]) focused on collision de-
formulate our main task as a quadratic programming (QP) tection; their post-collision behavior was varied, but lacked
optimization problem that allows for collision avoidance with the ability to anticipate contact. The framework presented in
reduced velocities and relaxed avoidance constraints to allow [23] is the most similar work to that of ours, in that their
for intentional contact. Second, we devise a novel contact framework allows for both avoidance and contact, but lacks
detection and reaction scheme to register anomalies in noisy a smooth transition between the two.
external force data and leverage environmental information
to become sensitive to contact. III. M ETHODS
The proposed method is experimentally evaluated on a In this work, we present a framework to enable safe phys-
real robot in three different scenarios, demonstrating multiple ical interaction between a human and robot manipulator. The
capabilities of our framework. Altogether, our results demon- system is designed to allow for a spectrum of interactions
strate that onboard sensing and dynamic contact thresholding to safely occur, including avoidance and soft contact. In this
can allow for a smooth transition between avoidance and section, we detail each component of our framework, which,
contact. By leveraging these contributions, we are one step at a broad level, constitutes of perception, avoidance, and
closer to ubiquitous and safe human–robot interactions. contact detection and reaction as seen in the four sections of
This paper is organized as follows: Section II provides Fig. 2.
background on environmental sensing and safety in HRI
though obstacle avoidance methods. Section III details the A. Development of Sensor Units
entire system setup for this framework. Specifically, Sec- As previously mentioned, the contribution can only be
tion III-F introduces the main contribution, dynamic contact fully realized via distributed, fast and whole-body sensing. To
thresholding for contact anticipation and identification. In achieve this first crucial step, we developed a self-contained,
Section IV, we outline our system evaluation with static and low-cost, low-power sensor unit (SU) capable of transmitting
dynamic environments, followed by results and discussion of real-time environmental data wirelessly—see Fig. 1. It is
our experimental scenarios in Section V. designed to be the most minimal and modular element to
perform prior-to-contact, nearby space perception. It utilizes
II. R ELATED W ORK the LSM6DS3 iNEMO 3D accelerometer and 3D gyroscope
Onboard sensing with proximity sensors has been utilized inertial measurement unit (IMU) for automatic kinematic
by [4] to enable whole–body avoidance, with large sensor calibration of the SUs (introduced in [7]) and a VL53L1X
modules placed in predetermined locations. To avoid fixed time-of-flight (ToF) sensor. Distance information is published
placed sensor units, [5], [6] created an automated calibration at a rate of 50 Hz, with a max distance of four meters from
algorithm that enabled sensors to be placed anywhere on the SU mounted location. For future mentions of proximity
a manipulator and located automatically. Our work in [7] units in this work, we are referring to the VL53L1X ToF
created a system that improved upon this method’s accuracy sensor.
by multiple factors to accurately localize sensor units. This 33 mm by 36 mm board is optimized for reduced
However, with any sensing capabilities, work in safe HRI area and affords the benefits of modularity with a built-in
has largely focused on complete obstacle avoidance, leading ESP8266 microcontroller with real-time wireless communi-
to zero contact between manipulators and collaborators. For cation capability, integrated 3.7 V, 1000 mAh lithium-ion bat-
example, [8], [9] demonstrated the use of designated safety tery power and charging, and on-board programming through
zones around the robot where the robot will slow down, the USB-to-UART bridge. Each SU costs approximately $36
or stop its movement entirely to avoid collision. There also and is operational for up to ten hours on a single charge with
exists dynamic avoidance methods [10]–[12] that utilize an current consumption between 110–160 mA. The low cost,
externally mounted 3D camera to inform the robot’s move- long operational life, and built-in wireless capabilities make
ment in dynamic environments. Approaches in [13], [14] these sensor units extremely suitable for minimal, distributed
embraced the idea of working in close proximity; however, sensing that can function on a robot for several hours.
these methods relied on external sensing and maintained a
safe distance away from the human at all times, allowing B. Identification of Obstacle Positions
close interaction without contact. The method proposed in With the capabilities of the aforementioned sensor units,
[15] used onboard sensors, but employed them to solely objects are perceived in the robots immediate surroundings.
inform avoidance, and not transition to contact. More recent To accomplish this, SUs are distributed along the robot body,
works in [4], [16] utilized optimization based frameworks for as seen in Fig. 1. Each sensor unit’s proximity sensor is
pure avoidance informed with onboard proximity sensing. positioned so that its distance measurement is parallel to
Furthermore, none of the methods in multiple surveys [1], the sensor unit’s z-axis. This allows to compute an object’s
[2] address how to approach the transition from avoidance to position h ∈ R3 as function of the proximity reading dobs ∈
contact through implicit anticipation, which is the core focus R: T
h = O ~rSUk + O RSUk 0 0 dobs ,

of this work. (1)

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

Fig. 2: Diagram of the presented framework for contact anticipation, detection, and reaction. Each module represents a
distinct interaction, from when an object enters the robot’s environment to when the repulsive velocity becomes zero, and
the initial trajectory is resumed.

where SUk is the kth sensor unit, O ~rSUk ∈ R3 is the its joint limits. In this term, q̇mid consists of the desired
position of SUk in relation to the robot base frame O, joint velocities that will move a joint towards its middle
and O RSUk ∈ R3×3 is the rotational matrix of SUk in position (as determined from the joint operational range from
relation to the robot base frame. This allows to compute any the robot’s manufacturer), and k is a scaling factor used to
object position in Cartesian space for each calibrated SUk . weight the middle joint term. Eq. (2) is then manipulated
Combined with knowledge of the robot’s state, expressive to conform with the quadratic programming notation and
collision avoidance behavior and contact thresholding are restrictions explicitly noted in [4]. The previous formulation,
now achievable. This concludes the object detection block which did not include the middle joint limit term, allowed the
of Fig. 2. manipulator to enter undesired configurations, and ultimately
error states. In the following two sections, the EE velocity
C. Motion Control with Quadratic Programming reduction and robot movement restrictions are described,
The main task and avoidance behaviors of the system which form the constraints of the QP formulation.
are implemented using a Cartesian velocity controller that D. End-Effector Velocity Reduction
solves for joint velocities in a unified QP expression. This
optimization technique is used due to its ability to incorporate When a robot detects a obstacle, its velocity is reduced
behavioral movement modifications into the optimization in order to maximize safety and prepare for contact, in-
problem. dependent of avoidance. As a collaborator approaches, the
To define the task, first, let q̇ ∈ Rn represent the joint robot should slow down execution of its main task, which
velocities of a kinematically redundant robot manipulator causes the corresponding avoidance motion to be slower; still
with n joints, and let the Cartesian velocity of the end- allowing contact to easily be made. Without any velocity
effector (EE) be represented as ẋ ∈ Rm , where ẋ = J(q)q̇, reduction behavior, purposeful contact is both difficult and
and J(q) ∈ Rm×n is the Jacobian of the robot. With this in unnatural. The velocity is reduced as follows: for each
mind, the QP control equation is written as: detected obstacle h within a selected range dmax (0.8 m
in our implementation), the norm of the distance is taken,
1 µ ||d||, between the object and end-effector (EE). The lowest
g(q̇) = (ẋd − Jq̇)> (ẋd − Jq̇)+ q̇> q̇
2 2 (2) norm is taken to ensure safest behavior. The manipulator’s
k > desired velocity is reduced based on the following equation:
+ (q̇mid − q̇) (q̇mid − q̇).
2
||d||
The first term in Eq. (2) represents the quadratic Cartesian ẋd = ẋd , (3)
error between the current EE velocity ẋ and the desired task dmax
velocity ẋd , whereas the second term is used as a damping above, dmax scales ||d|| to produce a normalization term
term to avoid singularities based on a robot manipulability that reduces the main task’s desired EE velocity, dependent
measure—we refer the reader to [24] for more details. on how close an object is. Note that objects detected beyond
Additionally, a third term is added to the main task, that dmax from the EE are discarded. Furthermore, to prevent
causes the robot to favor joint positions near the middle of jerky motions induced by vanishing obstacle readings, we

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

(a) Prior to Contact: A human enters the (b) Contact: When a object makes contact (c) Post Contact: After contact has been
robot’s workspace, and is sensed by onboard with the robot, we measure the external force made and the force threshold exceeded, a
sensor units, triggering both velocity scaling and determine if it exceeds the dynamically reactive behavior moves the robot in the
and avoidance behavior. computed threshold. direction that the force was applied.
Fig. 3: A human interacting with a robot to elicit reactive behavior while the robot is in motion. This interaction is outlined
in Section IV-C

apply a linear decay formula to simulate a slowly moving Va − Vmax : if d < dnotice and d < drepulse ,

away obstacle: ẋa = Vb : if d < dnotice and d ≥ drepulse ,

Drop Restriction : otherwise.
  
||d|| ||d|| lobs
ẋd = ẋd · + (1 − )· , (4) (7)
dmax dmax lmax
Where Vmax is the maximum repulsive velocity, d is the
where lobs , which starts at 0, is the lifespan of a current distance from the object to the closest point on the robot
obstacle after sensing, that linearly increases over time, and body, drepulse is the distance where repulsive behavior
lmax is the value of where the original EE velocity is begins, and dnotice is the distance at which we “notice”
completely restored, 200 in this work’s evaluation. an obstacle, and start to impose movement restrictions. This
ensures smooth approach velocities and transitions between
E. Robot Movement Restrictions repulsive and movement limiting. When an object is close,
Using the QP framework allows to construct expressive a small repulsive velocity can be applied—the robot will
avoidance behaviors via movement restrictions. To make it only slowly avoid the object, and contact can be easily made
easier for contact to be made, such utilization of the QP if desired. For our experimental scenarios, Vmax = 0.04,
framework additionally makes it possible to move around drepulse = 0.1, and dnotice = 0.6.
objects instead of directly moving away. F. Dynamic Contact Thresholding
For such behavior to be obtained, movement restrictions In order to smoothly transition between prior-to-contact
are placed on the part of the robot closest to each obstacle and post-contact behaviors, a controller must determine if
to allow for movement around obstacles. At a high level, external contact has been made, along with the direction and
a linear inequality constraint is constructed that limits the magnitude of that contact. To achieve such behavior, three
motion ẋc,i ∈ R3 , which is the Cartesian velocity of the qualities are desired: a) perfect external force data should not
closest point to obstacle i on the manipulator. This value be required, b) contact is more likely when an object is close
is computed as ẋc,i = Jc,i q̇, where Jc,i is the Jacobian to the robot, and c) a robot moving close to an object should
of the closest point on the manipulator to the obstacle. move slower than normal and have increased sensitivity to
Next, d̂ is computed, which describes the direction of the interaction. To this end, we propose a dynamic thresholding
obstacle hi ∈ R3 ; thus, the term d̂T Jc,i q̇ describes the algorithm that relies on obstacle readings and an estimation
approach velocity of the robot towards an object. Finally, of the robot’s force; together, both are fused in such a fashion
this value is constrained, for any number of obstacles, to a that will capture all three desired qualities.
desired approach velocity value, ẋa , which is parameterized To detect contact forces, the proposed algorithm does not
based on the distance to an obstacle. We compute ẋa , or require a perfect estimation of the force: the signal that
the desired approach velocity to an obstacle, as a smooth we utilize throughout this paper is the estimated external
and continuous function. Drawing inspiration from [12], our Cartesian contact force, provided through the Franka Control
movement restrictions are computed as follows: Interface [25]. This estimation is often volatile and far from
zero in all axes, especially when the robot is moving at high
Vmax
Va = ; (5) velocities. However, our algorithm is robust to noisy data
α(2 d d −1)
1+e crit and can be used to accurately determine if external contact
is made on the robot’s end-effector.
Vmax 1) Average External Force Calculation: In order to oper-
Vb = d−dcrit ; (6)
α(2 d −1) ate on only recent force data, a sliding window is used to
1+e notice −dcrit

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

determine the running average. The manner in which we add object with a greater y value than the current end-effector
new values to the window is detailed below: position will lead to a reduction in the required force to
( trigger contact behavior in the negative y direction.
αxt + (1 − α)xt−1 : if |xt − µt−1 |> λσt−1 ,
xt = (8) When contact behavior is triggered, the external force data
xt : otherwise, reading (from the sliding window) must exceed either the
where xt is the data point appended to the window, α is upper or lower threshold in any one or more of the x, y, z
the influence value of a detected outlier (exponential moving axes. If this occurs, only then will a repulsive force be
average), λ is a scaling factor that is multiplied by the applied. We calculate the direction and overall force applied
previous standard deviation σt−1 , and µt−1 is the previous to the EE to apply a velocity in the force’s direction, as
mean value. The term λσt−1 is used to determine if a given shown below.
value is an outlier and should be discounted when added
Fext = Freading − µ, ẋdes = C · Fext . (12)
to the signal history. This allows for data points which are
considered to be outliers to have less influence on the data, Freading ∈ R3 describes the contact force from the sliding
reducing its noise. Furthermore, a window size is selected window, and µ ∈ R3 is the mean force of the window, which
that only includes a small number of samples because a larger computes Fext ∈ R3 . C describes the Cartesian compliance
window size would make it difficult for actual contact to be matrix that proportionally multiplies each component of
detected. In our implementation, α = 0.1 and λ = 0.75. the external force to output a desired end-effector velocity,
2) Contact Threshold Calculation: Contact thresholding which temporarily overrides the main task. We then linearly
determines when contact is made, in what directions (x, y, decay the velocity to zero over a specified time period,
z) contact was made, and the magnitude of the contact. To similar to Eq. (4). For the experimental scenarios, our user-
start, from the constructed external force window, the running defined contact thresholding values were set to Fstd = 3,
average (µ) is used as a base value for the dynamic external Fd = 4, Fσ = 4, and Fb = 10.
contact thresholds, which can be expressed as:
(
Tu = µ + Fb + Fσ − F –obs , Ultimately, as a result of dynamic contact thresholding,
Contact Threshold = (9) the three aforementioned interaction qualities are achieved:
Tl = µ − Fb − Fσ + F +obs ,
1) Robustness to measurement noise: The method does
where Tu is the upper and Tl is the lower limit of the contact not rely on perfect external force data; instead, it only
threshold, Fb is the base additional force required from the requires data that is reasonably noticeable beyond the
mean to trigger contact behavior (negative and positive), and current mean, similar to anomaly detection.
Fσ increases the external force required to trigger contact
2) Anticipated contact from obstacles: As an obstacle
behavior, and is based on the external force signal’s standard
approaches the robot EE, the contact thresholds de-
deviation. This value is computed as
crease due to the Fobs value.
σ
Fσ = min( · Fstd , Fstd ), (10) 3) Increased contact sensitivity as velocity decreases:
σmax
For noisier data, we are less certain of contact being
where σmax is the max (user-defined) standard deviation. made, growing the threshold bounds. When a robot
σ is the data’s current standard deviation, and Fstd is the slows down due to obstacles, reduced noise is ob-
max amount of force (user-defined) that is applied based on served, and, in turn, a smaller Fσ value is added to
the σ value. From the above equation, when the standard the thresholds.
deviation of the current data is low, Fσ will be small, which
leads to a threshold that is sensitive to an external force. As IV. E XPERIMENT D ESIGN
the robot slows down, the external force data is less noisy,
making a slower robot more sensitive to movement. Next, To validate the effectiveness of our algorithm, experiments
F -obs and F +obs are force reductions of the upper and lower are conducted on a physical Franka Emika Panda robot, a
limits, respectively, based on the distance from an obstacle 7-DoF robotic arm. A computer, outfitted with a real time
to the robot EE. A closer object should make the thresholds kernel, updates the commanded robot joint velocities at a
more constrictive because contact is more likely. rate of 100 Hz, which are sent to the Panda’s control box.
F -obs and F +obs are calculated using the following equation: We utilize a kinematic calibration algorithm [7] and sensor
units detailed in the previous sections to detect objects. Each
dmax − d
Fobs = ( ) · Fd , (11) sensor unit is outfitted with an IMU and proximity sensor, the
dmax − dmin latter of which informs the avoidance and contact detection
where dmax is the distance from the robot EE where Fobs algorithms. The experimental pipeline is developed using
begins to be applied, and dmin is the distance at which the C++ and ROS [26]: we initially implemented our avoidance
maximum force reduction is applied. Fd is the max force algorithms in the Gazebo simulator [27], before moving to
reduction (user-defined), and d is the minimum distance from the real robot to test both avoidance and contact. We then
the robot EE on the positive side of the robot for F +obs and integrated our framework’s ability for contact anticipation
the negative for F -obs , relative to each axis. For example, an and evaluated it in multiple scenarios, as detailed below.

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

Fig. 4: Robot end-effector, moving in a circular path (blue

circle); when an object is detected (green dots), the end-
effector trajectory is modified to avoid collision (red) then
gradually recovers (green). We demonstrate this behavior in
Fig. 5: Force difference for contact detection, in the y-
two interactions on opposite sides of the robot; the objects
direction, between our method with proximity sensing (red
are noticed by independent proximity sensors.
line) and without proximity sensing (blue line). The shaded
regions represent the thresholds for contact. In the top graph,
A. Static Obstacle Collision
the robot is moving in a circular trajectory, and in the bottom,
The first scenario consists of a static obstacle collision; the a horizontal line, before contact is made.
robot was placed near a large static object and commanded
to move towards the object to cause a collision to occur. This C. Dynamic Obstacle Collision
scenario highlights the integration of proximity information Finally, multiple, dynamic obstacle collisions with a hu-
to slow down and enable soft contact. More specifically, man participant are shown to demonstrate the full capabilities
we position the object in two positions around the robot of our system. To showcase this interaction, obstacle sensing
to highlight the ability of our system to reduce contact from multiple sensors and contact from multiple directions
thresholds based on object location. In the first position, is present. After contact has been made and the reactive
the static object is placed in the negative y direction of behavior has been taken, the robot continues along its
the end-effector; in the second position, it is placed in the designated trajectory. A depiction of this interaction can be
positive y direction. Two different trajectories are used for seen in Fig. 3, where all three segments of the interaction
each collision – in the first position, the robot moves in a (prior to contact, contact, and post contact) are shown.
Cartesian circle with a radius of 0.25 m, centered at (0.5,
0, 0.25), and moving counter-clockwise only in the y and V. RESULTS AND DISCUSSION
z direction. For the second position, the robot moves in a A. Static Obstacle Collision
line from (0.4, 0.5, 0.5) to (0.4, -0.5, 0.5) with speeds up In the static contact scenario, Fig. 5 illustrates the force
to 0.3 m/s. These results are compared to that of the same reduction when proximity sensors are used to anticipate a
movement when not informed by onboard sensor units to collision. In both examples, the difference in the overall
anticipate contact and reduce movement speed. detected contact force is approximately two times smaller
when utilizing proximity sensors. The smaller forces demon-
B. Obstacle Avoidance
strate that proximity detection for collision anticipation is
In the next scenario, the robot’s avoidance capabilities are beneficial for safe human–robot interaction.
demonstrated by commanding the robot to move in the same Prior to contact, the contact threshold value is reduced, as
circular trajectory as the previous section, then having a seen most prominently in the concave red shaded region of
human enter the robot’s path. When the human enters the the bottom graph. This change in threshold is due to object
scene, they cause a slight deviation in the robot’s initial proximity and robot velocity scaling, which are effectively
circular trajectory, while it avoids them. An example of able to soften the collision force between the object. Two
this demonstration can be seen in Fig. 3a, where a human independent contacts in both the robot’s positive and negative
holds their hand up near the end-effector. We show the path y-axis are display to demonstrate that the system works inde-
deviation, as two separate proximity sensors are activated on pendent of direction. While we purposefully allow our robot
opposite sides of the manipulator. to make contact with a static object, the robot’s avoidance

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

Fig. 6: Three components of a continuous human–robot interaction over 30 seconds, during this time, three distinct contact
are made from multiple directions. Top graph: depth values of seven proximity sensors placed along the robot body and EE,
with the activation threshold as a dotted black line. Middle graph: external force exerted on the robot EE in the y-axis with
the dynamic thresholds shown above and below. Bottom graph: The manipulator’s EE velocity along the y-axis.

away from, or moving in the opposite direction of the

obstacle, the robot then returns to its original trajectory. We
showcase this behavior using multiple sensor units mounted
on opposite sides of the manipulator to emphasize that the
controller can simultaneously avoid collisions and anticipate
contact. Similar to the previous section, obstacle avoidance
parameters can be tuned to increase or decrease restrictive
behavior.

C. Dynamic Obstacle Collision

The final evaluation highlights multiple, dynamic contacts
Fig. 7: Top graph: distance values from six proximity sensors with a human as outlined in Fig. 3. These interactions are
mounted on the robot body and EE during interaction. The displayed in Fig. 6. Throughout this figure, three separate
region of interest label denotes the region at which sensed contacts are made with the manipulator. The robot’s move-
distances are meaningful for our algorithm. Bottom graph: ment is only affected by obstacles with distance less than a
external force exerted on the robot EE in the y direction and predetermined distance threshold of 0.5 m (the diameter of
associated contact thresholds above and below. its circular trajectory), seen as a black dotted line in Fig. 6.
When this distance condition is met, the robot’s velocity is
weights can easily be altered to impose harsher movement reduced, as seen in the blue shaded region of the bottom
restrictions that completely avoid the object. Dependent on graph in Fig. 6. In addition to the robot slowing down, the
the task, these parameters can be tuned to fit a certain use contact thresholds become more restrictive when objects are
case. nearby; this is most clearly shown in Fig. 7, which shows a
higher resolution view of the third interaction. During this
B. Obstacle Avoidance scenario, two proximity sensors are activated on opposite
Fig. 4 shows the robot’s EE trajectory altered with the sides of the manipulator, and this leads to a reduction of
system’s avoidance behavior while executing a circular path. both upper and lower thresholds, as seen in the red shaded
As an object appears in the robot’s operational space (green regions of Fig. 7. After slowing down, contact with the robot
dots), the controller scales the robot’s velocity, and causes is shown in the red shaded regions of Fig. 6 from multiple
a slight deviation in the circular path. Once far enough directions. The direction and magnitude of the applied force

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.
 CONFIDENTIAL. Limited circulation. For review only.

directly determines the direction and magnitude of the EE [8] C. Vogel, C. Walter, and N. Elkmann, “A projection-based sensor sys-
velocity when repulsive behavior is triggered. We can see the tem for safe physical human-robot collaboration,” in 2013 IEEE/RSJ
International Conference on Intelligent Robots and Systems. IEEE,
resulting velocity spikes from contact in the bottom graph 2013.
of Fig. 6 in the red shaded regions. Ultimately, through [9] P. Svarny, M. Tesar, J. K. Behrens, and M. Hoffmann, “Safe
this interaction, we illustrate that our system is able to physical hri: Toward a unified treatment of speed and separation
monitoring together with power and force limiting,” 2019 IEEE/RSJ
anticipate and react to contact with dynamic objects within International Conference on Intelligent Robots and Systems (IROS),
its environment from multiple directions. In fact, without our Nov 2019. [Online]. Available: http://dx.doi.org/10.1109/IROS40897.
thresholding behavior, such soft contact would not trigger the 2019.8968463
[10] A. Tulbure and O. Khatib, “Closing the loop: Real-time perception
reactive movement and not allow a human to easily interact and control for robust collision avoidance with occluded obstacles,”
in close proximity. The implicit anticipation opens a new in 2020 IEEE/RSJ International Conference on Intelligent Robots and
realm of behaviors for true human–robot collaboration in Systems (IROS), 2020.
[11] H. Nascimento, M. Mujica, and M. Benoussaad, “Collision avoidance
close proximity, where a robot can dynamically adapt to its in human-robot interaction using Kinect vision system combined with
changing environment. robot’s model and data,” in 2020 IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS), 2020.
VI. C ONCLUSIONS [12] F. Flacco, T. Kröger, A. De Luca, and O. Khatib, “A depth space
approach to human-robot collision avoidance,” in 2012 IEEE Interna-
In this work, we introduce a framework for contact antici- tional Conference on Robotics and Automation. IEEE, 2012.
[13] A. M. Zanchettin, N. M. Ceriani, P. Rocco, H. Ding, and B. Matthias,
pation during physical human–robot interaction. Through re- “Safety in human-robot collaborative manufacturing environments:
laxed avoidance constraints in a quadratic programming con- Metrics and control,” IEEE Transactions on Automation Science and
trol formulation, combined with a novel dynamic threshold- Engineering, vol. 13, no. 2, 2015.
[14] P. A. Lasota, G. F. Rossano, and J. A. Shah, “Toward safe close-
ing algorithm, our work addresses the unexplored gray area proximity human-robot interaction with standard industrial robots,”
between avoidance and contact. Our experiments demon- in 2014 IEEE International Conference on Automation Science and
strate the system’s robustness in multiple scenarios with both Engineering (CASE). IEEE, 2014.
[15] G. B. Avanzini, N. M. Ceriani, A. M. Zanchettin, P. Rocco, and
static and dynamic interactions. L. Bascetta, “Safety control of industrial robots based on a distributed
With respect to future work, we plan to develop a truly distance sensor,” IEEE Transactions on Control Systems Technology,
collaborative system through both hardware and software. vol. 22, no. 6, 2014.
[16] Y. Ding, F. Wilhelm, L. Faulhammer, and U. Thomas, “With proximity
Our current sensor units are outfitted with IMUs; while we servoing towards safe human-robot-interaction,” in 2019 IEEE/RSJ
used the IMUs for our calibration, they were not utilized International Conference on Intelligent Robots and Systems (IROS).
in this framework. IMU data can be leveraged to increase IEEE, 2019.
[17] M. Geravand, F. Flacco, and A. De Luca, “Human-robot physical
sensitivity to external forces acting on the robot. The next interaction and collaboration using an industrial robot with a closed
iteration of sensor units will include tactile sensors, allowing control architecture,” in 2013 IEEE International Conference on
for localization of contact from external forces. Additionally, Robotics and Automation. IEEE, 2013.
[18] E. Mariotti, E. Magrini, and A. De Luca, “Admittance control for
we plan to explore the effectiveness of our perception as we human-robot interaction using an industrial robot equipped with a f/t
add more sensor units along a robot’s exterior and combine sensor,” in 2019 International Conference on Robotics and Automation
this information with external vision to gain a holistic view (ICRA). IEEE, 2019.
[19] E. Magrini and A. De Luca, “Hybrid force/velocity control for physical
of the robot’s environment. Ultimately, we will continue to human-robot collaboration tasks,” in 2016 IEEE/RSJ International
improve our current sensor units and achieve whole–body Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.
awareness, effectively bringing the paradigms of avoidance [20] E. Magrini, F. Flacco, and A. De Luca, “Estimation of contact
forces using a virtual force sensor,” in 2014 IEEE/RSJ International
and contact together for physical human–robot interaction. Conference on Intelligent Robots and Systems. IEEE, 2014.
[21] S. Haddadin, A. De Luca, and A. Albu-Schäffer, “Robot collisions: A
R EFERENCES survey on detection, isolation, and identification,” IEEE Transactions
on Robotics, vol. 33, no. 6, 2017.
[1] V. Villani, F. Pini, F. Leali, and C. Secchi, “Survey on human–robot [22] S. Haddadin, A. Albu-Schaffer, A. De Luca, and G. Hirzinger,
collaboration in industrial settings: Safety, intuitive interfaces and “Collision detection and reaction: A contribution to safe physical
applications,” Mechatronics, vol. 55, 2018. human-robot interaction,” in 2008 IEEE/RSJ International Conference
[2] P. A. Lasota, T. Fong, J. A. Shah, et al., A survey of methods for safe on Intelligent Robots and Systems. IEEE, 2008.
human-robot interaction. Now Publishers, 2017. [23] A. De Luca and F. Flacco, “Integrated control for phri: Collision
[3] M. Melchiorre, L. S. Scimmi, S. P. Pastorelli, and S. Mauro, “Collison avoidance, detection, reaction and collaboration,” in 2012 4th IEEE
avoidance using point cloud data fusion from multiple depth sensors: RAS & EMBS International Conference on Biomedical Robotics and
a practical approach,” in 2019 23rd International Conference on Biomechatronics (BioRob). IEEE, 2012.
Mechatronics Technology (ICMT). IEEE, 2019. [24] Y. Nakamura and H. Hanafusa, “Inverse kinematic solutions with
[4] Y. Ding and U. Thomas, “Collision avoidance with proximity servoing singularity robustness for robot manipulator control,” 1986.
for redundant serial robot manipulators,” in 2020 IEEE International [25] “Franka control interface documentation.” [Online]. Available:
Conference on Robotics and Automation (ICRA). IEEE, 2020. https://frankaemika.github.io/docs/
[5] P. Mittendorfer, E. Yoshida, and G. Cheng, “Realizing whole-body [26] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs,
tactile interactions with a self-organizing, multi-modal artificial skin R. Wheeler, and A. Y. Ng, “ROS: an open-source robot operating
on a humanoid robot,” Advanced Robotics, vol. 29, no. 1, 2015. system,” in ICRA workshop on open source software, vol. 3, no. 3.2.
[6] P. Mittendorfer and G. Cheng, “Open-loop self-calibration of ar- Kobe, Japan, 2009.
ticulated robots with artificial skins,” in 2012 IEEE International [27] N. Koenig and A. Howard, “Design and use paradigms for gazebo, an
Conference on Robotics and Automation. IEEE, 2012. open-source multi-robot simulator,” in 2004 IEEE/RSJ International
[7] K. Watanabe, M. Strong, M. West, C. Escobedo, A. Aramburu, Conference on Intelligent Robots and Systems (IROS) (IEEE Cat.
K. Chaitanya, and A. Roncone, “Self-contained kinematic calibration No.04CH37566), vol. 3, 2004.
of a novel whole-body artificial skin for collaborative robotics,” 2021,
under review.

Manuscript 1646 submitted to 2021 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS). Received March 5, 2021.