NRI-Small: Improved safety and reliability of robotic systems by

faults/anomalies detection from uninterpreted signals of computation graphs

Andrea Censi (coPI) Richard M. Murray (PI)
California Institute of Technology
One of the main challenges to designing robots that can operate around humans is to create
systems that can guarantee safety and eectiveness, while being robust to the numerous nuisances
of unstructured environments, from hardware faults to software issues, erroneous calibration, and
less predictable anomalies, such as tampering and sabotage. Perception algorithms work assuming
a simplied model of the world and are notoriously fragile to such disturbances; because actions
are based on perception, this creates safety issues. In principle, each nuisance can be detected
using established techniques, but, in practice, this amounts to a huge design eort, and it is not
possible to anticipate everything. However, the fact that the streams of observations and commands
possess coherence properties suggests that it would be possible to detect and mitigate many of these
disturbances using relatively simple models that can be learned from the data. Currently, robotic
systems are developed as a directed computation graph using middleware such as ROS or Orocos.
In these architectures, every signal is available for logging and learning. The goal of this project is
to develop theoretical methods, an applicable design, and a reference implementation of a generic
faults/anomalies detection mechanism, based on black-box modeling of robotic sensorimotor signals.
The system, without any prior information about the robot conguration, should be able to learn
a model of the robot and the environment by passive observations of the signals exposed in the
computation graph, and, based on this model, instantiate faults/anomalies detection components
in an augmented computation graph. The system should need zero or minimal conguration and it
should be possible to plug-and-play into any existing robotic system.
Intellectual merit To achieve this projects goals, it will be necessary to push the state of
the art of machine learning and system identication. While machine learning methods are now
commonly used with massive datasets, they are not designed for high-dimensional, heterogeneous
real-time data streams, where most information is present in the dynamics of the data and the
causal relation between actions and changes in the observations. Moreover, only weak guarantees
can be given for the inference procedure. System identication, as studied in the control theory
eld, provides hard bounds and guarantees, but for very limited classes of systems that are not
suitable to represent robotic sensorimotor cascades. We aim at developing theoretically-grounded
generic black-box sensorimotor learning methods, using the problem of faults/anomalies detection
as a litmus test of their generality across classes of sensors and actuators.
Broader impact The immediate broad impact will be a standard plug-and-play faults/anomalies
detection layer, hopefully becoming a standard part of any robotic system. The sets of benchmarks
and datasets collected along the way are expected to have a broader impact beyond the robotics eld,
hopefully inviting non-roboticists (statisticians, neurobiologists, cognitive development experts) to
look at realistic instances of embodied learning scenarios and problems relevant to robotics. The
practical activities will involve the participation of undergraduate students, as part of class projects
and as paid research assistants.
1
1 Motivation
State of the art for autonomous robotics deployment in the civil sector Based on the
progress mainly seen in sensing and perception, the last years have witnessed the rst commercial
deployments of completely autonomous robotics systems in the civil sector. Examples of systems
currently deployed include Kivas autonomous storage room [1], autonomous waterfront opera-
tions [2] and mining [3]. In these applications, the robots and the humans do not share the same
environment. In fact, in the Autostrad autonomous waterfront system, composed by dozens of au-
tonomous trucks, the entire plant is shut down whenever an intruder crosses the barrier, primarily
for safety purposes. We have also seen the rst commercially successful domestic robots, such as
iRobots Roomba [4], and Evolution Roboticss Mint [5], which uses exteroceptive sensing to imple-
ment advanced navigation strategies. These robots do share peoples environment, because, due to
the small mass and not powerful actuators, they cannot cause any harm (the worst that they can
do is scaring the household pets).
Challenges ahead: ensuring safety and reliability in systems heavily relying on per-
ception The challenge for the future is to improve the safety properties of current prototype
systems to cooperate safely with humans, sharing the same unstructured environments. Examples
include autonomous cars: the VisLab team [6] has managed to drive autonomously from Italy to
China using mainly vision-based algorithms; the Google cars (whose perception is based on active
lidar sensing) have reportedly accumulated hundreds of thousands of miles of autonomous driving.
Willow Garages PR2 promises to be a prototype for the future robotic housekeeper. When evalu-
ating the safety of these systems, it is instructive to think of the children: eventually, we would
like to have enough faith in these systems to have an autonomous school bus, or an autonomous
housekeeper allowed to clean while the kids are playing around.
The diculty in ensuring the safe operation of these systems depends heavily on a correct
perception of the external world based on non-trivial data processing; if the perception mechanism
is confused, the robot might perform an unsafe action. By contrast, for industrial manipulators
one has accurate models of the system, rich proprioception, and powerful actuators that make it
possible to quickly detect and mitigate unsafe situations [7].
Dealing with a world full of unexpected nuisances Assuring safety of these systems will be
a grand enterprise; in this proposal, we focus on one particular aspect: dealing with unexpected
nuisances to sensory data. What makes the system fragile is the fact that a perception algorithm
typically uses a (possibly implicit) model of the world that is a simplied version of reality. Designers
try to anticipate possible nuisances based on their previous experience, but it is certainly not possible
to anticipate every possible nuisance that makes data unreliable.
For example, consider a localization/mapping algorithm using camera data and based on the
assumption of a planar world. If the robot drives over a bump, the modeling assumption is violated.
However, mapping algorithms are designed in such a way to minimize the eect of these rare
random events (perhaps by relaxing the odometry error model). This is an example of nuisance
that is anticipated and mitigated. In the same scenario, imagine a y landing over the camera. The
y creates a very salient and very stable feature for the robot to track, therefore possibly causing
a failure of the localization/mapping algorithms. Yet, it is clear that a human observing the data
would be able to decide that those pixels are unreliable, based on simple considerations of spatial
2
and temporal coherence with respect to the nearby pixels and the commands given to the actuators.
We want to equip robotic systems with this sort of low level common sense that might allow to
tolerate such nuisances.
Examples of faults and anomalies in robotic systems There are a variety of phenomena,
here collected under the generic labels of faults and anomalies, that would create nuisances for
perception algorithms that do not take special steps to anticipate them.
1. Static calibration and conguration issues. An example would be the presence of some xture
on the robot obstructing a sensor in a way that was not considered at design time; or an
erroneous calibration procedure.
2. Hardware faults/deterioration. Eventually all mechanical components of a robot deteriorate,
either suddenly or gradually. The ability to detect such events is critical to ensure the safety
of the system.
3. Innocent interference. An unexpected sensor occlusion due to an external agent (animal/child)
is perhaps the most intuitive example of a nuisance that is likely to confuse the robots
perception of the world.
4. Intentional tampering/sabotage. This is likely to be an important issue with the eventual
widespread deployment of autonomous systemswhat better publicity for a neo-Luddite anti-
robot group than a series of inexplicable incidents?
We remark that, in theory, all of these nuisances could be identied using standard methods, such as
Bayesian estimation. For example, one could certainly design a y detector that continuously checks
for the presence of ies on the sensor frame. However, this would imply an exorbitant design eort
and computational cost if done for all possible nuisancesand by denition, one cannot expect the
unexpected.
The basic premise of our work is that faults/anomalies detection can be done with minimal
design eort and in a very general way, because most nuisances signicantly alter the sensorimotor
properties of the robot, and such changes can be detected with relatively simple and ecient models,
directly learned from data.
2 Opportunity
Modern robotic software systems are described by computation graphs Modern robotic
systems are designed as sets of software components glued together by robotics-specic middle-
ware (Fig. 1a) such as ROS [8] and Orocos [9]. The robotic system can be considered a directed
graph, where nodes are software components and labeled edges represent information ow between
software components. We call this graph the computation graph of the robotic system. This com-
putation graph abstracts middleware-specic metaphors (such as topics, ports, sinks/sources,
blackboards, etc.). The computation graph is typically specied by a conguration le that de-
scribes the software components conguration parameters, their interconnection, and, in the case
of distributed systems, the allocation of components to specic computation nodes.
Some of the signals in the computation graph represent sensor data (they originate from sources
representing the low-level sensor drivers), some represent commands towards the actuators (they
3
sensor
sensor
sensor
actuator
actuator
actuator
world
(a) Robotics systems can be abstracted as computation graphs where nodes are components and edges are signals.
sensor
sensor
sensor
actuator
actuator
actuator
y u
learned
sensorimotor
model
(b) In a computation graph, every signal is available for monitoring and learning at no additional eort.
original system
augmented
system
(c) Computation graphs have a formal description that can be programmatically manipulated.
Figure 1: Designing robotic systems as a computation graph of software components opens up new
possibilities that are at the basis of this project.
4
go towards sinks representing the low-level actuators drivers), while the rest of the signals are
intermediate results. The external world can be considered as an additional component that closes
the loop between sensors and actuators.
Every signal of interest in the computation graph is now available for learning The
main motivations for describing a robotic application as a graph of software components were
its advantages for modularity, code reuse, ease of conguration, etc. A serendipitous side eect
is that in modern architectures all signals of interests are easily accessible for monitoring and
learning (Fig. 1b).
Our assumption is that, when considered as a whole, the signals of the computation graph
contain all necessary information to detect a wide class of faults and anomalies, and relatively
simple models are needed to interpret such information.
The computation graphs can be programmatically manipulated The conguration les
contain a formal description of the computation graph. Therefore, it is possible to treat the compu-
tation graph as an entity that can be programmatically manipulated (Fig. 1c). This allows to reason
about robotic systems at a more abstract level, and it is a key enabler for realizing a completely
generic plug-and-play solution for fault detection.
Opportunity: system-level learning and inference These recent changes in the design of
robotic systems enable us to think a new level of applications, whose primary object that they
manipulate is the robotic system as a whole: by observing the signals in the computation graph,
one can learn about the robot sensorimotor cascade; by manipulating the computation graph, one
can add new functionality to the system. In this project, we apply these ideas to the case of
fault/anomalies detection.
5
3 System overview
Our goal is developing theoretical methods, a reference design, and an example implementation of
a generic learning-based faults/anomalies detection and mitigation system. The main design goal
is to not impose any additional design or conguration eort on the user.
3.1 User manual
Ideally, using the system would be as easy as 1-2-3:
1. Collect logs of the system in nominal conditions.
2. Run the following command:
$ robustify -i system.launch logs collected_logs/ -o robust_system.launch
where system.launch is a conguration le describing the original computation graph; collected_logs
contains the collected logs; and the output le robust_system.launch describes an augmented
system, with added software components for faults/anomalies detection.
3. Enjoy a more robust robotic system. The added components automatically detect faults/anomalies,
and censor/replace the aected data, or otherwise mitigate the eects on the rest of the system.
A diagram of how this is supposed to work is shown in Fig. 2.
original
graph
new graph
learned
models
world
robustify logs
original system
augmented system
introduced components for
faults/anomalies detection
Figure 2: Prototype idea
3.2 Behind the scenes
The theory and methods to create such a system do not exist yet; we briey describe the main
challenges that must be confronted.
1) Learning from uninterpreted signals of computation graphs The inputs to the system
are a description of the robots computation graph and a set of logs taken during the operation
of the robot. Based on these logs, and no other prior information about the robot, the system
estimates a model of the robot sensorimotor cascades; i.e., how the robot commands (u) inuence
the observations (y). The model is used for fault detection in the following phase.
The challenges of this phase include:
6
Selecting a generic class of models that can represent a large class of sensor and actuators. For
this, we will be extending our previous work on bootstrapping [10, 11, 12], which showed that
relatively simple models can approximate the dynamics of a variety of sensors such as range-
nders, cameras, and eld samplers. The main extensions needed are dealing with articulated
bodies and second-order dynamics (for example, to apply the methods to systems such as
quadrotors).
Selecting which subset of signals to consider for modeling the sensor-sensor correlations or
the sensori-motor interaction. It is likely to be unfeasible to consider all observation signals y
as candidates for learning, as the data volume can be in the range of 1Gb/s for robotic
platforms with many rich sensors. However, the data available for learning is expected to be
highly redundant. For example, a dead pixel can be detected both from the raw camera data
as well as the undistorted camera image, and both signals are part of y. We will explore a
principled way to choose a relevant subsets of signals to use, with the goal of having something
treatable, for example by using submodular optimization [13].
2) Augmenting the computation graph with fault/anomalies detection components In
this second phase, based on the models learned, the initial computation graph is modied as to
include monitoring and ltering modules. Each monitoring module is assigned one of more signals
to monitor, and outputs a reliability signal which describes whether the data follows the model
previously learned. A ltering module is assigned to a particular signal and it has the ability to mark
data as invalid, in a way compatible with the semantics of the data. The preexisting components
are recongured to read from the ltered streams.
The challenges of this phase include:
Given a global CPU budget, describing the computational resources the system designer de-
cides to be worth using for faults/anomaly detection, decide in a principled way which signals
are worth monitoring.
Distribute the computation across dierent hosts in a way compatible with communication
constraints and local CPU budgets.
3) Faults/anomalies mitigation Assume that monitoring components are embedded in the
computation graph and correctly detecting faults, we have to decide what to do with them. Up to
now, all of the techniques described did not actually modify any signal or inuence the behavior of
the robot. There are two ways in which this could happen. The simplest way is for the lter modules
to mark the data as invalid based on the reliability measure given by the monitoring components.
For this to happen, the existing messages in the computation graph must support the semantics of
invalid data.
More generally, one would want to activate some emergency plan. While it is certainly possible
to identify in the computation graph which messages drive the actuators, it is not safe in general
to just shut o the actuators. For example, if the robot platform was a quadrotor, one might want
to activate some safe landing if a fault is detected. In conclusion, while the previous phases can be
engineered considering the system a black box, and no knowledge of the actual robot or task, this
functionality cannot be realized in a pure plug-and-play manner as it requires a slight modication
of the existing components.
7
The main challenges of this phase are:
Dening conventions for expressing the concept of invalid data for all common messages,
and understand what is the right level of granularity, both in space (should all the message
be marked as invalid, or is it useful to mark only part of the sensor data as invalid?) and in
time (should one reason message-by-message or work at a coarser time scale?).
Dene messages and protocols that allow to trigger a safe shutdown or emergency mode,
in such a way that is maximally compatible with the conventions in the existing systems.
4 Technical approach
Our design goal is to implement a one-click system that adds faults/anomalies detection and
mitigation to an arbitrary robotic system. We do not want the user to spend additional design or
conguration eort. This ambitious goal dictates that various forms of learning to be a central part
of this project. At the same time, because we want to improve the safety of robotic systems, it
is crucial to establish some form of theoretical guarantee regarding the faults/anomalies that can
be detected. To succeed, we will have to integrate the exibility of machine learning methods, the
rigor of control theory, and the ambitions of developmental learning/epigenetic robotics.
4.1 Related work
Classical control theory approaches to faults/anomalies detection and isolation In the
established approach to fault detection and isolation (see, e.g., [14, 15]) one denes faults of sen-
sors and actuators as deviation of their behavior from a nominal model, assumed to represent the
healthy system, which can be given or identied from the data. Classical control theory primarily
studies very limited classes of models, such as linear time-invariant systems, for which theoretical
guarantees can be given, in terms of the convergence of the learning procedure as well as rejecting
the eects of nuisances [16]. Unfortunately, these models are not powerful enough to represent the
dynamics of robotic sensorimotor cascades.
Universal approximators in machine learning and robotics Machine learning provides
many examples of methods whose goal is to be universal approximators. For example, various
avors of deep belief networks [17, 18] are designed as universal approximators of data distributions,
but they do not deal with actions. Established techniques such as reinforcement learning [19] gives a
mathematical framework to deal with actions in unknown state spaces, but it does not explain how
to build stateful representations from raw sensory values and partially observable environments.
Predictive State Representations [20] is an eort to introduce a exible idea of state induced by
tests, which are manually designed features of the systems dynamics. Learning the dynamics of an
arbitrary dynamical system is extremely dicult in general, with no additional constraints on the
system, as shown by the bounds of the complexity of the optimal agent given by Hutter [21].
In the robotics community, there have been several attempts to learning dynamics with methods
more or less tailored to the robotic context [22, 23, 24, 25]. All of these methods cannot start from
low-level sensorimotor interaction (i.e., the pixel level) and include a good amount of domain
knowledge or previous knowledge of the body schema.
8
Related work in developmental learning & epigenetic robotics A somewhat distinct line
of research is present in the developmental learning and epigenetic robotics community. The stated
goal is to obtain generic articial intelligence agents that learn to act in the world starting from
low-level sensorimotor interaction, and unknown sensors and actuators. The focus is to obtain
explicit generative models of the actuator/world/sensor dynamics, rather than simply an optimal
policy; in this context, direct learning of a policy as in reinforcement learning would be seen as an
inadmissible shortcut.
A bootstrapping agent builds progressively more complicated models of the world starting from
scratch [26]. The rst steps consist in understanding from the observed statistics of the data what is a
plausible grouping of observations by sensor [27], and recovering the geometry of the sensors [28, 29];
with these steps one can transform scrambled streams into spatially coherent sensor snapshots. The
next steps consist in understanding what is the eect of actions on the sensor readings, by progres-
sively abstracting the continuous dynamics into (possibly discrete) primitives [30, 31]. The agent is
often assumed to be able to explore the world, according to some form of intrinsic motivation [32].
While several aspects of what would be a complete bootstrapping architecture have been demon-
strated anecdotally, no agent that can provably work for general robotic sensors/actuators has been
designed. Part of our long-term research plan is adding the control theory in bootstrapping, to
obtain reliable methods that can be routinely used by roboticists. Faults/anomalies detection is a
low hanging fruit for bootstrapping, which allows us to harden the theory while being immediately
useful to the robotics community.
Other related work in faults/anomalies detection in robotics Relevant techniques were in-
vestigated in two European projects under the 6th and 7th Framework Programme, called Phriends [33]
(Physical Human-Robot Interaction: DepENDability and Safety), and its successor Saphari [34]
(Safe and Autonomous Physical Human-Aware Robot Interaction). In these projects, the researchers
focused on the safe physical interaction of a human and a manipulator in an industrial context. The
core of the issue is detecting potentially dangerous events, such as the collision of the robotic arm
with the human body. The collisions are monitored using anomaly detection, by monitoring if
the joint measurements dier with respect to the nominal dynamics. If a collision is detected, the
robot task is suspended, and the the dynamics of the arm is changed to absorb the collision energy.
This is possible as industrial manipulators have very precise models, and full observations of the
state through the joint measurements. Ideally, our generic faults/anomalies detection scheme would
obtain similar performance, but relying on models which are entirely learned from scratch.
4.2 Premise: the dynamics of heterogeneous sensorimotor cascades are more
similar than one might expect
The key observations that allows generic learning models is that the dynamics of robotic sensori-
motor cascades (the series of actuators and sensors, considered together) are more similar than one
might expect. In previous work [10], we have shown this with reference to three canonical robotic
sensors: eld-samplers, range-nders, and cameras. Let y
s
be the raw readings of such sensors;
where s S represents an index over the sensels (short for sensory elements) which ranges over the
sensel space S, and y
s
is, respectively, eld intensity, distance reading, and luminance at sensel s.
At this low level, the three sensor dynamics are quite similar, as shown in Table 1, which gives an
expression for y
s
as a function of the robot velocities.
9
This similarity justies looking for models that can represent the dynamics of potentially any
robotic sensor. Learning with no prior information is what enables a zero-conguration approach to
fault detection: all information about the system is in the data itself.
Table 1: Continuous dynamics of canonical robotic sensors
sensor S continuous dynamics (far from occlusions)
eld sampler R
3
y
s
=

i
y
s
v
i
+

i
(s y
s
)
i

i
camera S
2
y
s
=
s

i
y
s
v
i
+

i
(s y
s
)
i

i
range-nder S
2
y
s
=

i
(
i
log y
s
s

i
)v
i
+

i
(s y
s
)
i

i
In this table, v R
3
and R
3
are the robot linear and angular velocities; s is a continuous index ranging over the
sensel space S; y(s) is the raw sensor data (eld intensity value, pixel luminance, or range reading); i is the i-th
component of the spatial gradient with respect to s; (s) is the nearness (inverse of the distance to the obstacles).
4.3 Bootstrapping models of sensorimotor cascades
In a series of previous works, we have studied several dierent models that could represent the
dynamics of canonical robotic sensors. These models occupy dierent points of the design space,
as they have various tradeos in terms of the computational complexity and the required prior
knowledge about the system.
Bilinear dynamics systems (BDS) In particular, we proposed the class of bilinear dynam-
ics systems (BDS) as a possible candidate in [10]. Given a discretized observation vector y =
{y
i
}
1iny
R
ny
, and generic commands u R
nu
, the dynamics are assumed to be of the form:
y
i
t
=

a
M
i
ja
y
j
t
u
a
t
. (BDS) (1)
The model is parametrized by a n
y
n
y
n
u
tensor M
i
ja
.
Bilinear gradient dynamics systems (BGDS) In [12] we studied bilinear gradient dynamics
systems (BGDS), a more constrained class of models that uses explicitly the spatial organization
of the sensels. The observations are assumed to be a smooth function from the sensels space:
y = {y
s
}
sS
. The dynamics depend on the spatial gradient y. The model is parametrized by two
tensor elds G
ds
a
and B
s
a
that model a general bilinear/ane relation between the derivative of the
observations and the commands:
y
s
t
=

a
(G
ds
a

d
y
s
t
+ B
s
a
) u
a
t
, (BGDS) (2)
This model is slightly more complicated, and less general, but much more ecient, as the complexity
is linear in n
y
instead of quadratic.
Dieomorphisms dynamical systems (DDS) In [35] we studied dieomorphisms dynamical
systems (DDS). These are discrete-time dynamical systems, where the state is y
k
: S R, and the
commands u belong to a nite alphabet U = {u
1
, . . . , u
|U|
}. Each command u
j
is associated to a
dieomorphism
j
Di(S). The transition function from the state y
k
at time k to the state y
k+1
is given by
y
s
k+1
= y
(s)
k
, (DDS) (3)
10
Examples of detected
faults/anomalies
Models
Relevant trade-os
Single sensel
statistics
Inter-sensel
statistics
Sensorimotor
interaction
stateful
instantaneous
quantitative
qualitative
More learning data needed
More prior information
needed
More powerful
Actuator performance
deterioration
Gross fault of actuators
Dead pixels
Less learning data needed
Less prior information needed
Less powerful
Random readings
Mirror placed in front
of camera
able to capture the dierence between faults/anomalies and the normal regime. In this project,
rather than aim for maximum generality, we want to characterize a hierarchy of models, ordered
by their representation power (Fig. 3). To the best of our knowledge, formal methods to establish
this (partial) order are not available and they will have to be developed along the way. In parallel,
we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models that
allow to detect them.
Very simple models can still be useful Here, we use the word model to refer to whatever
description of the system that allows to make predictions on the observations. We remark that
models could be simpler than expected.
As a simple example, let y
t
= {y
i
t
}
n
i=1
be the observations at time t, where y
i
t
is a single sensor
reading, perhaps the pixel of a camera. Consider the model
y
i
t
Uniform([y
i
, y
i
]), (4)
which describes the data as generated independently by a simple random distribution. Learning of
this model consists in tting the upper and lower bounds [y
i
, y
i
]. This model is clearly not complex
enough to fully represent the sensor, however it allows to make prediction and detect simple faults,
such as a dead pixel: y
i
t
= 0 for all t t
0
is a very unlikely sequence to be generated by the
model (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillating
measurements, this model would be not powerful enough. To detect such fault, one could use the
model
corr(y
i
t
, y
j
t
) = R
ij
,
which describe the pairwise correlation between readings. This model is not generative, but just
descriptive of the typical sequences in the data. This model allows to detect a larger class of faults,
and implement mitigation strategies, such as in-painting of dead pixels.
More complex faults/anomalies need even more complex models which take into account the
sensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts a
mirror in front of the camera). To detect this anomaly, one should use a predictive model of the
observations based on the actuators commands, such as (1)(3).
4.5 Fault/anomalies mitigation
Suppose that, based on the learned models, the system has identied a fault or anomaly in the data.
We want to address the question of what should be happening now in the system. There are several
possibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,
the simplest actions would consist in altering the information ow:
Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-
ping the messages (estimation algorithms are usually designed to handle large temporal gaps
in the data).
Marking the data. A reliability measure could be attached to the data. This could be done
both at the level of the messages or with more granularity, for example by marking as invalid
the single reading of the sensor. This implies that the messages are augmented to add this
information, and that the original system components can use this reliability indication.
11
able to capture the dierence between faults/anomalies and the normal regime. In this project,
rather than aim for maximum generality, we want to characterize a hierarchy of models, ordered
by their representation power (Fig. 3). To the best of our knowledge, formal methods to establish
this (partial) order are not available and they will have to be developed along the way. In parallel,
we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models that
allow to detect them.
Very simple models can still be useful Here, we use the word model to refer to whatever
description of the system that allows to make predictions on the observations. We remark that
models could be simpler than expected.
As a simple example, let y
t
= {y
i
t
}
n
i=1
be the observations at time t, where y
i
t
is a single sensor
reading, perhaps the pixel of a camera. Consider the model
y
i
t
Uniform([y
i
, y
i
]), (4)
which describes the data as generated independently by a simple random distribution. Learning of
this model consists in tting the upper and lower bounds [y
i
, y
i
]. This model is clearly not complex
enough to fully represent the sensor, however it allows to make prediction and detect simple faults,
such as a dead pixel: y
i
t
= 0 for all t t
0
is a very unlikely sequence to be generated by the
model (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillating
measurements, this model would be not powerful enough. To detect such fault, one could use the
model
corr(y
i
t
, y
j
t
) = R
ij
,
which describe the pairwise correlation between readings. This model is not generative, but just
descriptive of the typical sequences in the data. This model allows to detect a larger class of faults,
and implement mitigation strategies, such as in-painting of dead pixels.
More complex faults/anomalies need even more complex models which take into account the
sensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts a
mirror in front of the camera). To detect this anomaly, one should use a predictive model of the
observations based on the actuators commands, such as (1)(3).
4.5 Fault/anomalies mitigation
Suppose that, based on the learned models, the system has identied a fault or anomaly in the data.
We want to address the question of what should be happening now in the system. There are several
possibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,
the simplest actions would consist in altering the information ow:
Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-
ping the messages (estimation algorithms are usually designed to handle large temporal gaps
in the data).
Marking the data. A reliability measure could be attached to the data. This could be done
both at the level of the messages or with more granularity, for example by marking as invalid
the single reading of the sensor. This implies that the messages are augmented to add this
information, and that the original system components can use this reliability indication.
11
This similarity justies looking for models that can represent the dynamics of potentially any
robotic sensor. Learning with no prior information is what enables a zero-conguration approach to
fault detection: all information about the system is in the data itself.
Table 1: Continuous dynamics of canonical robotic sensors
sensor S continuous dynamics (far from occlusions)
eld sampler R
3
y
s
=

i
y
s
v
i
+

i
(s y
s
)
i

i
camera S
2
y
s
=
s

i
y
s
v
i
+

i
(s y
s
)
i

i
range-nder S
2
y
s
=

i
(
i
log y
s
s

i
)v
i
+

i
(s y
s
)
i

i
In this table, v R
3
and R
3
are the robot linear and angular velocities; s is a continuous index ranging over the
sensel space S; y(s) is the raw sensor data (eld intensity value, pixel luminance, or range reading); i is the i-th
component of the spatial gradient with respect to s; (s) is the nearness (inverse of the distance to the obstacles).
4.3 Bootstrapping models of sensorimotor cascades
In a series of previous works, we have studied several dierent models that could represent the
dynamics of canonical robotic sensors. These models occupy dierent points of the design space,
as they have various tradeos in terms of the computational complexity and the required prior
knowledge about the system.
Bilinear dynamics systems (BDS) In particular, we proposed the class of bilinear dynam-
ics systems (BDS) as a possible candidate in [10]. Given a discretized observation vector y =
{y
i
}
1iny
R
ny
, and generic commands u R
nu
, the dynamics are assumed to be of the form:
y
i
t
=

a
M
i
ja
y
j
t
u
a
t
. (BDS) (1)
The model is parametrized by a n
y
n
y
n
u
tensor M
i
ja
.
Bilinear gradient dynamics systems (BGDS) In [12] we studied bilinear gradient dynamics
systems (BGDS), a more constrained class of models that uses explicitly the spatial organization
of the sensels. The observations are assumed to be a smooth function from the sensels space:
y = {y
s
}
sS
. The dynamics depend on the spatial gradient y. The model is parametrized by two
tensor elds G
ds
a
and B
s
a
that model a general bilinear/ane relation between the derivative of the
observations and the commands:
y
s
t
=

a
(G
ds
a

d
y
s
t
+B
s
a
) u
a
t
, (BGDS) (2)
This model is slightly more complicated, and less general, but much more ecient, as the complexity
is linear in n
y
instead of quadratic.
Dieomorphisms dynamical systems (DDS) In [35] we studied dieomorphisms dynamical
systems (DDS). These are discrete-time dynamical systems, where the state is y
k
: S R, and the
commands u belong to a nite alphabet U = {u
1
, . . . , u
|U|
}. Each command u
j
is associated to a
dieomorphism
j
Di(S). The transition function from the state y
k
at time k to the state y
k+1
is given by
y
s
k+1
= y
(s)
k
, (DDS) (3)
9
Second phase (approximately 1.5 years) In the second phase, there will be explicit eorts
to publicize the project with the ROS community; we will create an ocial website for increased
visibility. A data repository will be setup on Caltech-owned servers.
Post-project (5 years) At the end of the project, we will freeze the complete software envi-
ronment used as VMWare images for long-term preservation. The software is likely to be actively
maintained/extended in the near future, and the collected data kept available on Caltech servers.
Permanent storage and availability Caltech maintains a secure institutional repository that
provides access to material that is of long-term value to the academic community, including datasets.
The Caltech Collection of Open Digital Archives (CODA) (http://coda.caltech.edu) makes use of
the open source EPrints software that is installed on three high-availability Linux servers managed by
the Library and utilizing ber-attached SAN. CODA metadata conform to international standards.
Documents and metadata are backed up to disk and tape, with o-site tape storage in case of
disaster. All records are given Persistent URLs that do not change with system upgrades. Caltech
CODA is indexed by all major Internet search engines. All publications (articles, theses, technical
reports, conference papers) and software packages produced as part of the project will be archived
in Caltech CODA persistently.
u
j
= 0 y
j
= 0
2
Figure 3: Hierarchy of models
where is the dieomorphism associated to the command given at time k. Learning generic
dieomorphisms is typically much more expensive; the method that we use in [35] has computational
complexity which is cubic in the sensor resolution.
Extensions needed for this project We have shown that these models are useful to represent
the dynamics of a variety of robotic sensors. Their biggest limitation so far is how they represent
the robot platform dynamics. They work well if the commands u are kinematic velocities applied
to the same rigid body where the sensor is attached. They cannot represent second-order dynamics
(necessary for ying vehicles) nor articulated bodies (necessary for personal robots). These are
non-trivial extensions that we intend to pursue in this project.
4.4 Hierarchies of models
Selection criteria for models to be used in faults/anomalies detection Choosing the right
classes of models is a central point of this project. In machine learning and system identication,
one usually classies classes of models according to two criteria: representation power (a more
general class is better), and the computational resources necessary for learning and inference. In
this application the evaluation criteria are slightly dierent. For example, a more powerful model is
more likely to require more data to be learned; for example, learning the models (2)(3) requires data
streams whose length is on the order of hours to learn camera models from scratch (i.e. without
calibration information). Here, the problem is not the complexity of the learning phase, which
could possibly happen oine, but rather the storage of a large amount of data. The computational
constraints are stricter for what regards the inference procedure, as it must happen online, in parallel
with the original process.
Characterizing hierarchy of models by their representation power Therefore, one should
select the simplest model possible, that can be consistently learned from small amounts of data, and
that supports a computationally inexpensive inference procedure. At the same time, the models
11
must be complex enough to represent the phenomena of interest; in this case, the models must be
able to capture the dierence between faults/anomalies and the normal regime. In this project,
rather than aim for maximum generality, we want to characterize a hierarchy of models, ordered
by their representation power (Fig. 3). To the best of our knowledge, formal methods to establish
this (partial) order are not available and they will have to be developed along the way. In parallel,
we aim to enumerate classes of faults/anomalies, characterizing what are the simplest models that
allow to detect them.
Very simple models can still be useful Here, we use the word model to refer to whatever
description of the system that allows to make predictions on the observations. We remark that
models could be simpler than expected.
As a simple example, let y
t
= {y
i
t
}
n
i=1
be the observations at time t, where y
i
t
is a single sensor
reading, perhaps the pixel of a camera. Consider the model
y
i
t
Uniform([y
i
, y
i
]), (4)
which describes the data as generated independently by a simple random distribution. Learning of
this model consists in tting the upper and lower bounds [y
i
, y
i
]. This model is clearly not complex
enough to fully represent the sensor, however it allows to make prediction and detect simple faults,
such as a dead pixel: y
i
t
= 0 for all t t
0
is a very unlikely sequence to be generated by the
model (4). If, instead of a dead pixel, the camera had a faulty pixel giving randomly oscillating
measurements, this model would be not powerful enough. To detect such fault, one could use the
model
corr(y
i
t
, y
j
t
) = R
ij
,
which describe the pairwise correlation between readings. This model is not generative, but just
descriptive of the typical sequences in the data. This model allows to detect a larger class of faults,
and implement mitigation strategies, such as in-painting of dead pixels.
More complex faults/anomalies need even more complex models which take into account the
sensorimotor interaction. Suppose that the camera is mounted upside down (or a saboteur puts a
mirror in front of the camera). To detect this anomaly, one should use a predictive model of the
observations based on the actuators commands, such as (1)(3).
4.5 Fault/anomalies mitigation
Suppose that, based on the learned models, the system has identied a fault or anomaly in the data.
We want to address the question of what should be happening now in the system. There are several
possibilities that could be implemented. To mitigate faults/anomalies concerning the sensor data,
the simplest actions would consist in altering the information ow:
Censoring the data. For temporary faults, perhaps the data can be simply censored by drop-
ping the messages (estimation algorithms are usually designed to handle large temporal gaps
in the data).
Marking the data. A reliability measure could be attached to the data. This could be done
both at the level of the messages or with more granularity, for example by marking as invalid
the single reading of the sensor. This implies that the messages are augmented to add this
information, and that the original system components can use this reliability indication.
12
Filling-in the data. For some kinds of faults one can ll-in the data. For example, if a y
does land on the camera sensor, the interested area could be lled in by using the learned
models, either considering sensel-sensel correlation (this is called in-painting in computer
vision) or using the learned dynamics to predict what the data would have been based on
previous images and the observed commands.
In principle, the faults/anomalies detection system can also manipulate the commands. For example,
it can freeze the robot should there be a major fault detected. However, this is not always a safe
choicethink for example to the case of a ying platform. More generally, one might want some
platform-specic fault recovery mode to be triggered. This necessarily requires some modications
of the original system components. A systems engineering focus of this project is how to design such
a system; how messages could be augmented to include reliability data, and what minimal interface
should planning modules expose to participate in this faults/anomalies mitigation system.
5 Preliminary results
We have already implemented some of these ideas in a prototype system for faults detection [36].
Intrinsic fault detection using sensel usefulness In this work, we explored the idea that
some hardware faults can be described even without reference to a nominal model. The principle
is that a sensel is faulty if it is useless, in the sense that it gives no information on the state
of the system. For example, consider a sensor with some obstructed readings. The obstructed
readings can be considered faulty (even if, to be precise, they are working) because they give no
information about the robot motion (Fig. 4). For the i-th reading, we can formalize a notion of
usefulness, as the distance between p(u|y) and p(u|y\{y
i
}). If the distance is 0, the i-th reading
can be considered faulty.
Example of computation graph manipulation Figure 5 shows an example of the manipulation
of a computation graph with minimal impact on the original components. The computation graph
is augmented with a bootstrapping agent, estimating a model of the sensorimotor cascade according
to the BDS model (1), and computing the usefulness signal. A ltering component is introduced
between the robot sensors/actuators drivers and the controller. Based on the estimated usefulness
signal, it marks some of the readings as invalid. The original controller receives the sanitized
readings.
commands
not inuencing
the observations
observations
not revealing
the commands
x u
a
y
a
u
b
... ...
y
b
world
y
Figure 4: Some faults in sensorimotor cascades can be given a characterization independent of a
nominal model, from an information-theory perspective. Completely faulty sensels (y
b
in the
gure) are those that do not provide any information about the commands applied to the system.
13
sanitized y
y
y u
controller
robot
ltering
module
bootstrapping
agent
panic
signal
sensel
usefulness
u u y
robot
controller
Figure 5: Example manipulation of the computation graph
Prototype demonstration Figure 6 shows an example demonstration of the system. The read-
ings of a range-nder on a mobile robot are partially obstructed by the robots antennas. The
bootstrapping agent can detect those anomalies, as those readings are not predictive of the robot
motion. Based on the computed sensel usefulness, some of the readings are marked as invalid. To
avoiding committing to hard thresholds, there is an adaptive threshold. The controller communi-
cates to the ltering component with a panic signal that the data does not allow any feasible
plan; the ltering component progressively lowers the threshold on the sensel usefulness until all
obstructed readings are recognized as invalid.
wi- antennas
cover the range
nder
antennas vibrate
during motion
antenna antenna
IR readings
sensel #
Sensel usefulness
(a) Robotic platform and sensel usefulness
sensels
time (s)
sensels
marked
as invalid
panic mode normal operation
adaptive sensel
blocking starts
operating
time (s)
(b) Demonstration of faults mitigation
Figure 6: Prototype of generic anomalies/faults detection mechanism from zero prior information
about the system.
Previous accomplishments based on NSF support
Control Design for CyberPhysical Systems Using Slow Computing (0931746). Richard
Murray is the PI for a 2009-2012 CPS grant. The goal of this project is to develop new, systematic
methods for the design of control systems that can work in the presence of slow computing elements.
In the rst two years of the project we have developed new techniques for "bootstrapping" controllers
based on experimental data [10, 11, 12], designing delay-based controllers that approximate arbitrary
14
transfer functions [37], and exploring the use of learned data for generating trajectories for complex
systems [38, 39]. Two graduate students have been supported under this grant to date.
The Preliminary results presented in the previous section were developed with the support of
this grant.
6 Methodology
Robotic systems and collected data We plan to use a variety of robotic systems to demonstrate
the generality of the approach. The Kuka YouBot is the simplest robot that supports mobile
manipulation. It will be equipped with cameras and range-nders. The Skybotix helicopter is an
inexpensive ying platform. A ying platform gives the added diculty of having to use second-
order models, and for which the fault recovery program cannot be to just stop the actuators. The
TurtleBot, equipped with a Kinect camera, is meant to be a platform with which undergraduate
project participants can get started quickly.
These platforms will be used to gather a large set of heterogeneous data logs, both in nominal
conditions, as well with various faults/anomalies introduced. These logs will be eventually made
available as part of a benchmark, with annotated ground-truth, in ROS format as well as other
formats more suitable to long-term preservation.
Software products Several software components will be produced; these can be divided in
roughly three independent groups:
Components for learning and inference from sensorimotor data. This will be a library of
models for low-level sensorimotor learning. This part of the software is meant to be largely
independent of ROS, with standard dependencies, and reusable as part of future projects.
Plug-and-play fault/anomalies detection system (i.e., the robustify command introduced in
Section 3.1). This part is supposed to be largely ROS-specic and have a simple, well-dened
user interface that requires no expert knowledge of the systems internal workings.
Benchmarks, data annotation and evaluation scripts. A lasting contribution of this project
will be the creation of benchmarks for the problem of faults/anomalies detection. This in-
cludes annotation tools for attaching ground-truth faults/anomaly data, utilities for generating
synthetic faults, and evaluation scripts.
The evaluation of a particular model will involve learning from the variety of collected logs and
testing the model prediction against the annotated ground-truth. The number of platforms
and volume of data makes it necessary to process the data in a distributed way by cloud
computing. We are currently evaluating services such as PiCloud that oer a simple API to
access the Amazon EC2 platform.
As discussed in the Data Management Plan, we plan to distribute software and data as free software
(e.g., GPL/BSD), and to package it according to the conventions used by the ROS project.
15
7 Plan & milestones
Year 1
Theory Denition of a reference universe of models of interest. Develop a theory to frame the
intuition of hierarchy of models. Develop a parallel hierarchy of reference faults/anomalies
to consider in the analysis.
Platforms and data Assembling robotic platforms and implementation of reasonably complex
behaviors based on already available software components, partly as group projects of an
advanced robotics class. Initial collection of data logs in nominal and faulty conditions.
Software Prototype of a library of generic learning modules based on the proposers previous re-
search, as well as established machine learning methods. Prototype of fault/anomaly detection
benchmarks.
Year 2
Theory Extension of the previous theory on bootstrapping to cover all systems of interest (e.g., in-
clude articulated bodies and second-order systems). Extend/adapt relevant machine learning
methods. Give a solid theoretical foundations to benchmarks and performance measures.
Platforms and data Consolidation of the datasets informed by the theory developments. First
end-to-end demonstrations of the system, with the rst users/testers being the students of the
yearly robotics class.
Software Prototype of plug-and-play fault detection/mitigation system. Design extensions to com-
mon data formats/messages necessary for fault mitigation.
Dissemination The prototype software and benchmarks are available on the projects website.
Involve the robotics community for early feedback on design.
Year 3
Theory Work out the theoretical guarantees that the system oers in terms of data needed for
learning and bounds on the faults/anomalies detection reliability.
Platforms and data Update datasets as appropriate given the theory developments.
Software Plug-and-play fault/anomaly detection system ready for public release.
Dissemination Organize workshops in robotics conferences (ICRA, IROS, RSS) to disseminate
the theory and technology developed in the project. Publicize the faults/anomaly detection
problem and the availability of benchmarks/datasets to the machine learning community
through participation in relevant conferences.
16
References
[1] Kivas automated material handling order fulllment system.. (link).
[2] H. Durrant-Whyte, D. Pagac, B. Rogers, M. Stevens, and G. Nelmes, An autonomous strad-
dle carrier for movement of shipping containers: From research to operational autonomous
systems, IEEE Robotics and Automation Magazine, vol. 14, no. 3, 2007. DOI.
[3] S. Vasudevan, F. Ramos, E. Nettleton, and H. Durrant-Whyte, A mine on its own, IEEE
Robotics and Automation Magazine, vol. 17, no. 2, 2010. DOI.
[4] iRobot Roomba.. (link).
[5] Evolution Robotics Mint.. (link).
[6] M. Bertozzi, A. Broggi, E. Cardarelli, R. Fedriga, L. Mazzei, and P. Porta, The VIAC expe-
dition toward autonomous mobility, 2011. DOI.
[7] S. Haddadin, A. Albu-Schaer, and G. Hirzinger, Requirements for Safe Robots: Measure-
ments, Analysis and New Insights, The International Journal of Robotics Research, vol. 28,
no. 11-12, 2009. DOI.
[8] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng,
ROS: an open-source Robot Operating System, in ICRA Workshop on Open Source Software,
2009.
[9] The Orocos Project.. (link).
[10] A. Censi and R. M. Murray, Bootstrapping bilinear models of robotic sensorimotor cascades,
in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),
2011. (link).
[11] A. Censi and R. M. Murray, Uncertain semantics, representation nuisances, and necessary
invariance properties of bootstrapping agents, in Joint IEEE International Conference on
Development and Learning and Epigenetic Robotics, 2011.
[12] A. Censi and R. M. Murray, Bootstrapping sensorimotor cascades: a group-theoretic perspec-
tive, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011.
(link).
[13] A. Krause and C. Guestrin, Submodularity and its applications in optimized information
gathering, ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 4, 2011.
[14] S. Simani, C. Fantuzzi, and R. J. Patton, Model-based Fault Diagnosis in Dynamic Systems
Using Identication Techniques. Advances in Industrial Control, Springer, 2002.
[15] S. X. Ding, Model-based fault diagnosis techniques. Springer, 2008.
[16] P. Ioannou, Adaptive Control Tutorial (Advances in Design and Control). SIAM, 2006.
[17] G. E. Hinton, S. Osindero, and Y.-W. Teh, A fast learning algorithm for deep belief nets,
Neural Computation, vol. 18, no. 7, 2006. DOI.
17
[18] Y. Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning,
2009. DOI.
[19] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.
[20] B. Boots and G. J. Gordon, Predictive state temporal dierence learning, in Advances in
Neural Information Processing Systems (NIPS), 2011. (link).
[21] M. Hutter, Universal Articial Intelligence: Sequential Decisions based on Algorithmic Proba-
bility. Berlin: Springer, 2004. (link).
[22] J. Bongard, V. Zykov, and H. Lipson, Resilient Machines Through Continuous Self-Modeling,
Science, vol. 314, no. 5802, 2006. DOI.
[23] Z. Kira, Modeling cross-sensory and sensorimotor correlations to detect and localize faults in
mobile robots, in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), 2007. DOI.
[24] J. Sturm, C. Plagemann, and W. Burgard, Body schema learning for robotic manipulators
from visual self-perception, Journal of Physiology, 2009.
[25] J. Ko and D. Fox, Learning GP-BayesFilters via Gaussian process latent variable models,
Autonomous Robots, vol. 30, no. 1, 2010. DOI.
[26] B. Kuipers, An intellectual history of the Spatial Semantic Hierarchy, Robotics and cognitive
approaches to spatial mapping, vol. 38, 2008.
[27] D. Pierce and B. Kuipers, Map learning with uninterpreted sensors and eectors, Articial
Intelligence, vol. 92, no. 1-2, 1997. DOI.
[28] J. Stober, L. Fishgold, and B. Kuipers, Sensor map discovery for developing robots, in AAAI
Fall Symposium on Manifold Learning and Its Applications, 2009. (link).
[29] J. Modayil, Discovering sensor space: Constructing spatial embeddings that explain sensor
correlations, in Proceedings of the International Conference on Development and Learning
(ICDL), 2010. DOI.
[30] J. Stober and B. Kuipers, From pixels to policies: A bootstrapping agent, in Proceedings of
the International Conference on Development and Learning (ICDL), 2008. DOI.
[31] J. Stober, L. Fishgold, and B. Kuipers, Learning the sensorimotor structure of the foveated
retina, in Proceedings of the International Conference on Epigenetic Robotics (EpiRob), 2009.
(link).
[32] S. Singh, R. L. Lewis, A. G. Barto, and J. Sorg, Intrinsically Motivated Reinforcement Learn-
ing: An Evolutionary Perspective, IEEE Transactions on Autonomous Mental Development,
vol. 2, no. 2, 2010. DOI.
[33] PHRIENDS: Physical Human-Robot Interaction: DepENDability and Safety.. (link).
[34] The Saphari project: Safe and Autonomous Physical Human-Aware Robot Interaction..
(link).
18
[35] A. Censi and R. M. Murray, Learning dieomorphism models of robotic sensorimotor cas-
cades, in ICRA, 2012. (submitted). (link).
[36] A. Censi, M. Hakansson, and R. M. Murray, Fault detection and isolation from uninterpreted
data in robotic sensorimotor cascades, in ICRA, 2012. (submitted). (link).
[37] S. Sojoudi, J. Lavaei, and R. M. Murray, Fault-tolerant controller design with applications in
power systems and synthetic biology, in American Control Conference (ACC), 2011.
[38] A. Censi, S. Han, S. B. Fuller, and R. M. Murray, A bio-plausible design for visual attitude
stabilization, in Proceedings of the 48th IEEE Conference on Decision and Control, 2009. DOI.
[39] S. Han, A. Censi, A. D. Straw, and R. M. Murray, A bio-plausible design for visual pose sta-
bilization, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
2010. (Link is to the extended technical report). DOI.
19