Performance Metrics for Coverage of Cleaning

Robots with MoCap System

Kuisong Zheng, Guangda Chen, Guowei Cui, Yingfeng Chen, Feng Wu(B) ,
and Xiaoping Chen

Multi-Agent Systems Lab, School of Computer Science and Technology,

University of Science and Technology of China, Hefei 230027, Anhui, China
{kszheng,cgdsss,cuigw,chyf}@mail.ustc.edu.cn,
{wufeng02,xpchen}@ustc.edu.cn

Abstract. Nowadays there are a lot of kinds of cleaning robots which

producted by different manufacturers come into people’s lives. But it is
still a problem that how to evaluate each robot’s performance to check
whether the quality is acceptable. In this paper, we make the first trial
to evaluate the complete coverage path planning algorithm which is the
core algorithm of a cleaning robot with Mocap system, and three simple
metrics were proposed to evaluate overall performance of the algorithm.
Lastly, the comparisons between different kinds of robots are presented.

Keywords: Cleaning robot · Mocap system · Coverage algorithm

1 Introduction
A robotic vacuum cleaner (or robot cleaner or cleaning robot) is a service robot
[7,15] which designed to help people clean their houses efficiently, one of the
popular robot cleaners is Roomba [13]. A typical robot cleaner is composed
of a mobile base, cleaning units, collision sensors and attachments. Nowadays
many robot cleaners are equipped with lasers, they will build a map to help
them generate a more efficiently cleaning path. Robot cleaners from different
manufacturers vary in price, size, motors, sensors, efficiency and so on. There
are already many evaluation indexes of a robot cleaner such as price, noise,
suction. However, these evaluation indexes are often performed by unofficial third
party evaluation organizations, and the process of evaluation is often carried out
manually and the data are not recorded for post review. Therefore, a general
framework of performance evaluation of a robot cleaner is needed to address
this problem.
We will focus on evaluation the performance of the coverage algorithm of a
robot cleaner in this paper, because the algorithm has greatest contribution to
the intelligence of the robot. It is urgent to have a general framework to evaluate
the performance of a robot cleaner for the following reasons: (1) A customer
wants to know which type of robot cleaner is more efficient or more suitable for
their needs. (2) Currently evaluation methods rely on experience, for example,

c Springer International Publishing AG 2017
Y. Huang et al. (Eds.): ICIRA 2017, Part III, LNAI 10464, pp. 267–274, 2017.
DOI: 10.1007/978-3-319-65298-6 25
268 K. Zheng et al.

A tester sprinkles scraps on the ground and see how many scraps are left with
eye after the cleaning robot finish cleaning, this kind of method is not scientific.
(3) Different kinds of robot cleaners vary in hardware and software, therefore we
can only rely on external measurement to evaluate their performance.
In this paper, we proposed a general platform for evaluating the performance
of coverage of a cleaning robot. Then we discussed the three criterias for mea-
suring the performance. Lastly, we conduct the experiments to evaluate the per-
formance of complete coverage algorithms of three robots which manufactured
by different companies.

2 Related Work
Complete coverage algorithms have applications in floor cleaning [14], and lawn
mowing [1]. Obviously, the goal of the algorithm is to generate the shortest path
that cover the entire area as much as possible. A large body of algorithms are
developed [10–12,26]. Different kinds of coverage algorithms were compared in
[11], randomized coverage algorithms which don’t need a laser sensor are very
simple but inefficient, while sensor-based coverage algorithms use sensors infor-
mation to help the robot cleaners make a plan.
Cleaning robots in the market can be divided into two types according to
whether they are equipped with laser sensors. The strategies of randomized
robots are mostly pre-defined heuristic behaviors, for example, perform spiral
walking in free space or following a wall detailed in [2] and so on. While laser-
based robots need to build a map of the home environment, the map can be an
occupancy grid map or geometry map or others which detailed in [21]. They also
need to use algorithms such as those proposed in [20] to localize where they are
in the map. With the sensors and computational resources, they are also possible
to make high-level decision-making or long-term planning [4,25] in the future.
As a product, a cleaning robot has a lot of indexes of performance evaluation
criteria as detailed in [18]. However we focus on evaluation of the performance
of the coverage algorithm of different kinds of cleaning robots. Sylvia used two
simple methods (percentage of coverage and distance travelled) for measuring
the performance of complete coverage algorithms [22], the position of the robot
is calculated by computer vision, this method has the following disadvantages:
(1) The camera needs to be calibrated in advance. (2) The method cannot be
carried out in real home environment. (3) The accuracy of the position of the
robot can’t be guaranteed. (4) The process of evaluation needs a lot of manual
intervention. Therefore we use Motion Capture System (MCS) as our evaluation
tool in virtual of it can provide realtime, accurate movement data of measured
objects. Actually MCS has already been used in robotics in many tasks [8,9,19,
27,28].
Our goal in this paper is to develop benchmarks problems for comparing the
performance of different cleaning robots. Many benchmark problems have been
proposed to evaluate robot systems, e.g., robot soccer [3,5], search and rescue
tasks [16,17,23,24], etc. However, to the best of our knowledge, none has been
proposed for the cleaning robots in the literature.
 Performance Metrics for Coverage of Cleaning Robots with MoCap System 269

3 Performance Metrics for Coverage

Coverage task is difficult for a cleaning robot because the environment is dynamic
and complex. The coverage algorithm itself is not covered in this paper, we want
to evaluate the performance of the cleaning robots which can be buied off the
shell. The following three performance metrics are used.

Compute the coverage rate. To compute the coverage rate of a cleaning robot,
we build a map with SLAM [6] to compute the free space Af ree and get the area
Arobot robot covers with Algorithm 1.
Arobot
Rcover = (1)
Af ree

3.1 Computing the Repeat Coverage Area Ratio

We record positions of the cleanning robots with MoCap while they run freely in
the simulated home environment, and we de-sample the positions so that each
position is at least 2 cm or 2 deg away from its neighbors. Then we compute the
repeat coverage area ratio by feeding the positions to Algorithm 1.

3.2 Computing the Distance Travelled

This metric is very simple, we can compute the distance Drobot that the robot
travelled by summing up the distance of each two adjacent positions. Dmin is
the least length which a robot travels just covers the whole filed. The efficiency
of the coverage E can be calculated by:
Drobot
E= (2)
Dmin

Algorithm 1. Computing the repeat coverage area ratio

1: Each grid of map M has two filed, covered, repetition time
2: Let C indicate whether a grid of map is covered
3: Let R indicate times that a grid of map is covered
4: Input: Robot positions P
5: procedure ComputeRepetition(P )
6: I ← ∅
7: for p ∈ P do
8: Compute indexes I that the robot covers in position p
9: for i ∈ I − I  do
10: C [i] ← 1
11: R [i] ← R [i] + 1
12: I ← I
270 K. Zheng et al.

Fig. 1. Experiment site Fig. 2. (a) MoCap system (b) Optical

camera (c) Markset

4 Experimental Setup

We tested three popular cleaning robots among which two cleaning robots are
laser-based and one cleaning robot is collision-based. Experiment environment is
shown in Fig. 1, the area of test platform is about 9.5 m2 . MoCap system shown
in Fig. 2 can track markerset which fixed on the robot in realtime. The cleaning
robot will run with no interference once started. Every robot performances the
cleaning task several times and the trajectories are recorded.
The collected trajectories and coverage areas are shown in Fig. 3. The first
row of images are the coverage areas of the three cleaning robots, the second row
of pictures are the trajectories of the three cleaning robots. Obviously, robot A
travelled the farthest distance to finish the task, robot C is better than robot
A because it’s trajectory is more ordered, and the trajectory of robot B is as
ordered as robot C except that distance between sweeping lines is larger than
robot C.

4.1 Result and Comparisons

We compared these three cleaning robot in travelled distance, covered area,

sweeping time as shown in Fig. 4. Obviously, all robots successfully covered the
whole field, robot A can’t plan according to the environment in advance because
it is not equipped with a laser sensor, so it travelled according it’s designed
behavior which is almost randomly, therefore, it waste a lot of time and it’s
efficiency is worst among the three robots, and robot B finishes the cleaning
task with the least time among the three robots. The repetition times of the
three cleaning robots is shown in Fig. 5, we can see robot B has least repeation
ratio.
 Performance Metrics for Coverage of Cleaning Robots with MoCap System 271

Fig. 3. Areas are in the upper row and trajectories are in the lower row, the color of
the area which the cleaning robot covers many times is brighter.

Fig. 4. Comparisons of three robots

272 K. Zheng et al.

Fig. 5. Repetition times of the robots

5 Conclusions
In this paper, we proposed three performance evaluation indices of the complete
coverage algorithm of a cleanning robot, they are coverage area percentage,
repeat coverage area ratio, distance travelled respectively, these metrics show the
performance of coverage algorithm of cleaning robots from the whole level. We
use Mocap system as an external measurement because it is accurate, realtime
and easy to use. Then, we test the three robots buied in market to see how
well they perform actually, we can conclude that laser-based cleaning robots are
better than randomized cleaning robots from the result.

Acknowledgments. This work was supported in part by National Natural Science

Foundation of China under grant No. 61603368, the Youth Innovation Promotion Asso-
ciation of CAS (No. 2015373), and Natural Science Foundation of Anhui Province under
grant No. 1608085QF134.

References
1. Abramson, S., Levin, S., Tikochinsky, Y., Zur, E.: Robotic lawnmower. US Patent
D451,931 (2001)
2. Ando, Y., Yuta, S.: Following a wall by an autonomous mobile robot with a
sonar-ring. In: Proceedings of 1995 IEEE International Conference on Robotics
and Automation, vol. 3, pp. 2599–2606. IEEE (1995)
3. Bai, A., Wu, F., Chen, X.: Towards a principled solution to simulated robot
soccer. In: Chen, X., Stone, P., Sucar, L.E., Zant, T. (eds.) RoboCup 2012.
LNCS (LNAI), vol. 7500, pp. 141–153. Springer, Heidelberg (2013). doi:10.1007/
978-3-642-39250-4 14
 Performance Metrics for Coverage of Cleaning Robots with MoCap System 273

4. Bai, A., Wu, F., Chen, X.: Bayesian mixture modelling and inference based thomp-
son sampling in monte-carlo tree search. In: Proceedings of the Advances in Neural
Information Processing Systems (NIPS), pp. 1646–1654, Lake Tahoe, United States
(2013)
5. Bai, A., Wu, F., Chen, X.: Online planning for large markov decision processes
with hierarchical decomposition. ACM Trans. Intell. Syst. Technol. (ACM TIST)
6(4), 45 (2015)
6. Bailey, T., Durrant-Whyte, H.: Simultaneous localization and mapping (slam):
Part ii. IEEE Robot. Autom. Mag. 13(3), 108–117 (2006)
7. Chen, Y., Wu, F., Shuai, W., Wang, N., Chen, R., Chen, X.: KeJia robot–an
attractive shopping mall guider. In: Tapus, A., André, E., Martin, J.C., Ferland,
F., Ammi, M. (eds.) Social Robotics. LNCS, vol. 9388, pp. 145–154. Springer,
Cham (2015). doi:10.1007/978-3-319-25554-5 15
8. Chen, Y., Wu, F., Wang, N., Tang, K., Cheng, M., Chen, X.: KeJia-LC : a low-
cost mobile robot platform — champion of demo challenge on benchmarking service
robots at RoboCup 2015. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.)
RoboCup 2015. LNCS, vol. 9513, pp. 60–71. Springer, Cham (2015). doi:10.1007/
978-3-319-29339-4 5
9. Cheng, M., Chen, X., Tang, K., Wu, F., Kupcsik, A., Iocchi, L., Chen, Y.,
Hsu, D.: Synthetical benchmarking of service robots: a first effort on domestic
mobile platforms. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup
2015. LNCS (LNAI), vol. 9513, pp. 377–388. Springer, Cham (2015). doi:10.1007/
978-3-319-29339-4 32
10. Choi, Y.H., Lee, T.K., Baek, S.H., Oh, S.Y.: Online complete coverage path plan-
ning for mobile robots based on linked spiral paths using constrained inverse
distance transform. In: 2009 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pp. 5788–5793 (2009). doi:10.1109/IROS.2009.5354499
11. Choset, H., Pignon, P.: Coverage path planning: the boustrophedon cellular decom-
position. In: Zelinsky, A. (eds) Field and Service Robotics, pp. 203–209. Springer,
London (1998). doi:10.1007/978-1-4471-1273-0 32
12. Huang, W.H.: Optimal line-sweep-based decompositions for coverage algorithms.
In: Proceedings of 2001 ICRA IEEE International Conference on Robotics and
Automation (Cat. No.01CH37164), vol. 1, pp. 27–32 (2001). doi:10.1109/ROBOT.
2001.932525
13. Jones, J.L.: Robots at the tipping point: the road to irobot roomba. IEEE Robot.
Autom. Mag. 13(1), 76–78 (2006)
14. Jones, J.L., Mack, N.E., Nugent, D.M., Sandin, P.E.: Autonomous floor-cleaning
robot. US Patent 6,883,201 (2005)
15. Lu, D., Zhou, Y., Wu, F., Zhang, Z., Chen, X.: Integrating answer set programming
with semantic dictionaries for robot task planning. In: Proceedings of the 26th
International Joint Conference on Artificial Intelligence (2017)
16. Ramchurn, S.D., Huynh, T.D., Wu, F., Ikuno, Y., Flann, J., Moreau, L., Fischer,
J.E., Jiang, W., Rodden, T., Simpson, E., Reece, S., Roberts, S., Jennings, N.R.:
A disaster response system based on human-agent collectives. J. Artif. Intell. Res.
(JAIR) 57, 661–708 (2016)
17. Ramchurn, S.D., Wu, F., Fischer, J.E., Reece, S., Jiang, W., Roberts, S., Rodden,
T., Greenhalgh, C., Jennings, N.R.: Human-agent collaboration for disaster
response. J. Auton. Agents Multi-Agent Syst. (JAAMAS), pp. 1–30 (2015)
18. Rhim, S., Ryu, J.C., Park, K.H., Lee, S.G.: Performance evaluation criteria for
autonomous cleaning robots. In: International Symposium on Computational Intel-
ligence in Robotics and Automation, CIRA 2007, pp. 167–172. IEEE (2007)
274 K. Zheng et al.

19. Röwekämper, J., Sprunk, C., Tipaldi, G.D., Stachniss, C., Pfaff, P., Burgard, W.:
On the position accuracy of mobile robot localization based on particle filters
combined with scan matching. In: 2012 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS), pp. 3158–3164. IEEE (2012)
20. Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings
of the Third International Conference on 3-D Digital Imaging and Modeling, pp.
145–152. IEEE (2001)
21. Thrun, S., et al.: Robotic mapping: a survey. Exploring Artif. Intell. New
Millennium 1, 1–35 (2002)
22. Wong, S.C., Middleton, L., MacDonald, B.A., Auckland, N.: Performance metrics
for robot coverage tasks. In: Proceedings of Australasian Conference on Robotics
and Automation, vol. 27, p. 29 (2002)
23. Wu, F., Jennings, N.R.: Regret-based multi-agent coordination with uncertain task
rewards. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence
(AAAI), Quebec City, Canada, pp. 1492–1499 (2014)
24. Wu, F., Ramchurn, S.D., Jiang, W., Fischer, J.E., Rodden, T., Jennings, N.R.:
Agile planning for real-world disaster response. In: Proceedings of the 24th Interna-
tional Joint Conference on Artificial Intelligence (IJCAI), Buenos Aires, Argentina,
pp. 132–138 (2015)
25. Wu, F., Zilberstein, S., Chen, X.: Trial-based dynamic programming for multi-agent
planning. In: Proceedings of the 24th AAAI Conference on Artificial Intelligence
(AAAI), Atlanta, United States, pp. 908–914 (2010)
26. Yang, S.X., Luo, C.: A neural network approach to complete coverage path plan-
ning. IEEE Trans. Syst. Man, Cybern. Part B (Cybern.) 34(1), 718–724 (2004)
27. Zhang, H., Cao, R., Zilberstein, S., Wu, F., Chen, X.: Toward effective soft robot
control via reinforcement learning. In: Proceedings of the 10th International Con-
ference on Intelligent Robotics Applications (2017)
28. Zheng, K., Chen, Y., Wu, F., Chen, X.: A general batch-calibration framework of
service robots. In: Proceedings of the 10th International Conference on Intelligent
Robotics Applications (2017)