Skip to main content

Jorge Lobo

University of Coimbra, ISR, DEEC,, Faculty Member

Followers

35

Following

14

Co-authors

12

Public Views

Jorge Lobo (Jorge Nuno de Almeida e Sousa Almada Lobo), born 23 Sep 1971, Cambridge, UK. Five-year course in Electrical Eng. at Coimbra University (UC) in 1995, MSc in 2002 and PhD in 2007. Assistant Professor at DEEC UC, teaching Digital Design, Microprocessors and Computer Architecture. Researcher at Institute of Systems and Robotics, ISR-UC, working in computer vision, sensor fusion, and mobile robotics. Current research interests focus on inertial sensor data integration in computer vision systems, Bayesian models for multimodal perception of 3D structure and motion, and real-time performance using GPUs and reconfigurable hardware. He has been involved on several EU projects, including BACS - Bayesian Approach to Cognitive Systems (FP6-IST-027140) and HANDLE - Developmental pathway towards autonomy and dexterity in robot in-hand manipulation (FP7-2008-231640) and currently in BAMBI - Bottom-up Approaches to Machines dedicated to Bayesian Inference (FET - FP7-ICT-2013-C, 618024).

less

InterestsView All (6)

Uploads

Papers by Jorge Lobo

Tele-3D-developing a handheld scanner using structured light projection

Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission

Three-dimensional surface reconstruction using a handheld scanner is a process with great potenti... more Three-dimensional surface reconstruction using a handheld scanner is a process with great potential for use on different fields of research, commerce and industrial production. In this article we will describe the evolution of a project comprising the study and development of a system that implements the aforementioned process based on two-dimensional images. We will present our current work on the development of a fully portable, handheld system using cameras, projected structured light and attitude and positioning measuring sensors-the Tele-3D scanner.

The CHOPIN project: Cooperation between human and rObotic teams in catastrophic incidents

Mobile robots can be an invaluable aid to human first responders (FRs) in catastrophic incidents,... more Mobile robots can be an invaluable aid to human first responders (FRs) in catastrophic incidents, as they are expendable and can be used to reduce human exposure to risk in search and rescue (SaR) missions, as well as attaining a more effective response. Moreover, parallelism and robustness yielded by multi-robot systems (MRS) may be very useful in this kind of spatially distributed tasks, which are distributed in space, providing augmented situation awareness (SA). However, this requires adequate cooperative behaviors, both within MRS teams and between human and robotic teams. Collaborative context awareness between both teams is crucial to assess information utility, efficiently share information and build a common and consistent SA. This paper presents the foreseen research within the CHOPIN research project, which aims to address these scientific challenges and provide a proof of concept for the cooperation between human and robotic teams in SaR scenarios.

Integration of inertial information with vision towards robot autonomy

Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes h... more Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes harder if the goal is to recover the dynamics of the 3D world from the image flow. However, it is known that humans integrate and combine the information from different sensorial systems to perceive the world. For example, tlie human vision system has close links with the wstibular system to perform everyday tasks. A computational approach for sensorial data

Editorial: Special issue: 2nd Workshop on integration of vision and inertial sensors

International Journal of Robotics Research, 2007

Research is devoted to the topic of Integration of Visual and Inertial Sensors. For robots, as fo... more Research is devoted to the topic of Integration of Visual and Inertial Sensors. For robots, as for animals, a sense of self-motion is critical to performing useful tasks in a complex and non-priory mapped world. The problems with sensing, particularly reliability and robustness, are well known to robotocists and sensor-fusion is a standard technique. Animals also make use of multiple sensory modalities, not just vision, smell, hearing, touch and taste but also joint position, muscle exertion, balance and motion. Some have even developed task-or environment-specific sensors such as acoustic ranging, magnetic dead-reckoning and electric fields. In this special issue we discuss two sensors that are common to animals and robots-vision and inertial sensing-which when combined provide rich information about self-motion and also the structure of the world. The advantage of these sensors is that they require no external infrastructure such as GPS satellites or visual or acoustic navigation beacons. Neither vision nor inertial sensors alone are perfect; however they have nice complementarities where each compensates for a shortcoming in the other. While inertial sensing is able to follow motions of any speed, its uncertainty-in particular at slow motions-is high compared to visual input. On the other hand visual sensors have problems with fast motions and find it difficult to distinguish between rotational and translational motion. Visual and inertial sensors today have high performance and are low cost and compact. They require no external reference and emit no radiation. Digital camera chips and micromachined accelerometers and gyroscopes are now commodities, and when combined with today's available computing can provide robust estimates of self-motion as well 3D scene structure.

Robotic Implementation of Biological Bayesian Models Towards Visuo-inertial Image Stabilization and Gaze Control

2008 IEEE International Conference on Robotics and Biomimetics, ROBIO 2008, 2009

Robotic implementations of gaze control and image stabilization have been previously proposed, th... more Robotic implementations of gaze control and image stabilization have been previously proposed, that rely on fusing inertial and visual sensing modalities. They are bioinspired in the sense that human and biological system also combine the two sensing modalities for the same goal. In this work we build upon these previous results and, with the contribution of psychophysical studies, attempt a more biomimetic approach to the robotic implementation. Since Bayesian models have been successfully used to explain psychophysical experimental findings, we propose a robotic implementation using Bayesian inference.

Probabilistic representation of 3D object shape by in-hand exploration

IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings, 2010

This work presents a representation of 3D object shape using a probabilistic volumetric map deriv... more This work presents a representation of 3D object shape using a probabilistic volumetric map derived from inhand exploration. The exploratory procedure is based on contour following through the fingertip movements on the object surface. We first consider the simple case of having single hand exploration of a static object. The cumulative pose data provides a 3D point cloud that is quantized to the probabilistic volumetric map. For each voxel we have a probability distribution for the occupancy percentage. This is then extended to in-hand exploration of non-static objects. Since the object is moving during the in-hand exploration, and we also consider the use of the other hand for re-grasping, object pose has to be tracked. By keeping track of object motion we can register data to the initial pose to build a consistent object representation. An object centered representation is implemented using the computed object center of mass to define its frame of reference. Results are presented for in-hand exploration of both static and non-static objects that show that valid models can be obtained. The 3D object probabilistic representation can be used in several applications related with grasp generation tasks.

Sensors for mobile robot navigation

Lecture Notes in Control and Information Sciences, 1998

Bayesian 3D independent motion segmentation with IMU-aided RBG-D sensor

2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2012

In this paper we propose a two-tiered hierarchical Bayesian model to estimate the location of obj... more In this paper we propose a two-tiered hierarchical Bayesian model to estimate the location of objects moving independently from the observer. Biological vision systems are very successful in motion segmentation, since they efficiently resort to flow analysis and accumulated prior knowledge of the 3D structure of the scene. Artificial perception systems may also build 3D structure maps and use optical flow to provide cues for ego-and independent motion segmentation. Using inertial and magnetic sensors and an image and depth sensor (RGB-D) we propose a method to obtain registered 3D maps, which are subsequently used in a probabilistic model (the bottom tier of the hierarchy) that performs background subtraction across several frames to provide a prior on moving objects. The egomotion of the RGB-D sensor is estimated starting with the angular pose obtained from the filtered accelerometers and magnetic data. The translation is derived from matched points across the images and corresponding 3D points in the rotation-compensated depth maps. A gyro-aided Lucas Kanade tracker is used to obtain matched points across the images. The tracked points can also used to refine the initial sensor based rotation estimation. Having determined the camera egomotion, the estimated optical flow assuming a static scene can be compared with the observed optical flow via a probabilistic model (the top tier of the hierarchy), using the results of the background subtraction process as a prior, in order to identify volumes with independent motion in the corresponding 3D point cloud. To deal with the computational load CUDAbased solutions on GPUs were used. Experimental results are presented showing the validity of the proposed approach.

A Probabilistic Framework to Detect Suitable Grasping Regions on Objects

IFAC Proceedings Volumes, 2012

This work relies on a probabilistic framework to search for suitable grasping regions on objects.... more This work relies on a probabilistic framework to search for suitable grasping regions on objects. In this approach, the object model is acquired based on occupancy grid representation that deals with the sensor uncertainty allowing later the decomposition of the object global shape into components. Through mixture distribution-based representation we achieve the object segmentation where the outputs are the point cloud clustering. Each object component is matched to a geometrical primitive. The advantage of representing object components into geometrical primitives is due to the simplification and approximation of the shape that facilitates the search for suitable object region for grasping given a context. Human demonstrations of predefined grasp are recorded and then overlaid on the object surface given by the probabilistic volumetric map to find the contact points of stable grasps. By observing the human choice during the object grasping, we perform the learning phase. Bayesian theory is used to identify a potential object region for grasping in a specific context when the artificial system faces a new object that is taken as a familiar object due to the primitives approximation into known components.

World feature detection and mapping using stereovision and inertial sensors

Robotics and Autonomous Systems, 2003

This paper explores the fusion of inertial information with vision for 3D reconstruction. A metho... more This paper explores the fusion of inertial information with vision for 3D reconstruction. A method is proposed for vertical line segment detection and subsequent local geometric map building. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovery of 3D structure from images, increasing the capabilities of autonomous vehicles and enlarging the application potential of vision systems. From the inertial sensors, a camera stereo rig, and a few system parameters we can recover the 3D parameters of the ground plane and vertical lines. The homography between stereo images of ground points can be found. By detecting the vertical line segments in each image, and using the homography of ground points for the foot of each segment, the lines can be matched and reconstructed in 3D. The mobile robot then maps the detected vertical line segments in a world map as it moves. To build this map an outlier removal method is implemented and a statistical approach used, so that a simplified metric map can be obtained for robot navigation.

Bayesian Models for Multimodal Perception of 3D Structure and Motion

by Pierre Bessiere, Jorge Lobo, and Christian Laugier

In this text we will formalise a novel solution, the Bayesian Volumetric Map (BVM), as a framewor... more In this text we will formalise a novel solution, the Bayesian Volumetric Map (BVM), as a framework for a metric, short-term, egocentric spatial memory for multimodal perception of 3D structure and motion. This solution will enable the implementation of top-down mechanisms of attention guidance of perception towards areas of high entropy/uncertainty, so as to promote active exploration of the environment by the robotic perceptual system. In the process, we will to try address the inherent challenges of visual, auditory and vestibular sensor fusion through the BVM. In fact, it is our belief that perceptual systems are unable to yield truly useful descriptions of their environment without resorting to a temporal series of sensory fusion processed on a short-term memory such as the BVM.

A Bayesian Framework for Active Artificial Perception

by Pierre Bessiere and Jorge Lobo

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society, Jan 1, 2012

In this text, we present a Bayesian framework for active multimodal perception of 3D structure an... more In this text, we present a Bayesian framework for active multimodal perception of 3D structure and motion. The design of this framework finds its inspiration in the role of the dorsal perceptual pathway of the human brain. Its composing models build upon a common egocentric spatial configuration that is naturally fitting for the integration of readings from multiple sensors using a Bayesian approach. In the process, we will contribute with efficient and robust probabilistic solutions for cyclopean geometry-based stereovision and auditory perception based only on binaural cues, modelled using a consistent formalisation that allows their hierarchical use as building blocks for the multimodal sensor fusion framework. We will explicitly or implicitly address the most important challenges of sensor fusion using this framework, for vision, audition and vestibular sensing. Moreover, interaction and navigation requires maximal awareness of spatial surroundings, which in turn is obtained through active attentional and behavioural exploration of the environment. The computational models described in this text will support the construction of a simultaneously flexible and powerful robotic implementation of multimodal active perception to be used in real-world applications, such as human-machine interaction or mobile robot navigation.

Recovering 3D Structure from Images and Inertial Sensors

Advanced sensor systems, exploring high integrity and multiple sensorial modalities, have been si... more Advanced sensor systems, exploring high integrity and multiple sensorial modalities, have been signicantly increasing the capabilities of autonomous vehicles and enlarging the application potential of vision systems. The article describes the cooperation between two relevant sensors -image and inertial sensors. Inertial sensors coupled to the active vision system can provide valuable information, as happens with the vestibular system in human and other animals. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovering of three-dimensional structure f r om images.

Camera-Inertial Sensor Modeling and Alignment for Visual Navigation

This article presents a technique for modeling and calibrating a camera with integrated low-cost ... more This article presents a technique for modeling and calibrating a camera with integrated low-cost inertial sensors, three gyros and three accelerometers for full 3D sensing. Inertial sensors attached to a camera can provide valuable data about camera pose and movement. In biological vision systems, inertial cues provided by the vestibular system, are fused with vision at an early processing stage. Vision systems in autonomous vehicles can also benefit by taking inertial cues into account. Camera calibration has been extensively studied, and standard techniques established. Inertial navigation systems, relying on high-end sensors, also have established techniques. Nevertheless, in order to use off-the-shelf inertial sensors attached to a camera, appropriate modeling and calibration techniques are required. For inertial sensor alignment, a pendulum instrumented with an encoded shaft is used to estimate the bias and scale factor of inertial measurements. For camera calibration, a standard and reliable camera calibration technique is used, based on images of a planar grid. Having both the camera and the inertial sensors calibrated and observing the vertical direction at different poses, the rigid rotation between the two frames of reference is estimated, using a mathematical model based on unit quaternions. The technique for this alignment and consequent results with simulated and real data are presented at the end of this article.

Inertial Information On Artificial Vision And Mobile Robots

Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes h... more Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes harder if the goal is to recover the dynamics of the 3D world from the image flow. However, it is known that humans integrate and combine the information from different sensorial systems to perceive the world. For example, the human vision system has close links with the vestibular system to perform every-day tasks. A computational approach for sensorial data integration, inertial and vision, is presented for a mobile robot equipped with an active vision system and inertial sensors. The inertial information is a different sensorial modality and, in this article, we explain our initial steps to combine this information with other sensorial systems, namely vision. Some of the benefits of using inertial information for navigation and dynamic visual processing are described in the article. During the development of these studies a low-cost inertial system prototype was developed. A brief description of low-cost inertial sensors and their integration in an inertial system prototype is also described. The set of sensors used in the prototype include three piezoelectric vibrating gyroscopes, a three-axial capacitive acceleromete and a dual axis clinometer. As a first approach the clinometer is used to track camera's pan and tilt, relative to a plane normal to the gravity vector and parallel to the ground floor. This provides the orientation data that, combined with image data, enables the identification of the ground plane or others parallel to it. The article also gives an algorithm that segments the image, identifying the floor along which the vehicle can move.

SoPHIE: Social Robotic Platform for Human Interactive Experimentation

Short-range DGPS for mobile robots with wireless Ethernet links

AMC'98 - Coimbra. 1998 5th International Workshop on Advanced Motion Control. Proceedings (Cat. No.98TH8354), 1998

For outdoor mobile robot applications the satellite based GPS system is available for position es... more For outdoor mobile robot applications the satellite based GPS system is available for position estimation. Differential GPS can add to the precision but requires a fixed receiver at a know position and some communication link. Typically differential GPS data is sent over dedicated radio links or broadcast. For shortrange applications, were multiple robots move around an outdoors workspace, a wireless ethernet link with multiple access points, and supporting multiple robots, is a good and cost-effective solution. The net can be used, amongst other things, for sending the differential dataji-om a local fwced receiver to the robots.

Segmentation of dense depth maps using inertial data a real-time implementation

IEEE/RSJ International Conference on Intelligent Robots and System, 2002

ABSTRACT

Grasp type identification and gesture recognition using distributed accelerometers

RA 2006-Tutorial on integration of vision and inertial sensors for robotic applications: August 14-16, 2006, Honolulu, Hawaii, USA

Tele-3D-developing a handheld scanner using structured light projection

Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission

Three-dimensional surface reconstruction using a handheld scanner is a process with great potenti... more Three-dimensional surface reconstruction using a handheld scanner is a process with great potential for use on different fields of research, commerce and industrial production. In this article we will describe the evolution of a project comprising the study and development of a system that implements the aforementioned process based on two-dimensional images. We will present our current work on the development of a fully portable, handheld system using cameras, projected structured light and attitude and positioning measuring sensors-the Tele-3D scanner.

The CHOPIN project: Cooperation between human and rObotic teams in catastrophic incidents

Mobile robots can be an invaluable aid to human first responders (FRs) in catastrophic incidents,... more Mobile robots can be an invaluable aid to human first responders (FRs) in catastrophic incidents, as they are expendable and can be used to reduce human exposure to risk in search and rescue (SaR) missions, as well as attaining a more effective response. Moreover, parallelism and robustness yielded by multi-robot systems (MRS) may be very useful in this kind of spatially distributed tasks, which are distributed in space, providing augmented situation awareness (SA). However, this requires adequate cooperative behaviors, both within MRS teams and between human and robotic teams. Collaborative context awareness between both teams is crucial to assess information utility, efficiently share information and build a common and consistent SA. This paper presents the foreseen research within the CHOPIN research project, which aims to address these scientific challenges and provide a proof of concept for the cooperation between human and robotic teams in SaR scenarios.

Integration of inertial information with vision towards robot autonomy

Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes h... more Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes harder if the goal is to recover the dynamics of the 3D world from the image flow. However, it is known that humans integrate and combine the information from different sensorial systems to perceive the world. For example, tlie human vision system has close links with the wstibular system to perform everyday tasks. A computational approach for sensorial data

Editorial: Special issue: 2nd Workshop on integration of vision and inertial sensors

International Journal of Robotics Research, 2007

Research is devoted to the topic of Integration of Visual and Inertial Sensors. For robots, as fo... more Research is devoted to the topic of Integration of Visual and Inertial Sensors. For robots, as for animals, a sense of self-motion is critical to performing useful tasks in a complex and non-priory mapped world. The problems with sensing, particularly reliability and robustness, are well known to robotocists and sensor-fusion is a standard technique. Animals also make use of multiple sensory modalities, not just vision, smell, hearing, touch and taste but also joint position, muscle exertion, balance and motion. Some have even developed task-or environment-specific sensors such as acoustic ranging, magnetic dead-reckoning and electric fields. In this special issue we discuss two sensors that are common to animals and robots-vision and inertial sensing-which when combined provide rich information about self-motion and also the structure of the world. The advantage of these sensors is that they require no external infrastructure such as GPS satellites or visual or acoustic navigation beacons. Neither vision nor inertial sensors alone are perfect; however they have nice complementarities where each compensates for a shortcoming in the other. While inertial sensing is able to follow motions of any speed, its uncertainty-in particular at slow motions-is high compared to visual input. On the other hand visual sensors have problems with fast motions and find it difficult to distinguish between rotational and translational motion. Visual and inertial sensors today have high performance and are low cost and compact. They require no external reference and emit no radiation. Digital camera chips and micromachined accelerometers and gyroscopes are now commodities, and when combined with today's available computing can provide robust estimates of self-motion as well 3D scene structure.

Robotic Implementation of Biological Bayesian Models Towards Visuo-inertial Image Stabilization and Gaze Control

2008 IEEE International Conference on Robotics and Biomimetics, ROBIO 2008, 2009

Robotic implementations of gaze control and image stabilization have been previously proposed, th... more Robotic implementations of gaze control and image stabilization have been previously proposed, that rely on fusing inertial and visual sensing modalities. They are bioinspired in the sense that human and biological system also combine the two sensing modalities for the same goal. In this work we build upon these previous results and, with the contribution of psychophysical studies, attempt a more biomimetic approach to the robotic implementation. Since Bayesian models have been successfully used to explain psychophysical experimental findings, we propose a robotic implementation using Bayesian inference.

Probabilistic representation of 3D object shape by in-hand exploration

IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings, 2010

This work presents a representation of 3D object shape using a probabilistic volumetric map deriv... more This work presents a representation of 3D object shape using a probabilistic volumetric map derived from inhand exploration. The exploratory procedure is based on contour following through the fingertip movements on the object surface. We first consider the simple case of having single hand exploration of a static object. The cumulative pose data provides a 3D point cloud that is quantized to the probabilistic volumetric map. For each voxel we have a probability distribution for the occupancy percentage. This is then extended to in-hand exploration of non-static objects. Since the object is moving during the in-hand exploration, and we also consider the use of the other hand for re-grasping, object pose has to be tracked. By keeping track of object motion we can register data to the initial pose to build a consistent object representation. An object centered representation is implemented using the computed object center of mass to define its frame of reference. Results are presented for in-hand exploration of both static and non-static objects that show that valid models can be obtained. The 3D object probabilistic representation can be used in several applications related with grasp generation tasks.

Sensors for mobile robot navigation

Lecture Notes in Control and Information Sciences, 1998

Bayesian 3D independent motion segmentation with IMU-aided RBG-D sensor

2012 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2012

In this paper we propose a two-tiered hierarchical Bayesian model to estimate the location of obj... more In this paper we propose a two-tiered hierarchical Bayesian model to estimate the location of objects moving independently from the observer. Biological vision systems are very successful in motion segmentation, since they efficiently resort to flow analysis and accumulated prior knowledge of the 3D structure of the scene. Artificial perception systems may also build 3D structure maps and use optical flow to provide cues for ego-and independent motion segmentation. Using inertial and magnetic sensors and an image and depth sensor (RGB-D) we propose a method to obtain registered 3D maps, which are subsequently used in a probabilistic model (the bottom tier of the hierarchy) that performs background subtraction across several frames to provide a prior on moving objects. The egomotion of the RGB-D sensor is estimated starting with the angular pose obtained from the filtered accelerometers and magnetic data. The translation is derived from matched points across the images and corresponding 3D points in the rotation-compensated depth maps. A gyro-aided Lucas Kanade tracker is used to obtain matched points across the images. The tracked points can also used to refine the initial sensor based rotation estimation. Having determined the camera egomotion, the estimated optical flow assuming a static scene can be compared with the observed optical flow via a probabilistic model (the top tier of the hierarchy), using the results of the background subtraction process as a prior, in order to identify volumes with independent motion in the corresponding 3D point cloud. To deal with the computational load CUDAbased solutions on GPUs were used. Experimental results are presented showing the validity of the proposed approach.

A Probabilistic Framework to Detect Suitable Grasping Regions on Objects

IFAC Proceedings Volumes, 2012

This work relies on a probabilistic framework to search for suitable grasping regions on objects.... more This work relies on a probabilistic framework to search for suitable grasping regions on objects. In this approach, the object model is acquired based on occupancy grid representation that deals with the sensor uncertainty allowing later the decomposition of the object global shape into components. Through mixture distribution-based representation we achieve the object segmentation where the outputs are the point cloud clustering. Each object component is matched to a geometrical primitive. The advantage of representing object components into geometrical primitives is due to the simplification and approximation of the shape that facilitates the search for suitable object region for grasping given a context. Human demonstrations of predefined grasp are recorded and then overlaid on the object surface given by the probabilistic volumetric map to find the contact points of stable grasps. By observing the human choice during the object grasping, we perform the learning phase. Bayesian theory is used to identify a potential object region for grasping in a specific context when the artificial system faces a new object that is taken as a familiar object due to the primitives approximation into known components.

World feature detection and mapping using stereovision and inertial sensors

Robotics and Autonomous Systems, 2003

This paper explores the fusion of inertial information with vision for 3D reconstruction. A metho... more This paper explores the fusion of inertial information with vision for 3D reconstruction. A method is proposed for vertical line segment detection and subsequent local geometric map building. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovery of 3D structure from images, increasing the capabilities of autonomous vehicles and enlarging the application potential of vision systems. From the inertial sensors, a camera stereo rig, and a few system parameters we can recover the 3D parameters of the ground plane and vertical lines. The homography between stereo images of ground points can be found. By detecting the vertical line segments in each image, and using the homography of ground points for the foot of each segment, the lines can be matched and reconstructed in 3D. The mobile robot then maps the detected vertical line segments in a world map as it moves. To build this map an outlier removal method is implemented and a statistical approach used, so that a simplified metric map can be obtained for robot navigation.

Bayesian Models for Multimodal Perception of 3D Structure and Motion

by Pierre Bessiere, Jorge Lobo, and Christian Laugier

In this text we will formalise a novel solution, the Bayesian Volumetric Map (BVM), as a framewor... more In this text we will formalise a novel solution, the Bayesian Volumetric Map (BVM), as a framework for a metric, short-term, egocentric spatial memory for multimodal perception of 3D structure and motion. This solution will enable the implementation of top-down mechanisms of attention guidance of perception towards areas of high entropy/uncertainty, so as to promote active exploration of the environment by the robotic perceptual system. In the process, we will to try address the inherent challenges of visual, auditory and vestibular sensor fusion through the BVM. In fact, it is our belief that perceptual systems are unable to yield truly useful descriptions of their environment without resorting to a temporal series of sensory fusion processed on a short-term memory such as the BVM.

A Bayesian Framework for Active Artificial Perception

by Pierre Bessiere and Jorge Lobo

IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society, Jan 1, 2012

In this text, we present a Bayesian framework for active multimodal perception of 3D structure an... more In this text, we present a Bayesian framework for active multimodal perception of 3D structure and motion. The design of this framework finds its inspiration in the role of the dorsal perceptual pathway of the human brain. Its composing models build upon a common egocentric spatial configuration that is naturally fitting for the integration of readings from multiple sensors using a Bayesian approach. In the process, we will contribute with efficient and robust probabilistic solutions for cyclopean geometry-based stereovision and auditory perception based only on binaural cues, modelled using a consistent formalisation that allows their hierarchical use as building blocks for the multimodal sensor fusion framework. We will explicitly or implicitly address the most important challenges of sensor fusion using this framework, for vision, audition and vestibular sensing. Moreover, interaction and navigation requires maximal awareness of spatial surroundings, which in turn is obtained through active attentional and behavioural exploration of the environment. The computational models described in this text will support the construction of a simultaneously flexible and powerful robotic implementation of multimodal active perception to be used in real-world applications, such as human-machine interaction or mobile robot navigation.

Recovering 3D Structure from Images and Inertial Sensors

Advanced sensor systems, exploring high integrity and multiple sensorial modalities, have been si... more Advanced sensor systems, exploring high integrity and multiple sensorial modalities, have been signicantly increasing the capabilities of autonomous vehicles and enlarging the application potential of vision systems. The article describes the cooperation between two relevant sensors -image and inertial sensors. Inertial sensors coupled to the active vision system can provide valuable information, as happens with the vestibular system in human and other animals. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovering of three-dimensional structure f r om images.

Camera-Inertial Sensor Modeling and Alignment for Visual Navigation

This article presents a technique for modeling and calibrating a camera with integrated low-cost ... more This article presents a technique for modeling and calibrating a camera with integrated low-cost inertial sensors, three gyros and three accelerometers for full 3D sensing. Inertial sensors attached to a camera can provide valuable data about camera pose and movement. In biological vision systems, inertial cues provided by the vestibular system, are fused with vision at an early processing stage. Vision systems in autonomous vehicles can also benefit by taking inertial cues into account. Camera calibration has been extensively studied, and standard techniques established. Inertial navigation systems, relying on high-end sensors, also have established techniques. Nevertheless, in order to use off-the-shelf inertial sensors attached to a camera, appropriate modeling and calibration techniques are required. For inertial sensor alignment, a pendulum instrumented with an encoded shaft is used to estimate the bias and scale factor of inertial measurements. For camera calibration, a standard and reliable camera calibration technique is used, based on images of a planar grid. Having both the camera and the inertial sensors calibrated and observing the vertical direction at different poses, the rigid rotation between the two frames of reference is estimated, using a mathematical model based on unit quaternions. The technique for this alignment and consequent results with simulated and real data are presented at the end of this article.

Inertial Information On Artificial Vision And Mobile Robots

Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes h... more Reconstructing 3D data from images acquired by cameras is a difficult task. The problem becomes harder if the goal is to recover the dynamics of the 3D world from the image flow. However, it is known that humans integrate and combine the information from different sensorial systems to perceive the world. For example, the human vision system has close links with the vestibular system to perform every-day tasks. A computational approach for sensorial data integration, inertial and vision, is presented for a mobile robot equipped with an active vision system and inertial sensors. The inertial information is a different sensorial modality and, in this article, we explain our initial steps to combine this information with other sensorial systems, namely vision. Some of the benefits of using inertial information for navigation and dynamic visual processing are described in the article. During the development of these studies a low-cost inertial system prototype was developed. A brief description of low-cost inertial sensors and their integration in an inertial system prototype is also described. The set of sensors used in the prototype include three piezoelectric vibrating gyroscopes, a three-axial capacitive acceleromete and a dual axis clinometer. As a first approach the clinometer is used to track camera's pan and tilt, relative to a plane normal to the gravity vector and parallel to the ground floor. This provides the orientation data that, combined with image data, enables the identification of the ground plane or others parallel to it. The article also gives an algorithm that segments the image, identifying the floor along which the vehicle can move.

SoPHIE: Social Robotic Platform for Human Interactive Experimentation

Short-range DGPS for mobile robots with wireless Ethernet links

AMC'98 - Coimbra. 1998 5th International Workshop on Advanced Motion Control. Proceedings (Cat. No.98TH8354), 1998

For outdoor mobile robot applications the satellite based GPS system is available for position es... more For outdoor mobile robot applications the satellite based GPS system is available for position estimation. Differential GPS can add to the precision but requires a fixed receiver at a know position and some communication link. Typically differential GPS data is sent over dedicated radio links or broadcast. For shortrange applications, were multiple robots move around an outdoors workspace, a wireless ethernet link with multiple access points, and supporting multiple robots, is a good and cost-effective solution. The net can be used, amongst other things, for sending the differential dataji-om a local fwced receiver to the robots.

Segmentation of dense depth maps using inertial data a real-time implementation

IEEE/RSJ International Conference on Intelligent Robots and System, 2002

ABSTRACT

Grasp type identification and gesture recognition using distributed accelerometers

RA 2006-Tutorial on integration of vision and inertial sensors for robotic applications: August 14-16, 2006, Honolulu, Hawaii, USA