Visual Servoing

Visual Servoing is generally contained of control and feature tracking. Study of previous methods shows that no attempt has been made to optimize these two parts together. In kernel based visual servoing method, the main objective is to combine and optimize these two parts together and to make an entire control loop. This main target is accomplished by using Lyapanov theory. A Lyapanov candidate function is formed based on kernel definition such that the Lyapanov stability can be verified. The implementation is done in four degrees of freedom and Fourier transform is used for decomposition of the rotation and scale directions from 2D translation. In the present study, a new method in scale and rotation correction is presented. Log-Polar Transform is used instead of Fourier transform for these two degrees of freedom. Tracking in four degrees of freedom is synthesized to show the visual tracking of an unmarked object. Comparison between Log-Polar transform and Fourier transform shows the advantages of the presented method. KBVS based on Log-Polar transform proposed in this paper, because of its robustness, speed and featureless properties.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments during robotized laparoscopic surgical operations. The instrument is mounted on the end-effector of a surgical robot which is controlled by visual servoing. The goal of the automated task is to safely bring the instrument at a desired 3-D location from an unknown or hidden position. Light Emitting Diodes are attached on the tip of the instrument and a speci£c instrument-holder £tted with optical £bers is used to project laser dots on the surface of the organs. These optical markers are detected in the endoscopic image and allow to localize the instrument with respect to the scene. The instrument is recovered and centered in the image plane by means of a visual servoing algorithm using feature errors in the image. With this system, the surgeon can specify a desired relative position between the instrument and the pointed organ. The relationship between the velocity screw of the surgical instrument and the velocity of the markers in the image is estimated online and, for safety reasons, a multi-stages servoing scheme is proposed. Our approach has been successfully validated in a real surgical environment by performing experiments on living tissues in the surgical training room of IRCAD.

Appearance-based visual learning and recognition techniques that are based on models derived from a training set of 2D images are being widely used in computer vision applications. In robotics, they have received most attention in visual servoing and navigation. In this paper we discuss a framework for visual self-localization of mobile robots using a parametric model built from panoramic snapshots of the environment. In particular, we propose solutions to the problems related to robustness against occlusions and invariance to the rotation of the sensor. Our principal contribution is an "eigenspace of spinning-images", i.e., a model of the environment which successfully exploits some of the specific properties of panoramic images in order to efficiently calculate the optimal subspace in terms of principal components analysis (PCA) of a set of training snapshots without actually decomposing the covariance matrix. By integrating a robust recover-and-select algorithm for the computation of image parameters we achieve reliable localization even in the case when the input images are partly occluded or noisy. In this way, the robot is capable of localizing itself in realistic environments.

The fast update rate and good performance of new generation electronic sector scanning sonars is now allowing practicable use of temporal information for signal processing tasks such as object classification and motion estimation. Problems remain, however, as objects change appearance, merge, maneuver, move in and out of the field of view, and split due to poor segmentation. This paper presents an approach to the segmentation, two-dimensional motion estimation, and subsequent tracking of multiple objects in sequences of sector scan sonar images. Applications such as ROV obstacle avoidance, visual servoing, and underwater surveillance are relevant. Initially, static and moving objects are distinguished in the sonar image sequence using frequency-domain filtering. Optical flow calculations are then performed on moving objects with significant size to obtain magnitude and direction motion estimates. Matches of these motion estimates, and the future positions they predict, are then used as a basis for identifying corresponding objects in adjacent scans. To enhance robustness, a tracking tree is constructed storing multiple possible correspondences and cumulative confidence values obtained from successive compatibility measures. Deferred decision making is then employed to enable best estimates of object tracks to be updated as subsequent scans produce new information. The method is shown to work well, with good tracking performance when objects merge, split, and change shape. The optical flow is demonstrated to give position prediction errors of between 10 and 50 cm (1%-5% of scan range), with no violation of smoothness assumptions using sample rates between 4 and 1 frames/s.

This paper describes the visual control of a parallel robot called "RoboTenis". The system has been designed and built in order to carry out tasks in three dimensions and dynamical environments; the system is capable to interact with objects which move up to 1m/s. The control strategy is composed by two intertwined control loops: The internal loop is faster and considers the information from the joints, its sample time is 0.5 ms. Second loop represents the visual servoing system, is external to the first mentioned and represents our study purpose, it is based on predicting the object velocity which is obtained form visual information, its sample time is 8.3 ms. Lyapunov stability analysis, system delays and saturation components has been taken into account.

In this paper, we present a method to navigate a mobile robot using a webcam. This method determines the shortest path for the robot to transverse to its target location, while avoiding obstacles along the way. The environment is first captured as an image using a webcam. Image processing methods are then performed to identify the existence of obstacles within the environment. Using the Cell Decomposition method, locations with obstacles are identified and the corresponding cells are eliminated. From the remaining cells, the shortest path to the goal is identified. The program is written in MATLAB with the Image Processing toolbox. The proposed method does not make use of any other type of sensor other than the webcam.

... However, when the stack is sealed in the CPU housing, access is limited. ... also a base technology that will be used in future ALJV development in the Underwater Technologies Laboratory at ... collision avoidance system for use on the Tuvaaq vehicle is also underway at the UTL. ...

This paper reports on designing a simulation and control robotic harvesting platform robotic harvesting of sweet pepper. The objective was to provide a simulated environment for improvement of plant/fruit scanning and visual servoing task through easy testing and debugging of control algorithms with zero damage risk to the real robot and to the actual equipment. A simulated workspace, including an exact replica of the FANUC LR Mate 200iD robot manipulator (with six degrees of freedom), models of sweet pepper fruit and plant system, and different vision sensors were created in the Virtual Robot Experimentation Platform (V-REP). The Robot Operating System (ROS) was used for exchanging data between the simulated environment and the real workspace via it’s publish and subscribe architecture. Results provided a platform for testing new ideas in the robotic harvesting of sweet pepper, and for experimenting with different sensing instrumentation and different control strategies. Three summarized demos are provided for (i) fruit/plant scan (Demo link), (ii) for visual servo control task in detection, localization and tracking (Demo link), and (iii) for Ubuntu/VREP-ROS environment setup

Demo link: https://youtu.be/tKagjNQ9FW4).
Presentation file: https://youtu.be/sNBtU3YjGzk

The design of reliable navigation and control systems for Unmanned Aerial Vehicles (UAVs) based only on visual cues and inertial data has many unsolved challenging problems, ranging from hardware and software development to pure control-theoretical issues. This paper addresses these issues by developing and implementing an adaptive visionbased autopilot for navigation and control of small and mini rotorcraft UAVs. The proposed autopilot includes a Visual Odometer (VO) for navigation in GPS-denied environments and a nonlinear control system for flight control and target tracking. The VO estimates the rotorcraft ego-motion by identifying and tracking visual features in the environment, using a single camera mounted on-board the vehicle. The VO has been augmented by an adaptive mechanism that fuses optic flow and inertial measurements to determine the range and to recover the 3D position and velocity of the vehicle. The adaptive VO pose estimates are then exploited by a nonlinear hierarchical controller for achieving various navigational tasks such as take-off, landing, hovering, trajectory tracking, target tracking, etc. Furthermore, the asymptotic stability of the entire closed-loop system has been established using systems in cascade and adaptive control theories. Experimental flight test data over various ranges of the flight envelope illustrate that the proposed vision-based autopilot performs well and allows a mini rotorcraft UAV to achieve autonomously advanced flight behaviours by using vision.

Diverse image-based tracking schemes of robot moving in free motion have been proposed, and experimentally validated, whose position and velocity image tracking errors converge to zero. However, visual servoing for constrained motion robot tasks has not been addressed so as to provide control schemes that guarantee simultaneous tracking of position, velocity and contact force trajectories for dynamic robot models. The main difficulty may steam from the fact that camera information cannot be used to drive force trajectories.Recognizing this fact, in this paper a unique error manifold that includes position-velocity errors and force errors in orthogonal complements is proposed under the framework of passivity, to yield a synergetic scheme that fuses camera, encoder and force sensor signals. This seemingly fusion of all tracking errors into a unique error variable allows to propose a new control system which guarantees local exponential tracking of all error trajectories. A neural network, driven by an orthogonalized second order sliding mode surface is derived to compensate approximately for nonlinear robot dynamics. Residual errors that arises because of the finite size o the neural network are finally eliminated via two sliding modes. Simulations results are presented and discussed.

The methods required to make humans and robots interact and collaborate are the subject of research in physical human-robot interaction (pHRI). One of the main issues, particularly when physical contact can occur between the two, is safety. A second fundamental requirement to establish pHRI is intuitive control by the human operator, in particular when s/he is non expert. Sensor-based control can address both issues, by closing the perception-to-action loop, in a way that may well be assimilated to that of the lower motor neurons in our nervous system. Traditionally, heterogeneous sensor data was fed to fusion algorithms (e.g., Kalman or Bayesian-based), so as to provide state estimation for modeling the environment. However, since robot sensors generally measure different physical phenomena, it is preferable to use them directly in the low-level servo controller rather than to apply them to multi-sensory fusion or to design complex state machines. This idea, originally proposed in the hybrid position-force control paradigm, when extended to multiple sensors brings new challenges to the control design; challenges related to the task representation and to the sensor characteristics (synchronization, hybrid control, task compatibility, etc.). The rationale behind my work has precisely been to use sensor-based control as a means to facilitate the physical interaction between robots and humans. In particular, I have utilized vision, proprioceptive force, touch and distance to address pHRI case studies, targeting different applications and robot platforms. My research has followed four main axes: teach-and-repeat navigation of wheeled mobile robots, collaborative industrial manipulation with safe physical interaction, force and visual control for interacting with humanoid robots, and shared robot control. Each of these axes will be presented here, before concluding with a general view of the issues at stake, and on the research projects that I plan to carry out in the upcoming years.

In this paper, we describe the HRI (Human-Robot Interaction) system developed to operate a humanoid robot head capable of visual tracking and servoing of human face through image processing. The robotic humanoid head named Amir-II, equipped with a camera and servoing mechanism is used as the platform. The Amir-II tracks the human face within the field-of-vision (FOV) while the servoing mechanism ensures the detected human face remains at the center of its FOV. The algorithm developed in this research utilizes the capability offered by scientific computing program MATLAB along with its Image Processing Toolbox. The algorithm basically compares the locations of the face in the image plane that is detected from the static face image captured from real-time video stream. The calculated difference is then used to produce appropriate motion command for the servo mechanism to keep track of the human face moving within the range of its FOV.

We consider the task-oriented modeling of the differential kinematics of nonholonomic mobile manipulators (NMMs). A suitable NMM Jacobian is defined that relates the available input commands to the time derivative of the task variables, and can be used to formulate and solve kinematic control problems. When the NMM is redundant with respect to the given task, we provide an extension of two well-known redundancy resolution methods for fixed-base manipulators (Projected Gradient and Task Priority) and introduce a novel technique (Task Sequencing) aimed at improving performance, e.g., avoiding singularities. The proposed methods are applied then to the specific case of image-based visual servoing, where the NMM image Jacobian combines the interaction matrix and the kinematic model of the mobile manipulator. Comparative numerical results are presented for two case studies.

This paper presents a complete design methodology for Cartesian position based visual servo control for robots with a single camera mounted at the end-effector. Position based visual servo control requires the explicit calculation of the relative position and orientation (POSE) of the work-piece object with respect to the camera. This is accomplished using image plane measurements of a number of known feature points on the object, and then applying an extended Kalman filter to obtain a recursive solution of the photogrammetric equations, and to properly combine redundant measurements. The control is then designed by specifying the desired trajectories with respect to the object and forming the control error in the end-effector frame. The implementation using a distributed computer architecture is described. An experimental system has been built and used to evaluate the performance of the POSE estimation and the position based visual servo control. Several results for relative trajectory control and target tracking are presented. The results of experiments showing the effect of loss of some of the redundant features is also presented.

In this paper a nonlinear suboptimal stabilizing control strategy based on Control Lyapunov Functions (CLF) is synthesized and applied to a quadrotor helicopter. Sufficient conditions are obtained for this control law to ensure the This work was partially supported by CuMex.

This work presents first in-vivo results of beating heart tracking with a surgical robot arm in off-pump cardiac surgery. The tracking is performed in a 2D visual servoing scheme using a 500 frame per second video camera. Heart motion is measured by means of active optical markers that are put onto the heart surface. Amplitude of the motion is evaluated along the two axis of the image reference frame. This is a complex and fast motion that mainly reflects the influence of both the respiratory motion and the electro-mechanical activity of the myocardium. A model predictive controller is setup to track the two degrees of freedom of the observed motion by computing velocities for two of the robot joints. The servoing scheme takes advantage of the ability of predictive control to anticipate over future references provided they are known or they can be predicted. An adaptive observer is defined along with a simple cardiac model to estimate the two components of the heart motion. The predictions are then fed into the controller references and it is shown that the tracking behaviour is greatly improved.

In this paper, we describe the HRI (Human-Robot Interaction) system developed to operate a humanoid robot head capable of visual tracking and servoing of human face through image processing. The robotic humanoid head named Amir-II, equipped with a camera and servoing mechanism is used as the platform. The Amir-II tracks the human face within the field-of-vision (FOV) while the servoing mechanism ensures the detected human face remains at the center of its FOV. The algorithm developed in this research utilizes the ...

This paper proposes a method for the visual-based navigation of a mobile robot in indoor environments, using a single omni-directional (catadioptric) camera. The geometry of the catadioptric sensor and the method used to obtain a bird's eye (orthographic) view of the ground plane are presented. This representation significantly simplifies the solution to navigation problems, by eliminating any perspective effects.

In this paper a real-time image based visual servoing scheme for a nonholonomic mobile robot is presented. The proposed method takes into account the nonholonomic motion constraint of mobile robots. As a distinctive fact it does not require the estimation and decomposition of the homography or fundamental matrix. A switched image based controller is design, using point features, in order to solve the camera field-of-view (FOV) constraint. The proposed architecture was implemented using a real-time visual servoing system and multiple test were conducted. Experimental results are revealed and commented.

This paper presents the development of an Image Based Visual Servoing control law for a differential drive mobile robot navigation
using single camera attached on the robot platform. Four points image features have been used to compute the actuation control
signals: the angular velocities of the right wheel and the left wheel. The actuation control signals move the mobile robot at the
desired position such that the error vector in the image space has been minimised. The stability of the proposed IBVS control law
has been validated in the sense of the Lyapunov stability. Simulations and real-time experiments have been carried out to verify the
performances of the proposed control algorithm. Visual servoing platform (ViSP) libraries have been used to develop the simulation
program. Real-time experiments have been conducted using a differential drive mobile robot where a Beaglebone Black Board was
used as the main hardware controller.

This paper presents a visual servoing method based on 2D ultrasound images. The main goal is to guide a robot actuating a 2D ultrasound probe in order to reach a desired cross-section image of an object of interest. The method we propose allows the control of both in-plane and out-of-plane probe motions. Its feedback visual features are combinations of moments extracted from the observed image. The exact analytical form of the interaction matrix that relates the image moments time variation to the probe velocity is developed, and six independent visual features are proposed to control the 6 degrees of freedom of the robot. In order to endow the system with the capability of automatically interacting with objects of unknown shape, a model-free visual servoing is developed. For that, we propose an efficient on-line estimation method to identify the parameters involved in the interaction matrix. Results obtained in both simulations and experiments validate this work and show the robustness of the developed servo method to different errors and perturbations, especially those inherent to the noisy ultrasound images.

Previous homography-based visual servo controllers have been developed using an error system that contains a singularity resulting from the representation of the rotation matrix. For some aerospace applications such as visual servo control of satellites or air vehicles, the singularity introduced by the rotation representation may be restrictive. To eliminate this singularity, a homography-based visual servo controller is developed in this paper based on an error system composed of the unit quaternion representation. The proposed adaptive controller regulates a camera to a desired position and orientation that is determined from a desired image. A quaternion-based Lyapunov function is developed to facilitate the control design and the stability analysis.

This paper addresses the problem of visual servo control for a humanoid robot in an unstructured domestic environment. The important issues in this application are autonomous planning, robustness to camera and kinematic model errors, large pose errors, occlusions and reliable visual tracking. Conventional image-based or position-based visual servoing schemes do not address these issues, which motivated the proposed hybrid position-based scheme exploiting fusion of visual and kinematic measurements. Kinematic measurements provide robustness to visual distractions, and allow servoing to continue when the end-effector leaves the field of view. Visual measurements provide the complementary benefits of accurate pose tracking and online estimation of the hand-eye transformation for kinematic calibration. Furthermore, it is shown that calibration errors in the focal length and baseline can be approximated as an unknown scale of the end-effector, which can be estimated in the tracking filter to overcome camera calibration errors. The improved accuracy and robustness compared to conventional positionbased servoing is demonstrated experimentally.

The paper gives a survey on the new Expectationbased Multifocal Saccadic Vision (EMS-Vision) system for autonomous vehicle guidance developed at the Universität der Bundeswehr München (UBM). EMS-Vision is the third generation dynamic vision system following the 4-D approach. Its core element is a new camera arrangement, mounted on a high bandwidth pan-tilt head for active gaze control. Central knowledge representation and a hierarchical system architecture allow efficient activation and control of behavioral capabilities for perception and action. The system has been implemented on commercial off-the-shelf (COTS) hardware components in both UBM test vehicles VaMoRs and VaMP. Results from autonomous turnoff maneuvers, performed on army proving grounds, are discussed.

Intelligent autonomous vehicles have received a great degree of attention in recent years. Although the technology required for these vehicles is relatively advanced, the challenge is firstly to ensure that drivers can understand the capabilities and limitations of such systems and secondly to design a system that can handle the interaction between the driver and the automated intelligent system. In this study, we describe an approach using different strategies for an autonomous system and a driver to drive a vehicle cooperatively. The proposed strategies are referred to as cooperative planning and control and determine when and how the path projected by the autonomous system can be changed safely by the driver to a path that he wishes to follow. The first phase of the project is described, covering the design and implementation of an autonomous test vehicle. Experiments are carried out with a driver to test the cooperative planning and control concepts proposed here.

In this paper an image predictive controller for an eye-in-hand type servoing architecture, composed from a 6 degree of freedom robot and a camera mounted on the gripper, is presented. A novel architecture for integrating reference trajectory and image prediction is proposed for use in predictive control of visual servoing systems. In the proposed method, a new predictor is developed based on the relation between the camera velocity and the time variation of the visual features given by the interaction matrix. The image based predictor generates the future trajectories of a visual feature ensemble when past and future camera velocities are known. In addition, a reference trajectory is introduced to define the way how to reach the desired features over the prediction horizon starting from the current features. The advantages of the new architecture are the reference trajectory used for the first time in the sense of the predictive control and the predictor based on a local model. Simulations reveal the efficiency of the proposed architecture to control a 6 degrees of freedom robot manipulator.

A robot end-mounted camera is used to position the robot arm in a plane orthogonal to the axis, such that the fruit to be harvested is centered in the camera’s field of view. The system has no trajectory generator, instead a feedback loop, closed visually, is used to control the robots arm position. (Two control laws were simulated: join angles and joint velocity)

Visually guided robotic capturing of a moving object often requires long-term prediction of the object motion not only for a smooth capture but because visual feedback may not be continually available, e.g., due to vision obstruction by the robotic arm, as well. This paper presents a combined prediction and motion-planning scheme for robotic capturing of a drifting and tumbling object with unknown dynamics using visual feedback. A Kalman filter estimates the states and a set of dynamics parameters of the object needed for long-term prediction of the motion from noisy measurements of a vision system. Subsequently, the estimated states, parameters, and predicted motion trajectories are used to plan the trajectory of the robot's end-effector to intercept a grapple fixture on the object with zero relative velocity (to avoid impact) in an optimal way. The optimal trajectory minimizes a cost function, which is a weighted linear sum of travel time, distance, cosine of a line-of-sight angle (object alignment for robotic grasping), and a penalty function acting as a constraint on acceleration magnitude. Experiments are presented to demonstrate the robot-motion planning scheme for autonomous grasping of a tumbling satellite. Two robotics manipulators are employed: One arm drifts and tumbles the mockup of a satellite, and the other arm that is equipped with a robotic hand tries to capture a grapple fixture on the satellite using the visual guidance system.

Abstract— In this paper, we present a controller for visual navigation, which utilizes a time-independent varying reference in the feedback,law. The navigation,framework,relies on a monocular camera, and the path is represented as a series of key images. The varying reference is determined,using a vector field, derived from the previous and next key images. Results in a simulated environment, as well

We study visual servoing in a framework of detection and grasping of unknown objects. Classically, visual servoing has been used for applications where the object to be servoed on is known to the robot prior to the task execution. In addition, most of the methods concentrate on aligning the robot hand with the object without grasping it. In our work, visual servoing techniques are used as building blocks in a system capable of detecting and grasping unknown objects in natural scenes. We show how different visual servoing techniques facilitate a complete grasping cycle.

In this paper, we show that visual feedback reduces the complexity of parallel robot Cartesian control. Namely, 3D pose visual servoing, where the end-effector pose is indirectly measured and used for regulation, is shown to be well suited to this task since it relieves the control from the difficult forward kinematic problem. Moreover, this complexity reduction is not coming with an increase of the implementation complexity since off-the-shelf hardware and software are now available for visual servoing. It is also shown that such a control gets rid of joint sensors. All this makes 3D pose visual servoing the most straightforward Cartesian control for parallel robots. Experimental results are provided using an open source visual servoing C++ library.

A dynamic Newton's approximation based estimation scheme is proposed for image-based visual servoing (IBVS) of a redundant manipulator. The estimation scheme approximates the kinematic Jacobian from joint space to vision space online. The estimated Jacobian is used for redundancy resolution with weighted least norm solution. This work discusses the kinematic limit avoidance with estimated Jacobian for a 7 DOF PowerCube manipulator, hence presenting an approach with reduced computational burden compared to traditional model based visual servoing approaches.

This paper presents a new visual servoing method that is able to stabilize a moving area of soft tissue within an ultrasound B-mode imaging plane. The approach consists of moving the probe in order to minimize the relative position between a target imaging plane and the ultrasound plane observed by the probe of the moving tissue target. The problem is decoupled into motion out-of-plane and motion within plane. For the former, a new original method based on the speckle information contained in the images is developed. For the latter, an image region tracker is used to provide the in-plane motion. A visual servoing control scheme is then developed to perform the tracking robotic task. The method is validated on simulated motions of a probe on a static ultrasound volume acquired from a phantom.

Visual servoing, or the control of motion on the basis of image analysis in a closed loop, is more and more recognized as 9 an important tool in modern robotics. Here, we present a new model-driven approach to derive a description of the motion 10 of a target object. This method can be subdivided into an illumination invariant target detection stage and a servoing process 11 which uses an adaptive Kalman filter to update the model of the non-linear system. This technique can be applied to any 12 pan-tilt zoom camera mounted on a mobile vehicle as well as to a static camera tracking moving environmental features.

Resumen La percepción de profundidad se hace imprescindible en muchas tareas de manipulación, control visual y navegación de robots. Las cámaras de tiempo de vuelo (ToF: Time of Flight) generan imágenes de rango que proporcionan medidas de profundidad en tiempo real. No obstante, el parámetro distancia que calculan estas cámaras es fuertemente dependiente del tiempo de integración que se configura en el sensor y de la frecuencia de modulación empleada por el sistema de iluminación que integran. En este artículo, se presenta una metodología para el ajuste adaptativo del tiempo de integración y un análisis experimental del comportamiento de una cámara ToF cuando se modifica la frecuencia de modulación. Este método ha sido probado con éxito en algoritmos de control visual con arquitectura 'eye-in-hand' donde el sistema sensorial está compuesto por una cámara ToF. Además, la misma metodología puede ser aplicada en otros escenarios de trabajo.

In this paper we argue that the focus of much of the visual servoing literature has been on the kinematics of visual control and has ignored a number of fundamental and significant dynamic control issues. To this end the concept of visual dynamic control is introduced which is concerned with dynamic effects due to the manipulator and machine vision sensor which limit performance. These must be explicitly addressed in order to achieve high-performance control.

This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control