Work From The ACFR
An Overview of the work I performed during my PhD at the Australian Center for Field Robotics
My PhD Research
From 2012 to 2015, while working towards my PhD at the Australian Center for Field Robotics I performed research into the alignment and calibration of multi-modal sensors. The methods developed provide a means of calibrating 2D and 3D sensors, require no special targets or other calibration aids and give a measure of the uncertainty in the resulting calibration.
A few examples of my research from this period and its applications are shown in the images below:
A few examples of my research from this period and its applications are shown in the images below:
-
2D Feature Matching used in the motion estimation pipeline -
A lidar scan of the Sydney Operahouse that was colored using our automated registration approach and an image sourced from the internet. -
IR and color images being aligned using NMI -
A virtual camera generating an image from pointcloud data -
The final result of fusing calibrated lidar, camera and nav data in the KITTI dataset -
A lidar scan of the ACFR with intensity information from the visible bands of a hyperspectral camera -
Two bands of a hyperspectral camera image. The coloring of the sky and trees provides a good example of why simple error metrics such as NCC fail for these alignments -
The Sydney University Great Hall lidar scan being aligned and assigned image intensities -
Top: alignment using motion information. Bottom: result of intensity based refinement -
Sensor setup for obtaining lidar and hyperspectral data of an area -
The ACFR Shrimp robot collecting data in an agricultural field -
Verification of the lidar alignment of a new sensor vehicle
ACFR Publications
All of the papers I published while working on my PhD at the ACFR
This thesis presents a novel system for calibrating the extrinsic parameters of an array of cameras, 3D lidars and GPS/INS sensors without the requirement for any markers or other calibration aids.
To achieve this, a new multi-modal metric, the gradient orientation measure is first presented. This metric operates by minimising the misalignment of gradients between the outputs of two candidate sensors and is able to handle the inherent differences in how sensors of different modalities perceive the world.
This metric is successfully demonstrated on a range of calibration problems, however to calibrate the systems in a reliable manner the metric requires an initial estimate to the solution and a constrained search space. These constraints are required as repeated and similar structure in the environment in combination with the limited field of view of the sensors result in the metric's cost function being non-convex. This non-convexity is an issue that affects all appearance-based markerless methods.
To overcome these limitations a second cue to the sensors' alignment is taken, the motion of the system. By estimating the motion that each individual sensor observes, an estimate of the extrinsic calibration of the sensors can be obtained. In this thesis standard techniques for this motion-based calibration (often referred to as hand-eye calibration) are extended by incorporating estimates of the accuracy of each sensor's readings. This allows the development of a probabilistic approach that calibrates all sensors simultaneously. The approach also facilities the estimation of the uncertainty in the final calibration. Finally, this motion-based approach is combined with appearance-based information to build a novel calibration framework. This framework does not require initialisation and can take advantage of all available alignment information to provide an accurate and robust calibration for the system.
The ability to map clay minerals on vertical geological surfaces is important from perspectives of stratigraphic mapping and safety. Clay minerals were mapped from hyperspectral imagery using Automated Feature Extraction and their areal coverage estimated on a complex geological surface (a mine pit) by automatically registering hyperspectral to LiDAR data. The area of the mine pit covered by each identified mineral was under- or over-estimated by as much as a factor of 2 when derived from the hyperspectral imagery alone compared to imagery co-registered to LiDAR data. Hyperspectral imagery enabled the identification of clay layers on a mine face as a means of separating geological units of similar visual or spectral characteristics.
This paper formulates a new pipeline for automated extrinsic calibration of multi-sensor mobile platforms. The new method can operate on any combination of cameras, navigation sensors and 3D lidars. Current methods for extrinsic calibration are either based on special markers and/or chequerboards, or they require a precise parameters initialisation for the calibration to converge. These two limitations prevent them from being fully automatic. The method presented in this paper removes these restrictions. By combining information extracted from both, platform’s motion estimates and external observations, our approach eliminates the need for special markers and also removes the need for manual initialisation. A third advantage is that the motion-based automatic initialisation does not require overlapping field of view between sensors. The paper also provides a method to estimate the accuracy of the resulting calibration. We illustrate the generalisation of our approach and validate its performance by showing results with two contrasting datasets. The first dataset was collected in a city with a car platform, and the second one was collected in a tree-crop farm with a Segway platform.
ICRA 2015 Ag Workshop:
Real-Time Target Detection and Steerable Spray for Vegetable Crops
This paper presents a system for autonomously delivering a quantum of fluid to individual plants in a vegetable crop field, using a fine spray nozzle attached to a manipulator arm on a mobile robot. This can reduce input cost and environmental impact, while increasing productivity, wherever blanket spraying can be replaced with targeted spray, in applications including thinning, micro-nutrient provision and weeding. The Ladybird platform is introduced and a pipeline from image-based seedling detection, geometry and transformation including camera-arm calibration, inverse-kinematics and target sequence optimisation.
The ability to map clay minerals on vertical geological surfaces is important from perspectives of stratigraphical mapping of geological units and safety. Clay minerals represent lines of stratigraphical weakness along which landslides can occur. To map clay minerals on complex geological surfaces we use a combination of hyperspectral and LiDAR data. These data are automatically registered to provide a map of the distribution of clay minerals and their abundances at different spatial scales.
ICRA 2014 LTA Workshop:
Parameterless Automatic Extrinsic Calibration of Vehicle Mounted Lidar-Camera Systems
This paper presents a new method for automated extrinsic calibration of multi-modal sensors. In particular the paper presents and evaluates a pipeline for calibration of 3D lidar and cameras mounted on a sensor vehicle. Previous methods for multi-modal sensor calibration find the optimal parameters by aligning a set of observations from the different sensor modalities. The main drawback of these methods is the need for a good initialisation in order to avoid converging into a local minima. Our approach eliminates this limitation by combining external observations with motion estimates obtained with the individual sensors. The method operates by utilizing structure from motion based hand-eye calibration to constrain the search space of the optimisation.
ICRA 2014 MEPC Workshop:
Gradient Based Multi-Modal Sensor Calibration
This paper presents an evaluation of a new metric for registering two sensors of different modality. The metric operates by aligning gradients present in the two sensors’ outputs. This metric is used to find the parameters between the sensors that minimizes the misalignment of the gradients. The metric can be applied to a wide range of problems and has been successfully demonstrated on the extrinsic calibration of two different lidar-camera systems as well as the alignment of IR and RGB images. Unlike most of previous techniques, our method requires no markers to be placed in the scene and can operate on a single scan from each sensor.
This paper presents a new metric for automated registration of multi-modal sensor data. The metric is based on the alignment of the orientation of gradients formed from the two candidate sensors. Data registration is performed by estimating the sensors’ extrinsic parameters that minimises the misalignment of the gradients. The metric can operate in a large range of applications working on both 2D and 3D sensor outputs and is suitable for both (i) single scan data registration and (ii) multi-sensor platform calibration using multiple scans. Unlike traditional calibration methods, it does not require markers or other registration aids to be placed in the scene. The effectiveness of the new method is demonstrated with experimental results on a variety of camera-lidar and camera-camera calibration problems. The novel metric is validated through comparisons with state of the art methods. Our approach is shown to give high quality registrations under all tested conditions.
A novel technique for calibrating a multi-modal sensor system has been developed. Our calibration method is based on the comparative alignment of output gradients from two candidate sensors. The algorithm is applied to the calibration of the extrinsic parameters of several camera-lidar systems. In this calibration the lidar scan is projected onto the camera's image using a camera model. Particle swarm optimization is used to find the optimal parameters for this model. This method requires no markers to be placed in the scene. While the system can use a set of scans, unlike many existing techniques it can also automatically calibrate the system reliably using a single scan. The method presented is successfully validated on a variety of cameras, lidars and locations. It is also compared to three existing techniques and shown to give comparable or superior results on the datasets tested.
This paper presents a multi-class image segmentation approach to automate fruit segmentation. A feature learning algorithm combined with a conditional random field is applied to multi-spectral image data. Current classification methods used in agriculture scenarios tend to use hand crafted application-based features. In contrast, our approach uses unsupervised feature learning to automatically capture most relevant features from the data. This property makes our approach robust against variance in canopy trees and therefore has the potential to be applied to different domains. The proposed algorithm is applied to a fruit segmentation problem for a robotic agricultural surveillance mission, aiming to provide yield estimation with high accuracy and robustness against fruit variance. Experimental results with data collected in an almond farm are shown. The segmentation is performed with features extracted from multi-spectral (colour and infrared) data. We achieve a global classification accuracy of 88%.
This paper presents a method for calibrating the extrinsic and intrinsic parameters of a camera and a lidar scanner. The approach uses normalised mutual information to compare an image with a lidar scan. A camera model that takes into account orientation, location and focal length is used to create a 2D lidar image, with the intensity of the pixels defined by the angle of the normals in the lidar scan. Particle swarm optimisation is used to find the optimal model parameters. The method is successfully validated in a natural environment with images collected by a hyperspectral and a 3D lidar scanner.
This paper is about automatic calibration of a camera-lidar system. The method presented is designed to be as general as possible allowing it to be used in a large range of systems and applications. The approach uses normalized mutual information to compare the cameras image with an image generated by a lidar scan of the same area. A camera model that takes into account orientation, location and focal length is used to create the lidar image, with the intensity of the pixels in the image representing a feature of the lidar scan that is chosen depending on the application. Particle swarm optimization is used to find the optimal model parameters. The method presented is successfully validated on a variety of cameras, lidars and locations, including scans of both urban and natural environments.