Terrain Learning Vehicle
This project proposes a self-supervised machine learning architecture which samples and learns a vehicle and terrain relationship from experience for an off road vehicle. The original thesis can be downloaded from here and the source code and run instructions can be found here.
The motivation for this project is that mobile robots are no longer confined to just factory floors anymore and are increasingly being used in unstructured and unknown environments, such as urban, farmland and even other planets! (see figure 1) As such there is a requirement for accurate and robust vehicle-terrain prediction mechanisms.
Learning Architecture
The vehicle (real or virtual) makes some pass over the terrain, which is used to calculate a traversability label. This score is mapped back onto images of the terrain and used as labels to train a CNN to predict terrain traversal for new images (Figure 2).
Traversal Data from Simulation
The data used to train this model was collected using a computer simulation, but the same process could be applied to data from a real robot.
Simulation Setup - Simulated using Gazebo and ROS (Robot Operating System). The Pioneer 3-AT skid-steer robot was modelled (Figure 3) as it can later be used in real life.
DeepScene Forest Dataset - Contains 350 real RGB, depth and ground truth images (Figure 4).
Terrain Generation - Depth images converted into 3D meshes (Figure 5) and loaded into simulation.
Traversal Training - Robot drives over terrain mesh, and the traversal labels are mapped back onto terrain image (Figure 6). This traversal data is required to train the CNN (see section 4).
Convolutional Neural Network
Architecture -The CNN architecture is based upon a height map which uses a Keras front end and TensorFlow backend. This is a standard CNN structure which is well suited for image feature recognition (see the exact architecture on Figure 7). 
Training - The CNN was trained for 9 hours with 52,000 samples (around 13km of driving simulations), and eventually achieved an accuracy of around 70% (see Figure 8). 
Results
The CNN was trained for both uniform (𝜇 = 1.0) and variable terrain friction coefficients (see Table 1)
Path Planning - A simple Dijkstra algorithm is demonstrated with traversability weightings applied:
