Road defects detection

Automatic road defect detection software was developed for one of the leading russian research group.

Computer vision
Deep learning
Torch
CUDA

2015

In cooperation with

About project

Task

To automate manual work of operators who search and select road defects from road laboratories video footage, like cracks, holes and patches.

Solution

  • Software for creating training set of defects
  • Torch, C++ and CUDA-based software for defects detection model training
  • Near real time algorithm for quick detection of defects from video
  • Web dashboard for monitoring and controlling GPU cluster: model training, running defects detection, view result
  • 6000 km of roads have been already successfully processed

Long roads, big problems

As they say, there are two major problems in Russia, and one of them are bad roads. According to Wikipedia, the total length of Russia's road network is 1,396,000 km. Russia is ranked fifth in the world by length of the road network. Given that and the fact that climatic conditions are not always favorable, the task of maintaining the good condition of the entire road network in Russia is a non-trivial. It is important to understand that the problem requires a systemic solution. Investing in a large number of disparate repairs — it is a struggle with the symptoms rather than treating the disease.

Systematic approach assumes that in the beginning we have to understand the problem: what is the condition of the road, which roads need to be repaired first? Here's an example: suppose, there are two roads that need repair. A detailed diagnosis is made, and, as a result, it is concluded that the cost of repairing the two areas is approximately the same at the moment. Another conclusion on the results of the diagnosis may consist in the fact that after a year the costs of first road repair will require ten times more money than the second. Situations like this could be predicted as a result of a detailed analysis of the structure and nature of damage on the roadway. Thus, diagnosis becomes very important in terms of limited resources.

Pic 1. Road laboratory
Pic 2. Example of videoframe from road lab

Road surface diagnosis

The following steps has had to be made to complete road surface diagnosis. A special car (road laboratory), equipped with a variety of measuring instruments (in the simplest case, video cam) drives the road. Sensor's output are recorded and later handled by operators manually. As a result, we get some road-«passport» that shows the scope and urgency of repairs.

We have one serious bottleneck in the described method of diagnosis: «manual» data processing. The task of an operator is manual digitization of defects of road surface. Armed with a keyboard, mouse, and specialized software, operators must specify (ie circle) all defects in the target road section. Imagine, how people encircle patches and holes on the roads. Well, it is still possible, but how to circle all the cracks and other minor defects?

There are two solutions for this problem: either to evaluate defects «roughly», seriously impairing the quality of diagnosis, or to search for defects automatically, using modern methods of machine learning.


Road lab captures video of the road
Videoframes are the output
Operators start manual digitization
Final report of road status is sent to the client
Schema 1. Manual processing of road defects

Building model

The next step is to build a model that based on the portion of the image can predict the presence or absence of a defect. Machine learning is very extensive and offers a variety of possible architectures, models and algorithms of their training. In the case of image recognition, models based on artificial neural networks are the most effective. More specifically, we have used convolutional neural network.

Pic 3. Detecting cracks on road: before and after
Pic 4. Detecting patches on road: before and after

Methods

An important neural networks feature is capability of using the power of parallel computing. We cannot use this advantage on modern CPUs, optimized for sequential calculations. On the other hand, the graphics processor (GPU) from NVIDIA are specifically designed for the efficient operation of parallel computing. Typically, the performance of learning and recognition on GPU is 50-100 times higher than running the same model on CPU.

Integration

Our software has been successfully build into existing business processes. A scalable cluster solution was created to parallelize the process of training models and recognition of defects on several levels

Pic 5. Web dashboard page


Levels of parallelization:

  • Level 1: Separate model for each type of object (cars, road surface, cracks, patches, holes and etc.)
  • Level 2: Different hyperparameters during model training process
  • Level 3: Training specific model on GPU

Special software was developed. It allows operators to create, modify and populate training sample, to review the results of recognition, to perform post-processing of the results using a specially crafted script language, that provides reproducibility of results and allows you to quickly get a predetermined sequence of commands and modify it.

Pic 6. Training set and results manupulation software UI


Also, web dashboard was developed, which allows:

  • Control cluster status and running tasks progress
  • Browse training sets
  • Start model training and specify every possible parameter
  • Start model recognition and explore output results

Video

Next project

Recommendation system