Road scene recognition

Self-driving cars is emerging topic of near future. Together with Russian leader company we develop road scene recognition software.

Computer vision
Deep learning
Caffe
CUDA

2016

About project

Task

Develop software capable of scene recognition and object segmentation on drive scene for autonomous cars project integration.

Solution

  • Create and train neural network for road scene recognition
  • Create and train neural network for general object segmentation and classification on video
  • Develop software for data augmentation and input modification
  • Prepare the code and tools for usage in embedded devices

Future is near

One can say that the self-driving cars are among the most anticipated trends of near future. According to some experts, by the year 2025 cars without driver will be everywhere, and by 2030 will supersede the usual cars completely. We should prepare for a lot of changes:

A few years ago, the concept of the car without the driver, capable of «see» around and move on public roads, among other cars seemed impossible. But even today self-driving cars don't surprise people from USA city named Mountain View. Now every major car-maker has its own R&D facility in this area, and anyone can buy a Tesla car with elements of artificial intelligence.

Pic 1. Google self-driving cars

Autonoumus cars in Russian

Subject of self-driving cars has not gone unnoticed in Russia.

In March 2016 the State Duma there was a special meeting. They discussed the legal regulation aspects of unmanned vehicle systems (BPTS) in the Russian Federation. If the amendments to the legal acts will be adopted by the State Duma (which could happen before the end of the year), Russia will become the second country in the world after the United States, where self-driving cars will be allowed to drive on public roads.

As part of the this project, we have developed a program complex for one of the leading Russian company to «understand» the space surrounding vehicle.

Scene segmentation

If wee want self-driving car to make decisions during a motion, the on-board computer has to know where and what objects are there on the road. To achieve that, the first task is to complete the scene segmentation — i.e., separation of whole image into individual segments.

For example, by use of scene segmentation we can extract the road surface from surrounding objects and only analyze road.

Pic 2. Scene segmentation example
Рис 3. Scene segmentation example

Tasks like that one are perfect for Deep Learning algorithms with convolutional neural network layers. The input is video-frame and the output is probability for every pixel on image to be one class or another: road, pedestrian, car and so on.

Pic 4. Scheme of CNN we use for segmentation

That network has several good points:

  • Fast: results are obtained in one pass only once for all classes of interest
  • Found objects have very precise boundaries, rather than simply a rectangle frame
  • A neural network sees the whole picture at once, not just individual sections

Data augmentation

Training size is a key quality-factor for most machine learning algorithms: the size of source data used to train the model

Well, in our case we have rather big dataset from dashcams. But, there are some rare classes. For example, there are almost always class "Road on frame", but "Road sign" is rare object. Neural network will only recognize such an object, if network have seen it before many times. What can we do?

In cases like that we can make augmentation of dataset — i.e. extend it by adding artificial data, generated from the real one. In our case we have developed a tool for semi-automatic copy and paste saved objects in places, where it is most common to find them.

Pic 5. Road signs are places on other frames
Pic 6. We also use other objects

Pic 7. ...sometimes it's hard to stop

With using that idea we can dramatically improve the quality of algorithm.

Results

Developed neural networks have shown high efficiency in our tasks.

Pic 8. Recognition and segmentation

Field trials of self-driving cars are planned on the end of this year using our solutions. We hope that the experts' forecasts are true, and in the near future, humanity will qualitatively improve the whole industry. And, without doubt, the methods of machine learning will play an important role.

Next project

Trading patterns