About

 

Benchmark at IJCNN 2013


We plan on presenting the benchmark mentioned below on the IEEE International Joint Conference on Neural Networks. Due to shifts in the preparation schedule we currently do not have a definite acceptance of our benchmark proposal. The official notification date is December 19th, 2012. Since this would leave very little time for the training phase (see below) we decided to nonetheless publish our training data set, so interested teams can start to work on their detector algorithms. However, the mentioned details of the conference session are preliminary at this point and presume the acceptance of our proposal.

 

Motivation


In 2011 the German Traffic Sign Recognition Benchmark yielded the satisfying result that image sections containing traffic signs can be reliably recognized with state-of-the-art classification algorithms. With this in mind we gladly present today the German Traffic Sign Detection Benchmark (GTSDB).


In spite of strong advances in image processing research detection due to several studies on this topic, traffic sign detection is still a challenging real-world problem of high industrial relevance. A detailed comparison of different detector types and processing aproaches is, however, missing.


Traffic sign detection is a search problem in natural (outdoor) images. A useful detector must, therefore, be able to cope with rotation, different lighting conditions, perspective changes, occlusion and all kinds of weather conditions. During the creation of our database we took special care on a diverse and representative compilation of single image frames.

Humans are capable of detecting the large variety of existing road signs with close to perfect reliability in experimental setups. This does obviously not apply to real-world driving, where the drivers attention is regularly drawn to different tasks and situations.


Competition task


The competition task is a detection problem in natural traffic scenes. Participating algorithms need to pinpoint the location of given categories of traffic signs (prohibitory, mandatory or danger). This can and should be done for different parametrizations of the algorithm in order to receive different values for precision (i.e. the percentage of detection results that are actually traffic signs) and recall (i.e. the percentage of given traffic signs that were actually found). The performance is computed by an area-under-curve measure for the detector's precision-recall plot on the test dataset.


Although the problem domain of advanced driver assistance systems implies constraints on the runtime of employed algorithms, this competition will not take processing times into account, as this puts too much emphasis on technical aspects like implementation issues and choice of programming language. We want to keep technical barriers as low as possible in order to encourage as many people as possible to participate in the proposed competition.


We will provide example code concerning reading of images, writing results, and testing own implementations. In addition, there are many publicly available resources that can be used in order to access state-of-the-art methods useful for the competition, e. g., the OpenCV library for computer vision and the Shark library for machine learning.

 

Evaluation procedures


The competition begins with the public availability of the training data set, corresponding ground truth data, and additional technical documentation on the competition website (see Dataset and Schedule). Potential participants are invited to develop, train and test their solutions based on these data sets.


We will provide results of established algorithms as a baseline. The chosen algorithms exemplify several currently competing approaches on the problem of traffic sign detection.

Shortly before the competition ends, the test set, without ground truth, will be published on the competition website. Participants can compute results for this dataset in several differently parametrized runs and submit results online using the convenient web interface. The performance is directly evaluated and the results are displayed in a precision-recall curve as the computed total performance is shown in a leaderboard. This way participants can compare their performance immediately.


Ground-truth for the test set will be made available after the competition is closed to allow participants to access the full database.

 

Availability of datasets


The training data will be made publicly available on December 1, 2012. The test set will be made available on February 18, 2013. The submission website will be open until the IJCNN's paper submission deadline.

 

Awards


The best team in every category is invited to present their approaches during the special session on the IJCNN 2013. Each team is awarded one free conference and workshop registration.

 

 

 

Please wait while your results are processed