About
Benchmark at IJCNN 2013
We plan on presenting the benchmark mentioned below on the IEEE International Joint Conference on Neural Networks. Due to shifts in the preparation schedule we currently do not
have
a definite acceptance of our benchmark proposal. The official
notification date is December 19th, 2012. Since this would leave very
little time for the training phase (see below) we decided to nonetheless
publish our training data set,
so interested teams can start to work on their detector algorithms.
However, the mentioned details of the conference session are preliminary
at this point and presume the acceptance of our proposal.
Motivation
In 2011 the German Traffic Sign Recognition Benchmark yielded the satisfying result that image sections
containing traffic signs can be reliably recognized with state-of-the-art classification algorithms. With this in mind
we gladly present today the German Traffic Sign Detection Benchmark (GTSDB).
In spite of strong advances in image processing research detection due
to several studies on this topic, traffic sign detection is
still a challenging real-world problem of high industrial relevance. A
detailed comparison of different detector types and processing
aproaches is, however, missing.
Traffic sign detection is a search problem in natural (outdoor)
images. A useful detector must, therefore, be able to cope
with rotation, different lighting conditions, perspective changes,
occlusion and all kinds of weather conditions. During the creation of
our database we took special care on a diverse and representative
compilation of single image frames.
Humans are capable of detecting the large variety of existing road signs with close to perfect reliability in experimental setups. This does obviously not apply to real-world driving, where the drivers attention is regularly drawn to different tasks and situations.
Competition task
The competition task is a detection problem in natural traffic scenes.
Participating algorithms need to pinpoint the location of given
categories of traffic signs (prohibitory, mandatory or danger).
This can and should be done for different parametrizations of the
algorithm in order to receive different values for precision (i.e. the
percentage of detection results that are actually traffic signs)
and recall (i.e. the percentage of given traffic signs that were
actually found). The performance is computed by an area-under-curve
measure for the detector's precision-recall plot on the test dataset.
Although the problem domain of advanced driver assistance systems
implies constraints on the runtime of employed algorithms, this
competition will not take processing times into account, as this puts
too much emphasis on technical aspects like implementation issues and
choice of programming language. We want to keep technical barriers as
low as possible in order to encourage as many people as possible to
participate in the proposed competition.
We will provide example code
concerning reading of images, writing results, and testing own
implementations.
In addition, there are many publicly available resources that can be
used in order to access state-of-the-art methods useful for the
competition, e.
g., the OpenCV library for computer vision and the Shark library for machine learning.
Evaluation procedures
The competition begins with the public availability of the training data
set, corresponding ground truth data, and additional technical
documentation on the competition website (see Dataset and Schedule). Potential participants are invited to develop, train and test their solutions based on these data sets.
We will provide results of established algorithms as a baseline. The
chosen algorithms exemplify several currently competing approaches on
the problem of traffic sign detection.
Shortly before the competition ends, the test set, without ground truth, will be published on the competition website. Participants can compute results for this dataset in several differently parametrized runs and submit results online using the convenient web interface. The performance is directly evaluated and the results are displayed in a precision-recall curve as the computed total performance is shown in a leaderboard. This way participants can compare their performance immediately.
Ground-truth for the test set will be made available after the
competition is closed to allow participants to access the full database.
Availability of datasets
The training data will be made publicly available on December 1, 2012. The test set will be made available on February 18, 2013.
The submission website will be open until the IJCNN's paper submission deadline.
Awards
The best team in every category is invited to present their approaches
during the special session on the IJCNN 2013. Each team is awarded one free conference and workshop registration.
