The German Traffic Sign Recognition Benchmark

The details about the full GTSRB dataset and the results of the final competition that was held at IJCNN 2011 can be found in our paper "Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition" that was accepted for publication in a Neural Networks Special Issue.

Please cite this paper when when using or referring to the GTSRB dataset.

The paper is currently in print, but it is already available as online version here:
http://dx.doi.org/10.1016/j.neunet.2012.02.016

Preliminary citation:

J. Stallkamp, M. Schlipsing, J. Salmen, C. Igel, Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition, Neural Networks, Available online 20 February 2012, ISSN 0893-6080, 10.1016/j.neunet.2012.02.016. (http://www.sciencedirect.com/science/article/pii/S0893608012000457) Keywords: Traffic sign recognition; Machine learning; Convolutional neural networks; Benchmarking

BibTeX:

@article{Stallkamp2012,
title = "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition",
journal = "Neural Networks",
volume = "",
number = "0",
pages = " - ",
year = "2012",
note = "",
issn = "0893-6080",
doi = "10.1016/j.neunet.2012.02.016",
url = "http://www.sciencedirect.com/science/article/pii/S0893608012000457",
author = "J. Stallkamp and M. Schlipsing and J. Salmen and C. Igel",
keywords = "Traffic sign recognition",
keywords = "Machine learning",
keywords = "Convolutional neural networks",
keywords = "Benchmarking"
}

We will publish the updated citation details as soon as the article is published as print version.

The editor-in-chief unexpectedly moved our special issue to a later issue of Transactions on Intelligent Transportation Systems. Therefore, the submission schedule was changed accordingly.

First submission deadline: January 11, 2012
Notification of first decision: March 15, 2012
Revision submission deadline: May 15, 2012
Notification of final decision: August 15, 2012
Final manuscript (camera ready) submission deadline: September 1, 2012
Publication: Fourth issue 2012 (December)

The PDF version can be found here.

We apologize for any inconvenience.

The submission deadline for the IEEE ITS Special Issue on "Machine Learning for Traffic Sign Recognition" has been extended.

This is the updated schedule:

First submission deadline: November 11, 2011
Notification of first decision: December 20, 2011
Revision submission deadline: January 15, 2012
Notification of final decision: February 28, 2012
Final manuscript (camera ready) submission deadline: March 10, 2012
Publication: Second issue 2012 (June)

The result analysis tool that was presented at the conference is now available in the dataset section (both as source code and binary). It allows to compare results of different approaches and inspect the incorrectly classified images. The description provides some screenshots to give an idea of its capabilities.

The final test set is now available (including ground truth) in the dataset section. There, you will find information which paper you should reference when using the GTSRB dataset.

The final competition stage of GTSRB 2011 is over. We want to thank all participants of the competition (both online and on-site) and attendees of the special session. Futhermore, we want to thank the organizing committee of IJCNN, namely the competition chairs Sven Crone and Isabelle Guyon,  for making this competition possible.

This is the final ranking:

We congratulate all awardees.

We will soon publish the final test set and corresponding class IDs.

Some participants have processing power requirements that can only be fulfilled to a limited extent at the on-site competition, as it is not feasible to bring anything larger than a laptop. Therefore, we will allow remote processing. We are going prepare download links upon request, so that you do not have to rely on the conference wifi network to upload the test set to your computing server. Please let us know if you plan to make use of this opportunity.

Please note: the competition itself remains an on-site event. Completely remote participation will not be possible.

The final training set is available.It contains all data that was published as training and test data during the online competition in January. As before, both images and precalculated features are available.

This dataset is the official training set of GTSRB: for the final competition session as well as for any subsequent evaluation. Please note that this dataset does not contain new data, but only reorganizes previously published data for convenience. The joint data is organized like the training set for the online competition. The files and details about the directory structure can be found in the dataset section.

We are happy to announe a Special Issue of IEEE Transactions on Intelligent Transportation Systems on the topic Machine Learning for Traffic Sign Recognition. A PDF version of the CFP is available here.

Call for papers
---------------

IEEE Transactions on Intelligent Transportation Systems Special Issue:

Machine Learning for Traffic Sign Recognition

Recognition of traffic signs is a challenging real-world problem
relevant for intelligent transportation systems. It is a
multi-category classification problem with unbalanced class
frequencies. Traffic signs show a wide range of variations between
classes in terms of color, shape, and the presence of pictograms or
text. However, there exist subsets of classes (e.g., speed limit
signs) that are very similar to each other. Further, the classifier
has to cope with large variations in visual appearances due to
illumination changes, partial occlusions, rotations, weather
conditions etc.

Although first commercial systems have reached the market and several
studies on sign recognition have been published, systematic
comparisons of different approaches are scarce.

The special issue focuses on unbiased evaluation of new and existing
methods from computer vision, machine learning, and related fields for
traffic sign recognition. Empirical evaluation should be based on
freely available benchmark data.

The special issue is organized in the context of the German Traffic
Sign Recognition Benchmark (GTSRB), a competition at this year's IEEE
International Conference on Artificial Neural Networks (IJCNN 2011).
The GTSRB data is freely available from http://benchmark.ini.rub.de,
the final test data will be released in August.


Important dates
----------------------
Manuscript submissions due: October 15 2011:
Notification of acceptance: November 20 2011
Revised manuscripts due: December 15 2011
Publication: second issue 2012

Submission
-----------------
Manuscripts should be submitted at
http://mc.manuscriptcentral.com/t-its by selecting the manuscript type
`Special Issue on MLFTSR'.


Special issue editors
----------------------------
Johannes Stallkamp, Ruhr-Universität Bochum, Germany
Marc Schlipsing, Ruhr-Universität Bochum, Germany
Jan Salmen, Ruhr-Universität Bochum, Germany
Christian Igel, University of Copenhagen, Denmark


Contact
------------
tsr-benchmark@ini.ruhr-uni-bochum.de

The date for the final session is set. It will take place Tuesday, August 2, 3:20PM-5:20PM. It will be part of the IJCNN special session "Machine Learning for Traffic Sign Recognition".

At the beginning of this session, the test data (images and pre-calculated features) will be distributed to everyone who wants to participate. Participation in the online competition is NOT a requirement to join the final session. Everybody is welcome to compete.

There will be four oral presentations during which the data can be processed and the results be generated. After the talks, the result files are collected and evaluated by us.

We will then present results and some analysis and announce the winner(s).

Some more details and rules:

1. Bring you own hardware. Please contact us if this is a problem.
2. One submission per team. Use the training and test data published for the online competition to determine your best-performing classifier/parameter set.
3. You can use all published images (training/test for online competition) to train your final classifier.
4. As the processing runs in parallel to the talks, you have approx. 80 minutes to run your classifier.
5. Please make sure that your result file follows the required format (exactly as many lines as the test set, class ids from 0 to 42, refer to competition website for more details).

Please do not hesitate to contact us if you have any questions (tsr-benchmark@ini.rub.de).

The test set is now available with the same directory and file structure as the training set, i.e., files are sorted by class and track.

The class ids for the (online) test set are now available in the download section.

The submission time is over. No more submissions are possible. Thank you very much for your efforts and participation.

We just enabled the confusion matrix for your results. You should find a "Details"-button right next to your own results on the result page when your are logged in. In addition to viewing it online, you can download it as CSV file (link below the confusion matrix). You might need to scroll to the top of the screen to actually see the matrix. Clicking on non-zero entries gives the list of misclassified images. The complete list of misclassified images is available as CSV as well.
Note: There are some issues with the confusion matrix display on Google Chrome. Please try a different browser (Firefox, Opera, IE).

The current behaviour of some teams which massively submit results to optimize (and overfit) on the test set forces us to introduce a limitation of 10 (ten) submissions per team. If someone tries to get around this limitation by using different accounts, we reserve the right to disqualify the team. We apologize to all teams for this important change.

Furthermore, regarding the overfitting behaviour on the test set, we noticed the need to clarify that the conference registration is awarded to the best-performing teams under the premise of participation in the final competition during the conference (but independent of its outcome).

Finally, we announce an important change to the schedule. We decided to publish the ground truth data for the online test set right after the submission deadline. In addition to this, we will enable more features on the result page on our website which include a confusion matrix and the list of misclassified images. We hope that this decision provides you with some insight into your results that might be valuable for your paper submission.

The test set is now available (images and pre-calculated features) in the download section.
 
The site is now open for submissions! Please go to "Submissions" (discontinued in July 2020) to submit your results.

Please submit your results before January 21, 2011, 23:59 PST (Pacific Standard Time).
 
If you have any questions, or problems downloading the dataset or submitting your results,
please do not hesitate to contact us (tsr-benchmark@ini.rub.de) so that we can quickly
resolve any issues that might come up during this short time frame.

As the submission time window for the competition is coming up, we want to provide you with a few updates.

1. We are sorry two inform you that some of the annotation data in the image training set was flawed. However, only the total image size attributes (width/height) had errors, with one or both of them being 1 pixel off compared to the provided images. The other attributes are correct. Thanks to Alberto Escalante (INI, Ruhr-Universität Bochum) for pointing this out. You can find corrected annotation data in our download section, either as part of the image archive, or as a standalone archive that only contains the annotations.
   Furthermore, there were two minor issues with the provided C++ code that prevented successful compilation on non-Windows platforms. Thanks to Søren Hauberg (DIKU, Copenhagen) for pointing this out. The fixed version can be found in the download section as well. We apologize for the inconvenience.
2. We just uploaded baseline results based on the provided HOG feature set and a k-nearest neighbor algorithm (Euclidean distance). You can find the results here.
3. Third, a word on the submission: The test set will be available on Wednesday, January 19, 2011, at 0:00 h PST (Pacific Standard Time). At the same time, the result submission site (discontinued in July 2020) will be opened. The site will be closed for submissions on Friday, January 21, at 23:59 h PST. Result submission will not be possible after this date. Please keep in mind that you need to sign up (discontinued in July 2020) before you can submit your results. The results are uploaded as a text file as described here. Please note that the file must contain exactly one entry per test image/feature vector. The test set will be available both as images, as well as HOG, Haar and hue histogram features. The test set will contain the same annotations than the training set (except the ClassId). However, the testset will not contain the track information like the training set (i.e., 30 consecutive images per traffic sign) but a mixed set of approximately 12,500 single images.
4. Last but not least: You are all cordially invited to submit papers to IJCNN 2011 (deadline Feb 1). Your papers will be directed to a special poster session at the main conference dedicated to the competition program. In addition, there will be a workshop after the conference, dedicated to the competitions, where the winners and the best paper authors can present their methods. As results on the final test set will only be available during the conference, you should describe your methodology, results on other datasets, and results in the online part of the challenge. The notification for acceptance is April 1st and the revised paper submission is May 1st.
   
   
   Instructions for authors:
   Go to the the submission site and follow the instructions

   http://www.ijcnn2011.org/submission.php

   In the submission page

   http://ieee-cis.org/conferences/ijcnn2011/upload.php

   Under "Main Research Topic", select the appropriate section under the competition heading.
   Deadline: February 1st, 2011
   Please respect the deadline. You will have the opportunity to revise your paper and include more details about the challenge results at revision time (deadline for revised papers May 1st).

Two minor issues in the C++ example code that prevented successful compilation on non-Windows platforms have been fixed.

We uploaded baseline results from a linear classifier. The results are based on the provided HOG features.

New sets of pre-calculated features are available: Haar-like features and color histograms in HSV color space (hue values only)
Update: The broken download links are now fixed. Sorry for the inconvenience.

Python code to read the training images and C++ code to train a LDA classifier based on the precalculated features are now available.

Matlab code samples and pre-calculated features are now available.

The competition started! The training data is now available.