Welcome to the INI Benchmark Website!

Dear benchmark enthusiasts,

due to high server traffic, we are forced to temporarily shut down the download section. We are working on providing a solution soon, please be patient.

Update: The datasets have been transferred to an external hosting service and can be downloaded from there, check out the respective "Dataset" section.

Thank you for your understanding,

Your GTSRB / GTSDB team

Dear benchmark enthusiasts,

we have decided to shut down the submission as we consider it a now rarely used feature that meant a lot of maintenance effort from us. The datasets, software packages, and results submitted so far are still available as they used to.

Have fun, best regards,

Your GTSRB / GTSDB team

Dear benchmark enthusiasts,

after migrating the server and due to some other administrative work we had several errors regarding the full functionality of the benchmark server. It should be fixed by now. However, if you run into problems, do not hesitate to contact us at tsd-benchmark@ini.rub.de .

We apologize to all those that experienced problems using our server and were not answered (there had been some email issues as well).

Best regards,

Your GTSRB / GTSDB team

The migration is completed. We apologize for any inconvenience.

Due to server migration we will take down the benchmark website on

Wednesday, 08/20/2014, 8am MEST

Hopefully the server will be back online until 2pm.

The conference paper describing the dataset and the course of the competition can be cited by (BibTeX):

@inproceedings{Houben-IJCNN-2013,
   author = {Sebastian Houben and Johannes Stallkamp and Jan Salmen and Marc Schlipsing and Christian Igel},
   booktitle = {International Joint Conference on Neural Networks},
   title = {Detection of Traffic Signs in Real-World Images: The {G}erman {T}raffic {S}ign {D}etection {B}enchmark},
   number = {1288},
   year = {2013},
}

Our special session at the annual IJCNN was very well-attended. We want to thank the presenters for their interesting and fitting talks and the participants for the vivid disucssion afterwards.

(Xiaolin Hu (Teams wff and LITS1), Alioscia Petrelli (Team BolognaCVLab), Markus Matthias (Team VISICS), Sebastian Houben (Organizer), Alberto De Souza (Team lcad-ufes)

We put together the test and training dataset with full annotations for download. Please refer to the 'Dataset' section for the links.

To our information the submission system will be online until at least March 8, 2013, possibly longer. For those of you are late with their results or intended to write a paper but were to pressed for time this should be an ideal opportunity.

Due to anew (and unannounced) changes in IJCNN submission schedule, we decided to reopen the submission site. You will be able to submit results again and they will be displayed in the "Results" section as before. However, for fairness reasons you will not be able to compete for the competition prizes.

Please note the submission instructions below when submitting, as resubmission will not be possible.

Please do also contact us if you have further questions.

Dear teams,

the submission is closed. Thank you for participating in the German Traffic Sign Detection Benchmark. You can now see the final results evaluated on the entire test dataset in section "Results". In particular, we want to congratulate the teams

wgy@HIT501 (Prohibitive, Mandatory)

visics (Prohibitive, Danger)

LITS1 (Prohibitive)

who managed to achieve perfect results in a category.

To this time, the submission deadline of the IJCNN has not been extended and it is still uncertain whether it will be. Please do not forget to hand in your papers if you planned to do so. Please do also follow the submission instructions below.

Dear participants,

we added some details to the "Results" section that might be useful if you are writing a paper. Please log into your account and open the site "Results". Below the ranking you will find the precision-recall pairs of your own submissions in case you want to use them in a plot.

As a tie breaker we also introduced the average overlap of your true positive detections.

We (again) will have to revise the final date of the submission phase, since another deadline extension for the IJCNN 2013 is under discussion. If the deadline is postponed, we will accordingly adjust the end of the submission phase. However, teams that prepare a paper should be prepared to hand in their papers on the upcoming Friday.

As we discussed with several teams before, we cordially invite all participants to submit a paper on their respective algorithms to the IJCNN 2013. You will find the necessary information at http://www.ijcnn2013.org/paper-submission.php . On the submission form please select the Main Research Topic 'C02. German Traffic Sign Detection Competition' (under Cross-Disciplinary Topics) to make sure that your papers can be related to this competition.

Please pay attention to the firm deadline on Friday, March 1, 2013. We will close the competition on Friday at 8 am CET. At that point you will be able to see your results evaluated on the whole evaluation dataset (Section 'Results'). Logging into your account will make sure the precision-recall plots of your submissions are shown in the section 'Results'.

When you refer to the competition, please cite the following paper (BibTeX):

@inproceedings{Houben-IJCNN-2013,
author = {Sebastian Houben and Johannes Stallkamp and Jan Salmen and Marc Schlipsing and Christian Igel},
booktitle = {International Joint Conference on Neural Networks (submitted)},
title = {Detection of Traffic Signs in Real-World Images: The {G}erman {T}raffic {S}ign {D}etection {B}enchmark},
year = {2013},
}

@inproceedings{Houben-IJCNN-2013,
   author = {Sebastian Houben and Johannes Stallkamp and Jan Salmen and Marc Schlipsing and Christian Igel},
   booktitle = {International Joint Conference on Neural Networks},
   title = {Detection of Traffic Signs in Real-World Images: The {G}erman {T}raffic {S}ign {D}etection {B}enchmark},
   number = {1288},
   year = {2013},
}

By popular demand we have raised the submission limit to 10 submissions per team.

Dear participants in the IJCNN-2013-Traffic-Sign-Detection-Benchmark,

the submission phase is officially started. Please download the test dataset from section "Dataset", process it with your algorithm of choice, and submit your result file by means of the new section "Submission".

As we described in detail in the section "Dataset", you can (and should) assemble several result files with the specified format of several of your algorithm's runs with possibly different parametrization in a single zip file.
The result text files within are parsed on our server and the result is immediatly visible to you and all other participants.

Please note that the shown results are preliminary. The performance is evaluated on a subset of the whole test dataset for the time being. The final performance will be shown after the submission phase is over, which will be shorty before the IJCNN's paper submission deadline. By revealing the final performance to a later point of time we hope to prevent teams from overfitting their parametrizations to the data.

Please feel free to contact us with whatever questions or problems might occur during the competition.

Good luck!

After the IJCNN has officially announced our competition we decided to start the submission phase on Monday, February 18, 2013, at 10 am CET. The submission site will continue to stay open until shortly before the IJCNN's paper submission deadline.

After checking with the IJCNN's competition organizer, we decided to postpone the announced submission phase to most likely the first weeks of February. We will provide you with a definite date as soon as possible.

It is very likely that the current submission deadline (22 February, 2013) will be extended. We will take care that you will have enough time to prepare your papers after the competition. It will also be possible to hand in preliminary results.

We are sorry to inform you on such short notice, but in order to attract as many teams as possible we want to align with the IJCNN's schedule.

Thanks to several remarks from participating teams we have updated the ground truth of the training dataset. You can download the data package or the ground truth text file only. If you decide for the latter, please consider that the submission example file (ex.txt) and the image section subdirectories (00 - 42) have changed, too.

We added 6 further ground truth entries. 4 of the corresponding signs are at the limit of the minimal size of 16x16 pixels. 2 were well-hidden. Furthermore a class ID was incorrectly annotated. Therefore one sign of a non-relevant category changes to "mandatory".

Some teams aproached us with wishes for the date of the postponed submission phase. For example, the Chinese New Year holiday season is just around the corner. If you have problems with the submission being held in February, please let us know.

The submission site will be opened on Wednesday, 23 January, 2013, at 10 am CET.

We are currently discussing the schedule with the IJCNN organizers.

Due to delays in the review process of the IJCNN we decided to adapt the schedule of our submission phase. It will begin on January 23, 2013, as planned, but
will stay opened longer than previously announced. The final date is yet unclear, but we want to give a fair chance to teams that will be attracted by the IJCNN competition announcements.

The IJCNN paper submission deadline has been extended to February 22, 2013. This should allow all teams working on papers to finish in time with first results.

Unfortunately, the ReadMe.txt in the download package with the GTSDB training data (TrainIJCNN2013.zip) contains an error. Two class IDs have mistakenly been added to the prohibitive and the danger traffic sign category. The download package has been corrected, the correct categories are

  prohibitory = [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 15, 16] (circular, white ground with red border)
  mandatory = [33, 34, 35, 36, 37, 38, 39, 40] (circular, blue ground)
  danger = [11, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31] (triangular, white ground with red border)
 
Teams that used our C++ or Matlab function library will find the correct categories in the respective subroutines. This mistake only concerned the ReadMe.txt in the download package.
 
We thank the team that pointed this out to us and apologize for the inconvenience.

For your motivation we uploaded results from three different baseline detectors that are popular in the literature: a Viola-Jones detector based on Haar features, a Hough-like voting scheme that detects circles and triangles, and a LDA based on HOG features that is used as a detector by presenting it all possible image sections.

Refer to the new section "Results" for an overview.

You can now download code snippets in the section "Dataset" that facilitate your work with the training dataset. There are downloads for C++ and Matlab. Both packages contain functions to read ground truth data from the training dataset, to test and evaluate your self-implemented detector functions by presenting the downloaded images one by one, and to evaluate a text file that contains the results of your detector in the format you would need to submit during the competition.

Feel free to contact us with whatever questions you might have,

By popular demand we are glad to announce the German Traffic Sign Detection Benchmark (GTSDB) as a successor to the German Traffic Sign Recognition Benchmark.
Research teams with background in image processing and / or machine learning are cordially invited to compete with their algorithms in this new challenge.
For details, downloads, and regulations feel free to browse the sections About and Dataset.

The notification of acceptance for our benchmark proposal at the IEEE International Joint Conference for Neural Networks (IJCNN) 2013 will be put out on December 19th, 2012.

The details about the full GTSRB dataset and the results of the final competition that was held at IJCNN 2011 can be found in our paper "Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition" that was accepted for publication in a Neural Networks Special Issue.

Please cite this paper when when using or referring to the GTSRB dataset.

The paper is currently in print, but it is already available as online version here:
http://dx.doi.org/10.1016/j.neunet.2012.02.016

Preliminary citation:

J. Stallkamp, M. Schlipsing, J. Salmen, C. Igel, Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition, Neural Networks, Available online 20 February 2012, ISSN 0893-6080, 10.1016/j.neunet.2012.02.016. (http://www.sciencedirect.com/science/article/pii/S0893608012000457) Keywords: Traffic sign recognition; Machine learning; Convolutional neural networks; Benchmarking

BibTeX:

@article{Stallkamp2012,
title = "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition",
journal = "Neural Networks",
volume = "",
number = "0",
pages = " - ",
year = "2012",
note = "",
issn = "0893-6080",
doi = "10.1016/j.neunet.2012.02.016",
url = "http://www.sciencedirect.com/science/article/pii/S0893608012000457",
author = "J. Stallkamp and M. Schlipsing and J. Salmen and C. Igel",
keywords = "Traffic sign recognition",
keywords = "Machine learning",
keywords = "Convolutional neural networks",
keywords = "Benchmarking"
}

We will publish the updated citation details as soon as the article is published as print version.

The editor-in-chief unexpectedly moved our special issue to a later issue of Transactions on Intelligent Transportation Systems. Therefore, the submission schedule was changed accordingly.

First submission deadline: January 11, 2012
Notification of first decision: March 15, 2012
Revision submission deadline: May 15, 2012
Notification of final decision: August 15, 2012
Final manuscript (camera ready) submission deadline: September 1, 2012
Publication: Fourth issue 2012 (December)

The PDF version can be found here.

We apologize for any inconvenience.

The submission deadline for the IEEE ITS Special Issue on "Machine Learning for Traffic Sign Recognition" has been extended.

This is the updated schedule:

First submission deadline: November 11, 2011
Notification of first decision: December 20, 2011
Revision submission deadline: January 15, 2012
Notification of final decision: February 28, 2012
Final manuscript (camera ready) submission deadline: March 10, 2012
Publication: Second issue 2012 (June)

The result analysis tool that was presented at the conference is now available in the dataset section (both as source code and binary). It allows to compare results of different approaches and inspect the incorrectly classified images. The description provides some screenshots to give an idea of its capabilities.

The final test set is now available (including ground truth) in the dataset section. There, you will find information which paper you should reference when using the GTSRB dataset.

The final competition stage of GTSRB 2011 is over. We want to thank all participants of the competition (both online and on-site) and attendees of the special session. Futhermore, we want to thank the organizing committee of IJCNN, namely the competition chairs Sven Crone and Isabelle Guyon,  for making this competition possible.

This is the final ranking:

We congratulate all awardees.

We will soon publish the final test set and corresponding class IDs.

Some participants have processing power requirements that can only be fulfilled to a limited extent at the on-site competition, as it is not feasible to bring anything larger than a laptop. Therefore, we will allow remote processing. We are going prepare download links upon request, so that you do not have to rely on the conference wifi network to upload the test set to your computing server. Please let us know if you plan to make use of this opportunity.

Please note: the competition itself remains an on-site event. Completely remote participation will not be possible.

The final training set is available.It contains all data that was published as training and test data during the online competition in January. As before, both images and precalculated features are available.

This dataset is the official training set of GTSRB: for the final competition session as well as for any subsequent evaluation. Please note that this dataset does not contain new data, but only reorganizes previously published data for convenience. The joint data is organized like the training set for the online competition. The files and details about the directory structure can be found in the dataset section.

We are happy to announe a Special Issue of IEEE Transactions on Intelligent Transportation Systems on the topic Machine Learning for Traffic Sign Recognition. A PDF version of the CFP is available here.

Call for papers
---------------

IEEE Transactions on Intelligent Transportation Systems Special Issue:

Machine Learning for Traffic Sign Recognition

Recognition of traffic signs is a challenging real-world problem
relevant for intelligent transportation systems. It is a
multi-category classification problem with unbalanced class
frequencies. Traffic signs show a wide range of variations between
classes in terms of color, shape, and the presence of pictograms or
text. However, there exist subsets of classes (e.g., speed limit
signs) that are very similar to each other. Further, the classifier
has to cope with large variations in visual appearances due to
illumination changes, partial occlusions, rotations, weather
conditions etc.

Although first commercial systems have reached the market and several
studies on sign recognition have been published, systematic
comparisons of different approaches are scarce.

The special issue focuses on unbiased evaluation of new and existing
methods from computer vision, machine learning, and related fields for
traffic sign recognition. Empirical evaluation should be based on
freely available benchmark data.

The special issue is organized in the context of the German Traffic
Sign Recognition Benchmark (GTSRB), a competition at this year's IEEE
International Conference on Artificial Neural Networks (IJCNN 2011).
The GTSRB data is freely available from http://benchmark.ini.rub.de,
the final test data will be released in August.


Important dates
----------------------
Manuscript submissions due: October 15 2011:
Notification of acceptance: November 20 2011
Revised manuscripts due: December 15 2011
Publication: second issue 2012

Submission
-----------------
Manuscripts should be submitted at
http://mc.manuscriptcentral.com/t-its by selecting the manuscript type
`Special Issue on MLFTSR'.


Special issue editors
----------------------------
Johannes Stallkamp, Ruhr-Universität Bochum, Germany
Marc Schlipsing, Ruhr-Universität Bochum, Germany
Jan Salmen, Ruhr-Universität Bochum, Germany
Christian Igel, University of Copenhagen, Denmark


Contact
------------
tsr-benchmark@ini.ruhr-uni-bochum.de

The date for the final session is set. It will take place Tuesday, August 2, 3:20PM-5:20PM. It will be part of the IJCNN special session "Machine Learning for Traffic Sign Recognition".

At the beginning of this session, the test data (images and pre-calculated features) will be distributed to everyone who wants to participate. Participation in the online competition is NOT a requirement to join the final session. Everybody is welcome to compete.

There will be four oral presentations during which the data can be processed and the results be generated. After the talks, the result files are collected and evaluated by us.

We will then present results and some analysis and announce the winner(s).

Some more details and rules:

1. Bring you own hardware. Please contact us if this is a problem.
2. One submission per team. Use the training and test data published for the online competition to determine your best-performing classifier/parameter set.
3. You can use all published images (training/test for online competition) to train your final classifier.
4. As the processing runs in parallel to the talks, you have approx. 80 minutes to run your classifier.
5. Please make sure that your result file follows the required format (exactly as many lines as the test set, class ids from 0 to 42, refer to competition website for more details).

Please do not hesitate to contact us if you have any questions (tsr-benchmark@ini.rub.de).

The test set is now available with the same directory and file structure as the training set, i.e., files are sorted by class and track.

The class ids for the (online) test set are now available in the download section.

The submission time is over. No more submissions are possible. Thank you very much for your efforts and participation.

We just enabled the confusion matrix for your results. You should find a "Details"-button right next to your own results on the result page when your are logged in. In addition to viewing it online, you can download it as CSV file (link below the confusion matrix). You might need to scroll to the top of the screen to actually see the matrix. Clicking on non-zero entries gives the list of misclassified images. The complete list of misclassified images is available as CSV as well.
Note: There are some issues with the confusion matrix display on Google Chrome. Please try a different browser (Firefox, Opera, IE).

The current behaviour of some teams which massively submit results to optimize (and overfit) on the test set forces us to introduce a limitation of 10 (ten) submissions per team. If someone tries to get around this limitation by using different accounts, we reserve the right to disqualify the team. We apologize to all teams for this important change.

Furthermore, regarding the overfitting behaviour on the test set, we noticed the need to clarify that the conference registration is awarded to the best-performing teams under the premise of participation in the final competition during the conference (but independent of its outcome).

Finally, we announce an important change to the schedule. We decided to publish the ground truth data for the online test set right after the submission deadline. In addition to this, we will enable more features on the result page on our website which include a confusion matrix and the list of misclassified images. We hope that this decision provides you with some insight into your results that might be valuable for your paper submission.

The test set is now available (images and pre-calculated features) in the download section.
 
The site is now open for submissions! Please go to "Submissions" (service discontinued in July 2020) to submit your results.

Please submit your results before January 21, 2011, 23:59 PST (Pacific Standard Time).
 
If you have any questions, or problems downloading the dataset or submitting your results,
please do not hesitate to contact us (tsr-benchmark@ini.rub.de) so that we can quickly
resolve any issues that might come up during this short time frame.

As the submission time window for the competition is coming up, we want to provide you with a few updates.

1. We are sorry two inform you that some of the annotation data in the image training set was flawed. However, only the total image size attributes (width/height) had errors, with one or both of them being 1 pixel off compared to the provided images. The other attributes are correct. Thanks to Alberto Escalante (INI, Ruhr-Universität Bochum) for pointing this out. You can find corrected annotation data in our download section, either as part of the image archive, or as a standalone archive that only contains the annotations.
   Furthermore, there were two minor issues with the provided C++ code that prevented successful compilation on non-Windows platforms. Thanks to Søren Hauberg (DIKU, Copenhagen) for pointing this out. The fixed version can be found in the download section as well. We apologize for the inconvenience.
2. We just uploaded baseline results based on the provided HOG feature set and a k-nearest neighbor algorithm (Euclidean distance). You can find the results here.
3. Third, a word on the submission: The test set will be available on Wednesday, January 19, 2011, at 0:00 h PST (Pacific Standard Time). At the same time, the result submission site (service discontinued in July 2020) will be opened. The site will be closed for submissions on Friday, January 21, at 23:59 h PST. Result submission will not be possible after this date. Please keep in mind that you need to sign up (service discontinued in July 2020) before you can submit your results. The results are uploaded as a text file as described here. Please note that the file must contain exactly one entry per test image/feature vector. The test set will be available both as images, as well as HOG, Haar and hue histogram features. The test set will contain the same annotations than the training set (except the ClassId). However, the testset will not contain the track information like the training set (i.e., 30 consecutive images per traffic sign) but a mixed set of approximately 12,500 single images.
4. Last but not least: You are all cordially invited to submit papers to IJCNN 2011 (deadline Feb 1). Your papers will be directed to a special poster session at the main conference dedicated to the competition program. In addition, there will be a workshop after the conference, dedicated to the competitions, where the winners and the best paper authors can present their methods. As results on the final test set will only be available during the conference, you should describe your methodology, results on other datasets, and results in the online part of the challenge. The notification for acceptance is April 1st and the revised paper submission is May 1st.
   
   
   Instructions for authors:
   Go to the the submission site and follow the instructions

   http://www.ijcnn2011.org/submission.php

   In the submission page

   http://ieee-cis.org/conferences/ijcnn2011/upload.php

   Under "Main Research Topic", select the appropriate section under the competition heading.
   Deadline: February 1st, 2011
   Please respect the deadline. You will have the opportunity to revise your paper and include more details about the challenge results at revision time (deadline for revised papers May 1st).

Two minor issues in the C++ example code that prevented successful compilation on non-Windows platforms have been fixed.

We uploaded baseline results from a linear classifier. The results are based on the provided HOG features.

New sets of pre-calculated features are available: Haar-like features and color histograms in HSV color space (hue values only)
Update: The broken download links are now fixed. Sorry for the inconvenience.

Python code to read the training images and C++ code to train a LDA classifier based on the precalculated features are now available.

Matlab code samples and pre-calculated features are now available.

The competition started! The training data is now available.