Welcome to the GQA Challenge 2020!
!
Please notice updated timeline. We are pleased to announce the GQA Challenge 2020 for Real-World Visual Reasoning! Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. The challenge will be hosted on EvalAI.
March 1, 2020:
GQA Challenge 2020 launched!
August, 2020:
Test Phase starts
Sep 24, 2020:
Submission deadline at 23:59:59 UTC
Oct, 2020:
Winners' announcement
The GQA dataset, with more than 110K images and 22M questions, is available on the download page. Each image is associated with a scene graph of the image's object, attributes and relations. Each question is associated with a structured representation of its semantics, a functional program that specifies the reasoning steps have to be taken to answer it. All annotations on the training and validation sets are publicly available.

Many of the GQA questions involve multiple reasoning skills, spatial understanding and multi-step inference, thus are generally more challenging than previous visual question answering datasets used in the community. We made sure to balance the dataset, tightly controlling the answer distribution for different groups of questions, in order to prevent educated guesses using language and world priors. The dataset is complemented with a suite of new metrics to test not only the accuracy, but also the consistency, validity and plausibility of models' responses, shedding much more light on their behavior.

After the challenge deadline, all challenge participant results on the test split will be made public on the test leaderboard.
The GQA challenge has started on March 1, 2020. In this year's challenge we begin with a development phase that takes place in March and will then start the Test phase in August. It will end on Sep 24, 2020, 23:59 UTC with winners being announced in October.

Note that in this year's challenge we will also release new and updated test and challenge question sets in early August. When attempting the challenge please make sure to answer each question in the dataset independently, based on the question and its corresponding image only.

Evaluation

Results must be submitted to the evaluation server by the challenge deadline. The competitors will be evaluated according to the metrics described on the evaluation page. We encourage people to first submit to the validation phase to make sure that you understand the submission procedure, as it is identical to the test and challenge submission procedure. Note that the validation and challenge evaluation servers do not have public leaderboards.

Submission

To enter the competition, first you need to create an account on EvalAI. We allow people to enter our challenge either privately or publicly. Any submissions to the challenge phase remain private and will be considered to be participating in the challenge. For submissions to the test phase, only those that were submitted before the challenge deadline and posted to the public leaderboard will be considered to be participating in the challenge.

Before uploading your results to EvalAI, you will need to create a JSON file which provides an answer to each question in submission_all_questions.json, and conforms to the following format:
results = [result]
result = {
  "questionId": str,
  "prediction": str
}
To submit your JSON file to the GQA evaluation servers, click on the Submit tab on the GQA Challenge 2020 EvalAI. Select the phase (Validation, Test or Challenge) and the JSON file to upload, fill in the required fields (e.g. method name and method description), and finally click Submit. After the file is uploaded, the evaluation server will begin processing. To view the status of your submission please go to My Submissions tab and choose the phase to which the results file has been uploaded. Please be patient, the evaluation may take quite some time to complete. If the status of your submission is Failed please check your stderr file for the corresponding submission.

After evaluation is complete and the server shows a status of Finished, you will have the option to download your evaluation results by selecting Result File for the corresponding submission. The result file will contain the aggregated accuracy for the corresponding phase. If you want your submission to appear on the public leaderboard, please submit to the Test phase and check the box under Show on Leaderboard for the corresponding submission.

Please limit the number of entries to the challenge evaluation server to a reasonable number, e.g., one entry per paper. To avoid overfitting, the number of submissions per user is limited to 1 upload per day (according to UTC timezone) and a maximum of 5 submissions per user. It is not acceptable to create multiple accounts for a single project to circumvent this limit. The exception to this is if a group publishes two papers describing unrelated methods, in this case both sets of results can be submitted for evaluation. However, the Validation phase allows for 10 submissions per day.

Resources

The download page contains links to all GQA images, questions, and for train/val splits, also the associated annotations. Please specify any and all external data used for training in the method description when uploading results to the evaluation server.

We provide an off-line evaluation script as well as baselines for the GQA datasets. To download the GQA evaluation script, please visit our evaluation page. Baselines can be found in our GitHub repo.
For any questions or suggestions regarding the GQA challenge or dataset, please contact dorarad@cs.stanford.edu. In case of technical questions related to EvalAI, please post a message on the GQA Challenge forum. For further clarity, we answer some common questions below:

Q: What do I do if I want to make my test results public and participate in the challenge?
A: Making your results public (i.e., visible on the leaderboard) on the test phase implies that you are participating in the challenge.

Q: What do I do if I want to make my test results public, but I do not want to participate in the challenge?
A: We do not allow this option.

Q: What do I do if I want to participate in the challenge, but I do not want to make my test results public yet?
A: Submit results to the challenge phase, which was created for this scenario.

Q: When will I find out my accuracies on the challenge split?
A: We will reveal challenge results some time after the deadline. Results will first be announced in October.

Q: Can I participate from more than one EvalAI team in the GQA challenge?
A: No, you are allowed to participate from one team only.

Q: Can I add other members to my EvalAI team?
A: Yes, you can add other members to your group.

Q: Is the daily/overall submission limit for each user or for the whole team?
A: The submission limit is for the whole team.