Submission Format & Evaluation Metrics
Submission
We require every participant to submit the Docker images for our evaluation. This method enables us to keep the testset secret and to easily reproduce the result. Each team is allowed to submit the results of at most 3 runs, selecting one run as the primary run of the submission. When submitting, you should specify 3 URLs at most for us to download your images.
The format of the output CSV file as a result of the execution is the same as that for last year’s challenge: each line contains an image id followed by 7 predicted labels sorted by confidence in descending order. It looks as follows:
<Test Image ID id#1>, <Training Image ID id#1_1>, <Training Image ID id#1_2>, ..., <Training Image ID id#1_6>, <Training Image ID id#1_7>
<Test Image ID id#2>, <Training Image ID id#2_1>, <Training Image ID id#2_2>, ..., <Training Image ID id#2_6>, <Training Image ID id#2_7>
…
<Test Image ID id#N>, <Training Image ID id#N_1>, <Training Image ID id#N_2>, ..., <Training Image ID id#N_6>, <Training Image ID id#N_7>
Preparation of the Image
We suggest you base your image on ubuntu:16.04 or above and test your solution on it. Once you are ready to submit the image, perform the following steps:
1. docker commit --change 'ENTRYPOINT ["<main_executable>"]' <container_id> <your_image>
2. docker save <your_image> | gzip -c > <saved_image_file>
3. upload the file to Google Drive and submit the downloadable URL to us
In the first command , the <main_executable> should be an executable, e.g., bash script, which accepts 2 arguments described below.
We provide a sample image at https://hub.docker.com/r/aimeetsbeauty/sample-submission.
The ENTRYPOINT in the sample is set to ["/challenge/predict"]. The /challenge/predict script calls random_predict.py to actually perform the prediction and write the output to the given path.
How We Run the Image
1. curl -L -o <downloaded_image> <url>
2. docker load < <downloaded_image>
3. docker run –-network none -ti -v <testset_path>:/testset:ro -v /result --name <container_name> <your_image> /testset /result/predictions.csv
4. docker cp <container_name>:/result/predictions.csv <all_results_path>
Note that:
Please test your image with these commands before submitting.
The total execution time must not be over 1 hour. The score will be 0 if the execution can not produce the expected output or not finish within 1 hour.
Hardware Used to Run the Image
AWS EC2 instance of type r4.4xlarge:
- Dual socket Intel Xeon E5 Broadwell Processors (2.3 GHz), 16 vCPUs
- DDR4 memory, 122GB
Evaluation Metrics
For each test image, the evaluation metric is Mean Average Precision@7 (MAP@7). The performance of each individual test image is then averaged to evaluate the submissions to this challenge.
