Download the SeaDronesSee Multi-Object Tracking (all-objects-in-water) dataset on the dataset page. Upload a json-file with predictions on the test set on the
The SeaDronesSee benchmark was recently
published in WACV 2022. The goal of this benchmark is to
advance computer vision algorithms in maritime search and rescue missions. It is
aimed at detecting humans, boats and other objects in open
water. See the Explore page for some annotation examples.
The task of airborne multi-object tracking in the maritime domain is far from solved. The small sizes of objects
in conjunction with hardly visible objects due to waves and sun reflections make the detection as well as tracking
hard. What is more, gimbal movement and altitude change cause objects to move quickly within the video. The occasional
partial occlusion of objects are on top of the other challenges.
We provide tracking id labels alongside the bounding box labels for
several videos where the task is to track swimmers and
boats. This is a short-term tracking task, i.e. objects that disappear from the scene do not to be tracked anymore.
Create a multi-object tracker that, given a video, outputs bounding boxes and tracking ids for every object
(boats, swimmer, floater). We group all classes into a single class. Also see the dataset page for an
overview. This task is identical to the MOT-All-Objects-In-Water benchmark introduced in the original paper with minor changes
in the annotation file. This is a short-term tracking task, i.e. objects that disappear from the scene need not be tracked anymore.
We provide 21 clips in the train set, 17 clips in the validation set and 19 clips in the test set with a total of 54,105 frames and
403,192 annotated instances. Note that the video clip ids in the annotations files are not necessarily increasing integers (some integers are missing).
Furthermore, there are some images having a video id that is not contained in the train,val and set annotation files. However, this applies to few images only.
Each image has precise meta data labels about altitude, angles of the UAV and the gimbal, GPS, and more.
Admittedly, the annotation files are not super clean and you might have to adapt a little bit here and there. Feel free to contact me any time if you find
some irregularities (firstname.lastname@example.org).
We evaluate your predictions on HOTA, MOTA, IDF1, MOTP, MT, ML, FP, FN, Recall, Precision, ID Switches, Frag. The determining
metric for winning is HOTA. In case of tie, MOTA is the tiebreaker.
Furthermore, we require every participant to submit information on the speed of their method measured in frames per second walltime.
Please also indicate the hardware that you used. Lastly, you should indicate which data sets (also for pretraining) you used during training
and whether you used meta data.
In order to participate, you can perform the following steps:
Download the dataset SeaDronesSee Multi-Object Tracking on the dataset page.
Train a multi-object trackinger of your choice on the dataset.
We try to provide you with a sample multi-object tracking pipeline on Git soon. We also provide public detections that you can use so that
you do not need to train your own detector. It's from a Yolov7 model trained on SeaDronesSee-MOT train set for 8 epochs with a AP of roughly 0.5. For reference, the same model (except for the number of class outputs) has a AP of 0.4181 on Object Detection v2. So, it's not the best, but solid.
You need to create a json-prediction file for upload. In the following, we will illustrate the structue of the json based on an example:
See the MOT_example_submission.json. You can load it in Python via
This yields you a list where you find that this list is of length = number of frames in entire test set (all videos), i.e. 18,253. You can compare that to the length of the list in the corresponding test-json on Nextcloud.
The frames of the videos should be ordered according to the following list:
These videos were previously not listed but ARE included.
Also find the respective information on the video names in the test set json-annotation. Note the difference between DJI_0001.MOV and DJI_0001.mov. Furthermore note, that the number of videos is 19, not 22 or 20!
Each entry in the above list contains a single list. This list contains a single list or multiple lists.
If you found no object on that frame, then there is only a single empty list, so that the entry is:
For example, in the sample submission the first frame does not contain any prediction, i.e.
If you found objects, then there is one or more lists contained. For example, in the sample submission the 101th frame contains three predicted objects:
The object id must be an integer (not the datatype but after rounding to the next integer it should be the same value). The confidence should be a float between 0 and 1.
The coordinates should be self-explanatory.
Let us know if you need further information.
Upload your json-file along with all the required information in the form here. You need to register first.
If you provided a valid file and sufficient information, you should see your method on the leaderboard. Now, you can reiterate. You may upload
at most three times per day (independent of the challenge track). Note that the upload can take a few minutes. You can leave the page although the loading
sign of your browser indicates otherwise.
Terms and Conditions
Submissions must be made before the deadline as listed on the dates page
You may submit at most 3 times per day independent of the challenge
The winners are determined by the HOTA (tiebreaker MOTA) metric
You are allowed to use any publicly available data for training but you must list them at the time of upload. This also applies to pretraining.
Note that we (as organizers) may upload models for this challenge BUT we do not compete for a winning position
(i.e. our models do not count on the leaderboard and merely serve as references). Thus, if your method is worse (in any metric)
than one of the organizer's, you are still encouraged to submit your method as you might win.
In case of any questions regarding the challenge modalities or uploads, please direct them to Benjamin Kiefer.