Productivity #6: Image Queuing Strategy

In large industrial scale annotation projects, there are millions of images with multiple annotators participating. Images and videos can be distributed among the annotators using different distribution strategies.  All the image & video assets to be labeled are stored in a queue. There are two separate queues for labeling and reviewing:

Labeling Queue

The items can be distributed among annotators in following ways:

1. Distribute all assets to all annotators with 100% duplication. This strategy is useful for projects like radiology. In MRI and CT-scans consecutive images are related to each other, and all images need to be viewed to generate annotations like a liver tumor segmentation.
2. Distribute unique assets of each annotator. In this case each annotator, when they log in, receive a unique image to annotate.
3. Duplicate X% of assets among annotators. Some of the images are shown to multiple annotators based on value set by the admin
4. Individual assignment of each image to a specific annotator. There is an option for admin to assign images to specific annotators

Review Queue

There is an option to show only annotated images to the reviewers

There is an option to allow reviewer to modify annotations

Admin can specify if only certain percentage of annotated assets need to be reviewed.