Consensus Workflows

Consensus, in Encord, provides the ability for one or more reviewers to directly compare the labels on a data unit, generated by 2 or more annotators. Annotators do not see what other annotators have labeled. Reviewers can, depending on the settings configured in a Consensus block, select the best individual labels/classifications on a data unit or for agreement/correctness on their labels across an entire data unit (bulk label selection). In both cases, many annotation branches come in and the reviewer decides on the best annotations (individually or in bulk) and one branch comes out.

So in short, multiple annotators label a data unit (a data unit being an image, an image group, an image sequence, a video, or a DICOM series), and a reviewer can decide on the best labels individually (Refine and Review) or in bulk (Determine consensus).

Consensus supports Object labels and Classifications on images, image groups, image sequences, videos, and DICOM series.

Consensus basics

Workflows direct who and how annotations, and reviews of those annotations, are performed on data units. The addition of the Consensus and Archive blocks brings the ability to get consensus on your annotations.

Consensus blocks include an ANNOTATE and REVIEW stage in a single block.

ANNOTATE blocks require:

• # of annotators required for task: Specify the number of annotators required to label a task (data unit) for the task to move to the REVIEW stage in the Consensus block.

• Advanced - Allow task reassignment: Specifies whether, in subsequent blocks, an annotator can annotate a task they have already annotated.

• Add and designate annotators to label the data. This could be from a pool of all annotators (All) on a Project or by selectively choosing annotators (Selective).

REVIEW blocks require:

• Specify the type of review by selecting either Review and Refine or Determine consensus. Review and refine allows reviewers to individually select any of the best labels/classifications on a data unit from any annotator. Reviewers can also edit any Approved labels or create new labels on the data unit. Determine consensus allows reviewers to select the best labels/classifications on a data unit from 1 annotator (bulk selection) and to determine if there is agreement on the labels/classifications across annotators on the data unit.

• # of annotators required to agree (Determine consensus): Specify the number of annotators that a reviewer decides have labels that are correct and in agreement on a task.

• Advanced - Allow task reassignment: Specifies whether, in subsequent blocks, a reviewer can review a task they have already worked reviewed.

Archive blocks provide a location for any task that reviewers deem “not in consensus” or “rejected”.

Tasks move through Consensus Projects in the following way:

1. When the number of annotators specified in # of annotators required for task label the task, the task moves to REVIEW.

2. Reviewers compare the labels on the task:

   • Review and refine : Reviewers select the best individual labels/classifications from any annotator across the data unit. Reviewers can edit any Approved labels and add new labels to the data unit.

   • Determine consensus : Reviewers decide on the correctness and annotator agreement specified in # of annotators required to agree. The task moves to the next stage based on label correctness and whether consensus is reached.

3. Adding multiple Consensus blocks in sequence, means all labels from all previous Consensus blocks display for the reviewers in each subsequent Consensus block.

Encord provides the following default Consensus Workflow, when creating a Project:

The default Consensus Workflow Consensus uses 2 Consensus blocks. The first block uses Determine consensus with the second using Review and Refine.

When using any Consensus Workflow in a Project, do not forget to add your annotators and reviewers to the ANNOTATE and REVIEW sections. Also, if needed, edit the values for # of annotators required for task and # of annotators required to agree, for each Consensus block.

Create a Consensus Project

1. Log in to Encord. The Encord homepage appears.

2. Go to Annotate > Projects > Annotation projects. The Annotation projects page appears.

3. Click the New annotation project button. The Create new project page appears.

4. Click the Load workflow template button. The Load from workflow template modal appears.

5. Select a Workflow template that uses Consensus blocks from the workflow template list. Consensus, the default Consensus Workflow, appears at the top of the template list.

6. Click Load template. The Workflow template appears in the Project workspace.

7. Specify which type of consensus you want to use from the REVIEW block:

   • Review and refine (default value)
   • Determine consensus

8. (Optional) Selectively add your annotators and reviewers to the ANNOTATE and REVIEW sections of the Consensus block.

9. Specify the required value for # of annotators required for task.

10. Specify the required value for # of annotators required to agree if you selected Determine consensus.

11. Specify the Advanced - Allow task reassignment for both the Annotate and Review blocks. The default settings are Disabled for Annotate blocks and Enabled for Review blocks.

12. Complete configuration of the Project.

13. Click Create project. The Project appears in the Annotation project list.

Consensus Advanced

When the default Consensus Workflow template does not suit your requirements, build your own Consensus Workflow template. You can add multiple Consensus, Annotate, Review, and Router blocks to a Workflow.

Here are a couple of examples.

Consensus - Determine consensus - Workflow

In this Consensus Workflow:

• Total # of annotators: 6
• Total # of reviewers: 2
• # of annotators required for task: 4
• # of annotators required to agree: 3

The following diagrams follows a task through the Consensus Workflow until the labels on the task are correct and in consensus or incorrect and not in consensus.

Consensus - Review and Refine - Workflow

In this Consensus Workflow:

Consensus Block:

• Total # of annotators: 6
• Total # of reviewers: 2
• # of annotators required for task: 4
• Reviewers see labels from all 4 annotators.

Consensus Block 2:

• Total # of expert annotators: 2
• Total # of expert reviewers: 2
• # of annotators required for task: 1
• Expert reviewers see labels from all 5 annotators (4 from Consensus Block 1 and 1 from Consensus Block 2).

The following diagrams follow a task through the Consensus Workflow until the labels on the task are correct and in consensus or incorrect and not in consensus. If consensus happens in the first Consensus block, the task goes directly to Complete.

Create a Consensus Workflow Template

To create a consensus Workflow template, your template must include one (or more) Consensus block and one Archive block. Consensus blocks can be used with Annotate Percentage Router blocks and Review blocks to build out your Workflow template.

For more information on creating Workflow templates see our documentation here.

To create a Consensus Workflow template:

1. Log in to Encord. The Encord homepage appears.

2. Go to Annotate > Workflow templates. The Workflow templates page appears.

3. Click the New workflow template button. The Create Workflow template page appears.

4. Click the Invite collaborators icon. The Workflow Collaborators dialog appears.

5. Add the required collaborators for the template.

   Remember that consensus requires at least 2 annotators for the first Consensus block, and we recommend at least 2 reviewers per Consensus block.

6. Click the Add stage icon. The Add stage dialog appears.

7. Drag and drop at least 1 Consensus block and 1 Archive block into the template creation workspace.

8. Specify the REVIEW settings for the block (Review and refine or Determine consensus).

9. Connect the No Consensus output of the Consensus block (the final Consensus block if you have more than one Consensus block) to the input of the Archive block.

10. Add other blocks to the template as you require.

11. In each block, specify collaborators to perform tasks. You can add all annotators and reviewers as pools (All) or you can specify specific annotators and reviewers for each block (Selective).

12. Specify a value for # of annotators required for task for the Annotate portion of the Consensus blocks.

# of annotators required for task is the number of annotators required to annotate a task (data unit) before the task (data unit) moves to the Review stage in the Consensus block. If you use 2 or more Consensus blocks in your workflow, we recommend specifying only 1 annotator for your second and subsequent Consensus blocks. This is because all annotators, and their annotations, from previous blocks appear in the review of subsequent reviews.

12. (Determine consensus only) Specify a value for # of annotators required to agree for the Review stage in the Consensus blocks.

# of annotators required to agree is the number of annotators required to agree on ALL labels on a data unit before consensus is reached. Reviewers decide if annotators have correctly labeled a data unit and if the annotators are in agreement.

13. Complete configuration of the template.

14. Save the template.

What do different user roles see?

Certain tabs can only be accessed by specific users roles.

Tab	Annotator	Reviewer*	Annotator + Reviewer*	Team Manager	Admin
Summary	❌	❌	❌	✅	✅
Explore	❌	❌	❌	✅	✅
Queue	✅	✅	✅	✅	✅
Workflow	❌	❌	❌	✅	✅
Labels & Export	❌	❌	❌	❌	✅
Performance	✅	✅	✅	✅	✅
Models	❌	❌	❌	✅	✅
Settings	❌	❌	❌	✅	✅

Annotators

When working on consensus Projects, annotators have no changes to their workflow or within the Encord platform. Annotators simply need to annotate the tasks (data units) in their queue.

Annotators do not see the work of other annotators as they annotate their tasks in Consensus Projects.

Advanced - Allow task reassessment: When this feature is disabled, annotators CANNOT see this task again as the moves through the workflow. This means they CANNOT annotate a task again, once they have annotated the task. Disabled is the default setting.

Reviewers

When reviewing labels from a REVIEW block on Consensus Projects, reviewers have a new UI when they initiate a task for review.

The only annotator’s work/labels that display in audit Review blocks or the Complete block, is the annotator you specify as the “best annotator”.

When reviewing and refining labels from a REVIEW and REFINE block, reviewers see all the labels from all annotators.

Advanced - Allow task reassessment: When this feature is enabled, reviewers CAN see this task again as the moves through the workflow. This means they CAN review a task multiple times as the task moves through the workflow. Enabled is the default setting.

Annotator & Reviewer

Users with the Annotator & Reviewer role can see both the ANNOTATE and REVIEW blocks in a Consensus. As an annotator, the user cannot see the labels from other annotators. As a reviewer the user can view all labels from all annotators (including themselves) because the Advanced - Allow task reassessment feature is Enabled by default.

Advanced - Allow task reassessment: What a user with the Annotator & Reviewer role can see or do depends on the block they are in.

• ANNOTATE block: When this feature is enabled, annotators CAN see this task again as the task moves through the workflow. This means they CAN review a task multiple times as the task moves through the workflow. Disabled is the default setting.

• REVIEW block: When this feature is enabled, reviewers CAN see this task again as the task moves through the workflow. This means they CAN review a task multiple times as the task moves through the workflow. This also means if the user annotated the task, they can review their own task. Enabled is the default setting.

Admins and Team Managers

Admins and Team Managers can do the following:

• Act as a reviewer on Review tasks
• View tasks
• Monitor the entire queue (Annotate and Review tasks). This includes which tasks are in progress in terms of the number of annotators working on a task with the number of annotators required to finish a task.
• Reassign annotation tasks

Admins and Team Managers CANNOT annotate tasks in a Consensus Workflow.

Team Managers CANNOT access the Labels & Export page.

Export Labels from Consensus Projects

Consensus Projects export ONLY the labels that come from the annotator’s that are marked as “Best” on each data unit by reviewers. This means that only the best labels are exported for your ML training.

To export labels from Consensus Projects

1. Log in to Encord. The Encord homepage appears.

2. Go to Annotate > Projects > Annotate projects. The Annotate projects page appears.

3. Click a Consensus Project from the Project list. The Summary page for the Project appears.

4. Click Labels & Export. The Data tab appears on the Labels & Export page.

5. Filter the data to only display data from the Complete stage. Only data from the Complete stage appears in the list.

6. Select the data units you want to export the labels from.

7. Click the Select all XX rows with labels button.

8. Click Export & Save. The Export labels and save a version dialog appears.

9. Provide a meaningful name for the version of the export.

10. Specify the format, label types for export, whether to generate signed URLs, and specify the Label approval status.

11. Click Export and save. A message appears showing the export progress.

12. Click the Download when the collection of labels completes or go to the Versions page and download the labels from Versions page.

Reopen/Reassign Consensus Annotate tasks

As annotators work through their queue of annotation tasks, each task is “locked” to the annotator that first opened that specific task. The task is no longer visible to other annotators in the Project. In a Project that uses a Consensus Workflow, tasks get locked once multiple annotators (as specified by the number of annotators required to annotate a task in a Consensus block) open a specific task. All annotators must complete the annotation task before the task can move to the Review stage. Admins and Team Managers can reassign/reopen a task. Reopening the task makes the task visible to all annotators in the Project, and any annotator in the Project can then work on the task. Reassigning the task allows Admins or Team Managers to specify the annotator that works on the task.

To reopen/reassign an Annotate task:

1. Log in to Encord. The Encord homepage appears.

2. Go to Annotate > Projects > Annotate projects. The Annotate projects page appears.

3. Click a Consensus Project from the Project list. The Summary page for the Project appears.

4. Click the Annotate block with the task you want reassigned.

5. Expand the task you want to reassign. The row expands to display the users currently assigned to the task.

6. To reopen the task to any annotator on the Project:

   a. Click the 3 dots icon next to the annotator you want the task reopened from. The Reopen button appears.

   b. Click Reopen. The task becomes visible to all annotators in the Annotate queue. Any annotator can then “lock” the task (including the annotator the task was reopened from).

7. To assign a specific annotator to the task:

   a. Click the down arrow next to the annotator you want the task reassigned from. A drop-down menu appears with all annotators available on the Project

   b. Click the down arrow next to the user you want changed. A drop-down menu appears with other annotators available to perform the task.

   c. Select the annotator you want the task assigned to.

Reopen Complete or Archive Consensus Tasks

In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.