Workflows and Templates

Workflows are a powerful tool for designing and building Projects, allowing you to control how tasks move through different stages and how those stages interact.

Creating Workflows

Workflows are created when creating new annotation Projects, and when creating Workflow templates.

• Manage Collaborators allows you to seamlessly add users to your Workflow as collaborators, which can be added to different stages of the workflow.

• Add stage allows you to add more stages to your Workflow.

---

Workflow Stages

This section contains details for all Workflow library components.

Only ANNOTATE, CONSENSUS, and AGENT nodes can be the first node after the START node in a workflow.

To add library components to the canvas, click the Add stage button and then drag-and-drop the components onto the canvas.

---

Start

Start

Start represents the beginning an annotation task through your Workflow, and can therefore only be used once in a Project. It must have a connector linking it to the next stage. It displays the Datasets connected to the Project, and the selected Ontology.

Annotate

Annotate

Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.

Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.

Click the Edit icon to edit the name of the annotation stage.

Annotation instructions can be added in the Project’s Settings after the Project is created.

The stage contains all available annotators selected by default. Specific annotators can be selected in the Selective tab. Type the email address of users you want to add and select them from the list. Only collaborators that have been added to the Workflow are available for selection. Click Add to confirm the selection.

Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can’t be added, or removed.

A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Click the Edit icon next to Webhook and add the URL of the server you want to receive notifications on.

Routers

Routers

A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:

• Percentage router.
• Collaborator router.

Percentage router

Pathways that data units take through the Workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.

Once the router selects a route for a given task, it is routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it is routed to review stage A on each consecutive pass.

Routing percentages can be changed after Project creation. Changing routing percentages resets the percentage router, and may alter the pathway taken by a given task.

Collaborator router

Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.

The Or else option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.

1. Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.

2. Click Add to finish a pathway.

Collaborators can only be added to the Router after an input is connected.

Review

Review

Review represents a review stage where tasks labeled in an annotation stage are reviewed.

To attach the Rejected node of the review stage to another review stage, contact [e-mail]

Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.

Click the Edit icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.

The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.

Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all review stages and cannot be removed.

Strict review

Strict review is enabled by default, but can be disabled under Advanced settings.

Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.

Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.

Video Tutorial - Strict review & change review judgment

User assignment restriction

The User assignment restriction feature prevents team members working on tasks in the current stage that they have done in previous stages. This can prevent individuals with the Annotator & Reviewer role from being assigned their own annotations to review.

This feature does not apply to users with the Admin or Team Manager roles, since they are added to each stage by default and cannot be removed.

Enable the toggle to implement User assignment restriction.

Agent

Agent

The Agent component represents a Task Agent. It triggers a custom operation each time a task passes through the Agent stage. The exact operation being triggered is configured using the Encord SDK.

To fully connect an Agent to other Workflow components you must add pathways that allow you to create connections from the Agent stage to other components. To add a pathway, click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add.

Learn more about Task Agents in our Task Agent section

Complete

Complete

Complete represents the stage of a Project at which a task is marked as complete, and can therefore only be used once per project.

Having passed though your Workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.

Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Click the Edit icon next to ‘Webhook’ and add the URL of the server you would like to receive notifications on.

In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.

---

Managing collaborators

---

User permissions

Permission	Admin	Team Manager	Reviewer	Annotator	Annotator & Reviewer
Attach / Detach datasets	✅	❌	❌	❌	❌
Attach / Switch ontology	✅	❌	❌	❌	❌
Delete	✅	❌	❌	❌	❌
Invite team members	✅	✅	❌	❌	❌
Manage team permissions	✅	✅	❌	❌	❌
Manage admins	✅	❌	❌	❌	❌
Annotate & review tasks	✅	✅	Review only	Annotate only	✅
Confirm annotations	✅	✅	❌	❌	❌
Control assignments and status	✅	✅	❌	❌	❌

Workflow stage level collaborators

Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.

Learn how to add collaborators to different stages of a Project by visiting the relevant sections in this document:

• Annotate collaborators.
• Review collaborators.
• Collaborator type Router.

If no collaborators are specified at a given stage, all users with permissions are added by default.

For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles will be added by default:

• Annotator
• Annotator & Reviewer
• Team Manager
• Admin

---

Templates

Workflow templates allow you to save and reuse your most common workflows. You can then use these templates to create new annotation Projects, ensuring consistency and efficiency.

Create Templates

New Workflow templates can be created each time a new Workflow is created.

1. Navigate to the Projects section of the Encord platform.
2. Navigate to the Templates tab.
3. Click Create template.

4. Create your Workflow. Ensure your Workflow has a title.
5. Click the Create workflow template button. The template is saved in the custom templates section.

Edit templates

You can edit Workflow templates after they have been created.

1. Navigate to the Projects section of the Encord platform.
2. Navigate to the Templates tab.
3. Select the template you want to Edit.
4. Click the Edit button in the top-right.
5. Edit the template.
6. Click Save changes.

You can not edit the start node.

Template Collaborators

Template collaborators are annotators and reviewers that can be assigned to Workflow stages.

Invite Collaborators

1. Click the Collaborators button.

2. Select a role. You can assign the following roles to collaborators:

• Annotator: Labels tasks in the Project.
• Reviewer: Reviews and validates labeled tasks.
• Annotator & Reviewer: Performs both annotation and review tasks.
• Team Manager: Assigns tasks to users and manages Project collaborators.
• Admin: Grants full administrative control over the Project. This action is irreversible.

3. Type the email address of the user you want to add and select the user from the list.

4. Click the Add button to add the user with the specified role.

Manage Users

1. Click the Collaborators button.

2. Click the Manage tab.

• Change collaborator role: Click the dropdown and select a new role.
• Remove collaborator: Click the red Remove icon in the remove column.

Manage Template Access

Template Access determines which users can edit, share, and delete the template.

Invite Users

1. Click the Manage template access button.

2. Search for the collaborator or the group you want to add. Multiple collaborators and groups can be added at once.

Only group members are added when a group is assigned to a Project. Group managers are not included.

3. Select a role. You can assign the following roles to collaborators:

   • User: Can view the template.
   • Admin: Grants full administrative control over the template. This action is irreversible.

4. Click Add.

Edit User Role

Only template admins can modify collaborator roles. Admin roles cannot be changed, not even by other admins.

1. Click the Manage template access button.

2. Click the user role in the Role column
3. Select a new role from the dropdown.

Remove User

1. Click the Manage template access button.

2. Click the red Remove icon in the remove column.

Delete Templates

To delete a template, click the red Delete icon next to the template.