Follow these steps to set up annotator training in Encord. The annotator training workflow enables you to assess the accuracy and performance of your annotation workforce.

​
Overview

  1. Add your training files to Encord.

  2. Create a Benchmark Project Establish ground truth labels by having a trusted expert annotate the data. This project must be completed before annotator training begins.

  3. Set up Annotator Training Projects You must create one training Project per annotator. Use the same Dataset as the Benchmark Project in each annotator training Project. Annotators label the data, and their work is compared against the gold standard created in the benchmark.

  4. Annotators Label Training Tasks Annotators must complete all the training tasks assigned to them.

  5. Evaluate Annotator Performance Use the provided SDK script to compare annotator labels with the benchmark. Analyze the results to assess accuracy and provide targeted feedback.

​
STEP 1: Add Files to Encord

You must first add your data Encord. These files are used to train your annotators.

1

Create a Cloud Integration

2

Create a Folder to Store your Files

  1. Navigate to Files under the Index heading in the Encord platform.
  2. Click the + New folder button to create a new folder. A dialog to create a new folder appears.

  1. Give the folder a meaningful name and description.

  2. Click Create to create the folder. The folder is listed in Files.

3

Create JSON file for Registration

To register files from cloud storage into Encord, you must create a JSON file specifying the files you want to upload.

While you can use a CSV file, we strongly recommend using JSON files for uploading cloud data to Encord for better compatibility and performance.

Find helpful scripts for creating JSON files for the data registration process here.

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

For a list of supported file formats for each data type, go here
Encord supports file names up to 300 characters in length for any file or video for upload.

Encord enforces the following upload limits for each JSON file used for file registration:

  • Up to 1 million URLs
  • A maximum of 500,000 items (e.g. images, image groups, videos, DICOMs)
  • URLs can be up to 16 KB in size

Optimal upload chunking can vary depending on your data type and the amount of associated metadata. For tailored recommendations, contact Encord support. We recommend starting with smaller uploads and gradually increasing the size based on how quickly jobs are processed. Generally, smaller chunks result in faster data reflection within the platform.

4

Import your Files

​
STEP 2: Create a Benchmark Project

The benchmark Project contains reference labels used to evaluate your annotators’ labels. These gold standard labels should be created by a trusted expert to ensure accurate assessment.

1

Create a Training Dataset

Create a Dataset containing tasks designed to establish ground truth labels. These files are used to generate ‘gold-standard’ labels against which annotator performance can be evaluated. Give the Dataset a meaningful name.

Learn how to create Datasets here.
2

Create an Ontology

Create an Ontology to label your data. The same Ontology must be used in the Benchmark Project AND the Annotator Training Project.

Learn how to create Ontologies here.
3

Create the Benchmark Project

Ensure that you attach ONLY the Training Dataset to the Project.

  1. Go to Annotate > Projects.
  2. Click the + New annotation project button to create a new Project.
  1. Give the Project a meaningful title and description. For example “Benchmark Labels”.
  2. Click the Attach ontology button and attach the Ontology you created.
  3. Click the Attach dataset button and attach the Benchmark Dataset you created.

  1. Click Invite collaborators. Add collaborators to the Project and add them to the relevant Workflow stages. You annotators should be experts you trust to create gold-standard labels.

  2. Click Create project to finish creating the Project. You have now created the Project to Establish ground-truth labels.

​
STEP 3: Create Annotator Training Projects

Create a Project where your annotation workforce labels data and is evaluated against benchmark labels.

1

Create an Annotator Training Workflow Template

Create a Workflow template and give it a meaningful name like “Annotator Training”.

Creating templates makes creating one Project per annotator quicker and easier.

Create the following Workflow template for your Annotator Training Projects. Documentation on how to create new Workflow templates can be found here.

2

Create Annotator Training Projects

You must create one Annotator Training Project per annotator. This step must be repeated for each annotator.

Ensure that you:

  • Attach the Training Dataset you created in Step 2.1 for the Benchmark Project.
  • Attach the SAME Ontology you created in Step 2.2 for the Benchmark Project.
  • Attach the Annotator Training Workflow Template to the Project.
  1. Go to Annotate > Projects.
  2. Click the + New annotation project button to create a new Project.
  3. Give the Project a meaningful title and description. For example “Annotator Training - Alex” for an annotator named Alex.
  4. Click the Attach ontology button and attach the Ontology you created. Attach the SAME Ontology you created in Step 2.2 for the Benchmark Project.
  5. Click the Attach dataset button and attach the training Dataset you created in Step 2.1.
  6. Click the Load from template button to attach the “Annotator Training” template you created in Step 3.1.
  7. Click Invite collaborators. Add the annotator you want to train in this Project to the annotation stage.
  8. Click Create Project to create the Project. You have now created the Project to train the selected annotator.

​
STEP 4: Annotator Training

Your annotators must now complete all tasks in the Annotator Training Project they are assigned to. Only tasks in the Complete stage are evaluated.

Information on how to label can be found here.

​
STEP 5: Evaluate Annotators

This example only evaluates Bounding Boxes.

Save and run the following script to evaluate annotator performance. The script must be run once for each Annotator Training Project. It outputs a CSV file called iou_results.csv containing the results. The evaluation metrics used are Intersection over Union (IoU) and Class score.

  • IoU (Intersection over Union): Quantifies the overlap between predicted labels and the ground truth. It ranges from 0 to 1: 1.0: Indicates a perfect overlap between the predicted label and the ground truth. 0.0: Indicates no overlap between the predicted label and the ground truth. Values between 0 and 1: Represent the percentage of overlap. For example, an IoU of 0.6 signifies that 60% of the predicted label area overlaps with the ground truth label area.

  • Class Score (0 or 1): 1: The label was created using the correct class. 0: The label was created using the wrong class.

Ensure that you:

  • Replace <private_key_path> with the full path to your private SSH key.
  • Replace <benchmark-project-id> with the id of your Benchmark Project.
  • Replace <training-project-id> with the id of the Training Project you want to evaluate.
from encord import EncordUserClient
from encord.objects.common import Shape
from encord.objects.coordinates import BoundingBoxCoordinates
import pandas as pd
from encord.user_client import EncordUserClient
import os

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

training_project_id = "<training-project-id>"
benchmark_project_id = "<benchmark-project-id>"

training_project = user_client.get_project(training_project_id)
benchmark_project = user_client.get_project(benchmark_project_id)

training_label_rows = training_project.list_label_rows_v2(workflow_graph_node_title_eq='Complete')
benchmark_label_rows = benchmark_project.list_label_rows_v2(workflow_graph_node_title_eq='Complete')

# Match by data_hash
benchmark_dict = {lr.data_hash: lr for lr in benchmark_label_rows}
paired_label_rows = [
    (benchmark_dict[lr.data_hash], lr)
    for lr in training_label_rows
    if lr.data_hash in benchmark_dict
]

# Initialise labels
with training_project.create_bundle() as bundle:
    for _, prod_lr in paired_label_rows:
        prod_lr.initialise_labels(bundle=bundle, overwrite=True)

with benchmark_project.create_bundle() as bundle:
    for bm_lr, _ in paired_label_rows:
        bm_lr.initialise_labels(bundle=bundle, overwrite=True)

# IoU calculation
def calculate_iou(bbox1: BoundingBoxCoordinates, bbox2: BoundingBoxCoordinates) -> float:
    x_left = max(bbox1.top_left_x, bbox2.top_left_x)
    y_top = max(bbox1.top_left_y, bbox2.top_left_y)
    x_right = min(bbox1.top_left_x + bbox1.width, bbox2.top_left_x + bbox2.width)
    y_bottom = min(bbox1.top_left_y + bbox1.height, bbox2.top_left_y + bbox2.height)
    intersection = max(0, x_right - x_left) * max(0, y_bottom - y_top)
    union = bbox1.width * bbox1.height + bbox2.width * bbox2.height - intersection
    return intersection / union if union > 0 else 0.0

# Compare labels and extract information
results = []
for bm_lr, prod_lr in paired_label_rows:
    prod_instances = [oi for oi in prod_lr.get_object_instances() if oi.ontology_item.shape == Shape.BOUNDING_BOX and oi.get_annotation(0)]
    bm_instances = [oi for oi in bm_lr.get_object_instances() if oi.ontology_item.shape == Shape.BOUNDING_BOX and oi.get_annotation(0)]

    training_data_unit_name = prod_lr.data_title
    training_label_id = prod_lr.label_hash

    for prod_obj in prod_instances:
        best_iou = 0.0
        best_match_hash = None
        prod_bbox = prod_obj.get_annotation(0).coordinates
        training_email = prod_obj.get_annotation(0).created_by

        for bm_obj in bm_instances:
            bm_bbox = bm_obj.get_annotation(0).coordinates
            iou = calculate_iou(prod_bbox, bm_bbox)
            if iou > best_iou:
                best_iou = iou
                best_match_hash = bm_obj.feature_hash

        class_score = 1.0 if best_match_hash == prod_obj.feature_hash and best_match_hash is not None else 0.0

        results.append({
            'training_email': training_email,
            'data_unit_name': training_data_unit_name,
            'label_id': training_label_id,
            'iou_score': best_iou,
            'class_score': class_score
        })

# Output the results to a CSV file
if results:
    df_results = pd.DataFrame(results)
    script_dir = os.path.dirname(os.path.abspath(__file__))
    csv_file_path = os.path.join(script_dir, "iou_results.csv")
    df_results.to_csv(csv_file_path, index=False)
    print(f"Results saved to: {csv_file_path}")
else:
    print("No matching label rows found for comparison.")

Was this page helpful?