Tabular Data Projects

We recommend reviewing our end-to-end example to get a practical grasp of using Tabular Data Projects.

Tabular Data currently supports CONSENSUS Projects only.

Tabular data Projects work a little differently than typical Projects in Encord. Annotators and Reviewers select from options columns for each row. You can use multiple columns for selection.

Create Ontology for Tabular Data

Modify the following script example to create your Ontology.

Items of Interest	Notes
READ_ONLY_COLUMNS	Specifies the columns you want your Annotators and Reviewers to see in the Label Editor. Column count starts at 0. Omit the columns in your CSV you do not want your Annotators and Reviewers to see.
ANNOTATION_COLUMNS	Specifies the columns your Annotators and Reviewers use to label data from. Your Annotators and Reviewers select answers from a drop down in these columns. Specify the options available to Annotators and Reviewers using the files in MAPPING_FIELD_OPTION_PATHS. These files are single column files with one option available on each row.
ONTOLOGY_NAME	Specifies the name for your Ontology.
OBJECT_NAME	Specifies the name of the text region for each row in your CSV file. The script applies a label to each row in your CSV file using this text region.

tabular_create_ontology script

import pandas as pd
from encord.objects import OntologyStructure, Shape, TextAttribute
from encord.objects.attributes import RadioAttribute
from encord.user_client import EncordUserClient

# --- Configuration ---
ENCORD_SSH_KEY = "/Users/chris-encord/ssh-private-key.txt" # Replace with the file path to your SHH private key
TASK_CSV_PATH = "/file/path/to/video_game_annotation_1.csv" # Replace with the file path to any of the video_game_annotation_X.csv files

READ_ONLY_COLUMNS = [0, 1, 2]
ANNOTATION_COLUMNS = [3, 4]

# Replace these paths with actual mapping column name > options file
MAPPING_FIELD_OPTION_PATHS = {
    "genre": "/file/path/to/genre-options.csv",
    "platform": "/file/path/to/platform-options.csv",
}

ONTOLOGY_NAME = "E2E - Tabular Data - Ontology"
OBJECT_NAME = "Game Row"


def parse_csv():
    csv_df = pd.read_csv(TASK_CSV_PATH)
    readonly_columns = csv_df.columns[READ_ONLY_COLUMNS].tolist()
    mapping_columns = csv_df.columns[ANNOTATION_COLUMNS].tolist()

    return mapping_columns, readonly_columns


def create_ontology(text_attribute_names, radio_option_names):
    ontology_structure = OntologyStructure()
    text_object = ontology_structure.add_object(name=OBJECT_NAME, shape=Shape.TEXT)

    for attribute in text_attribute_names:
        text_object.add_attribute(TextAttribute, attribute)

    for column_name in radio_option_names:
        options_path = MAPPING_FIELD_OPTION_PATHS.get(column_name)
        if options_path is None:
            raise ValueError(f"No options file defined for column '{column_name}'")

        options = pd.read_csv(options_path).iloc[:, 0].dropna().astype(str).tolist()

        radio_attribute = text_object.add_attribute(RadioAttribute, column_name, required=True)
        for option in options:
            radio_attribute.add_option(option)

    user_client = EncordUserClient.create_with_ssh_private_key(
        ssh_private_key_path=ENCORD_SSH_KEY,
        domain="https://api.encord.com",
    )
    return user_client.create_ontology(ONTOLOGY_NAME, structure=ontology_structure)


if __name__ == "__main__":
    mapping_columns, readonly_columns = parse_csv()
    ontology = create_ontology(readonly_columns, mapping_columns)
    print(f"Created ontology {ontology.title}, id: {ontology.ontology_hash}")

Create Tabular Data Project

Create a Project adding your Ontology and Dataset for Tabular Data.

• Tabular data currently supports CONSENSUS Projects only.
• An AGENT block must be the first block in the Workflow for Tabular Data Projects.
• The AGENT block and AGENT pathway MUST be the exact name specified below.

Run the Agent script

The tabular_run_agent.py populates tasks in the AGENT block in your workflow.

Create the following Python scripts. Both scripts must be in the same directory.

• tabular_run_agent.py
• tabular_utils.py

After creating the scripts, run the tabular_run_agent.py script.

After running the script, tasks that were in the AGENT stage are now in the CONSENSUS - ANNOTATE stage.

Items of Interest	Notes
AGENT_STAGE	Specifies the name of the AGENT block in your Tabular Data Project. This name must exactly match the name of the AGENT block in your Project.
AGENT_PATHWAY	Specifies the name of the Pathway in your AGENT block. This name must exactly match the name of the pathway in the AGENT block in your Project.

tabular_run_agent script

from typing import Annotated
from pathlib import Path
import os

from encord_agents.tasks import Runner
from encord.objects.ontology_labels_impl import LabelRowV2
from encord.project import Project
from encord_agents.tasks.dependencies import dep_asset
from encord_agents.core.dependencies import Depends
from encord.objects.common import Shape

from tabular_utils import parse_csv_and_add_objects

# --- Configuration ---
ENCORD_SSH_KEY = "/Users/chris-encord/ssh-private-key.txt" # Replace with the file path to your SSH private key
PROJECT_HASH = "00000000-0000-0000-0000-000000000000" # Replace with unique Project ID of the tabular data Project
AGENT_STAGE = "Pre-label"
AGENT_PATHWAY = "Labelled"

# Inject into environment so Encord Agents can pick it up
os.environ["ENCORD_SSH_KEY_FILE"] = ENCORD_SSH_KEY

runner = Runner(project_hash=PROJECT_HASH)

@runner.stage(stage=AGENT_STAGE)
def agent_logic(
    lr: LabelRowV2, project: Project, asset: Annotated[Path, Depends(dep_asset)]
):
    ontology = project.ontology_structure
    text_object = ontology.objects[0]
    if text_object is None:
        raise Exception("No objects found")
    elif text_object.shape is not Shape.TEXT:
        raise Exception("Text object required")

    parse_csv_and_add_objects(text_object, lr, asset)

    return AGENT_PATHWAY

if __name__ == "__main__":
    runner.run()

import pandas as pd
from pathlib import Path

from encord.exceptions import OntologyError
from encord.objects.attributes import Attribute
from encord.objects.coordinates import TextCoordinates
from encord.objects.frames import Range
from encord.objects.ontology_labels_impl import LabelRowV2
from encord.objects.ontology_object import Object

def parse_csv_and_add_objects(
    text_object: Object, label_row: LabelRowV2, asset_link: Path
):
    csv_df = pd.read_csv(asset_link)
    filtered_columns = csv_df.columns

    read = None
    with asset_link as f:
        read = f.read_bytes()

    for index, row in csv_df.iloc[0:].iterrows():
        byte_range = get_byte_range_for_row(read, index + 1)
        if byte_range is None:
            raise Exception("Row does not exist")
        (start, end) = byte_range

        row_dict = row[filtered_columns].to_dict()

        add_text_object(text_object, label_row, Range(start=start, end=end), row_dict)

    label_row.save()


def get_byte_range_for_row(csv_content: bytes, row: int) -> tuple[int, int] | None:
    # Find all newline positions
    newline_positions = []
    current_pos = 0

    while True:
        newline_pos = csv_content.find(b"\n", current_pos)
        if newline_pos == -1:
            break
        newline_positions.append(newline_pos)
        current_pos = newline_pos + 1

    # Check if requested row exists
    if row > len(newline_positions):
        return None

    # Calculate start and end positions
    if row == 0:
        start_byte = 0
    else:
        start_byte = newline_positions[row - 1] + 1

    if row < len(newline_positions):
        end_byte = newline_positions[row]
    else:
        # Last row might not have a trailing newline
        end_byte = len(csv_content)

    return (start_byte, end_byte)


def add_text_object(
    text_object: Object,
    lr: LabelRowV2,
    byte_range: Range,
    csv_column_to_value: dict[str, str],
):
    new_instance = text_object.create_instance()
    new_instance.set_for_frames(coordinates=TextCoordinates(range=[byte_range]))
    for key, value in csv_column_to_value.items():
        try:
            attribute = text_object.get_child_by_title(title=key, type_=Attribute)
            if attribute.required:
                # If required attribute then this is the one they are writing so ignore
                continue
            new_instance.set_answer(attribute=attribute, answer=str(value))
        except OntologyError:
            # Ignore columns that don't have attributes
            pass
    lr.add_object_instance(new_instance)

Label and Review Tabular Data

Annotators

Annotators use drop downs to select the genre and platform for each row.

Reviewers

Reviewers verify that the labels are correct.

Use any column in a row to select correct answers.

When there is an issue with labels/classifications, Reviewers can:

• Reject the task and add a comment about why a task was rejected. Rejected tasks go back to the person who added the labels/classifications.
• Edit labels directly using the Edit labels button and then approve the task.