Scene Annotation End-to-End

This tutorial shows the shortest production-style path for working with point cloud and multimodal Scenes in Encord: prepare sensor assets, register Scenes, create a Project, annotate, review, and reuse the Scene data.

Enable hardware acceleration in Chrome before annotating large point clouds or multimodal Scenes.

What You Need

• An MCAP recording, either on your computer or in cloud storage connected through an integration. MCAP support is in Beta. Contact your Encord support representative to enable it for your Organization.
• For a composite Scene, point cloud and image files accessible through a cloud storage integration.
• A folder in Files & Folders for the registered Scenes.
• A Dataset and Project for annotation work.
• An Ontology with the Scene tools you need, such as cuboids, polylines, keypoints, or classifications.

1. Prepare Scene Assets

For MCAP, keep the recording as one .mcap file and check its supported topics and transforms.

For a composite Scene, put the separate sensor files in a stable cloud prefix. For example:

s3://my-bucket/scenes/scene-001/
  lidar/000000.pcd
  lidar/000001.pcd
  front/000000.jpg
  front/000001.jpg

Use the same frame index for files that should appear together. If your sensors are asynchronous, use sensor timestamps instead and keep the timestamps consistent across streams.

2. Register Scenes

For MCAP, upload the recording directly in Files & Folders (see MCAP Support), or use the SDK to upload a local file or register a file from cloud storage:

Upload an MCAP file to Encord

# Import dependencies
from encord import EncordUserClient
from encord.storage import StorageFolder

# User input
SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"  # Replace with the file path to your SSH private key
FOLDER_ID = "00000000-0000-0000-0000-000000000000"  # Replace with the unique ID for the folder in Files & Folders
MCAP_FILE_PATH = "/Users/chris-encord/recording.mcap"  # Replace with the file path to your MCAP recording
SCENE_TITLE = "Robot recording"  # Replace with the title for the Scene

# Create user client using ssh key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
    # For US platform users use "https://api.us.encord.com"
    domain="https://api.encord.com",
)

# Get the folder
folder: StorageFolder = user_client.get_storage_folder(FOLDER_ID)

# Upload the MCAP recording as a Scene
scene_uuid = folder.upload_scene_file(file_path=MCAP_FILE_PATH, title=SCENE_TITLE)
print(f"Uploaded Scene item {scene_uuid}")

Replace the values in the # User input section with your own. You can then continue to Project creation below.

For separate sensor files, use the Python SDK to build a composite Scene rather than hand-authoring Scene JSON. The SDK validates stream names, references, camera links, and frame-of-reference relationships before registration.

Build and Upload a Scene

from uuid import UUID

from encord.beta.scene import SceneBuilder, identity_pose, intrinsics_pinhole
from encord.storage import StorageFolder


def build_scene() -> SceneBuilder:
    scene = SceneBuilder()

    # Anchor the sensor streams to the moving platform.
    ego = scene.add_for_stream("ego")
    ego.add_pose(identity_pose(), timestamp=0)

    # Matching timestamps make the point cloud and image appear together.
    scene.add_pcd_stream("lidar", frame_of_reference=ego).add_pcd(
        uri="s3://my-bucket/scenes/scene-001/lidar/000000.pcd",
        timestamp=0,
    )

    # Inline camera parameters create a camera stream named "front/camera".
    scene.add_image_stream(
        "front",
        width=1920,
        height=1080,
        intrinsics=intrinsics_pinhole(fx=1000, fy=1000, ox=960, oy=540),
        timestamp=0,
        frame_of_reference=ego,
    ).add_image(
        uri="s3://my-bucket/scenes/scene-001/front/000000.jpg",
        timestamp=0,
    )

    return scene


def register_scene(storage_folder: StorageFolder, integration_id: str) -> UUID:
    scene = build_scene()

    # Pass the SceneBuilder itself. upload_scene validates and serializes it internally.
    return storage_folder.upload_scene(
        scene=scene,
        title="scene-001",
        integration_id=integration_id,
        client_metadata={"source": "sample"},
    )

For sequences, additional sensors, and distortion models, see Create and Register Scenes and Scene SDK Workflows.

3. Create the Annotation Project

1. Add the registered Scene items to a Dataset.
2. Create or reuse an Ontology for the labels you need.
3. Create a Project from the Dataset and Ontology.
4. Assign work through your Workflow as usual.

Use metadata such as route, split, capture_date, or scenario on Scene items if you need to filter, prioritize, or route tasks later.

4. Annotate Scenes

Open a task and use the Scene editor to inspect the point cloud, camera views, and synchronized frames.

Recommended first pass:

• Confirm the point cloud orientation and scale look correct.
• Check that camera images align with the point cloud if calibration is present.
• Use the built-in Scene navigation tutorial to choose mouse or trackpad controls.
• Create the first few labels, move through frames, and verify object continuity.

For editor workflows, see Annotating Scenes.

5. Review and Reuse Scene Data

After annotation, spot-check a few completed tasks in review. Pay special attention to frame alignment and camera projection quality; most Scene issues are easiest to catch visually.

For MCAP labels, see MCAP Scene labels.

For composite Scenes, use SceneReader when you need to inspect, copy, or migrate registered Scene assets.

Read and Copy a Scene

from uuid import UUID

from encord.beta.scene import SceneReader
from encord.orm.storage import DataUploadItems, StorageItemType
from encord.storage import StorageFolder, StorageItem


def find_scene(storage_folder: StorageFolder, title: str) -> StorageItem:
    scene_items = storage_folder.list_items(
        search=title,
        item_types=[StorageItemType.SCENE],
        page_size=1,
    )
    scene_item = next(iter(scene_items), None)
    if scene_item is None:
        raise ValueError(f"Scene with title '{title}' not found.")
    return scene_item


def get_lidar_signed_url(scene_item: StorageItem, timestamp: float = 0) -> str:
    scene = SceneReader(scene_item).read()
    lidar = scene.get_stream("lidar", kind="point_cloud")
    return lidar.get_event(timestamp=timestamp).signed_url


def get_image_signed_urls(scene_item: StorageItem, timestamp: float = 0) -> list[tuple[str, str]]:
    scene = SceneReader(scene_item).read()
    return [(stream_id, event.signed_url) for stream_id, event in scene.get_images_at_timestamp(timestamp)]


def copy_scene_to_folder(
    scene_item: StorageItem,
    target_storage_folder: StorageFolder,
    *,
    integration_id: str,
    title: str,
    source_uri_prefix: str,
    target_uri_prefix: str,
) -> UUID:
    payload = SceneReader(scene_item).to_upload_payload(
        title=title,
        uri_mapper=lambda uri: uri.replace(source_uri_prefix, target_uri_prefix),
    )

    return target_storage_folder.add_private_data_to_folder_start(
        integration_id=integration_id,
        private_files=DataUploadItems(scenes=[payload]),
    )

Troubleshooting

Symptom	Check
Scene does not load	Confirm the integration can access every URI used in the Scene.
Camera view is offset	Check camera intrinsics, sensor pose, and world/camera conventions.
Streams advance out of sync	Check the timestamps assigned to each stream event.
Editor feels slow	Enable browser hardware acceleration and test with a smaller sample Scene.