Active Custom Embeddings

TLDR;

Do you already know what you are doing and only want to look over a Jupyter Notebook example? We provide one here.

Active’s Default Embeddings

Currently in Active, we calculate and display embeddings using a generic CLIP model. This model is excellent for a wide range of tasks and across the board performance. However, a generic CLIP model may struggle with highly specialized tasks. The embeddings generated from the generic model are used for: Natural Language Search (not supported for custom embeddings), image similarity search and the embeddings view (where you view reduced embeddings).

Custom Embeddings Support

We currently support embeddings of dimensions ranging from 1 to 4096 for Index, and 1 to 2000 for Active, following on from our in-house clip Embeddings.

We support embeddings for images, image sequences and image groups. Support for Videos is coming soon.

Use Custom Embeddings in Active

To use custom embeddings in Active:

1. Add an Embedding type to your custom metadata schema.
2. Upload your embeddings as custom metadata.
3. Create an Annotate Project.
4. Select your custom embeddings as you import the Project to Active.

Step 1: Create an Embedding Field

A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.

If you do not include a key in your metadata schema, your imported embeddings are treated as strings.

Embedding key names can contain alphanumeric (a-z, A-Z, 0-1) characters, hyphens, and underscores.

Use add_embedding to add an embedding to your metadata schema.

Key	Description	Display Benefits
embedding	1 to 4096 for Index. 1 to 2000 for Active	Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon)

Import schema template

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()


# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

Step 2: Upload Embeddings

With the key in the custom metadata schema ready, we can now import our embeddings.

Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.

You can import embeddings after you have added your data or during your data registration.

Your key frames (frames specified with or without embeddings) always appear in Index, regardless of what sampling rate you specify.

Embedding key names can contain alphanumeric (a-z, A-Z, 0-1) characters, hyphens, and underscores.

If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.

Specifying a sampling_rate of 0 only imports the first frame and all keyframes of your video into Index.

Import while registering images

Import while importing images

This JSON file imports embeddings while registering your data with Index from a cloud integration.

Template

{
  "images": [
    {
      "objectUrl": "file/path/to/images/file-name-01.file-extension",
      "title": "image-title.file-extension",
      "clientMetadata": {"metadata-1": "value", "metadata-2": "value"}
    },
  ],
  "skip_duplicate_urls": true
}

Update specific images

Import specific images

The custom embeddings format for images follows the same format as importing custom metadata.

# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

# Define a dictionary with item UUIDs and their respective metadata updates
updates = {
    "<data-ID-1>": {"<my-embedding>": [1.0, 2.0, 3.0]},
    "<data-ID-2>": {"<my-embedding>": [1.0, 2.0, 3.0]}
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

Import while registering videos

Import while importing videos

This JSON file imports embeddings while registering your data with Index from a cloud integration.

config is optional when importing your custom embeddings:

"config": {
    "sampling_rate": "<samples-per-second>",
    "keyframe_mode": "frame" or "seconds",
},

If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.

Specifying a sampling_rate of 0 only imports the first frame and all keyframes of your video into Index.

Import Embeddings Template

{
    "videos": [
        {
            "objectUrl": "<cloud-file-path-to-your-video-1>",
            "title": "<title-for-your-video-1>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<samples-per-second>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        },
        {
            "objectUrl": "<cloud-file-path-to-your-video-2>",
            "title": "<title-for-your-vide-2>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<frames-per-seccond>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        },
        {
            "objectUrl": "<cloud-path-to-your-video-3>",
            "title": "<title-for-your-video-3>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<frames-per-second>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        }
    ],
    "skip_duplicate_urls": true
}

Update specific videos

Update specific videos

Boilerplate

# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle
from encord.orm.storage import StorageFolder, StorageItem, StorageItemType, FoldersSortBy

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

updates = {
    "<data-hash-1>": {
        "$encord": {
            "frames": {
                "<frame-number-1>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                },
                "<frame-number-2>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                }
            }
        }
    },
    "<data-hash-2>": {
        "$encord": {
            "config": {
                "sampling_rate": <samples-per-second>,  # VIDEO ONLY (optional default = 1 sample/second)
                "keyframe_mode": "frame" or "seconds",  # VIDEO ONLY (optional default = "frame")
            },
            "frames": {
                "<frame-number-1>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                },
                "<frame-number-2>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                }
            }
        }
    },
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

Step 3 : Create a Project in Annotate

The following task outlines the basics of creating an Annotate Project. For detailed instructions, refer to the documentation here.

To create an Annotate Project:

1. Go to Annotate > Projects. The Annotate Projects list appears.

2. Click New annotation project. The Create New Project page appears.

3. Provide a meaningful title and description.

4. Select an Ontology.

5. Select one or more Datasets.

6. Load a Workflow template.

7. Add collaborators to the Project.

8. Click Create Project.

Step 4: Import Project with Custom Embeddings

Before you can use your custom embeddings in Encord Active Projects, you need to import the custom embeddings. This is performed while you import your Annotate Project into Active.

For existing Active Projects, you can import custom embeddings for your Project import if your Project imported to Data or Labels. Importing to Metrics & Embeddings requires deleting the Project in Active and re-importing the Project with your custom embeddings.

After updating your embeddings, sync your Active Project to automatically apply the new embeddings.

To import a Project with custom embeddings:

1. Log in to the Encord platform. The landing page for the Encord platform appears.

2. Create a Project (Annotation Project or Training Project) in Encord Annotate.

3. Click Active from the main menu. The landing page for Active appears.

4. Click the Import Annotate Project button. The Import an Annotate Project to Encord Active dialog appears.

5. Click the Import button for the Annotate Project you want to import. The Confirm Project Import dialog appears.

   .

6. Select the custom embeddings you want to import to the Project, under Metrics & Embeddings.

7. Select Import under Metrics & Embeddings.

8. Click Proceed. The Annotate Project imports with your custom embeddings.

End-to-End Custom Embeddings Example

We provide an end-to-end example using a Jupyter Notebook here.