Collections for Data Curation

Learn the basics of using the Collections for Data Curation with the Encord SDK.

Collections are saved groups of data units or labels that let you curate subsets of your data and perform bulk actions on them, such as sending items to annotation, running bulk classifications, or exporting a curated Dataset.

Filter Presets

Filter presets are saved groups of filter criteria that you can reuse across Data Curation.

Create Filter Preset

Create a Filter Preset based on raw JSON.

Template

# Import dependencies
from encord import EncordUserClient

SSH_PATH = "file-path-to-ssh-private-key"


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create a preset

preset = user_client.create_preset(
    name="name-of-my-preset",
    description="A useful and meaningful description for the preset.",
    filter_preset_json={
        "global_filters": {
            "filters": [
                {
                    # define your filter here
                }
            ]
        }
    },
)

print(preset.name, preset.uuid)

Update Filter Presets

All arguments are optional when updating a filter preset.

Template

# Import dependencies
from encord import EncordUserClient

SSH_PATH = "file-path-to-ssh-private-key"

# Preset ID to update
PRESET_ID = "unique-preset-filter-id" 


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Update a preset

preset = user_client.get_filter_preset(preset_uuid=PRESET_ID)
preset.update_preset(
    name="updated-name-of-my-preset",
    description="Updated useful and meaningful description for the preset.",
    filter_preset_json={ ... }
)

print(preset.name, preset.uuid)

Delete Filter Presets

Delete Preset template

# Import dependencies
from encord import EncordUserClient

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"

# Preset ID to delete
PRESET_ID = "unique-preset-filter-id"  

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Delete the preset by ID
user_client.delete_preset(PRESET_ID)

List Filter Presets

List all Filter Presets in Curate by name with their unique identifier.

Template

# Import dependencies
from encord import EncordUserClient

SSH_PATH = "file-path-to-ssh-private-key"


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Get Preset Filters
presets = user_client.list_presets()

# List Preset Filters by name and UUID
for preset in presets:
    print(preset.name, preset.uuid)

List Filters in Filter Preset

This code lists the filters that make up the filter preset into a JSON file format.

Filters are of the following types:

• Global filters: Global filters can apply to the data in any top-level Folder. Examples: Data type, Storage Location, Quality Metrics.

• Local filters: Local filters can apply ONLY to specific top-level Folders. Examples: Data title, File ID, Collection.

Returned results are a combination of Global and Local filters. While Global filters provide information about filter, Local filters provide a raw filter ID.

Template

# Import dependencies
from encord import EncordUserClient

from pprint import pprint

from encord.orm.filter_preset import FilterPreset as OrmFilterPreset

SSH_PATH = "/file/path/to/your/ssh-private-key.txt"
PRESET_ID = "00000000-0000-0000-0000-000000000000"

user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

filters = user_client.get_filter_preset(preset_uuid=PRESET_ID).get_filter_preset_json()

filters_dict = filters.to_dict() 

print("\nFormatted Output:\n")
pprint(filters_dict)

Add Filter Preset Contents to Collection

Adds all storage items that match the criteria specified by a filter preset to a Collection.

This runs in the background. The items might not appear in the Collection when your script finishes, and large filter presets can take some time to complete.

Template

# Import dependencies
from encord import EncordUserClient
from encord.collection import Collection
from encord.orm.filter_preset import FilterPreset

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"
COLLECTION_HASH = "unique-collection-id"

# Preset ID to add to the collection
PRESET_ID = "unique-preset-filter-id"  

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

# Add preset storage items to the collection by preset ID
collection.add_preset_items(PRESET_ID)

print(f"Submitted request to add items matching preset {PRESET_ID} to collection {COLLECTION_HASH}. This can take some time to complete.")

Remove Filter Preset Contents from Collection

Removes all storage items that match the criteria specified by a filter preset from a Collection.

This runs in the background. The items might not be removed from the Collection when your script finishes, and large filter presets can take some time to complete.

Template

# Import dependencies
from encord import EncordUserClient
from encord.collection import Collection
from encord.orm.filter_preset import FilterPreset

# Define constants
SSH_PATH = "file-path-to-key-private-key"
COLLECTION_HASH = "collection-unique-id"

# Preset ID to remove contents from the collection
PRESET_ID = "preset-filter-unique-id"  

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

# Remove preset storage items from the collection by preset ID
collection.remove_preset_items(PRESET_ID)

print(f"Submitted request to remove items matching preset {PRESET_ID} from collection {COLLECTION_HASH}. This can take some time to complete.")

Collections

Collections are groups of data units you selected by filtering, sorting, inspecting, and analyzing the data units in Curate.

List Collections

List all Collections in Curate by name with their unique identifier.

Template

# Import dependencies
from encord import EncordUserClient

SSH_PATH = "file-path-to-your-ssh-private-key"


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Get Collection

collections = user_client.list_collections()

# Print out the names of the Collections
for collection in collections:
    print(collection.name, collection.uuid)

Create Collection

Create a Collection with a meaningful name and description in a Folder.

Template

# Import dependencies
from encord import EncordUserClient

SSH_PATH = "file-path-to-ssh-private-key"


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create a Collection

collection = user_client.create_collection(top_level_folder_uuid="root-folder-uuid", name="name-of-my-collection", description="A useful and meaningful description for the Collection.")

print(collection.name, collection.uuid)

Add Storage Items to a Collection using UUIDs

Add storage items to a Collection using the UUID of the storage items.

Template

# Import dependencies
from encord import EncordUserClient
from encord.collection import Collection
from encord.orm.filter_preset import FilterPreset

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"
COLLECTION_HASH = "collection-unique-id"

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

updates = [
    "<storage-item-id-01>",
    "<storage-item-id-02>",
    "<storage-item-id-03>"
]


collection.add_items(updates)

print(f"Added storage items {updates} to collection {COLLECTION_HASH}.")

Add Storage Items to a Collection using Data Titles

Add storage items to a Collection using their Data Titles, without first looking up each storage item’s UUID. Encord resolves the titles to storage items server-side.

Data Titles are not unique, so a title that matches more than one storage item is not added. Instead, it is returned in ambiguous_titles so you can resolve it, for example by adding the correct item using its UUID. Titles that match no storage item are returned in unmatched_titles.

• Requires encord SDK version 0.1.208 or later.
• You can pass any number of titles. Large sets are split into batches automatically, and the returned counts cover the whole set.
• Duplicate titles are sent once.
• Pass a list, even for a single title (for example, ["image-0001.jpg"]). Passing a bare string raises a TypeError.
• If any single title is too long to send, a ValueError is raised before anything is sent, and no items are added.

Template

# Import dependencies
from encord import EncordUserClient
from encord.collection import Collection

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"
COLLECTION_HASH = "collection-unique-id"

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

data_titles = [
    "<data-title-01>",
    "<data-title-02>",
    "<data-title-03>"
]

# Add storage items to the collection by Data Title
result = collection.add_items_by_title(data_titles)

print(f"Added {result.items_added} storage items to collection {COLLECTION_HASH}.")
print(f"Titles requested: {result.requested}, matched: {result.matched_titles}")

# Titles matching more than one storage item are not added
if result.ambiguous_titles:
    print(f"Not added (matches more than one storage item): {result.ambiguous_titles}")

# Titles matching no storage item
if result.unmatched_titles:
    print(f"Not added (no matching storage item): {result.unmatched_titles}")

# Storage items that matched but could not be added
if result.failed_items:
    print(f"Failed to add storage items: {result.failed_items}")

Remove Storage Items from a Collection using UUIDs

Remove storage items from a Collection using UUIDs.

Template

# Import dependencies
from encord import EncordUserClient
from encord.collection import Collection
from encord.orm.filter_preset import FilterPreset

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"
COLLECTION_HASH = "collection-unique-id"

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

updates = [
    "<storage-item-id-01>",
    "<storage-item-id-02>",
    "<storage-item-id-03>"
]


collection.remove_items(updates)

print(f"Removed storage items {updates} from collection {COLLECTION_HASH}.")

List all Storage Items in a Collection

Lists all storage items in a Collection.

list_items() only returns storage items you have access to. To also include items you cannot access, use collection.list_items_include_inaccessible().

Template

# Import dependencies
from encord import EncordUserClient
from encord.orm.storage import StorageItemType
from encord.collection import Collection
from encord.orm.filter_preset import FilterPreset

# Define constants
SSH_PATH = "file-path-to-ssh-private-key"
COLLECTION_HASH = "collection-unique-id"

# Initialize the user client using the SSH private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Fetch the specific collection by hash
collection: Collection = user_client.get_collection(COLLECTION_HASH)

# Initialize an empty list to store items
items = []

# Iterate over storage items and append them to the list
for item in collection.list_items():
    items.append(item)
    print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")