Skip to main content

Cloud-Synced Folders

Cloud-synced folders automatically stay in sync with connected cloud storage, regularly updating to ensure all new and existing files are available in Encord without manual intervention. Cloud-synced Folders require that you have at least one integration set up.
  • Cloud-synced folders support images, videos, PDFs, text, HTML, and audio files.
Image groups, image sequences, and DICOM series are not currently supported by Cloud-synced folders.
  • A single cloud synced folder can contain a maximum of 10 million files.

Create a Cloud-synced Folder

Create Cloud-synced Folder

Sync Cloud-synced Folder with Cloud Storage

The following code syncs a Cloud-sync Folder with the cloud storage bucket. The sync_private_data_with_cloud_synced_folder_get_result time out value can be adjusted to your needs.
Sync Cloud-synced Folder

Inspect a Folder

Read a folder’s type and, for cloud-synced folders, the integration and remote bucket it is mounted on. folder_type is GENERIC for a standard Encord-managed folder and CLOUD_LINKED for a cloud-synced one; integration_hash and remote_url are populated only for cloud-synced folders.

Enable Metadata Ingestion

Enable metadata sidecar ingestion on a cloud-synced folder. Every item whose path ends with the matcher suffix is treated as a metadata sidecar and paired with the data file that shares its path once the suffix is stripped. The default suffix is .metadata.json.
Requires the API key’s user to have the folder-admin role, and applies to cloud-synced folders only.

Create New Standard Folder

The following script creates a new folder in the root directory of Data > Files & Folders. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
  • Replace A folder to store my files with a meaningful description for your folder.
  • Replace my: folder_metadata with any metadata you want to add to the folder. Remove the line if you do not want to add any metadata to the folder.

Create a Subfolder

Use StorageFolder.create_subfolder() to create a folder inside an existing folder. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace <parent_folder_name> with the name of the folder you want to create the subfolder in.
  • Replace <subfolder_name> with the name you want to give the subfolder.
Create a subfolder

Browse Subfolders

Use list_subfolders() to list the folders directly inside a folder, and find_subfolders() to search all nested folders below it. The path_to_root property returns the folders between a folder and the root of Data > Files & Folders.

Search Files in Nested Folders

StorageFolder.find_items() searches a folder and all of its subfolders for files. You must provide search, item_types, or both.
find_items() is slower than list_items(), which only returns the files directly inside a folder. Use list_items() when you know which folder holds the files.
Search files recursively

Move Files

Use the following script to move various types of files to a different folder in Data > Files & Folders. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace <data_unit_name> with the name of the file you want to move to a new folder.
  • Replace <target_folder_name> with the name of the folder you want to move the file from.
  • Replace <target_folder_name> with the name of the folder you want to move the file to.

Move Multiple Files at Once

StorageFolder.move_items_to_folder() moves several files in one call. All the files must be directly inside the folder you call it on.
By default, files linked to a mirrored Dataset cannot be moved, because moving them removes the data units from that Dataset. Pass allow_mirror_dataset_changes=True to move them anyway.
Move multiple files

Move Folders

You can move folders between different parent folders in Data > Files & Folders, including moving them to the root (no parent). The following script demonstrates how a folder with the name Folder Name 3 is moved between 2 different target folders. The script must be modified to suit your needs. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace Folder Name 1 with the name of a target folder.
  • Replace Folder Name 2 with the new name of another target folder.
  • Replace Folder Name 3 with the name of the folder that is moved.
Folders

Update URL for Cloud Storage Files

Use the following script to update the cloud storage path of your files in Encord when, for example:
  • You move files in cloud storage
  • Update your folder structure in cloud storage
  • Update URLs to use multi-region access points

Change Cloud Storage Provider

Using the SDK you can change from one cloud storage provider (for example AWS) to another cloud storage provider (for example GCP) while maintaining data integrity.
Change Cloud Storage Provider

Update Folder Properties

Use the following script to update the name and description of an existing folder in Data > Files & Folders. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace Cat Videos with the name of the folder you want to update properties for.
  • Replace Cat videos and images with the new name you want to give the folder.
  • Replace Images and videos of cats with the description you want to give the folder.
Use the list_storage_folders and find_storage_folders methods to search for specific folders.
If you are updating many folders at once, pass a Bundle to folder.update to batch the changes into a single server call. See Bulk Action Best Practices and keep bundle sizes under 1000 operations.

Filter Folders by Organization Tag

Both find_storage_folders and list_storage_folders accept organisation_tags_allof, which keeps only folders tagged with all of the given organization tags. Pass tag name strings, or OrganisationTag instances from list_organisation_tags. Unknown tag names raise a ResourceNotFoundError that lists every name that did not match.

Delete Folders

Use the following script to delete a specific folder from Data > Files & Folders. Ensure that you:
  • Replace <private_key_path> with the path to your private key.
  • Replace Specific Folder Name with the name of the folder you want to delete.

Delete Files

Deleting individual files (storage items) is separate from deleting a folder. Use StorageItem.delete() to delete a single item, or StorageFolder.delete_storage_items() to delete several items in one call by their UUIDs. Deleted items are removed permanently.
remove_unused_frames defaults to True, which also removes individual images or DICOM files that belonged to an image group or DICOM series and are not used by any other item.

Get a Signed URL for a File

StorageItem.get_signed_url() returns a temporary, signed URL for downloading a file. This is useful for fetching the raw data of an item you have located by UUID or by listing a folder.
Composite items have no single downloadable URL, so get_signed_url() returns None for them. For a Scene, use SceneReader(item); for an image group or DICOM series, use get_child_items() to get a signed URL per frame.
Get a signed URL for a storage item