Skip to main content

Cloud-Synced Folders

Cloud-synced folders automatically stay in sync with connected cloud storage, regularly updating to ensure all new and existing files are available in Encord without manual intervention. Cloud-synced folders require that you have at least one integration set up.
  • Cloud-synced folders support images, videos, PDFs, text, HTML, and audio files.
  • Image groups, image sequences, and DICOM series are not currently supported by cloud-synced folders.
  • A single cloud-synced folder can contain a maximum of 10 million files.
Your cloud bucket controls the contents of a cloud-synced folder. Change the files in the bucket, then sync the folder. In a cloud-synced folder, you can’t:
  • Upload files from your computer. Add the files to the bucket and sync the folder instead.
  • Delete individual files. Remove the files from the bucket and sync the folder instead.
  • Change client metadata with patch_client_metadata(). Update the metadata sidecar file in the bucket and sync the folder instead. See Enable Metadata Ingestion.
  • Create Data Groups. Create them in a standard folder instead.

Create a Cloud-synced Folder

Create Cloud-synced Folder

Sync Cloud-synced Folder with Cloud Storage

The following code syncs a cloud-synced folder with its cloud storage bucket. The sync_private_data_with_cloud_synced_folder_get_result time out value can be adjusted to your needs.
Sync Cloud-synced Folder

Inspect a Folder

Read a folder’s type and, for cloud-synced folders, the integration and remote bucket it is mounted on. folder_type is GENERIC for a standard Encord-managed folder and CLOUD_LINKED for a cloud-synced one; integration_hash and remote_url are populated only for cloud-synced folders.

Enable Metadata Ingestion

Enable metadata sidecar ingestion on a cloud-synced folder. Every item whose path ends with the matcher suffix is treated as a metadata sidecar and paired with the data file that shares its path once the suffix is stripped. The default suffix is .metadata.json.
Requires the API key’s user to have the folder-admin role, and applies to cloud-synced folders only.

Create New Standard Folder

The following script creates a new folder in the root directory of Data > Files & Folders. To create the folder inside another folder, pass parent_folder, or use create_subfolder(). Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
  • Replace A folder to store my files with a meaningful description for your folder.
  • Replace my: folder_metadata with any metadata you want to add to the folder. Remove the line if you do not want to add any metadata to the folder.

Create a Subfolder

Use StorageFolder.create_subfolder() to create a folder inside an existing folder. Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace <parent_folder_name> with the name of the folder you want to create the subfolder in.
  • Replace <subfolder_name> with the name you want to give the subfolder.
Create a subfolder

Browse Subfolders

Use list_subfolders() to list the folders directly inside a folder, and find_subfolders() to search all nested folders below it. The path_to_root property returns the folders between a folder and the root of Data > Files & Folders.

Search Files in Nested Folders

StorageFolder.find_items() searches a folder and all of its subfolders for files. You must provide search, item_types, or both.
find_items() is slower than list_items(), which only returns the files directly inside a folder. Use list_items() when you know which folder holds the files.
Search files recursively

Move Files

Use the following script to move a file to a different folder in Data > Files & Folders. The script works for every file type, including images, videos, image groups, image sequences and DICOM series. Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace <start_folder_name> with the name of the folder you want to move the file from.
  • Replace <target_folder_name> with the name of the folder you want to move the file to.
  • Replace <data_unit_name> with the name of the file you want to move.
  • Optionally, change ITEM_TYPES to the type of the file, for example StorageItemType.DICOM_SERIES.
Moving a file has these rules:
  • You need Admin access to both folders.
  • Files must be inside a folder. You can’t move a file to the root level.
  • Both folders must be in the same organization.
  • If the start folder is mirrored by a Dataset, the move fails unless you pass allow_mirror_dataset_changes=True. The file is then removed from that Dataset. If the target folder is mirrored by a Dataset, the file is added to that Dataset.
Encord moves files. It has no copy operation, and each file lives in exactly one folder. To use the same file in more than one Project, add it to more than one Dataset instead. Client metadata belongs to the file, so every Dataset and Project that uses the file sees the same metadata.
Move a file

Move Multiple Files at Once

StorageFolder.move_items_to_folder() moves several files in one call. All the files must be directly inside the folder you call it on.
By default, files linked to a mirrored Dataset cannot be moved, because moving them removes the data units from that Dataset. Pass allow_mirror_dataset_changes=True to move them anyway.
Move multiple files

Move Folders

You can move folders between different parent folders in Data > Files & Folders, including moving them to the root (no parent). A folder moves with all its subfolders and files. You need Admin access to the folder and to the target folder, and you can’t move a folder into itself or into one of its own subfolders. The following script demonstrates how a folder with the name Folder Name 3 is moved between 2 different target folders. The script must be modified to suit your needs. Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace Folder Name 1 with the name of a target folder.
  • Replace Folder Name 2 with the new name of another target folder.
  • Replace Folder Name 3 with the name of the folder that is moved.
Folders

Update URL for Cloud Storage Files

Use the following script to update the cloud storage path of your files in Encord when, for example:
  • You move files in cloud storage
  • Update your folder structure in cloud storage
  • Update URLs to use multi-region access points
Updating file URLs or changing cloud storage providers requires Admin access to each file and access to every cloud integration involved, including each file’s current integration and any source or destination integration you specify. If you cannot access an involved integration, the entire request fails even when skip_missing=True.

Change Cloud Storage Provider

Using the SDK you can change from one cloud storage provider (for example AWS) to another cloud storage provider (for example GCP) while maintaining data integrity.
Change Cloud Storage Provider

Update Folder Properties

Use the following script to update the name and description of an existing folder in Data > Files & Folders. Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace Cat Videos with the name of the folder you want to update properties for.
  • Replace Cat videos and images with the new name you want to give the folder.
  • Replace Images and videos of cats with the description you want to give the folder.
To find folders, use find_storage_folders, which searches folders at every level, or list_storage_folders, which only lists top-level folders. The search value filters by folder name and can match more than one folder.
If you are updating many folders at once, pass a Bundle to folder.update to batch the changes into a single server call. See Bulk Action Best Practices and keep bundle sizes under 1000 operations.

Filter Folders by Organization Tag

Both find_storage_folders and list_storage_folders accept organisation_tags_allof, which keeps only folders tagged with all of the given organization tags. Pass tag name strings, or OrganisationTag instances from list_organisation_tags. Unknown tag names raise a ResourceNotFoundError that lists every name that did not match.

Delete Folders

Use the following script to delete a specific folder from Data > Files & Folders.
Deleting a folder permanently deletes everything inside it, including all subfolders and files. The files are removed from every Dataset that uses them, and Datasets that mirror the deleted folders are deleted. You need Admin access to the folder.The deletion fails if a file in the folder belongs to an image group, DICOM series or Data Group stored outside the folder, or if the folder is used by a live Dataset’s continuous dataflow.
Ensure that you:
  • Replace the SSH_PATH value with the path to your private key.
  • Replace Specific Folder Name with the name of the folder you want to delete.

Delete Files

Deleting individual files (storage items) is separate from deleting a folder. Use StorageItem.delete() to delete a single item, or StorageFolder.delete_storage_items() to delete several items in one call by their UUIDs. Deleted items are removed permanently. Deleting files has these rules:
  • You need Admin access to the folder.
  • A deleted file is removed from every Dataset that uses it, and so from every Project that uses those Datasets.
  • You can’t delete a file that is part of an image group, DICOM series or Data Group, unless you delete that group in the same call. Delete the group first.
  • You can’t delete individual files from a cloud-synced folder. Remove the files from the bucket and sync the folder instead.
remove_unused_frames defaults to True, which also removes individual images or DICOM files that belonged to an image group or DICOM series and are not used by any other item.
You can’t remove a single DICOM file (slice) from a DICOM series. To leave out a slice, delete the series and upload it again without that file.

Get a Signed URL for a File

StorageItem.get_signed_url() returns a temporary, signed URL for downloading a file. This is useful for fetching the raw data of an item you have located by UUID or by listing a folder.
Composite items have no single downloadable URL, so get_signed_url() returns None for them and shows a deprecation warning. A future SDK version will raise a ValueError instead. For a Scene, use SceneReader(item) from encord.beta.scene. For an image group or DICOM series, use get_child_items() to get a signed URL per frame.
Signed URL requests for items in deleted Amazon S3 buckets return a 404 RESOURCE_NOT_FOUND_ERROR.
Get a signed URL for a storage item