Cloud-Synced Folders
Cloud-synced folders automatically stay in sync with connected cloud storage, regularly updating to ensure all new and existing files are available in Encord without manual intervention. Cloud-synced Folders require that you have at least one integration set up.- Cloud-synced folders support images, videos, PDFs, text, HTML, and audio files.
Image groups, image sequences, and DICOM series are not currently supported by Cloud-synced folders.
- A single cloud synced folder can contain a maximum of 10 million files.
Create a Cloud-synced Folder
Create Cloud-synced Folder
Sync Cloud-synced Folder with Cloud Storage
The following code syncs a Cloud-sync Folder with the cloud storage bucket. Thesync_private_data_with_cloud_synced_folder_get_result time out value can be adjusted to your needs.
Sync Cloud-synced Folder
Inspect a Folder
Read a folder’s type and, for cloud-synced folders, the integration and remote bucket it is mounted on.folder_type is GENERIC for a standard Encord-managed folder and CLOUD_LINKED for a cloud-synced one; integration_hash and remote_url are populated only for cloud-synced folders.
Enable Metadata Ingestion
Enable metadata sidecar ingestion on a cloud-synced folder. Every item whose path ends with the matcher suffix is treated as a metadata sidecar and paired with the data file that shares its path once the suffix is stripped. The default suffix is.metadata.json.
Requires the API key’s user to have the folder-admin role, and applies to cloud-synced folders only.
Create New Standard Folder
The following script creates a new folder in the root directory of Data > Files & Folders. Ensure that you:- Replace
<private_key_path>with the path to your private key. - Replace
<folder_name>with the name you want to give your folder. We recommend using unique folder names. - Replace
A folder to store my fileswith a meaningful description for your folder. - Replace
my: folder_metadatawith any metadata you want to add to the folder. Remove the line if you do not want to add any metadata to the folder.
Create a Subfolder
UseStorageFolder.create_subfolder() to create a folder inside an existing folder. Ensure that you:
- Replace
<private_key_path>with the path to your private key. - Replace
<parent_folder_name>with the name of the folder you want to create the subfolder in. - Replace
<subfolder_name>with the name you want to give the subfolder.
Create a subfolder
Browse Subfolders
Uselist_subfolders() to list the folders directly inside a folder, and find_subfolders() to search all nested folders below it. The path_to_root property returns the folders between a folder and the root of Data > Files & Folders.
Search Files in Nested Folders
StorageFolder.find_items() searches a folder and all of its subfolders for files. You must provide search, item_types, or both.
Search files recursively
Move Files
Use the following script to move various types of files to a different folder in Data > Files & Folders. Ensure that you:- Replace
<private_key_path>with the path to your private key. - Replace
<data_unit_name>with the name of the file you want to move to a new folder. - Replace
<target_folder_name>with the name of the folder you want to move the file from. - Replace
<target_folder_name>with the name of the folder you want to move the file to.
Move Multiple Files at Once
StorageFolder.move_items_to_folder() moves several files in one call. All the files must be directly inside the folder you call it on.
By default, files linked to a mirrored Dataset cannot be moved, because moving them removes the data units from that Dataset. Pass
allow_mirror_dataset_changes=True to move them anyway.Move multiple files
Move Folders
You can move folders between different parent folders in Data > Files & Folders, including moving them to the root (no parent). The following script demonstrates how a folder with the nameFolder Name 3 is moved between 2 different target folders. The script must be modified to suit your needs.
Ensure that you:
- Replace
<private_key_path>with the path to your private key. - Replace
Folder Name 1with the name of a target folder. - Replace
Folder Name 2with the new name of another target folder. - Replace
Folder Name 3with the name of the folder that is moved.
Folders
Update URL for Cloud Storage Files
Use the following script to update the cloud storage path of your files in Encord when, for example:- You move files in cloud storage
- Update your folder structure in cloud storage
- Update URLs to use multi-region access points
Change Cloud Storage Provider
Using the SDK you can change from one cloud storage provider (for example AWS) to another cloud storage provider (for example GCP) while maintaining data integrity.Change Cloud Storage Provider
Update Folder Properties
Use the following script to update the name and description of an existing folder in Data > Files & Folders. Ensure that you:- Replace
<private_key_path>with the path to your private key. - Replace
Cat Videoswith the name of the folder you want to update properties for. - Replace
Cat videos and imageswith the new name you want to give the folder. - Replace
Images and videos of catswith the description you want to give the folder.
Filter Folders by Organization Tag
Bothfind_storage_folders and list_storage_folders accept organisation_tags_allof, which keeps only folders tagged with all of the given organization tags. Pass tag name strings, or OrganisationTag instances from list_organisation_tags. Unknown tag names raise a ResourceNotFoundError that lists every name that did not match.
Delete Folders
Use the following script to delete a specific folder from Data > Files & Folders. Ensure that you:- Replace
<private_key_path>with the path to your private key. - Replace
Specific Folder Namewith the name of the folder you want to delete.
Delete Files
Deleting individual files (storage items) is separate from deleting a folder. UseStorageItem.delete() to delete a single item, or StorageFolder.delete_storage_items() to delete several items in one call by their UUIDs. Deleted items are removed permanently.
remove_unused_frames defaults to True, which also removes individual images or DICOM files that belonged to an image group or DICOM series and are not used by any other item.Get a Signed URL for a File
StorageItem.get_signed_url() returns a temporary, signed URL for downloading a file. This is useful for fetching the raw data of an item you have located by UUID or by listing a folder.
Composite items have no single downloadable URL, so
get_signed_url() returns None for them. For a Scene, use SceneReader(item); for an image group or DICOM series, use get_child_items() to get a signed URL per frame.Get a signed URL for a storage item

