Data Management and Curation

Data Management and Curation

STEP 1: Import Data into Encord

Import Data into Encord

STEP 2: Curate your Data

Curation of your data consists of sorting, filtering, using analytics, and embedding plots to create one or more Collections. Collections can then be made into a Dataset for use with your Annotate and Active Projects.

1

Filter and Sort Data

You can sort your data using a number of data unit merics. The metrics available to sort your data depends on the data available in the Folder. You can sort your data in ascending and descending order.

Index has a number of criteria available to filter your data. You can only access filtering criteria that is available from the top-level Folder. For example, if you do not have any video or audio data in the top-level Folder or any sub-Folders, you cannot filter your data using those filtering criteria.

Adding key frames (for videos), custom metadata, and custom embeddings) to your data can all significantly improve the speed and effectiveness when filtering your data. We recommend importing customr metadata and custom embeddings with your data, but you can import custom metadata and embeddings after importing your data.

File Quality Metrics

Criteria	Description
File Size	Filters files (video) based on their size in bytes.
Random Value	Filters files (video) based on a random value assigned as the file imports into Index.

Video Quality Metrics

Criteria	Description
Area	Filters videos based on the area of the frame.
Aspect Ratio	Filters videos based on the aspect ratio of the video.
Clip Duration	Filters videos based on the duration of the video by hundredths of seconds.
Frame per Second	Filters videos based on the frames per second of the video.
Height	Filters videos based on the height of the video frame.
No. of Video Frames	Filters videos based on the number of frames in the video.
Random Value	Filters videos based on a random value assigned as the file imports into Index.
Width	Filters videos based on the width of the video.

Frame Quality Metrics

Criteria	Description
Area	Filters images/video frames based on the area of the frame.
Aspect Ratio	Filters images/video frames based on the aspect ratio of the video.
Frame Number	Filters images/video frames based on the height of the video frame.
Height	Filters images/video frames based on the height of the video frame.
Random Value	Filters images/video frames based on a random value assigned as the file imports into Index.
Width	Filters images/video frames based on the width of the video.

Audio Quality Metrics

Criteria	Description
Bit Depth	Filters audio files based on the size of each sample (int) in bits.
Duration	Filters audio files based on the duration of the audio file by hundredths of seconds.
Number of Channels	Filters audio files based on number of audio channels in the audio file.
Random Value	Filters audio files based on a random value assigned as the file imports into Index.
Samples per Second	Filters audio files based on the number of samples per second.

Custom Metadata, Custom Embeddings, Key Frames

Criteria	Description
Custom Metadata	Filters files by any custom metadata available on your data. For more information about custom metadata with Index, refer to our documentation
Custom Embeddings	Filters files by any custom embeddings available on your data. For more information about custom embeddings with Index, refer to our documentation
Key Frames	VIDEO ONLY Filters files by any key frames specified on frames in your video data. Specifying key frames on videos significantly speeds up importing videos in to Index. Index imports videos at 1 FPS by default.

Other Filtering Criteria

Criteria	Description
Collection	Filters files by the Collection the files belong to.
Data Title	Filters files their file name.
Data Type	Filters files their file type: Video, Audio, Image, Image Group, Image Sequence, DICOM Series
File ID	Filters files their unique file ID.
Folder	Filters files by the Folders that contain the files.
Integration	Filters files by the Integrations that contain the files.
Mime	Filters files the file’s Mime type.
Storage Location	Filters files by the storage location that contains the files.
Uploaded At	Filters files the time and date of their upload.

---

2

Use Custom Analytic Dashboards

You can create custom analytic dashboards using Distribution and Correlation charts. For more information on analytics dashboards, refer to our documentation.

Adding custom metadata to your data can significantly improve the insights you can gather from your data. We recommend importing customr metadata with your data, but you can import custom metadata after importing your data.

---

3

Custom Embeddings and Similarity Search

Adding custom embeddings to your data can significantly improve the insights you can gather from your data. We recommend importing customr embeddings with your data, but you can import custom embeddings after importing your data.

You must Upgrade the top-level Folder and select the custom embeddings to use embedding plot graphs and similarity search.

For information on upgrading your Folder and importing custom embeddings, refer to our documentation.

Custom Embeddings

Similarity Search

---

4

Create a Collection

1. Log in to the Encord platform. The landing page for the Encord platform appears.

2. Go to Index > Files. The All folders page appears with a list of all folders in Encord.

3. Click in to a Folder. The landing page for the Folder appears and the Explorer button is enabled.

4. Click the Explorer button. The Index Explorer page appears.

5. Search, sort, and filter your data until you have the subset of the data you need.

6. Select one or more of the images/frames/audio files in the Explorer workspace. A ribbon appears at the top of the Explorer workspace.

   Selecting a video frame selects the entire video. Specific frames from a video cannot be selected.

7. Click Select all to select all the images in the subset.

8. Click Add to a Collection.

9. Click New Collection.

10. Specify a meaningful title and description for the Collection.

    The title specified here is applied as a tag/label to every selected image.

11. Click Collections to verify the Collection appears in the Collections list.

STEP 3: Create a Dataset

Once you have a Collection, you can create a Dataset from your Collection.