> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encord.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agents Reference Core

# Core

#### b64\_encode\_image

```python theme={"dark"}
def b64_encode_image(img: NDArray[np.uint8],
                     format: Base64Formats = ".jpg") -> str
```

Encode an image to a base64 string.

**Arguments**:

* `img` - The image to encode. Expects \[RGB] channels
* `format` - The format of the image.

**Returns**:

The base64 encoded image.

### TaskSpeedColumn Objects

```python theme={"dark"}
class TaskSpeedColumn(ProgressColumn)
```

Renders human readable transfer speed.

#### render

```python theme={"dark"}
def render(task: Task) -> Text
```

Show data transfer speed.

### LabelRowMetadataIncludeArgs Objects

```python theme={"dark"}
class LabelRowMetadataIncludeArgs(BaseModel)
```

Warning, including metadata via label rows is good for *reading* metadata
**not** for writing to the metadata.

If you need to write to metadata, use the `dep_storage_item` dependencies instead.

### LabelRowInitialiseLabelsArgs Objects

```python theme={"dark"}
class LabelRowInitialiseLabelsArgs(BaseModel)
```

Arguments used to specify how to initialize labels via the SDK.

The arguments are passed to `LabelRowV2.initialise_labels`.

### FrameData Objects

```python theme={"dark"}
class FrameData(BaseModel)
```

Holds the data sent from the Encord Label Editor at the time of triggering the agent.

#### project\_hash

The identifier of the given project.

#### data\_hash

The identifier of the given data asset.

#### frame

The frame number. If single image, it's default 0.

#### object\_hashes

Object hashes if the request was made on particular objects from the App

### Frame Objects

```python theme={"dark"}
@dataclass(frozen=True)
class Frame()
```

A dataclass to hold the content of one frame in a video.

#### frame

The frame number within the video

#### content

An \[h,w,c] np.array with color channels RGB.

#### b64\_encoding

```python theme={"dark"}
def b64_encoding(
    image_format: Literal[".jpeg", ".jpg", ".png"] = ".jpeg",
    output_format: Literal["url", "openai", "anthropic", "raw"] = "url"
) -> str | dict[str, str | dict[str, str]]
```

Get a base64 representation of the image content.

This method allows you to convert the content into a base64 representation
based on various different image encodings.
This is useful, e.g., for prompting LLMs with image content.

Please see details for formats below.

**Arguments**:

* `image_format` - Which type of image encoding to use.

* `output_format` - Different common formats.
  * `raw`: the image content as a raw b64 string
  * `url`: url encoded image content. Compatible with, e.g., `<img src="<the_encoding>" />`
  * `openai`: a dict with `type` and `image_url` keys
    \_ `anthropic`: a dict with `media_type`, `type`, and `data` keys.

* `Returns` - a dict or string depending on `output_format`.

### InstanceCrop Objects

```python theme={"dark"}
@dataclass(frozen=True)
class InstanceCrop(Frame)
```

A dataclass to hold the frame content of one object instance in a video or image.

#### instance

The [ObjectInstance](/sdk-documentation/sdk-references/objects.ontology_object_instance#objectinstance) associated to the crop.

### EditorAgentResponse Objects

```python theme={"dark"}
class EditorAgentResponse(BaseModel)
```

A base class for all return types of Custom Agent functions.

#### message

A message to be displayed to the user.

#### get\_user\_client

```python theme={"dark"}
def get_user_client(settings: Settings | None = None) -> EncordUserClient
```

Generate an user client to access Encord.

**Returns**:

An EncordUserClient authenticated with the credentials from the encord\_agents.core.settings.Settings.

#### get\_initialised\_label\_row

```python theme={"dark"}
def get_initialised_label_row(
        frame_data: FrameData,
        include_args: LabelRowMetadataIncludeArgs | None = None,
        init_args: LabelRowInitialiseLabelsArgs | None = None) -> LabelRowV2
```

Get an initialized label row from the frame\_data information.

**Arguments**:

* `frame_data` - The data pointing to the data asset.

**Raises**:

* `Exception` - If the `frame_data` cannot be matched to a label row

**Returns**:

The initialized label row.

#### download\_asset

```python theme={"dark"}
@contextmanager
def download_asset(storage_item: StorageItem,
                   frame: int | None = None) -> Generator[Path, None, None]
```

Download the asset associated to a label row to disk.

This function is a context manager. Data will be cleaned up when the context is left.

Example usage:

```python theme={"dark"}

with download_asset(storage_item, 10) as asset_path:
## In here the file exists
pixel_values = np.asarray(Image.open(asset_path))

## outside, it will be cleaned up
```

**Arguments**:

* `storage_item` - The Storage item for which you want to download the associated asset.
* `frame` - The frame that you need. If frame is none for a video, you will get the video path.

**Raises**:

* `NotImplementedError` - If you try to get all frames of an image group.
* `ValueError` - If you try to download an unsupported data type (e.g., DICOM).

**Yields**:

The file path for the requested asset.

#### get\_frame\_count

```python theme={"dark"}
def get_frame_count(storage_item: StorageItem) -> int
```

Get the number of frames in a video.

#### batch\_iterator

```python theme={"dark"}
def batch_iterator(iterator: Iterable[T],
                   batch_size: int) -> Iterable[List[T]]
```

Yield batches of items from an iterator.

**Arguments**:

* `iterator` - The source iterator
* `batch_size` - Size of each batch > 0

**Returns**:

Iterable of lists, each containing up to batch\_size items

### GenericFieldModel Objects

```python theme={"dark"}
class GenericFieldModel(BaseModel)
```

#### set\_answer

```python theme={"dark"}
def set_answer(instance: ClassificationInstance) -> None
```

This function will be called from the parsing loop to allow the model to set it self as answer
on the classification instance.

#### FieldType

Field from pydantic can be anything so hard to type. This is supposed to indicate that you should use the
`pydantic.Field` function to construct this var.

### OntologyDataModel Objects

```python theme={"dark"}
class OntologyDataModel(Generic[OntologyType])
```

Class to create a pydantic model equivalent to an arbitrary classification ontology.

The model can be used to form a json schema based on the ontology. This is useful if
you are, e.g., trying to get a structured response from an LLM.

**Example:**

```python theme={"dark"}
from pydantic import ValidationError

classifications = project.ontology_structure.classifications
objects = project.ontology_structure.classifications

data_model = OntologyDataModel([objects])
## or
data_model = OntologyDataModel([classifications])

## Get a json schema for the ontology
print(data_model.model_json_schema_str)

## Parse json following the schema into label instances
json_str = my_favourite_llm(
    f"what is this? pls follow {schema}", img
)
try:
    instances = data_model(json_str)
except ValidationError:
    # invalid json
    ...

for ins in instances:
    label_row.add_classification_instance(ins)

label_row.save()
```

**Attributes**:

ontology:
DataModel:

#### \_\_call\_\_

```python theme={"dark"}
def __call__(answer: str) -> list[ClassificationInstance] | ObjectInstance
```

Validate a json response in accordance to the pydantic model.

This function allows you to convert from a json object (e.g., coming from an llm)
back to the encord "instance format".

**Arguments**:

* `answer` - The json object as a raw string.

* `Returns` - a list of classification / object instances that you will then
  have to add to a label row.

#### validate\_json

```python theme={"dark"}
def validate_json(
        answer_str: str) -> list[ClassificationInstance] | ObjectInstance
```

Validate a json response in accordance to the pydantic model.

This function allows you to convert from a json object (e.g., coming from an llm)
back to the encord "instance format".

**Arguments**:

* `answer_str` - The json object as a raw string.

* `Returns` - a list of classification / object instances that you will then
  have to add to a label row.

Settings used throughout the module.

Note that central settings will be read via environment variables.

### Settings Objects

```python theme={"dark"}
class Settings(BaseSettings)
```

#### ssh\_key\_file

The path to the private access key file to authenticate with Encord.

Either this or the `ENCORD_SSH_KEY` needs to be set for most use-cases.
To setup a key with Encord, please see
[the platform docs](/platform-documentation/General/general-access-keys).

#### ssh\_key\_content

The content of the private access key file to authenticate with Encord.

Either this or the `ENCORD_SSH_KEY` needs to be set for most use-cases.
To setup a key with Encord, please see
[the platform docs](/platform-documentation/General/general-access-keys).

#### get\_frame

```python theme={"dark"}
def get_frame(video_path: Path, desired_frame: int) -> NDArray[np.uint8]
```

Extract an exact frame from a video.

**Arguments**:

* `video_path` - The file path to where the video is stored.
* `desired_frame` - The frame to extract

**Raises**:

* `Exception` - If the video cannot be opened properly or the requested
  frame could not be retrieved from the video.

**Returns**:

Numpy array of shape \[h, w, c] where channels are BGR.

#### write\_frame

```python theme={"dark"}
def write_frame(frame_path: Path, frame: NDArray[np.uint8]) -> None
```

Write a frame to a file.

**Arguments**:

* `frame_path` - The file path to write the frame to.
* `frame` - The frame to write.

#### iter\_video

```python theme={"dark"}
def iter_video(video_path: Path) -> Iterator[Frame]
```

Iterate video frame by frame.

**Arguments**:

* `video_path` - The file path to the video you wish to iterate.

**Raises**:

* `Exception` - If the video file could not be opened properly.

**Yields**:

Frames from the video.

#### iter\_video\_with\_indices

```python theme={"dark"}
def iter_video_with_indices(video_path: Path,
                            frame_indices: Iterable[int]) -> Iterator[Frame]
```

Iterate video frame by frame with specified frame indices.

**Arguments**:

* `video_path` - The file path to the video you wish to iterate.
* `frame_indices` - The frame indices to iterate over.

**Yields**:

Frames from the video.

This module defines dependencies available for injection within serverless Custom Agents. These dependencies can be used independently, even when reliant on other dependencies.

Note: The injection mechanism necessitates the presence of type annotations for the following parameters to ensure proper resolution.

* [`FrameData`](/agents-documentation/Reference/Core/Agents-Reference-Core#framedata-objects) is automatically injected via the api request body.
* [`Project`](/sdk-documentation/sdk-references/project) is automatically loaded based on the frame data.
* `label_row_v2` is automatically loaded based on the frame data.

```python theme={"dark"}
from encord.project import Project
from encord.objects.ontology_labels_impl import LabelRowV2
from encord_agents import FrameData
...
@editor_agent()
def my_agent(
    frame_data: FrameData,
    project: Project,
    label_row: LabelRowV2,
):
    ...
```

#### dep\_client

```python theme={"dark"}
def dep_client() -> EncordUserClient
```

Dependency to provide an authenticated user client.

**Example:**

```python theme={"dark"}
from encord.user_client import EncordUserClient
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_client
...
@editor_agent()
def (
    client: Annotated[EncordUserClient, Depends(dep_client)]
):
    # Client will authenticated and ready to use.
    client.get_dataset("")
```

#### dep\_single\_frame

```python theme={"dark"}
def dep_single_frame(storage_item: StorageItem,
                     frame_data: FrameData) -> NDArray[np.uint8]
```

Dependency to inject the first frame of the underlying asset.

The downloaded asset will be named `lr.data_hash.{suffix}`.
When the function has finished running, the downloaded file is removed from the file system.

**Example:**

```python theme={"dark"}
from encord_agents import FrameData
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_single_frame
...

@editor_agent()
def my_agent(
    frame: Annotated[NDArray[np.uint8], Depends(dep_single_frame)]
):
    assert frame.ndim == 3, "Will work"
```

**Arguments**:

* `storage_item` - The Storage item. Automatically injected (see example above).

**Returns**:

Numpy array of shape \[h, w, 3] RGB colors.

#### dep\_asset

```python theme={"dark"}
def dep_asset(storage_item: StorageItem) -> Generator[Path, None, None]
```

Returns a local file path to the data asset, temporarily stored for the duration of the agent's execution.

This dependency fetches the underlying data asset using a signed URL.

The asset is temporarily stored on disk for the duration of the task and is automatically removed once the task
completes.

**Example:**

```python theme={"dark"}
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_asset
...
runner = Runner(project_hash="<project_hash_a>")

@editor_agent()
def my_agent(
    asset: Annotated[Path, Depends(dep_asset)]
) -> None:
    asset.stat()  # read file stats
    ...
```

**Returns**:

The path to the asset.

**Raises**:

* `ValueError` - if the underlying assets are not videos, images, or audio.
* `EncordException` - if data type not supported by SDK yet.

#### dep\_video\_iterator

```python theme={"dark"}
def dep_video_iterator(
        storage_item: StorageItem) -> Generator[Iterator[Frame], None, None]
```

Dependency to inject a video frame iterator for performing operations over many frames.

**Example:**

```python theme={"dark"}
from encord_agents import FrameData
from encord_agents.gcp import editor_agent
from encord_agents.gcp.dependencies import dep_video_iterator
...

@editor_agent()
def my_agent(
    video_frames: Annotated[Iterator[Frame], Depends(dep_video_iterator)]
):
    for frame in video_frames:
        print(frame.frame, frame.content.shape)
```

**Arguments**:

* `storage_item` - Automatically injected storage item dependency.

**Raises**:

* `NotImplementedError` - Fails for data types other than video.

**Yields**:

An iterator.

#### dep\_data\_lookup

```python theme={"dark"}
def dep_data_lookup(
        lookup: Annotated[DataLookup,
                          Depends(DataLookup.sharable)]) -> DataLookup
```

Returns a lookup for easily retrieving data rows and storage items associated with the given task.

!!! warning "Deprecated"
`dep_data_lookup` is deprecated and will be removed in version 0.2.10.
Use `dep_storage_item` instead for accessing storage items.

**Migration Guide:**

```python theme={"dark"}
## Old way (deprecated)
from encord_agents.core.dependencies.serverless import dep_data_lookup, DataLookup

@editor_agent()
def my_agent(
    frame_data: FrameData,
    lookup: Annotated[DataLookup, Depends(dep_data_lookup)]
):
    storage_item = lookup.get_storage_item(frame_data.data_hash)
    ...

## New way (recommended)
from encord_agents.gcp.dependencies import dep_storage_item

@editor_agent()
def my_agent(
    frame_data: FrameData,
    storage_item: Annotated[StorageItem, Depends(dep_storage_item)]
):
    # storage_item is directly available
    print(storage_item.client_metadata)
    ...
```

**Arguments**:

* `lookup` - The object that you can use to lookup data rows and storage items. Automatically injected.

**Returns**:

The (shared) lookup object.

#### dep\_storage\_item

```python theme={"dark"}
def dep_storage_item(storage_item: StorageItem) -> StorageItem
```

Get the storage item associated with the underlying agent task.

The [`StorageItem`](/sdk-documentation/sdk-references/storage#storageitem-objects)
is useful for multiple things like

* Updating client metadata
* Reading file properties like storage location, fps, duration, DICOM tags, etc.

**Example**

````python theme={"dark"}

```python
from typing_extensions import Annotated
from encord.storage import StorageItem
from encord_agents.gcp import editor_agent, Depends
from encord_agents.gcp.dependencies import dep_storage_item

@editor_agent()
def my_agent(storage_item: Annotated[StorageItem, Depends(dep_storage_item)]):
    print("uuid", storage_item.uuid)
    print("client_metadata", storage_item.client_metadata)
    ...
````

#### dep\_object\_crops

```python theme={"dark"}
def dep_object_crops(
    filter_ontology_objects: list[Object | str] | None = None
) -> Callable[[FrameData, LabelRowV2, NDArray[np.uint8]], list[InstanceCrop]]
```

Returns a list of object instances and frame crops associated with each object.

One example use-case is to run each crop against a model.

**Example:**

```python theme={"dark"}
@editor_agent
def my_agent(crops: Annotated[list[InstanceCrop], Depends[dep_object_crops(filter_ontology_objects=["eBw/75bg"])]]):
    for crop in crops:
        crop.content  # <- this is raw numpy rgb values
        crop.frame    # <- this is the frame number in video
        crop.instance # <- this is the object instance from the label row
        crop.b64_encoding()  # <- a base64 encoding of the image content
    ...
```

**Arguments**:

* `filter_ontology_objects` - Specify a list of ontology objects to include.
  If provided, only instances of these object types are included.
  Strings are matched against `feature_node_hashes`.

* `Returns` - The dependency to be injected into the cloud function.

#### DEncordClient

Get an authenticated user client.

#### DObjectsInstances

Get all object instances that the agent was triggered on.
No pixels, just the annotation.

#### DObjectCrops

Get all object crops that the agent was triggered on.
The instance crop contains the object instance, the frame content (pixel values), and the frame.

#### DSingleFrame

Get the single frame that the agent was triggered on.

#### DAssetPath

Get a local file path to data asset temporarily stored till end of agent execution.

#### DVideoIterator

Get a video frame iterator for doing things over many frames.

#### DStorageItem

Get the storage item associated with the underlying agent task to, for example, read/write client metadata or read data properties.

### DataLookup Objects

```python theme={"dark"}
class DataLookup()
```

!!! warning "Deprecated"
`DataLookup` is deprecated and will be removed in version 0.2.10.

**Migration Guide:**

* For accessing storage items, use `dep_storage_item` instead:
  ```python theme={"dark"}
  # Old way (deprecated)
  from encord_agents.core.dependencies.shares import DataLookup
  lookup: Annotated[DataLookup, Depends(dep_data_lookup)]
  storage_item = lookup.get_storage_item(data_hash)

  # New way (recommended)
  from encord_agents.tasks.dependencies import dep_storage_item
  # or from encord_agents.aws.dependencies import dep_storage_item
  # or from encord_agents.gcp.dependencies import dep_storage_item
  # or from encord_agents.fastapi.dependencies import dep_storage_item
  storage_item: Annotated[StorageItem, Depends(dep_storage_item)]
  ```

#### backing\_item\_uuids

```python theme={"dark"}
@property
def backing_item_uuids() -> list[UUID]
```

Get all backing item uuids for all data rows in the data lookup.

!!! warning "Deprecated"
This property is deprecated and will be removed in version 0.2.10.
Use the EncordUserClient directly to access backing item UUIDs from label rows.

#### get\_storage\_item

```python theme={"dark"}
def get_storage_item(data_hash: str | UUID,
                     dataset_hash: str | UUID | None = None,
                     sign_url: bool = False) -> StorageItem
```

!!! warning "Deprecated"
This method is deprecated and will be removed in version 0.2.10.
Use `dep_storage_item` dependency instead.

**Arguments**:

* `data_hash` - Data hash for the asset for which you need the underlying storage item.
* `dataset_hash` - If you didn't provide the associated dataset hash in the constructor,
  this is your last chance.
* `sign_url` - If `True`, pre-fetch a signed URLs for the items (otherwise the URLs will be signed on demand).

**Raises**:

* `ValueError` - Mainly if underlying data row cannot be found.

**Returns**:

The underlying storage item from which, e.g., client metadata can be updated.

#### get\_storage\_items

```python theme={"dark"}
def get_storage_items(data_hashes: list[str | UUID],
                      dataset_hash: str | UUID | None = None,
                      sign_urls: bool = False) -> list[StorageItem]
```

!!! warning "Deprecated"
This method is deprecated and will be removed in version 0.2.10.
Use the EncordUserClient directly for bulk storage item access.

**Arguments**:

* `data_hashes` - Data hashes for the assets for which you need the underlying storage items.
* `dataset_hash` - If you didn't provided the associated dataset hash in the constructor,
  this is your last chance.
* `sign_urls` - If `True`, pre-fetch a signed URLs for the items (otherwise the URLs will be signed on demand).

**Raises**:

* `ValueError` - Mainly if underlying data row cannot be found.

**Returns**:

list of underlying storage items from which, e.g., client metadata can be updated.