GCP Storage

There are multiple parts to setting up a GCP integration in Encord:

1. Start setting up the integration in Encord and choose an authentication type.
2. Give Encord access to your bucket, using one of:
   • The Encord service account (default).
   • Workload Identity Federation (WIF). Encord uses a service account in your own GCP project, without any service account keys.
3. Set up a CORS configuration.
4. Test the integration.

Ensure all objects in the bucket use the Standard Storage class.

1. Start Setting Up The Integration

1. Navigate to Workspace Settings > Cloud Buckets and click +New integration.

2. Select Google Cloud Storage.

3. Give your integration a meaningful title.

4. In Step 2 of the integration, choose an authentication type:

   • Encord service account: grant an Encord-managed service account access to your bucket. Continue to 2A. Grant Service Account Access.
   • Workload Identity Federation (WIF): Encord uses a service account in your own GCP project. Your project trusts Encord as an identity provider, so there are no service account keys or shared secrets to store or rotate. Continue to 2B. Set Up Workload Identity Federation.

2A. Grant Service Account Access

Use this method if you selected Encord service account in Step 2 of the integration.

1. Log into your Google Cloud account and navigate to the GCP bucket you want to integrate with Encord.

2. Click Grant Access to add a service account for Encord, in the bucket’s PERMISSION tab.

3. Add the service account shown in Step 3 of the integration in Encord as a New Principal. This is [e-mail] unless your Workspace has its own Encord service account. Assign the following roles:

• Storage Object Viewer
• Storage Object Creator

Storage Object Creator is needed for features that require write permissions, including re-encoding data and creating image sequences.

2A.1 Register the Cloud Bucket with Encord

Paste the name of your GCP bucket into Step 3.1 of the integration in Encord.

Continue to 3. Create a CORS Configuration.

2B. Set Up Workload Identity Federation

Use this method if you selected Workload Identity Federation (WIF) in Step 2 of the integration. Step 3 of the integration lists each sub-step and shows the values to copy into the Google Cloud console. See Service integrations to learn how WIF works.

You need permission to create workload identity pools, service accounts, and IAM bindings in your GCP project.

2B.1 Create a GCP Service Integration in Encord

Only Workspace Admins can create service integrations. If Service integrations does not appear in your Workspace settings, contact Encord to enable it for your Workspace.

1. In Encord, go to Workspace Settings and select Service integrations.

2. Click New service integration.

3. Give the service integration a meaningful Title. For example, Azure WIF - production storage.

4. Under Used for, select the cloud provider you are connecting: Azure, GCP, or AWS.

   Encord sets the signing algorithm to ES256 and fills in the target domain for that provider. You do not need to enter either.

5. Click Create service integration.

The service integration appears in the list with its UUID. You choose it by name when you create the cloud integration.

Create one service integration for each cloud integration. Your cloud provider trusts a single service integration UUID, so keeping them separate lets you revoke access to one bucket or storage account without affecting the others.

Select GCP under Used for. Then, in Step 3.1 of the GCP integration, select the service integration you created.

2B.2 Enable the Required APIs

In the GCP project that contains your bucket, enable the following APIs. Step 3.2 of the integration has a link to enable each one.

• IAM API (iam.googleapis.com)
• Security Token Service API (sts.googleapis.com)
• IAM Service Account Credentials API (iamcredentials.googleapis.com)

2B.3 Create a Workload Identity Pool

1. In the Google Cloud console, go to IAM & Admin > Workload Identity Federation and click Create pool.
2. Give the pool any name. For example, use encord as the pool ID.

2B.4 Add an OIDC Provider to the Pool

Add a provider to the pool with the following settings. Copy the values from Step 3.4 of the integration in Encord.

Setting	Value
Provider type	OpenID Connect (OIDC)
Provider name / ID	encord
Issuer (URL)	Copy from Encord. It has the form https://api.encord.com/v2/public/organisation/<Workspace UUID>/service-integrations.
JWK file	Leave empty. Google fetches the keys from the issuer.
Audiences	Default audience
Attribute mapping	google.subject = assertion.sub
Attribute conditions	assertion.sub == 'service-integration:<service integration UUID>'

2B.5 Register the Provider with Encord

1. Open the provider’s details page in the Google Cloud console and copy its resource name.

2. Paste it into Step 3.5 of the integration in Encord, in the form:

   projects/<project number>/locations/global/workloadIdentityPools/<pool ID>/providers/<provider ID>
   

Use the project number, not the project ID, and remove the https://iam.googleapis.com/ prefix if it is present.

2B.6 Create a Service Account that Can Read Your Bucket

1. In the Google Cloud console, create a service account. For example, encord-access. Do not create any keys for it.
2. Open your bucket’s Permissions tab, click Grant access, and grant the service account the following roles:
   • Storage Object Viewer
   • Storage Object Creator (optional). This is needed for features that write to your bucket, including re-encoding data and creating image sequences.
3. Paste the service account’s email into Step 3.6 of the integration in Encord.

The service account’s permissions are the only access Encord gets. Grant it access to exactly the buckets Encord should use.

2B.7 Let Encord Impersonate the Service Account

1. In the Google Cloud console, open the service account you created and go to Permissions > Grant access.

2. Copy the New principal from Step 3.7 of the integration in Encord. It has the form:

   principal://iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool ID>/subject/service-integration:<service integration UUID>
   

3. Assign the role Service Account Token Creator and click Save.

Service Account Token Creator is required, not just Workload Identity User, because Encord signs URLs to your files as the service account.

Set Up WIF Using the gcloud CLI

You can also set up WIF using the gcloud CLI. Copy the issuer URL and subject from the integration in Encord.

PROJECT_ID=<your project ID>
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
BUCKET=<your bucket name>
ISSUER=<issuer URL from Encord>
SUBJECT=service-integration:<service integration UUID>

gcloud services enable iam.googleapis.com sts.googleapis.com iamcredentials.googleapis.com --project=$PROJECT_ID

gcloud iam workload-identity-pools create encord --location=global --project=$PROJECT_ID
gcloud iam workload-identity-pools providers create-oidc encord \
  --location=global --workload-identity-pool=encord --project=$PROJECT_ID \
  --issuer-uri="$ISSUER" \
  --attribute-mapping="google.subject=assertion.sub" \
  --attribute-condition="assertion.sub == '$SUBJECT'"

gcloud iam service-accounts create encord-access --project=$PROJECT_ID
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:encord-access@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/storage.objectViewer
gcloud iam service-accounts add-iam-policy-binding encord-access@$PROJECT_ID.iam.gserviceaccount.com \
  --project=$PROJECT_ID \
  --member="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/encord/subject/$SUBJECT" \
  --role=roles/iam.serviceAccountTokenCreator

In Encord, enter:

• Provider resource name: projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/encord/providers/encord
• Service account email: encord-access@<PROJECT_ID>.iam.gserviceaccount.com

Revoke Access

Remove the service account’s Token Creator binding, the OIDC provider, or the pool to stop Encord getting new tokens. Tokens already issued stay valid for up to an hour, and URLs already signed stay valid until they expire. To cut off access immediately, also remove the service account’s access to your buckets.

3. Create a CORS Configuration

If you use Infrastructure as Code tools like Terraform, they can overwrite your manual CORS settings. To prevent this, always include the correct CORS policy in your Terraform configuration file.

Applying Encord required CORS settings may overwrite existing configurations on your cloud storage bucket. If your bucket already has CORS settings, make sure to append Encord’s required settings instead of replacing them.

A CORS configuration must be applied to the bucket you want to integrate with Encord. A CORS configuration enables Encord to request resources from the specified bucket.

The only way to create a CORS configuration is using the command line or terminal tool.

Install Google’s Cloud CLI, and use the following gcloud command in your command line or terminal to create a CORS configuration.

gcloud storage buckets update  gs://<YOUR_BUCKET_NAME> --cors-file="/path/to/you/cors.json"

• Replace <YOUR_BUCKET_NAME> with the name of the bucket you want to integrate.
• Replace /path/to/you/cors.json with the full path to your JSON CORS file. A sample JSON file is provided below.

Click Create in Encord to finish creating the integration, after creating a CORS configuration.

JSON configuration file

The following JSON file can be used to set up the CORS configuration in Step 4 of your GCP integration.

Global

[
    {
      "origin": ["https://app.encord.com", "https://api.encord.com","https://app.us.encord.com", "https://api.us.encord.com"],
      "method": ["GET", "HEAD"],
      "responseHeader": ["If-Modified-Since", "If-None-Match", "If-Range", "Range", "Accept-Ranges", "Content-Length", "Content-Range", "Content-Type", "ETag", "Last-Modified"],
      "maxAgeSeconds": 3600
    }
]

• Preflight requests are valid for 1 hour. Use the maxAgeSeconds variable to adjust the number of seconds the browser is allowed to make requests before it must repeat the preflight request.
• Successful browser requests return the Content-Type of the resource in the response, as set by the responseHeader.
• Byte-range requests (GET/HEAD) require the Range request header and the exposed response headers shown above so large scene files (such as MCAP) can stream in chunks in the browser.

Consult the GCP documentation, to customize the file further.

4. Test Integration

Click the Run a test button on the integration to test the integration.

Paste the URL of any object in the bucket and click Check Encord can access this URL. If the test is successful a green tick appears next to Encord infrastructure and This machine.

---

Register GCP Data

See the Register cloud data page for guidance on how to register data stored in GCP.

We recommend setting the expiration time for pre-signed URLs (found in your cloud storage settings) to be greater than the time it takes to complete an annotation task. More information can be found in the GCP documentation.

GCP with Unsigned URLs (Securely)

To use unsigned URLs, you must set up a Direct Access integration in Encord.