---
title: Sync data from external storage
short: Add project storage
type: guide
tier: all
order: 151
order_enterprise: 151
meta_title: Cloud and External Storage Integration
meta_description: "Label Studio Documentation for integrating Amazon AWS S3, Google Cloud Storage, Microsoft Azure, Redis, and local file directories with Label Studio."
section: "Import & Export"
---
Integrate popular cloud and external storage systems with Label Studio to collect new items uploaded to the buckets, containers, databases, or directories and return the annotation results so that you can use them in your machine learning pipelines.
| Storage | Community | Enterprise |
|---|---|---|
| [Amazon S3](storage_s3) | ✅ | ✅ |
| [Amazon S3 with IAM role](storage_s3#Amazon-S3-with-IAM-role) | ❌ | ✅ |
| [Google Cloud Storage](storage_gcp) | ✅ | ✅ |
| [Google Cloud Storage WIF Auth](storage_gcp#Google-Cloud-Storage-with-Workload-Identity-Federation-WIF) | ❌ | ✅ |
| [Google Cloud Storage with service account impersonation for GKE](storage_gcp#Google-Cloud-Storage-with-service-account-impersonation-for-GKE) | ❌ | ✅ |
| [Microsoft Azure Blob Storage](storage_azure) | ✅ | ✅ |
| [Microsoft Azure Blob Storage with Service Principal](storage_azure#Azure-blob-storage-with-Service-Principal) | ❌ | ✅ |
| [Databricks Files (UC Volumes)](storage_databricks) | ❌ | ✅ |
| [Databricks Files (UC Volumes) with Service Principal](storage_databricks) | ❌ | ✅ |
| [Redis database](storage_redis)| ✅ | ✅ |
| [Local storage](storage_local) | ✅ | ✅ |
You can add source storage connections to sync data from an external source to a Label Studio project, and add target storage connections to sync annotations from Label Studio to external storage.
Each source and target storage setup is project-specific. You can connect multiple buckets, containers, databases, or directories as source or target storage for a project.
## Source storage
Label Studio does not automatically sync data from source storage. If you upload new data to a connected cloud storage bucket, sync the storage connection using the UI to add the new labeling tasks to Label Studio without restarting. You can also use the API to set up or sync storage connections. See [Label Studio API](https://api.labelstud.io/api-reference/introduction/getting-started) and locate the relevant storage connection type.
Label Studio does not import media files from your source storage. Media stays in your bucket and is loaded through pre-signed URLs or the storage proxy when users open a task.
What Label Studio does store depends on the **Import method**. With **Files**, each task stores only a reference (URI) to an object in your bucket. With **Tasks**, Label Studio reads task files in JSON, JSONL, or Parquet format and saves the task data, such as text or metadata, to its database. See [Import method](#Import-method) for details.
You can also secure access to cloud storage using VPC and IP restrictions for your storage. For details, see [Secure access to cloud storages](security.html#Secure-access-to-cloud-storages).
### Source storage permissions
* If you set the import method to "Files", Label Studio backend will only need LIST permissions and won't download any data from your buckets.
* If you set the import method to "Tasks", Label Studio backend will require GET permissions to read JSON files and convert them to Label Studio tasks.
When your users access labeling, the backend will attempt to resolve URI (e.g., s3://) to URL (https://) links. URLs will be returned to the frontend and loaded by the user's browser. To load these URLs, the browser will require HEAD and GET permissions from your Cloud Storage. The HEAD request is made at the beginning and allows the browser to determine the size of the audio, video, or other files. The browser then makes a GET request to retrieve the file body.
### Source storage Sync and URI resolving
Source storage functionality can be divided into two parts:
* Sync - when Label Studio scans your storage and imports tasks from it.
* URI resolving - when the Label Studio backend requests Cloud Storage to resolve URI links (e.g., `s3://bucket/1.jpg`) into HTTPS (`https://aws.amazon.com/bucket/1.jpg`). This way, user's browsers are able to load media.
### Import method
!!! info
The "Treat every bucket object as a source file" option was renamed and reintroduced as the "Import method" dropdown.
Label Studio Source Storages feature an "Import method" dropdown. This setting enables two different methods of loading tasks into Label Studio.
##### Tasks
When set to "Tasks", tasks in JSON, JSONL/NDJSON or Parquet format can be loaded directly from storage buckets into Label Studio. This approach is particularly helpful when dealing with complex tasks that involve multiple media sources.
You may put multiple tasks inside the same JSON file, but not mix task formats inside the same file.
{% details Example with bare tasks %}
`task_01.json`
```
{
"image": "s3://bucket/1.jpg",
"text": "opossums are awesome"
}
```
`task_02.json`
```
{
"image": "s3://bucket/2.jpg",
"text": "cats are awesome"
}
```
Or:
`tasks.json`
```
[
{
"image": "s3://bucket/1.jpg",
"text": "opossums are awesome"
},
{
"image": "s3://bucket/2.jpg",
"text": "cats are awesome"
}
]
```
{% enddetails %}
{% details Example with tasks, annotations and predictions %}
`task_with_predictions_and_annotations_01.json`
```
{
"data": {
"image": "s3://bucket/1.jpg",
"text": "opossums are awesome"
},
"annotations": [...],
"predictions": [...]
}
```
`task_with_predictions_and_annotations_02.json`
```
{
"data": {
"image": "s3://bucket/2.jpg",
"text": "cats are awesome"
}
"annotations": [...],
"predictions": [...]
}
```
Or:
`tasks_with_predictions_and_annotations.json`
```
[
{
"data": {
"image": "s3://bucket/1.jpg",
"text": "opossums are awesome"
},
"annotations": [...],
"predictions": [...]
},
{
"data": {
"image": "s3://bucket/2.jpg",
"text": "cats are awesome"
}
"annotations": [...],
"predictions": [...]
}
]
```
{% enddetails %}
{% details Example with JSONL %}
`tasks.jsonl`
```
{ "image": "s3://bucket/1.jpg", "text": "opossums are awesome" }
{ "image": "s3://bucket/2.jpg", "text": "cats are awesome" }
```
{% enddetails %}
In Label Studio Enterprise and Starter Cloud editions, Parquet files can also be used to import tasks in the same way as JSON and JSONL.
##### Files
When set to "Files", Label Studio automatically lists files from the storage bucket and constructs tasks. This is only possible for simple labeling tasks that involve a single media source (such as an image, text, etc.).*
### Pre-signed URLs vs. Storage proxies
There are two secure mechanisms in which Label Studio fetches media data from cloud storage: via pre-signed URLS and via proxy. Which one you use depends on whether you have **Use pre-signed URLs** toggled on or off when setting up your source storage. **Use pre-signed URLs** is used by default. Proxy storage is enabled when **Use pre-signed URLs** is OFF.
The permissions required for this are already included in the cloud storage configuration documentation below.
#### Proxy storage
When in proxy mode, the Label Studio backend fetches objects server-side and streams them directly to the browser.
This has multiple benefits, including:
- **Security**
- Access to media files is further restricted based on Label Studio user roles and project access.
- This access is applied to cached files. This means that even if the media is cached, access will be restricted to that file if a user's access to the task is revoked.
- Data stays within the Label Studio network boundary. This is especially useful for on-prem environments who want to maintain a single entry point for their network traffic.
- **Configuration**
- No CORS settings are needed.
- No pre-signed permissions are needed.
To allow proxy storage, you need to ensure your permissions include the following:
{% details AWS S3 %}
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
```
{% enddetails %}