aid: cloud-storage-and-data-acquisition name: Cloud Storage and Data Acquisition description: >- Cloud Storage and Data Acquisition is a topic profile in the API Evangelist Network covering APIs and tooling for ingesting, moving, and persisting bulk and streaming data into cloud-resident storage. It groups object storage services, data-lake foundations, managed ingestion pipelines, change-data-capture connectors, transfer appliances, and data-broker APIs that provide source material for cloud-storage workloads. The topic is intended as an entry point for developers and architects evaluating how data lands in cloud storage from on-premises systems, SaaS APIs, IoT devices, public data sources, and partner exchanges. type: Index image: https://kinlane-productions.s3.amazonaws.com/apis-json/apis-json-logo.jpg url: https://raw.githubusercontent.com/api-evangelist/cloud-storage-and-data-acquisition/refs/heads/main/apis.yml created: '2024-01-01' modified: '2026-04-23' specificationVersion: '0.19' kind: topic tags: - Bulk Transfer - Change Data Capture - Cloud Storage - Data Acquisition - Data Ingestion - Data Lake - ETL - Object Storage - Pipelines - Streaming apis: - aid: cloud-storage-and-data-acquisition:object-storage name: Object Storage Surface tags: - Object Storage - REST image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://aws.amazon.com/s3/ properties: - url: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html type: Documentation - url: https://cloud.google.com/storage/docs/json_api/v1 type: Documentation - url: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api type: Documentation description: >- The object-storage surface includes Amazon S3, Google Cloud Storage, and Azure Blob Storage REST APIs. These APIs provide the canonical landing zone for cloud data acquisition and are the most common targets for ingestion pipelines. - aid: cloud-storage-and-data-acquisition:streaming-ingest name: Streaming Ingest Surface tags: - Pub/Sub - Real-time - Streaming image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://aws.amazon.com/kinesis/data-streams/ properties: - url: https://docs.aws.amazon.com/kinesis/latest/APIReference/ type: Documentation - url: https://cloud.google.com/pubsub/docs/reference/rest type: Documentation - url: https://learn.microsoft.com/en-us/rest/api/eventhub/ type: Documentation description: >- The streaming-ingest surface covers Amazon Kinesis Data Streams, Google Cloud Pub/Sub, and Azure Event Hubs. These services ingest high-volume event streams and make them durable for downstream landing into object storage and analytics systems. - aid: cloud-storage-and-data-acquisition:managed-ingestion name: Managed Ingestion Pipelines tags: - Data Pipelines - ETL - Managed image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://aws.amazon.com/glue/ properties: - url: https://docs.aws.amazon.com/glue/latest/webapi/Welcome.html type: Documentation - url: https://cloud.google.com/dataflow/docs/reference/rest type: Documentation - url: https://learn.microsoft.com/en-us/rest/api/datafactory/ type: Documentation description: >- Managed ingestion pipelines such as AWS Glue, Google Cloud Dataflow, and Azure Data Factory expose REST APIs for orchestrating extract-transform-load and extract-load-transform jobs that land source data in cloud storage. - aid: cloud-storage-and-data-acquisition:change-data-capture name: Change Data Capture Connectors tags: - CDC - Connectors - Replication image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://debezium.io/ properties: - url: https://debezium.io/documentation/ type: Documentation - url: https://docs.aws.amazon.com/dms/ type: Documentation description: >- Change Data Capture connectors (Debezium, AWS DMS, Fivetran, Striim) replicate row-level changes from operational databases to cloud storage and warehouses, providing a low-latency feed for analytics and data-lake hydration. - aid: cloud-storage-and-data-acquisition:transfer-services name: Bulk Transfer Services tags: - Bulk Transfer - Migration - Offline image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://aws.amazon.com/datasync/ properties: - url: https://docs.aws.amazon.com/datasync/latest/userguide/API_Reference.html type: Documentation - url: https://cloud.google.com/storage-transfer/docs/reference/rest type: Documentation - url: https://learn.microsoft.com/en-us/azure/databox/ type: Documentation description: >- Bulk transfer services (AWS DataSync and Snow family, Google Storage Transfer Service, Azure Data Box) move large datasets from on-premises and edge locations into cloud storage over network or via offline appliances. - aid: cloud-storage-and-data-acquisition:data-marketplaces name: Data Marketplaces tags: - Datasets - Marketplace - Open Data image: https://kinlane-productions2.s3.amazonaws.com/apis-json/apis-json-logo.jpg humanURL: https://aws.amazon.com/data-exchange/ properties: - url: https://aws.amazon.com/data-exchange/ type: Documentation - url: https://cloud.google.com/marketplace type: Documentation - url: https://registry.opendata.aws/ type: Open Data Registry description: >- Data marketplaces and open-data registries expose third-party datasets through REST APIs, subscription delivery, and shared buckets. They are an increasingly common acquisition channel for cloud-storage data lakes. common: - type: Topic url: https://apievangelist.com/topics/cloud-storage-and-data-acquisition/ - type: API Evangelist url: https://apievangelist.com/ - type: Network url: https://network.apievangelist.com/ - type: GitHub url: https://github.com/api-evangelist - type: JSONLD url: json-ld/cloud-storage-and-data-acquisition-context.jsonld - type: Spectral url: rules/cloud-storage-and-data-acquisition-rules.yml maintainers: - FN: Kin Lane email: kinlane@gmail.com