---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: rawduck
excerpt: |
DuckDB Community Extensions
Schema-less JSON and OTEL ingestion shredder for DuckDB
extension:
build: cmake
description: Schema-less JSON and OTEL ingestion shredder for DuckDB
excluded_platforms: windows_amd64;windows_arm64;windows_amd64_mingw;windows_amd64_rtools
language: C++
license: MIT
maintainers:
- lmangani
name: rawduck
requires_toolchains: parser_tools
version: '0.0.4'
repo:
github: quackscience/rawduck
ref: a14e474c1bae1459cdd22ee8d9eaf5a540ed8fa5
canonical_name: rawduck
docs:
hello_world: |
-- Attach a rawduck db/catalog to your session
ATTACH 'rawduck:store.db' AS raw;
-- no table 'events' exists yet
INSERT INTO raw.ingest.events VALUES
('{"id": 1, "action": "click", "ts": "2024-01-15T10:30:00", "user": {"name": "alice"}}'),
('{"id": 2, "action": "view", "ts": "2024-01-15T10:31:00", "user": {"name": "bob", "plan": "pro"}}');
DESCRIBE raw.events;
-- id BIGINT, action VARCHAR, ts TIMESTAMP, user.name VARCHAR, user.plan VARCHAR
SELECT "user.name", count(*) FROM raw.events GROUP BY 1;
extended_description: |
RawDuck brings the RawMergeTree "ingest first, schema later" model to DuckDB: point raw JSON,
NDJSON files, or OTLP telemetry at tables that don't exist yet — RawDuck creates them, types them,
flattens nested objects into real columns, transforms and evolves the schema as the data changes.
RawDuck ships with a full-featured API to ingest/query data including OpenTelemetry (OTLP) formats.
Check out the [RawDuck README](https://github.com/quackscience/rawduck) for more examples and details.
extension_star_count: 39
extension_star_count_pretty: 39
extension_download_count: 846
extension_download_count_pretty: 846
image: '/images/community_extensions/social_preview/preview_community_extension_rawduck.png'
layout: community_extension_doc
---
### Compatibility Matrix
This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ❌ | ❌ | ✅ | ✅ | ❌ |
| Linux ARM64 | ❌ | ❌ | ✅ | ✅ | ❌ |
| macOS AMD64 | ❌ | ❌ | ✅ | ✅ | ❌ |
| macOS ARM64 | ❌ | ❌ | ✅ | ✅ | ❌ |
| Windows AMD64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ✅ | ✅ | ❌ |
### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```
{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}
{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions
| function_name | function_type | description | comment | examples |
|----------------------|---------------|-------------|---------|----------|
| raw_flush | table | NULL | NULL | |
| raw_infer | scalar | NULL | NULL | |
| raw_ingest | table | NULL | NULL | |
| raw_ingest_file | table | NULL | NULL | |
| raw_optimize | table | NULL | NULL | |
| raw_project | table | NULL | NULL | |
| raw_projections | table | NULL | NULL | |
| raw_records | table | NULL | NULL | |
| raw_serve | table | NULL | NULL | |
| raw_serve_grpc | table | NULL | NULL | |
| raw_serve_grpc_stop | table | NULL | NULL | |
| raw_serve_stop | table | NULL | NULL | |
| raw_stats | table | NULL | NULL | |
| raw_stats_load | table | NULL | NULL | |
| raw_stats_save | table | NULL | NULL | |
| raw_transform_define | scalar | NULL | NULL | |
| raw_transforms | table | NULL | NULL | |
| raw_type | scalar | NULL | NULL | |
### Overloaded Functions
This extension does not add any function overloads.
### Added Types
This extension does not add any types.
### Added Settings
| name | description | input_type | scope | aliases |
|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------|---------|
| rawduck_async_busy_timeout_ms | Async insert buffer flush age threshold | BIGINT | GLOBAL | [] |
| rawduck_async_flush_threads | Async flush worker count: due tables flush concurrently, one worker per table at a time (0 = auto from hardware concurrency) | BIGINT | GLOBAL | [] |
| rawduck_async_insert | Buffer ingestion calls and flush asynchronously | BOOLEAN | GLOBAL | [] |
| rawduck_async_max_data_size | Async insert buffer flush threshold in bytes | BIGINT | GLOBAL | [] |
| rawduck_insert_transform | Transform name or explode path applied by INSERTs into ingest tables | VARCHAR | GLOBAL | [] |
| rawduck_overlap_flush | Flush completed row groups during a large multi-threaded ingest while the schema is stable, overlapping parse with compression/IO. Faster on large stable-schema imports at the cost of higher peak memory; off by default (drain-free) | BOOLEAN | GLOBAL | [] |
| rawduck_overlap_flush_auto | When true, overlap parse/flush on stable re-ingest batches that meet pool_min_rows (without setting rawduck_overlap_flush globally) | BOOLEAN | GLOBAL | [] |
| rawduck_pipeline_consumers | Parallel ingest consumers for raw_ingest_file (default 1; >1 overlaps parse with append on stable schemas only — wide schema churn should stay at 1) | BIGINT | GLOBAL | [] |
| rawduck_pipeline_depth | Bounded queue depth between reader, parse workers, and ingest consumer(s) | BIGINT | GLOBAL | [] |
| rawduck_pipeline_threads | NDJSON file parse thread count (0 = auto, up to 8 on large imports) | BIGINT | GLOBAL | [] |
| rawduck_pool_min_rows | Minimum rows in a batch before the multi-threaded append pool activates | BIGINT | GLOBAL | [] |
| rawduck_pool_threads | Append-pool worker count (0 = auto from batch size and hardware concurrency) | BIGINT | GLOBAL | [] |
| rawduck_use_projections | Rewrite eligible count(*) aggregations onto fresh materialized projections (append-only workloads) | BOOLEAN | GLOBAL | [] |