---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: lance_conversion
excerpt: |
  DuckDB Community Extensions
  Stream any DuckDB query into Lance datasets with Blob v2 storage and scalar, vector, text, and Bloom filter indexes

extension:
  name: lance_conversion
  description: Stream any DuckDB query into Lance datasets with Blob v2 storage and scalar, vector, text, and Bloom filter indexes
  version: 0.2.1
  language: C++ & Rust
  build: cmake
  license: MIT
  excluded_platforms: "wasm_mvp;wasm_eh;wasm_threads;windows_amd64;windows_amd64_mingw;osx_amd64"
  requires_toolchains: rust
  maintainers:
    - dentiny

repo:
  github: dentiny/duckdb_lance_conversion
  ref: 46a105d76e886df3bac24d47322345664225b733

docs:
  hello_world: |
    INSTALL lance_conversion FROM community;
    LOAD lance_conversion;

    COPY (
      SELECT *
      FROM read_parquet('events/**/*.parquet')
    ) TO 'events.lance' (FORMAT LANCE);

  extended_description: |
    `lance_conversion` registers Lance as a native DuckDB `COPY` destination.
    DuckDB reads, filters, joins, casts, and projects the source query, then
    streams its output into the Rust Lance writer without an intermediate
    Parquet export or full-input materialization.

    Any source that DuckDB can query can be written to Lance, including DuckDB
    tables and views, Parquet, CSV, JSON, and S3-backed data:

    ```sql
    COPY events TO 'events.lance' (FORMAT LANCE);

    COPY (
      SELECT *
      FROM read_json_auto('events.json')
    ) TO 'events_from_json.lance' (FORMAT LANCE);
    ```

    The extension also provides `read_huggingface` for Hugging Face Parquet
    datasets and `read_warc` for local or S3 WARC/WARC.GZ files:

    ```sql
    COPY (
      SELECT * FROM read_huggingface('lhoestq/demo1')
    ) TO 'demo1.lance' (FORMAT LANCE);

    COPY (
      SELECT id, type, date, content_type, body
      FROM read_warc('archive.warc.gz')
    ) TO 'archive.lance' (FORMAT LANCE);
    ```

    `read_huggingface` fetches only the projected Parquet columns and decodes
    row groups in the background. `max_read_parallelism` bounds concurrent
    row-group reads, and `read_ahead_bytes` caps the decoded data each one may
    queue (`0` disables read-ahead):

    ```sql
    SELECT id, package_name
    FROM read_huggingface(
      'lhoestq/demo1',
      max_read_parallelism = 8,
      read_ahead_bytes = 67108864
    );
    ```

    Storage timeouts and retries for S3 and Hugging Face I/O are configured
    through the `lance_conversion_storage_*` settings:

    ```sql
    SET lance_conversion_storage_io_timeout_ms = 30000;
    SET lance_conversion_storage_retry_max_times = 5;
    ```

    Lance writes support Create (the default), Append, and Overwrite modes:

    ```sql
    COPY new_events TO 'events.lance' (FORMAT LANCE, APPEND);
    COPY replacement TO 'events.lance' (FORMAT LANCE, OVERWRITE);
    ```

    When row order need not be preserved, every DuckDB thread runs its own
    Lance writer, and all writers commit as one dataset version. Use
    `PRESERVE_ORDER false` for the fastest writes:

    ```sql
    COPY events TO 'events.lance' (FORMAT LANCE, PRESERVE_ORDER false);
    ```

    `SAMPLE_PERCENT` and `SAMPLE_ROWS` write a random sample of the source
    rows instead of every row:

    ```sql
    COPY events TO 'events_10pct.lance' (FORMAT LANCE, SAMPLE_PERCENT 10);
    COPY events TO 'events_sample.lance' (FORMAT LANCE, SAMPLE_ROWS 1000);
    ```

    `GENERATED_COLUMNS` adds columns computed by DuckDB SQL expressions from
    each source row. Lance stores each expression in the column's metadata,
    and `APPEND` must use the same definitions:

    ```sql
    COPY (
      SELECT * FROM read_warc('archive.warc.gz') WHERE type = 'response'
    ) TO 'responses.lance' (FORMAT LANCE, GENERATED_COLUMNS {
      'doc_id': $$ sha256(id || ':' || target_uri) $$,
      'body_size': 'octet_length(body)'
    });
    ```

    Blob v2 layout, data file sizing, and indexes are configurable directly
    from the `COPY` statement:

    ```sql
    COPY (
      SELECT id, event_id, category, description, embedding, asset_uri
      FROM read_parquet('catalog/**/*.parquet')
    ) TO 'catalog.lance' (
      FORMAT LANCE,
      TARGET_FILE_SIZE 536870912,
      BLOB_INLINE_SIZE_THRESHOLD 2097152,
      BLOB_DEDICATED_SIZE_THRESHOLD 16777216,
      BLOB_COLUMNS (asset_uri),
      SCALAR_INDEX_COLUMNS (id, category),
      VECTOR_INDEX_COLUMNS (embedding),
      TEXT_INDEX_COLUMNS (description),
      BLOOM_FILTER_INDEX_COLUMNS (event_id)
    );
    ```

    Local and S3-compatible Lance destinations are supported. S3 credentials,
    endpoint, region, TLS, and URL style are resolved from DuckDB secrets and
    passed to the OpenDAL-backed Lance object store.

extension_star_count: 0
extension_star_count_pretty: 0
extension_download_count: 2767
extension_download_count_pretty: 2.8k
image: '/images/community_extensions/social_preview/preview_community_extension_lance_conversion.png'
layout: community_extension_doc
---
### Compatibility Matrix

This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

<div class="extension_compatibility_table"></div>

| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Linux ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| macOS AMD64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| macOS ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Windows AMD64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ❌ | ❌ | ❌ |

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions

<div class="extension_functions_table"></div>

|  function_name   | function_type |                                                          description                                                           | comment |                      examples                      |
|------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------|---------|----------------------------------------------------|
| read_atif        | table         | Reads ATIF agent trajectories from a local, S3, or HTTP(S) JSON or JSONL file, optionally gzipped, embedding referenced media. | NULL    | [SELECT * FROM read_atif('trajectories.jsonl');]   |
| read_huggingface | table         | Reads a Hugging Face dataset's Parquet files.                                                                                  | NULL    | [SELECT * FROM read_huggingface('lhoestq/demo1');] |
| read_warc        | table         | Reads records from a local, S3, or HTTP(S) WARC file or Common Crawl crawl or segment.                                         | NULL    | [SELECT * FROM read_warc('archive.warc.gz');]      |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|                    name                     |                            description                            | input_type | scope  | aliases | typed_value |
|---------------------------------------------|-------------------------------------------------------------------|------------|--------|---------|-------------|
| lance_conversion_storage_io_timeout_ms      | IO operation and body method timeout per attempt (ms)             | BIGINT     | GLOBAL | []      | 10000       |
| lance_conversion_storage_retry_factor       | Storage exponential retry backoff multiplier for reads and writes | DOUBLE     | GLOBAL | []      | 2.0         |
| lance_conversion_storage_retry_max_delay_ms | Maximum exponential retry backoff (ms)                            | BIGINT     | GLOBAL | []      | 60000       |
| lance_conversion_storage_retry_max_times    | Maximum retries after the initial attempt (0 disables retries)    | BIGINT     | GLOBAL | []      | 3           |
| lance_conversion_storage_retry_min_delay_ms | Initial exponential retry backoff (ms)                            | BIGINT     | GLOBAL | []      | 1000        |
| lance_conversion_storage_timeout_ms         | Control operation timeout per attempt (ms)                        | BIGINT     | GLOBAL | []      | 60000       |
| temp_directory                              | Set the directory to which temporary files are written.           | VARCHAR    | GLOBAL | []      | post.db.tmp |