---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: warc
excerpt: |
  DuckDB Community Extensions
  Parse WARC (Web ARChive) records for Common Crawl data processing

extension:
  name: warc
  description: Parse WARC (Web ARChive) records for Common Crawl data processing
  version: 0.1.0
  language: Rust
  build: cargo
  license: MIT
  excluded_platforms: "wasm_mvp;wasm_eh;wasm_threads;linux_amd64_musl"
  requires_toolchains: "rust"
  maintainers:
    - onnimonni

repo:
  github: midwork-finds-jobs/duckdb_warc
  variegata: 7184683eec41abd8dbe34b8457c780cd80648cae
  ref: 954cbe42e5fdb6f90d6e3ad20879fc36bcaeff3b

docs:
  hello_world: |
    -- Parse a WARC record from a gzip-compressed file
    SELECT parse_warc(content) FROM read_blob('record.warc.gz');
    ┌─────────────────────────────────────────────────────────────────────────────┐
    │                              parse_warc(content)                            │
    │ struct(warc_version varchar, warc_headers varchar, http_version varchar,    │
    │        http_status integer, http_headers varchar, http_body blob)           │
    ├─────────────────────────────────────────────────────────────────────────────┤
    │ {'warc_version': '1.0', 'warc_headers': '{"WARC-Type": "response", ...}',   │
    │  'http_version': 'HTTP/1.1', 'http_status': 200,                            │
    │  'http_headers': '{"content-type": "text/html", ...}',                      │
    │  'http_body': <!doctype html>...}                                           │
    └─────────────────────────────────────────────────────────────────────────────┘

    -- Extract specific fields
    SELECT
        (parse_warc(content)).http_status,
        (parse_warc(content)).http_body
    FROM read_blob('record.warc.gz');

  extended_description: |
    The WARC extension parses WARC (Web ARChive) records, the standard format used by Common Crawl
    and web archiving tools. It enables efficient processing of web archive data directly in DuckDB.

    ## Function

    ### `parse_warc(data)`

    Parse a WARC record and return a struct with all components.

    **Parameters:**
    - `data` (BLOB or VARCHAR): WARC record data (auto-detects gzip compression)

    **Returns:** STRUCT with fields:
    - `warc_version` (VARCHAR): WARC format version (e.g., "1.0")
    - `warc_headers` (VARCHAR): JSON object of WARC headers
    - `http_version` (VARCHAR): HTTP version (e.g., "HTTP/1.1")
    - `http_status` (INTEGER): HTTP status code (e.g., 200)
    - `http_headers` (VARCHAR): JSON object of HTTP headers (lowercase keys)
    - `http_body` (BLOB): Response body content

    ## Common Crawl Workflow

    The recommended workflow for processing Common Crawl data:

    1. **Query the columnar index** (Parquet) to find records of interest
    2. **Fetch only the specific byte ranges** you need using HTTP Range requests
    3. **Parse with this extension**

    ```sql
    -- Example: Parse a downloaded Common Crawl record
    -- First download: curl -r"46376769-46377713" "https://data.commoncrawl.org/crawl-data/..." > record.warc.gz
    SELECT
        (parse_warc(content)).http_status,
        decode((parse_warc(content)).http_body) as html
    FROM read_blob('record.warc.gz');
    ```

    ## Features

    - Auto-detects gzip compression
    - Handles binary content (skips body for non-text responses)
    - HTTP header keys are lowercased for consistent access
    - Works with both BLOB and VARCHAR input types

extension_star_count: 7
extension_star_count_pretty: 7
extension_download_count: 684
extension_download_count_pretty: 684
image: '/images/community_extensions/social_preview/preview_community_extension_warc.png'
layout: community_extension_doc
---
### Compatibility Matrix

This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

<div class="extension_compatibility_table"></div>

| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ❌ | ✅ | ❌ | ❌ | ✅ |
| Linux ARM64 | ❌ | ✅ | ❌ | ❌ | ✅ |
| macOS AMD64 | ❌ | ✅ | ❌ | ❌ | ✅ |
| macOS ARM64 | ❌ | ✅ | ❌ | ❌ | ✅ |
| Windows AMD64 | ❌ | ✅ | ❌ | ❌ | ✅ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ❌ | ❌ | ❌ |

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions

<div class="extension_functions_table"></div>

| function_name | function_type | description | comment | examples |
|---------------|---------------|-------------|---------|----------|
| parse_warc    | scalar        | NULL        | NULL    |          |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|      name      |                       description                       | input_type | scope  | aliases | typed_value |
|----------------|---------------------------------------------------------|------------|--------|---------|-------------|
| temp_directory | Set the directory to which temporary files are written. | VARCHAR    | GLOBAL | []      | post.db.tmp |