---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: web_archive
excerpt: |
  DuckDB Community Extensions
  Query Common Crawl and Wayback Machine web archive CDX APIs directly from SQL

extension:
  name: web_archive
  description: Query Common Crawl and Wayback Machine web archive CDX APIs directly from SQL
  version: '2026102301'
  language: C++
  build: cmake
  license: MIT
  excluded_platforms: windows_amd64_mingw
  maintainers:
    - onnimonni

repo:
  github: midwork-finds-jobs/duckdb-web-archive
  andium: 33968f9147d851ff565b19546650162c73b85f5e
  variegata: fa55e897d42ad8859f9d707d793b58807a8c8e84
  ref: 57e51a7e4a89e8ad6fd7f00700095ca3baf1cd39

docs:
  hello_world: |
    -- Find archived pages from Wayback Machine
    SELECT url, timestamp, statuscode
    FROM wayback_machine()
    WHERE url LIKE 'example.com/%'
      AND statuscode = 200
    LIMIT 10;

    -- One snapshot per year using DISTINCT ON pushdown
    SELECT DISTINCT ON(year) url, timestamp
    FROM wayback_machine()
    WHERE url = 'github.com/duckdb/duckdb'
      AND statuscode = 200
    LIMIT 5;

    -- Query Common Crawl index
    SELECT url, timestamp, mimetype
    FROM common_crawl_index()
    WHERE url LIKE '%.example.com/%'
      AND statuscode = 200
    LIMIT 10;

  extended_description: |
    This extension provides two table functions to query web archive CDX APIs:

    - **wayback_machine()**: Query Internet Archive Wayback Machine (1996-present)
    - **common_crawl_index()**: Query Common Crawl archive (2008-present, monthly snapshots)

    Features:
    - **DISTINCT ON pushdown**: `DISTINCT ON(year)` becomes `&collapse=timestamp:4`
    - **Filter pushdown**: WHERE clauses for statuscode/mimetype pushed to CDX API
    - **LIMIT pushdown**: Only fetches requested number of records
    - **SELECT pushdown**: Only fetches columns you need via `&fl=` parameter
    - **Response fetching**: Download archived page content via `response.body`
    - **Virtual columns**: `year` and `month` extracted from timestamp

    For full documentation, visit the [extension repository](https://github.com/midwork-finds-jobs/duckdb-web-archive).

extension_star_count: 24
extension_star_count_pretty: 24
extension_download_count: 925
extension_download_count_pretty: 925
image: '/images/community_extensions/social_preview/preview_community_extension_web_archive.png'
layout: community_extension_doc
---
### Compatibility Matrix

This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

<div class="extension_compatibility_table"></div>

| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Linux ARM64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| macOS AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| macOS ARM64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Windows AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ✅ | ❌ | ✅ | ✅ | ✅ |

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions

<div class="extension_functions_table"></div>

|   function_name    | function_type | description | comment | examples |
|--------------------|---------------|-------------|---------|----------|
| common_crawl_index | table         | NULL        | NULL    |          |
| wayback_machine    | table         | NULL        | NULL    |          |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|      name      |                       description                       | input_type | scope  | aliases | typed_value |
|----------------|---------------------------------------------------------|------------|--------|---------|-------------|
| temp_directory | Set the directory to which temporary files are written. | VARCHAR    | GLOBAL | []      | post.db.tmp |