---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: crawler
excerpt: |
  DuckDB Community Extensions
  SQL-native web crawler with HTML extraction and MERGE support

extension:
  name: crawler
  description: SQL-native web crawler with HTML extraction and MERGE support
  version: '2026102301'
  language: C++
  build: cmake
  license: MIT
  maintainers:
    - onnimonni

  requires_toolchains: rust
  excluded_platforms: "windows_amd64_mingw;wasm_mvp;wasm_eh;wasm_threads"

repo:
  github: midwork-finds-jobs/duckdb-crawler
  andium: f0aad857435a2ce138c567e8af11f9d4f8ae0c25
  variegata: ce2679bc70f65b0d3525262066ece014b8d8f1c9
  ref: d846ba2074e029a1747853e6bf5baa75e3a5e05c

docs:
  hello_world: |
    SELECT url, jq(html.document, 'h1').text as title
    FROM crawl(['https://example.com']);
  extended_description: |
    The crawler extension provides SQL-native web crawling capabilities for DuckDB.

    Features:
    - `crawl()` table function with automatic rate limiting and robots.txt compliance
    - LATERAL join support: `FROM urls u, LATERAL crawl(u.url)`
    - `sitemap()` for XML sitemap parsing
    - `read_html()` for Google Sheets-style IMPORTHTML (tables, lists, JS variables)
    - `jq()` and `htmlpath()` functions for CSS selector-based extraction
    - `html.readability` for article extraction
    - `html.schema` for JSON-LD/microdata parsing
    - `CRAWLING MERGE INTO` syntax for upsert operations

    Example with read_html (like Google Sheets =IMPORTHTML):
    ```sql
    SELECT * FROM read_html('https://en.wikipedia.org/wiki/...', 'table.wikitable', 1);
    SELECT * FROM read_html('https://example.com/page', 'js=jobs');
    ```

    Example with extraction:
    ```sql
    SELECT
        url,
        jq(html.document, '.price', 'data-amount') as price,
        html.readability.title as article_title
    FROM crawl(['https://example.com/products']);
    ```

    Example with MERGE:
    ```sql
    CRAWLING MERGE INTO pages
    USING crawl(['https://example.com']) AS src
    ON (src.url = pages.url)
    WHEN MATCHED THEN UPDATE BY NAME
    WHEN NOT MATCHED THEN INSERT BY NAME;
    ```

    For full documentation see: https://github.com/midwork-finds-jobs/duckdb-crawler

extension_star_count: 16
extension_star_count_pretty: 16
extension_download_count: 7986
extension_download_count_pretty: 8.0k
image: '/images/community_extensions/social_preview/preview_community_extension_crawler.png'
layout: community_extension_doc
---
### Compatibility Matrix

This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

<div class="extension_compatibility_table"></div>

| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ✅ | ❌ | ✅ | ❌ | ❌ |
| Linux ARM64 | ✅ | ❌ | ✅ | ❌ | ❌ |
| macOS AMD64 | ✅ | ❌ | ✅ | ❌ | ❌ |
| macOS ARM64 | ✅ | ❌ | ✅ | ❌ | ❌ |
| Windows AMD64 | ✅ | ❌ | ✅ | ❌ | ❌ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ❌ | ❌ | ❌ |

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions

<div class="extension_functions_table"></div>

|     function_name     | function_type | description | comment | examples |
|-----------------------|---------------|-------------|---------|----------|
| crawl                 | table         | NULL        | NULL    |          |
| css_select            | scalar        | NULL        | NULL    |          |
| htmlpath              | scalar        | NULL        | NULL    |          |
| jq                    | scalar        | NULL        | NULL    |          |
| page_info             | scalar        | NULL        | NULL    |          |
| read_html             | table         | NULL        | NULL    |          |
| sitemap               | table         | NULL        | NULL    |          |
| stream_merge_internal | table         | NULL        | NULL    |          |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|            name            |                     description                     | input_type | scope  | aliases |
|----------------------------|-----------------------------------------------------|------------|--------|---------|
| crawler_default_delay      | Default crawl delay in seconds if not in robots.txt | DOUBLE     | GLOBAL | []      |
| crawler_max_response_bytes | Maximum response body size in bytes (0 = unlimited) | BIGINT     | GLOBAL | []      |
| crawler_respect_robots     | Whether to respect robots.txt directives            | BOOLEAN    | GLOBAL | []      |
| crawler_timeout_ms         | HTTP request timeout in milliseconds                | BIGINT     | GLOBAL | []      |
| crawler_user_agent         | User agent string for crawler HTTP requests         | VARCHAR    | GLOBAL | []      |