---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: sitemap
excerpt: |
  DuckDB Community Extensions
  Parse XML sitemaps from websites with automatic discovery via robots.txt

extension:
  name: sitemap
  description: Parse XML sitemaps from websites with automatic discovery via robots.txt
  version: '2026102301'
  language: C++
  build: cmake
  license: MIT
  excluded_platforms: "windows_amd64_mingw"
  requires_toolchains: "libxml2;zlib"
  maintainers:
    - onnimonni

repo:
  github: midwork-finds-jobs/duckdb-sitemap
  andium: 4c873c85c3f7c8dcab6154de51aa218f61677634
  variegata: 15e96dfd6e5dfb6d70a9b7cc5a2675b3969e16b3
  ref: ab4c5718d2eda09b6123a7e7915ebef2266b3e3b

docs:
  hello_world: |
    -- Get all URLs from a sitemap (auto-prepends https://)
    SELECT * FROM sitemap_urls('example.com');

    -- Direct sitemap URL (skips discovery)
    SELECT * FROM sitemap_urls('https://example.com/sitemap.xml');

    -- Get URLs from multiple sites
    SELECT * FROM sitemap_urls(['example.com', 'google.com']);

    -- Filter specific URLs
    SELECT * FROM sitemap_urls('example.com')
    WHERE url LIKE '%/blog/%';

    -- Bruteforce find sitemap (tries 587+ patterns)
    SELECT bruteforce_find_sitemap('example.com') as sitemap_url;

    -- Ignore errors for invalid domains
    SELECT * FROM sitemap_urls(['valid.com', 'invalid.com'], ignore_errors := true);

    -- Set custom user agent
    SET sitemap_user_agent = 'MyBot/1.0';

    -- Count URLs by type
    SELECT
        CASE
            WHEN url LIKE '%/product/%' THEN 'product'
            WHEN url LIKE '%/blog/%' THEN 'blog'
            ELSE 'other'
        END as type,
        count(*) as count
    FROM sitemap_urls('https://example.com')
    GROUP BY type;

  extended_description: |
    The sitemap extension provides a table function for parsing XML sitemaps from websites.
    It automatically discovers sitemaps via multiple fallback methods and supports recursive sitemap index traversal.

    Features:
    - Multi-fallback sitemap discovery (robots.txt, /sitemap.xml, /sitemap_index.xml, HTML meta tags)
    - Session caching for discovered sitemap locations
    - Bruteforce finder (bruteforce_find_sitemap) - tries 587+ common sitemap URL patterns
    - Custom user agent via SET sitemap_user_agent
    - Direct sitemap URL support (skips discovery)
    - Sitemap index support (nested sitemaps)
    - Array support for processing multiple domains
    - Retry logic with exponential backoff
    - Respects Retry-After header on 429 responses
    - Gzip decompression for .xml.gz files
    - Multiple namespace support (standard + Google schemas)
    - SQL filtering with WHERE clauses
    - Optional error ignoring for batch processing

    The function returns a table with columns:
    - url: Page URL (VARCHAR)
    - lastmod: Last modification date (VARCHAR, optional)
    - changefreq: Change frequency hint (VARCHAR, optional)
    - priority: Priority hint 0.0-1.0 (VARCHAR, optional)

    Options:
    - follow_robots (BOOLEAN): Parse robots.txt first (default: true)
    - max_depth (INTEGER): Max sitemap index nesting (default: 3)
    - max_retries (INTEGER): Max retry attempts (default: 5)
    - backoff_ms (INTEGER): Initial backoff in ms (default: 100)
    - max_backoff_ms (INTEGER): Max backoff cap in ms (default: 30000)
    - ignore_errors (BOOLEAN): Don't throw on failed fetches (default: false)

    Example with options:
    ```sql
    SELECT * FROM sitemap_urls(
        'https://example.com',
        follow_robots := true,
        max_depth := 5,
        max_retries := 10
    ) WHERE url LIKE '%/product/%';
    ```

    Bruteforce sitemap discovery (scalar function):
    ```sql
    -- Find sitemap by trying 587+ common patterns
    SELECT bruteforce_find_sitemap('https://example.com') as sitemap_url;
    -- Returns first working sitemap URL or NULL

    -- Patterns tested include:
    -- /sitemap.xml, /sitemap_index.xml
    -- /sitemap/sitemap.xml, /sitemaps/sitemap-index.xml
    -- /en/sitemap.xml, /de/sitemap.xml
    -- /pub/media/sitemap.xml
    -- And 580+ more variations
    ```

    Compose with http_request to fetch page content:
    ```sql
    SELECT s.url, h.body
    FROM sitemap_urls('https://example.com') s
    JOIN LATERAL (SELECT * FROM http_get(s.url)) h ON true
    WHERE s.url LIKE '%/product/%'
    LIMIT 10;
    ```

extension_star_count: 3
extension_star_count_pretty: 3
extension_download_count: 893
extension_download_count_pretty: 893
image: '/images/community_extensions/social_preview/preview_community_extension_sitemap.png'
layout: community_extension_doc
---
### Compatibility Matrix

This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.

<div class="extension_compatibility_table"></div>

| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Linux ARM64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| macOS AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| macOS ARM64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Windows AMD64 | ✅ | ❌ | ✅ | ✅ | ✅ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ✅ | ❌ | ✅ | ✅ | ✅ |

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions

<div class="extension_functions_table"></div>

|      function_name      | function_type | description | comment | examples |
|-------------------------|---------------|-------------|---------|----------|
| bruteforce_find_sitemap | scalar        | NULL        | NULL    |          |
| sitemap_urls            | table         | NULL        | NULL    |          |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|        name        |                       description                       | input_type | scope  | aliases |    typed_value     |
|--------------------|---------------------------------------------------------|------------|--------|---------|--------------------|
| sitemap_max_size   | Maximum (decompressed) sitemap size in bytes            | UBIGINT    | GLOBAL | []      | 67108864           |
| sitemap_user_agent | User agent string for sitemap HTTP requests             | VARCHAR    | GLOBAL | []      | DuckDB-Sitemap/1.0 |
| temp_directory     | Set the directory to which temporary files are written. | VARCHAR    | GLOBAL | []      | post.db.tmp        |