---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: panduck
excerpt: |
DuckDB Community Extensions
Read documents natively into the duck_block vocabulary -- DOCX, ODT, EPUB, RTF, LaTeX, Org, RST, ipynb, MediaWiki and Textile -- and write them back as Pandoc JSON, without linking Pandoc
extension:
name: panduck
description: Read documents natively into the duck_block vocabulary -- DOCX, ODT, EPUB, RTF, LaTeX, Org, RST, ipynb, MediaWiki and Textile -- and write them back as Pandoc JSON, without linking Pandoc
version: 0.7.0
language: C++
build: cmake
license: MIT
requires_toolchains: vcpkg
maintainers:
- teaguesterling
# The default pin from extension-ci-tools v1.5.5, which is what panduck's own
# distribution pipeline builds green against on every platform including wasm.
vcpkg_commit: 84bab45d415d22042bd0b9081aea57f362da3f35
repo:
github: teaguesterling/duckdb_panduck
ref: dc094ee3261251a69f3f394b775709a682fb0640
# variegata pins the v1.5.6 track. scripts/build.py sends a v1.5.6 build here when
# present and to `ref` otherwise, and unlike ref_next it has no skip branch -- so
# once `ref` is 2.0-only a v1.5.6 build would fail rather than be skipped. Equal to
# `ref` while both lines build; advance only to a commit v1.5.6 can still build.
variegata: dc094ee3261251a69f3f394b775709a682fb0640
# THE SAME COMMIT AS `ref`, measured rather than defaulted. MEASURED for 0.7.0 on this
# exact commit, in panduck's post-merge run 41124226755. It is red, for known reasons:
# v1.5.6 and v2.0.0 linux_amd64: 3160 of 3163 assertions. The 3 failures
# (doc_namespace.test:26, multidoc.test:435, register_reader.test:56) INSTALL
# duck_block_utils FROM community. The served duck_block_utils still registers the
# converter names that 0.7.0 now owns, and the second extension to load fails its
# init. They pass once the duck_block_utils release that drops those names is served.
# macOS, Windows and wasm were skipped, because they wait on the linux job. So this
# registry build is their first build of 0.7.0.
# An advisory job that compares against duck_block_utils' converter copy got a 404,
# because that copy is gone. The canary's linux_arm64 failed on an empty yum mirror.
# Before the merge, on longbottom: the v2.0.0 (d54cbbe) suite gave the same 3160/3163, and
# 0.7.0 loads with the new duck_block_utils in both orders.
#
# ref_next STAYS SET even when it equals ref: without it scripts/build.py prints "Skipping
# prerelease validation" and test_against_latest never runs -- a green that means "did not
# look". panduck v0.2.0 merged that way, and the job has since caught two real defects.
# A STALE ref_next is worse than an absent one: it validates the PREVIOUS release
# against the next line while reporting green, so it moves with ref every release.
ref_next: dc094ee3261251a69f3f394b775709a682fb0640
docs:
hello_world: |
LOAD panduck;
-- Read a document straight into flattened duck_block rows
SELECT * FROM read_docx_blocks('report.docx');
SELECT * FROM read_epub_blocks('book.epub');
SELECT * FROM read_latex_blocks('paper.tex');
-- Every reader goes through DuckDB's filesystem, so any path DuckDB can open
-- works: httpfs, s3://, encrypted, in-memory
LOAD httpfs;
SELECT * FROM read_docx_blocks('https://example.org/report.docx');
-- Or let the path pick the reader
SELECT * FROM panduck_read_blocks('notes.org');
SELECT panduck_format_for('paper.tex'); -- 'latex'
SELECT panduck_can_read('archive.docx'); -- true
-- Formats with no file: parse a string directly
SELECT * FROM read_mediawiki_blocks_string('== Heading ==');
-- Outline any supported document
SELECT * FROM doc_toc('report.docx');
-- Pull one section out, by heading or by a fragment of one
SELECT * FROM doc_section('report.docx', 'Methods');
SELECT * FROM doc_section('report.docx', 'Meth', match := 'contains');
-- ...or find the section by what it SAYS rather than what it is called
SELECT * FROM doc_search_sections('report.docx', 'p < 0.05');
-- Render one back out (md, html or text)
SELECT doc_render('report.docx', 'md');
-- Write back out as Pandoc JSON that a real pandoc accepts
SELECT panduck_write_pandoc_ast('doc.json',
(SELECT list(b) FROM read_odt_blocks('doc.odt') b)
);
-- Or get the AST as a struct without touching the filesystem
SELECT panduck_blocks_to_pandoc_ast(
(SELECT list(b) FROM read_rtf_blocks('memo.rtf') b)
);
extended_description: |
Panduck reads document formats natively into the `duck_block` vocabulary shared
with `duck_block_utils`, `markdown` and `webbed` -- so a DOCX, an EPUB and a
LaTeX paper all land in the same seven-column shape and can be queried together.
**It does not link Pandoc.** Pandoc is a Haskell program; the only C bindings
that ever existed went unmaintained in 2017, and a Cabal foreign-library request
has been open upstream since 2020. Linking the GHC runtime into a dlopen'd DuckDB
extension would be a bad neighbour even if it existed. Panduck is instead
compatible with Pandoc's **data model**: every reader is hand-written, and a
checked mapping table pins the correspondence to Pandoc's AST constructors.
## Readers
| Function | Format |
|----------|--------|
| `read_docx_blocks(path)` | Word OOXML |
| `read_odt_blocks(path)` | OpenDocument Text |
| `read_epub_blocks(path)` | EPUB 2/3 |
| `read_rtf_blocks(path)` | Rich Text Format |
| `read_latex_blocks(path)` | LaTeX |
| `read_org_blocks(path)` | Org mode |
| `read_rst_blocks(path)` | reStructuredText |
| `read_ipynb_blocks(path)` | Jupyter notebooks |
| `read_mediawiki_blocks(path)` | MediaWiki wikitext |
| `read_textile_blocks(path)` | Textile |
| `read_pandoc_blocks(path)` | Pandoc JSON AST (reaches all 43 formats pandoc reads) |
Every text format also has a `_string` variant that parses content directly.
DOCX, ODT, EPUB and RTF read lists, blockquotes, tables, images and footnotes;
document metadata is extracted for every format that has any.
## Dispatch
| Function | Description |
|----------|-------------|
| `panduck_read_blocks(path)` | Read any supported path, reader chosen by extension |
| `panduck_format_for(path)` | Which format a path resolves to |
| `panduck_can_read(path)` | Whether a path is supported |
| `panduck_supported_extensions()` | The registry, as a table |
| `panduck_register_doc_reader(...)` | Register a reader at runtime |
## Document helpers
| Function | Description |
|----------|-------------|
| `doc_toc(path)` | Table of contents for any supported document |
| `doc_section(path, heading)` | The blocks under one heading (`match := 'contains'` for a fragment) |
| `doc_search_sections(path, pattern)` | The section whose CONTENT matches, not its heading |
| `doc_container(path, id)` | The blocks inside one container |
| `doc_render(path, fmt)` | Render a document to `md`, `html` or text |
Each of these accepts `expand_embedded := true`, which parses a format embedded in a
document -- a markdown cell inside a `.ipynb` -- so a notebook is navigable rather
than a wall of `raw` blocks. Opt-in: it needs the `markdown` extension, and an
unchanged call reads identically without it.
## Writing Pandoc JSON
| Function | Description |
|----------|-------------|
| `panduck_blocks_to_pandoc_ast(blocks)` | duck_block list to a Pandoc AST document |
| `panduck_blocks_to_pandoc_blocks(blocks)` | duck_block list to a Pandoc block array |
| `panduck_blocks_to_pandoc_json(blocks)` | duck_block list to Pandoc JSON text |
| `panduck_write_pandoc_ast(path, blocks)` | Write Pandoc JSON to a file |
| `panduck_pandoc_ast_map()` | The constructor mapping, as a queryable table |
Since 0.7.0, panduck also registers the converter names that `duck_block_utils` used
to register: `pandoc_ast_to_blocks`, `read_pandoc_ast`, `write_pandoc_ast`,
`duck_blocks_to_pandoc_ast`, `duck_blocks_to_pandoc_blocks`, `pandoc_ast`,
`pandoc_inlines_to_db_inlines`, `pandoc_inlines_to_text` and
`duck_blocks_inlines_to_pandoc`. Load it with a `duck_block_utils` release that no
longer registers them.
The governing rule is that panduck may be **more** faithful than pandoc -- richer
attributes, better block types, metadata pandoc does not extract -- so long as the
mapping back to valid Pandoc JSON stays total. Every fixture and a set of
hand-built constructs are round-tripped through a real pandoc in CI to hold that.
## Scope
This is an early release. Ten native readers are implemented and tested against
reference implementations -- RTF, DOCX, ODT, EPUB, LaTeX, Org, RST, ipynb, MediaWiki,
Textile -- plus a Pandoc AST reader, with 3163 test assertions, differential validation
against a real pandoc on every fixture, and CI that builds and tests against DuckDB
v1.5.6, v2.0.0 and main. PDF, Markdown and HTML
are read by delegating to the pdf, markdown and webbed extensions rather than by a
reader here.
The readers were audited over five rounds before this release, each round asking a
question the previous one could not answer -- does a construct come back at all, do
any words go missing, do all ten readers agree on one logical document. That found
18 defects, including silent text loss in DOCX where a hyperlink's anchor text was
dropped outright. All are fixed and pinned by tests. Three findings remain open and
are recorded in the documentation rather than omitted.
Known divergences from pandoc are declared and reasoned in the repository's
roundtrip ledger rather than hidden -- including several where pandoc is the one
losing information.
extension_star_count: 5
extension_star_count_pretty: 5
extension_download_count: 906
extension_download_count_pretty: 906
image: '/images/community_extensions/social_preview/preview_community_extension_panduck.png'
layout: community_extension_doc
---
### Compatibility Matrix
This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Linux ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| macOS AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| macOS ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Windows AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ✅ | ✅ | ✅ |
### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```
{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}
{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions
| function_name | function_type | description | comment | examples |
|----------------------------------|---------------|--------------------------------------------------------------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------|
| doc_container | table_macro | NULL | NULL | |
| doc_render | macro | NULL | NULL | |
| doc_search_sections | table_macro | NULL | NULL | |
| doc_section | table_macro | NULL | NULL | |
| doc_toc | table_macro | NULL | NULL | |
| duck_blocks_inlines_to_pandoc | scalar | NULL | NULL | |
| duck_blocks_to_pandoc_ast | scalar | Convert a list of duck_blocks into a Pandoc AST document struct. | NULL | [duck_blocks_to_pandoc_ast(blocks)] |
| duck_blocks_to_pandoc_blocks | scalar | Convert a list of duck_blocks into a JSON array of Pandoc blocks. | NULL | [duck_blocks_to_pandoc_blocks(blocks)] |
| pandoc_ast | table | Return a list of duck_blocks as one Pandoc AST row. | NULL | [SELECT * FROM pandoc_ast(blocks)] |
| pandoc_ast_to_blocks | scalar | Convert a Pandoc AST JSON string into a list of duck_blocks. | NULL | [pandoc_ast_to_blocks('{"pandoc-api-version": [1,23,1], "meta": {}, "blocks": []}')] |
| pandoc_inlines_to_db_inlines | scalar | Parse a Pandoc inlines JSON array into inline duck_blocks. | NULL | [pandoc_inlines_to_db_inlines('[{"t":"Str","c":"hi"}]')] |
| pandoc_inlines_to_text | scalar | NULL | NULL | |
| panduck_block_cols | macro | NULL | NULL | |
| panduck_blocks_to_pandoc_ast | scalar | Convert a list of duck_blocks into a Pandoc AST document struct. | NULL | [panduck_blocks_to_pandoc_ast(blocks)] |
| panduck_blocks_to_pandoc_blocks | scalar | Convert a list of duck_blocks into a JSON array string of Pandoc block elements. | NULL | [panduck_blocks_to_pandoc_blocks(blocks)] |
| panduck_blocks_to_pandoc_json | scalar | Convert a list of duck_blocks into a Pandoc AST JSON string. | NULL | [panduck_blocks_to_pandoc_json(blocks)] |
| panduck_builtin_format_for | scalar | Return the native builtin format name for a file path or URI. | NULL | [panduck_builtin_format_for('doc.docx')] |
| panduck_can_read | scalar | Check if panduck can read a given file path or URI. | NULL | [panduck_can_read('doc.docx')] |
| panduck_dependencies | table_macro | NULL | NULL | |
| panduck_duck_block_spec_at_least | macro | NULL | NULL | |
| panduck_duck_block_type | scalar | Return the SQL type definition string for duck_blocks. | NULL | [panduck_duck_block_type()] |
| panduck_ensure_extension | scalar | Ensure that a required DuckDB extension is loaded. | NULL | [panduck_ensure_extension('fts')] |
| panduck_expand_embedded | macro | NULL | NULL | |
| panduck_expand_embedded_impl | macro | NULL | NULL | |
| panduck_format_for | scalar | Return the document format name for a file path or URI. | NULL | [panduck_format_for('doc.docx')] |
| panduck_function_exists | scalar | Check if a scalar or table function exists in the catalog. | NULL | [panduck_function_exists('read_docx_blocks')] |
| panduck_glob | scalar | Glob files matching pattern. | NULL | [panduck_glob('docs/*.docx')] |
| panduck_is_glob | macro | NULL | NULL | |
| panduck_latex_tokens | table | Tokenize a LaTeX string and return the sequence of LaTeX tokens. | NULL | [SELECT * FROM panduck_latex_tokens('\textbf{hello}')] |
| panduck_pandoc_api_version | scalar | Return the Pandoc API version supported by the panduck extension as [major, minor, patch]. | NULL | [panduck_pandoc_api_version()] |
| panduck_pandoc_ast_json | scalar | Convert a list of duck_blocks into a Pandoc AST JSON string (alias for panduck_blocks_to_pandoc_json). | NULL | [panduck_pandoc_ast_json(blocks)] |
| panduck_pandoc_ast_map | table | Return the complete constructor set of Pandoc AST types mapped to DuckDB duck_block element types. | NULL | [SELECT * FROM panduck_pandoc_ast_map()] |
| panduck_pandoc_ast_to_blocks | scalar | Convert a Pandoc AST JSON string into a list of duck_blocks. | NULL | [panduck_pandoc_ast_to_blocks('{"pandoc-api-version": [1,23,1], "meta": {}, "blocks": []}')] |
| panduck_pdf_blocks_impl | table_macro | NULL | NULL | |
| panduck_policy_format | macro | NULL | NULL | |
| panduck_quote | macro | NULL | NULL | |
| panduck_read_arms | macro | NULL | NULL | |
| panduck_read_arms_opt | macro | NULL | NULL | |
| panduck_read_blocks | macro | NULL | NULL | |
| panduck_reader_enabled | scalar | Check if the reader for a specified format is currently enabled. | NULL | [panduck_reader_enabled('docx')] |
| panduck_reader_extension_for | scalar | Return the required extension name to read a file path or URI. | NULL | [panduck_reader_extension_for('doc.docx')] |
| panduck_reader_function_for | scalar | Return the reader table function name that handles a file path or URI. | NULL | [panduck_reader_function_for('doc.docx')] |
| panduck_reader_kind_for | scalar | Return the reader kind ('doc' or 'table') for a file path or URI. | NULL | [panduck_reader_kind_for('doc.docx')] |
| panduck_reader_option_for | scalar | Look up a reader configuration option for a given format. | NULL | [panduck_reader_option_for('docx', 'toc', NULL)] |
| panduck_reader_registry | table | Return the table of all registered format readers and handlers. | NULL | [SELECT * FROM panduck_reader_registry()] |
| panduck_register_doc_reader | table | Register a custom document block reader for a file format or extension. | NULL | [SELECT * FROM panduck_register_doc_reader('custom', 'read_custom', ['.custom'])] |
| panduck_register_table_reader | table | Register a custom table reader for a file format or extension. | NULL | [SELECT * FROM panduck_register_table_reader('custom', 'read_custom_tbl', ['.custom'])] |
| panduck_registry_key_for | scalar | Return the registry match key for a file path or URI. | NULL | [panduck_registry_key_for('doc.docx')] |
| panduck_render_format | macro | NULL | NULL | |
| panduck_render_params | scalar | Render named parameters map into SQL argument string. | NULL | [panduck_render_params(MAP {'opt': 'val'})] |
| panduck_renumber_blocks | macro | NULL | NULL | |
| panduck_resolved_format | macro | NULL | NULL | |
| panduck_source_list | macro | NULL | NULL | |
| panduck_supported_extensions | table | Return list of supported document formats and extensions in panduck. | NULL | [SELECT * FROM panduck_supported_extensions()] |
| panduck_supported_paths | macro | NULL | NULL | |
| panduck_version | scalar | Return the version of the panduck extension. | NULL | [panduck_version()] |
| panduck_wrap_expand | macro | NULL | NULL | |
| panduck_write_pandoc_ast | scalar | Write a list of duck_blocks out to disk as a Pandoc AST JSON file. | NULL | [panduck_write_pandoc_ast('output.json', blocks)] |
| read_docx_blocks | table | Read a DOCX document and return structured document blocks. | NULL | [SELECT * FROM read_docx_blocks('document.docx')] |
| read_epub_blocks | table | Read an EPUB document and return structured document blocks. | NULL | [SELECT * FROM read_epub_blocks('book.epub')] |
| read_ipynb_blocks | table | Read a Jupyter Notebook (.ipynb) file and return structured document blocks. | NULL | [SELECT * FROM read_ipynb_blocks('notebook.ipynb')] |
| read_ipynb_blocks_string | table | Parse a Jupyter Notebook JSON string and return structured document blocks. | NULL | [SELECT * FROM read_ipynb_blocks_string('{"cells": [], "metadata": {}, "nbformat": 4, "nbformat_minor": 2}')] |
| read_latex_blocks | table | Read a LaTeX document and return structured document blocks. | NULL | [SELECT * FROM read_latex_blocks('document.tex')] |
| read_latex_blocks_string | table | Parse a LaTeX string and return structured document blocks. | NULL | [SELECT * FROM read_latex_blocks_string('\section{Title}\n\textbf{Bold}')] |
| read_mediawiki_blocks | table | Read a MediaWiki wikitext document and return structured document blocks. | NULL | [SELECT * FROM read_mediawiki_blocks('document.wiki')] |
| read_mediawiki_blocks_string | table | Parse a MediaWiki wikitext string and return structured document blocks. | NULL | [SELECT * FROM read_mediawiki_blocks_string('== Section ==\n'''Bold'''')] |
| read_odt_blocks | table | Read an ODT document and return structured document blocks. | NULL | [SELECT * FROM read_odt_blocks('document.odt')] |
| read_org_blocks | table | Read an Emacs Org-mode document and return structured document blocks. | NULL | [SELECT * FROM read_org_blocks('document.org')] |
| read_org_blocks_string | table | Parse an Emacs Org-mode string and return structured document blocks. | NULL | [SELECT * FROM read_org_blocks_string('* Heading\nParagraph')] |
| read_pandoc_ast | scalar | Read a Pandoc AST JSON file into a list of duck_blocks. | NULL | [read_pandoc_ast('document.json')] |
| read_pandoc_blocks | table | Read a Pandoc AST JSON file and return structured document blocks. | NULL | [SELECT * FROM read_pandoc_blocks('document.json')] |
| read_pandoc_blocks_string | table | Parse a Pandoc AST JSON string and return structured document blocks. | NULL | [SELECT * FROM read_pandoc_blocks_string('{"pandoc-api-version": [1,23,1], "meta": {}, "blocks": []}')] |
| read_panduck_doc | table_macro | NULL | NULL | |
| read_panduck_table | table_macro | NULL | NULL | |
| read_pdf_blocks | table_macro | NULL | NULL | |
| read_rst_blocks | table | Read a reStructuredText (RST) document and return structured document blocks. | NULL | [SELECT * FROM read_rst_blocks('document.rst')] |
| read_rst_blocks_string | table | Parse a reStructuredText (RST) string and return structured document blocks. | NULL | [SELECT * FROM read_rst_blocks_string('Title\n=====\n\nParagraph')] |
| read_rtf_blocks | table | Read an RTF file and return structured document blocks. | NULL | [SELECT * FROM read_rtf_blocks('document.rtf')] |
| read_textile_blocks | table | Read a Textile document and return structured document blocks. | NULL | [SELECT * FROM read_textile_blocks('document.textile')] |
| read_textile_blocks_string | table | Parse a Textile string and return structured document blocks. | NULL | [SELECT * FROM read_textile_blocks_string('h1. Header\n\n*bold*')] |
| write_pandoc_ast | scalar | Write a list of duck_blocks out as a Pandoc AST JSON file. | NULL | [write_pandoc_ast('output.json', blocks)] |
### Overloaded Functions
This extension does not add any function overloads.
### Added Types
This extension does not add any types.
### Added Settings
| name | description | input_type | scope | aliases | typed_value |
|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------|---------|-------------|
| panduck_allow_registration | Allow panduck_register_doc_reader/panduck_register_table_reader to add readers at runtime | BOOLEAN | GLOBAL | [] | true |
| panduck_disabled_readers | Comma-separated denylist of reader FORMATS, applied after panduck_enabled_readers. 'code' turns off the fallback that returns a parse tree for sources no reader claimed | VARCHAR | GLOBAL | [] | |
| panduck_enabled_readers | Comma-separated allowlist of reader FORMATS ('*' for all). Applied before panduck_disabled_readers | VARCHAR | GLOBAL | [] | * |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |