---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github/duckdb/community-extensions repository, check README there
title: hdfs
excerpt: |
  DuckDB Community Extensions
  Read and write hdfs:// URLs via a native HDFS filesystem — no JVM or libhdfs required

extension:
  name: hdfs
  description: Read and write hdfs:// URLs via a native HDFS filesystem — no JVM or libhdfs required
  version: 0.1.0
  language: C++
  build: cmake
  license: MIT
  requires_toolchains: "rust"
  excluded_platforms: "wasm_mvp;wasm_eh;wasm_threads;windows_amd64;windows_amd64_mingw"
  maintainers:
    - casperhart

repo:
  github: casperhart/duckdb-hdfs
  ref: 3fb20ef426a65b78c620ba79a8c23da6d8d45bc2

docs:
  hello_world: |
    -- Read any format DuckDB supports directly from HDFS
    SELECT * FROM 'hdfs://namenode:8020/data/events/*.parquet';
    SELECT * FROM read_csv('hdfs://namenode:8020/data/users.csv');

    -- Write with COPY ... TO, including PARTITION_BY
    COPY (SELECT * FROM big_table)
    TO 'hdfs://namenode:8020/out/table.parquet' (FORMAT parquet);

    -- List directories with full FileStatus metadata
    SELECT name, type, size, owner, permissions
    FROM hdfs_ls('hdfs://namenode:8020/data');

    -- Metadata for a single path, and existence checks
    SELECT * FROM hdfs_stat('hdfs://namenode:8020/data');
    SELECT hdfs_exists('hdfs://namenode:8020/data/events');

  extended_description: |
    The `hdfs` extension adds a native HDFS filesystem to DuckDB: read and write
    `hdfs://` URLs directly — no JVM, no `libhdfs`. It is backed by the pure-Rust
    [`hdfs-native`](https://github.com/Kimahriman/hdfs-native) client through a thin
    C FFI bridge.

    **Features**:
    - Reading any file format DuckDB supports (Parquet, CSV, JSON, ...) over `hdfs://`,
      with concurrent positional reads for DuckDB's parallel Parquet reader.
    - Writing via `COPY ... TO`, including `PARTITION_BY` (HDFS is append-only).
    - Globbing (`*`, `?`, `[abc]`, `**`, plus Hadoop-style `{a,b}` alternation),
      directory listing, file metadata, and directory/file management.
    - Metadata SQL functions: `hdfs_ls`, `hdfs_stat`, `hdfs_exists`.
    - HDFS encryption zones (Transparent Data Encryption): data is decrypted/encrypted
      client-side, unwrapping keys via the KMS over TLS.
    - Kerberos-secured clusters are supported via the standard ticket/keytab mechanism.

    **Configuration**: connection config is resolved from the standard Hadoop
    environment, exactly as the Hadoop CLI tools do — `HADOOP_CONF_DIR` (or
    `HADOOP_HOME/etc/hadoop`) for `core-site.xml`/`hdfs-site.xml`, and
    `HADOOP_USER_NAME` for the effective user on non-secure clusters. The scheme-only
    form `hdfs:///path` uses `fs.defaultFS` from your Hadoop config.

    For details, see the [extension repository](https://github.com/casperhart/duckdb-hdfs).

extension_star_count: 0
extension_star_count_pretty: 0
extension_download_count: 589
extension_download_count_pretty: 589
image: '/images/community_extensions/social_preview/preview_community_extension_hdfs.png'
layout: community_extension_doc
---

### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```

{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}

{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}

### Added Functions

<div class="extension_functions_table"></div>

| function_name | function_type | description | comment | examples |
|---------------|---------------|-------------|---------|----------|
| hdfs_exists   | scalar        | NULL        | NULL    |          |
| hdfs_ls       | table         | NULL        | NULL    |          |
| hdfs_stat     | table         | NULL        | NULL    |          |

### Overloaded Functions

<div class="extension_functions_table"></div>

This extension does not add any function overloads.

### Added Types

<div class="extension_types_table"></div>

This extension does not add any types.

### Added Settings

<div class="extension_settings_table"></div>

|            name             |                                                                                                  description                                                                                                  | input_type | scope  | aliases |
|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------|---------|
| hdfs_include_hidden         | Return hidden HDFS entries (names starting with '_' or '.', e.g. _temporary, _SUCCESS) from listings, globs and scans; by default they are excluded unless a glob component names them explicitly (e.g. '_*') | BOOLEAN    | GLOBAL | []      |
| hdfs_list_parallelism       | Maximum number of concurrent HDFS directory-listing RPCs used by listings and globs that walk more than one directory (e.g. hdfs_ls('/path/**')); 1 disables parallelism                                      | UBIGINT    | GLOBAL | []      |
| hdfs_skip_permission_errors | Prune subtrees whose listing fails with a permission error during HDFS globs and recursive listings instead of failing the query; an error on the listed path or glob root itself still fails                 | BOOLEAN    | GLOBAL | []      |