---
warning: DO NOT CHANGE THIS MANUALLY, THIS IS GENERATED BY https://github.com/duckdb/community-extensions repository, check README there
title: gorz
excerpt: |
DuckDB Community Extensions
Read and write GORpipe .gorz files and .gord dictionary folders as native DuckDB tables
extension:
name: gorz
description: Read and write GORpipe .gorz files and .gord dictionary folders as native DuckDB tables
version: 1.0.0
language: C++
build: cmake
license: MIT
maintainers:
- gorfather
opt_in_platforms: "osx_amd64"
repo:
github: gorfather/duckdb-gorz
ref: 5c51c4734e432cf645232bc6a59899ace464b7ed
variegata: 40b5ec3be5dfe17b460566528aa0ad3aab6daeed
docs:
hello_world: |
-- Write a query result out as a GORZ file (rows must be in GOR order),
-- then read it back — with an optional GOR -p style range filter.
COPY (SELECT * FROM (VALUES ('chr1', 1000, 'A', 'G'), ('chr1', 2000, 'C', 'T'))
t(chrom, pos, ref, alt) ORDER BY chrom, pos)
TO 'variants.gorz' (FORMAT gorz);
SELECT * FROM read_gor('variants.gorz', range := 'chr1:1500-2500');
-- Equivalently, in plain SQL on the bare file name (the chrom / pos
-- predicates are pushed down into a block seek):
SELECT * FROM 'variants.gorz' WHERE chrom = 'chr1' AND pos BETWEEN 1500 AND 2500;
-- Write a .gord dictionary folder: one sorted .gorz per genomic range (here
-- 1 Mb per file) plus thedict.gord. The input needs no ORDER BY.
COPY (SELECT * FROM 'variants.gorz') TO 'variants.gord' (FORMAT gord, SPLIT 1000000);
SELECT * FROM 'variants.gord' WHERE chrom = 'chr1';
-- Read a partitioned .gord dictionary table (not shown here), keeping only
-- the partitions tagged sample1 or sample99 (GOR's -f). See the GORpipe paper
-- (Guðbjartsson et al., Bioinformatics 2016, doi:10.1093/bioinformatics/btw199)
-- and "Ultra-fast joint-genotyping with SparkGOR" (Guðbjartsson et al.,
-- bioRxiv 2022, doi:10.1101/2022.10.25.513331):
SELECT * FROM read_gord('example.gord', f := ['sample1', 'sample99']);
extended_description: |
The `gorz` extension reads and writes [GORpipe](https://github.com/gorpipe/gor)
genomic files — block-compressed `.gorz` and `.gord` dictionary folders — as
native DuckDB tables. Files it writes are read and seeked natively by `gorpipe`, and
vice-versa.
### Reading
- `read_gor(path)` — auto-detects `.gorz` vs `.gord` by extension.
- `read_gorz(path)` / `read_gord(path)` — force the kind.
- A **replacement scan**, so a bare literal works: `SELECT count(*) FROM 'variants.gord';`
- **`pgor … | write x.gord` folders** — pass the folder itself (`FROM 'x.gord'`); its `x.gord/thedict.gord` is read.
- **Projection & parallel scan** — only selected columns are read; large files scan across threads.
- **`WHERE chrom/pos` → block seek** — range predicates seek into the right block instead of scanning the whole file.
- **`range := 'chrN:start-end'`** — GOR's `-p` semantics as a hard positional filter (also `'chrN'`, `'chrN:start-'`, `'chrN:pos'`).
- **Partition filters** — GOR's `-f` / `-ff` semantics prune a `.gord`'s file list and expose a `Source` column: `f := ['A','B']` is an inline tag list, `ff := 'tags.txt'` is a *tag file* (one tag per line / first tab-delimited column, `#` lines skipped). `source := 'PN'` renames the exposed column.
- **Object stores** — paths resolve through DuckDB's FileSystem, so `s3://…` works for reads and writes when `httpfs` is loaded.
### Writing
```sql
COPY (SELECT chrom, pos, ref, alt FROM my_variants ORDER BY chrom, pos)
TO 'out.gorz' (FORMAT gorz); -- or (FORMAT gor) for plain sorted TSV
```
Single-threaded, standard block-zip; validates GOR order (chromosomes ascend
lexicographically, positions non-decreasing) and errors out otherwise. `META`
also writes the `.gorz.meta` sidecar (columns, types, line count, range, MD5),
which `read_gor` uses to hand the column types back.
```sql
COPY (SELECT * FROM my_variants) TO 'out.gord' (FORMAT gord, SPLIT 100);
```
Writes a dictionary **folder**, as `pgor … | write out.gord` does: one sorted
`.gorz` + `.gorz.meta` per genomic range and `out.gord/thedict.gord`. The query
needs no `ORDER BY`; it runs once and each range is sorted on its own, so
memory follows the size of a range rather than the size of the input.
- `SPLIT 'chrom'` (default) — one file per chromosome.
- `SPLIT n`, `n` ≤ 1000 — `n` genome partitions (a step of 3,000,000,000 / `n` bases within each chromosome, as `pgor -split=n`).
- `SPLIT n`, `n` > 1000 — `n` bases per file.
- `OVERWRITE` — replace an existing `.gord` folder.
- `STAGING 'dir'` — local directory for the temporary files (default: next to the output; DuckDB's `temp_directory` for an object-store target).
### Object stores
```sql
LOAD httpfs;
CREATE SECRET (TYPE s3, KEY_ID '…', SECRET '…', REGION 'eu-west-1');
COPY (SELECT * FROM my_variants) TO 's3://bucket/out.gord' (FORMAT gord, SPLIT 100);
SELECT count(*) FROM 's3://bucket/out.gord' WHERE chrom = 'chr17';
```
`.gorz` files and `.gord` folders can be read from and written to
S3-compatible stores (`s3://…`). For a folder write the temporary files stay
on local disk and only the finished `.gorz`, `.meta` and `thedict.gord`
objects are uploaded, the dictionary last.
### References
- Guðbjartsson H. et al. [GORpipe: a query tool for working with sequence data based on a Genomic Ordered Relational (GOR) architecture](https://doi.org/10.1093/bioinformatics/btw199). *Bioinformatics* 32(20):3081–3088, 2016.
- Guðbjartsson H. et al. [Ultra-fast joint-genotyping with SparkGOR](https://doi.org/10.1101/2022.10.25.513331). *bioRxiv*, 2022.
extension_star_count: 1
extension_star_count_pretty: 1
extension_download_count: 883
extension_download_count_pretty: 883
image: '/images/community_extensions/social_preview/preview_community_extension_gorz.png'
layout: community_extension_doc
---
### Compatibility Matrix
This table covers all DuckDB releases that have not reached [end-of-life](https://duckdb.org/release_calendar), including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Linux AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Linux ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| macOS AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| macOS ARM64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Windows AMD64 | ❌ | ❌ | ✅ | ✅ | ✅ |
| Windows ARM64 | ❌ | ❌ | ❌ | ❌ | ❌ |
| WebAssembly | ❌ | ❌ | ✅ | ✅ | ✅ |
### Installing and Loading
```sql
INSTALL {{ page.extension.name }} FROM community;
LOAD {{ page.extension.name }};
```
{% if page.docs.hello_world %}
### Example
```sql
{{ page.docs.hello_world }}```
{% endif %}
{% if page.docs.extended_description %}
### About {{ page.extension.name }}
{{ page.docs.extended_description }}
{% endif %}
### Added Functions
| function_name | function_type | description | comment | examples |
|---------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| read_gor | table | Reads a GORpipe .gorz file or .gord dictionary (a dictionary file or a .gord folder) as a table; the kind is taken from the path. Named parameters: range := 'chrN[:start-end]' keeps one genomic range (GOR's -p); for a .gord, f := ['tag', ...] and ff := 'tag-file' keep only the partitions with those tags (GOR's -f / -ff) and source := 'name' renames the Source column. | NULL | [SELECT * FROM read_gor('variants.gorz', range := 'chr1:1500-2500'), SELECT * FROM read_gor('cohort.gord', f := ['sample1', 'sample99'])] |
| read_gord | table | Reads a GORpipe .gord dictionary (a dictionary file or a .gord folder) as one table, with a trailing Source column holding each row's partition tag. Named parameters: f := ['tag', ...] and ff := 'tag-file' keep only the partitions with those tags (GOR's -f / -ff), source := 'name' renames the Source column, range := 'chrN[:start-end]' keeps one genomic range (GOR's -p). | NULL | [SELECT * FROM read_gord('cohort.gord', f := ['sample1', 'sample99']), SELECT count(*) FROM read_gord('cohort.gord', ff := 'tags.txt', range := 'chr7')] |
| read_gorz | table | Reads a GORpipe .gorz file as a table, whatever the path's extension. chrom / pos predicates in the WHERE clause seek to the matching blocks; range := 'chrN[:start-end]' keeps one genomic range (GOR's -p). | NULL | [SELECT chrom, pos, ref, alt FROM read_gorz('variants.gorz') WHERE chrom = 'chr1' AND pos BETWEEN 1500 AND 2500, SELECT count(*) FROM read_gorz('variants.gorz', range := 'chr7')] |
### Overloaded Functions
This extension does not add any function overloads.
### Added Types
This extension does not add any types.
### Added Settings
| name | description | input_type | scope | aliases | typed_value |
|-----------------------|--------------------------------------------------------------------------------------------------------------|------------|--------|---------|-------------|
| gorz_scan_chunk_bytes | Byte size of each parallel .gorz full-scan range (0 = automatic: one range per thread, at least 16 MiB each) | BIGINT | GLOBAL | [] | 0 |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |