--- layout: docu redirect_from: - /docs/internals/storage - /docs/stable/internals/storage - /internals/storage - /storage - /docs/current/internals/storage - /docs/lts/internals/storage - /docs/preview/internals/storage title: Storage Versions and Format --- ## Compatibility ### Backward Compatibility _Backward compatibility_ refers to the ability of a newer DuckDB version to read storage files created by an older DuckDB version. Version 0.10 is the first release of DuckDB that supports backward compatibility in the storage format. DuckDB 0.10 can read and operate on files created by the previous DuckDB version – DuckDB 0.9. For future DuckDB versions, our goal is to ensure that any DuckDB version released **after** can read files created by previous versions, starting from this release. We want to ensure that the file format is fully backward compatible. This allows you to keep data stored in DuckDB files around and guarantees that you will be able to read the files without having to worry about which version the file was written with or having to convert files between versions. ### Forward Compatibility _Forward compatibility_ refers to the ability of an older DuckDB version to read storage files produced by a newer DuckDB version. DuckDB 0.9 is [**partially** forward compatible with DuckDB 0.10]({% post_url 2024-02-13-announcing-duckdb-0100 %}#forward-compatibility). Certain files created by DuckDB 0.10 can be read by DuckDB 0.9. Forward compatibility is provided on a **best effort** basis. While stability of the storage format is important – there are still many improvements and innovations that we want to make to the storage format in the future. As such, forward compatibility may be (partially) broken on occasion. > Warning DuckDB versions before 2.0 cannot read database files that use the v2.0.0 storage format, which DuckDB 2.0 uses by default for new database files. See [Default Storage Version](#default-storage-version) and the [migration guide]({% link docs/2.0/operations_manual/migration.md %}#storage-format). ## How to Move between Storage Formats When you update DuckDB and open an old database file, you might encounter an error message about incompatible storage formats, pointing to this page. To move your database(s) to newer format you only need the older and the newer DuckDB executable. Open your database file with the older DuckDB and run the SQL statement `EXPORT DATABASE 'tmp'`. This allows you to save the whole state of the current database in use inside folder `tmp`. The content of the `tmp` folder will be overridden, so choose an empty/non yet existing location. Then, start the newer DuckDB and execute `IMPORT DATABASE 'tmp'` (pointing to the previously populated folder) to load the database, which can be then saved to the file you pointed DuckDB to. A Bash script to achieve this (to be adapted with the file names and executable locations) is the following ```bash /older/duckdb mydata.old.db -c "EXPORT DATABASE 'tmp'" /newer/duckdb mydata.new.db -c "IMPORT DATABASE 'tmp'" ``` After this, `mydata.old.db` will remain in the old format, `mydata.new.db` will contain the same data but in a format accessible by the more recent DuckDB version, and the folder `tmp` will hold the same data in a universal format as different files. Check [`EXPORT` documentation]({% link docs/2.0/sql/statements/export.md %}) for more details on the syntax. ### Default Storage Version > New Starting with DuckDB 2.0, new database files use the v2.0.0 storage format by default. DuckDB versions v1.0 to v1.5 create a DuckDB database file with [version 64](#storage-version-table), corresponding to v1.0.0. Starting with DuckDB 2.0, new database files are created with [version 69](#storage-version-table), corresponding to v2.0.0. On older versions you can opt in to a newer format by setting an [explicit storage version](#explicit-storage-versions). > Warning A file created with the v2.0.0 storage format cannot be opened by DuckDB versions prior to v2.0.0. To write a file that older versions can still read, create it with an [explicit storage version](#explicit-storage-versions). ### What's New in the v2.0.0 Format > New The v2.0.0 storage format, used by default from DuckDB 2.0, introduces several improvements over earlier versions. Compared to earlier storage versions, the v2.0.0 format: * loads column metadata lazily, so databases with wide tables and large indexes open faster and use less memory; * enables the `DICT_FSST` string compression method by default (see [Compression](#compression)); * stores row deletions more compactly; and * performs stronger corruption validation when reading data. ### Explicit Storage Versions [DuckDB 1.2.0 introduced the `STORAGE_VERSION` option]({% post_url 2025-02-05-announcing-duckdb-120 %}#explicit-storage-versions), which allows explicitly specifying the storage version. Since DuckDB 2.0, new database files use the latest storage version by default, so the main use of this option is to create database files that older DuckDB versions can still read: ```sql ATTACH 'file.db' (STORAGE_VERSION 'v1.2.0'); ``` To initialize a database with the latest storage version, use: ```sql ATTACH 'file.db' (STORAGE_VERSION 'latest'); ``` With the [command line client]({% link docs/2.0/clients/cli/overview.md %}), you can use the `-storage-version` argument: ```batch duckdb -storage-version v1.2.0 my_database.duckdb ``` To create a database with the latest storage version, use: ```batch duckdb -storage-version latest my_database.duckdb ``` The storage version setting specifies the minimum DuckDB version that should be able to read the database file. When database files are written with this option, the resulting files cannot be opened by older DuckDB released versions than the specified version. They can be read by the specified version and all newer versions of DuckDB. If you attach to DuckDB databases, you can query the storage versions using the following command: ```sql SELECT database_name, tags FROM duckdb_databases(); ``` This shows the storage versions: ```text ┌───────────────┬───────────────────────────────────┐ │ database_name │ tags │ │ varchar │ map(varchar, varchar) │ ├───────────────┼───────────────────────────────────┤ │ file1 │ {storage_version=v1.2.0} │ │ file2 │ {storage_version=v1.0.0 - v1.1.3} │ │ ... │ ... │ └───────────────┴───────────────────────────────────┘ ``` This means that `file2` can be opened by past DuckDB versions while `file1` is compatible only with `v1.2.0` (or future versions). ### Setting the Storage Compatibility The `storage_compatibility_version` [configuration option]({% link docs/2.0/configuration/overview.md %}#configuration-reference) can also be used to specify the storage version to use. It can be specified in various ways. In the Python client, you have to specify it when connecting to a new database: ```python import duckdb duckdb.connect("file.db", config={'storage_compatibility_version': 'latest'}) # or duckdb.connect("file.db", config={'storage_compatibility_version': 'v1.4.0'}) ``` In the CLI and some other clients, set the configuration option as follows: ```sql SET storage_compatibility_version = 'latest'; -- or SET storage_compatibility_version = 'v1.4.0'; ``` In the CLI client, the storage version can also be specified using the `-storage-version` command line argument for the entire CLI session. ### Converting between Storage Versions To convert from the new format to the old format for compatibility, use the following sequence in DuckDB 1.2.0+: ```sql ATTACH 'file1.db'; ATTACH 'converted_file.db' (STORAGE_VERSION 'v1.0.0'); COPY FROM DATABASE file1 TO converted_file; ``` ## Storage Header DuckDB files start with a `uint64_t` which contains a checksum for the main header, followed by four magic bytes (`DUCK`), followed by a version number in a `uint64_t`. Files created in the v2.0.0 format hold 999 there. In other files it is a storage version, but it can be older than the format the file was upgraded to, which only the database headers record. ```batch hexdump -n 20 -C mydata.db ``` ```text 00000000 01 d0 e2 63 9c 13 39 3e 44 55 43 4b 2b 00 00 00 |...c..9>DUCK+...| 00000010 00 00 00 00 |....| 00000014 ``` Because the number after the magic bytes does not always name the storage version, let DuckDB read the headers: attach the file read-only and query `duckdb_databases()`, as in the following Python example. ```python import duckdb con = duckdb.connect() con.execute("ATTACH 'file.db' AS file_db (READ_ONLY)") print(con.execute( "SELECT tags['storage_version'] FROM duckdb_databases() WHERE database_name = 'file_db'" ).fetchone()[0]) ``` ```text v2.0.0+ ``` ## Storage Version Table For changes in each given release, check out the [change log](https://github.com/duckdb/duckdb/releases) on GitHub. To see the commits that changed each storage version, see the [commit log](https://github.com/duckdb/duckdb/commits/main/src/storage/storage_info.cpp). | Storage version | DuckDB version(s) | |----------------:|---------------------------------| | 69 | v2.0.x | | 68 | v1.5.x | | 67 | v1.4.x | | 66 | v1.3.x | | 65 | v1.2.x | | 64 | v0.9.x, v0.10.x, v1.0.0, v1.1.x | | 51 | v0.8.x | | 43 | v0.7.x | | 39 | v0.6.x | | 38 | v0.5.x | | 33 | v0.3.3, v0.3.4, v0.4.0 | | 31 | v0.3.2 | | 27 | v0.3.1 | | 25 | v0.3.0 | | 21 | v0.2.9 | | 18 | v0.2.8 | | 17 | v0.2.7 | | 15 | v0.2.6 | | 13 | v0.2.5 | | 11 | v0.2.4 | | 6 | v0.2.3 | | 4 | v0.2.2 | | 1 | v0.2.1 and prior | ## Compression DuckDB uses [lightweight compression]({% post_url 2022-10-28-lightweight-compression %}). By default, compression is only applied to persistent databases and is **not applied to in-memory instances**. To turn on compression for in-memory databases, use `ATTACH` with the [`COMPRESS` option]({% link docs/2.0/sql/statements/attach.md %}#options). Note that available compression algorithms depend on the storage version used, so you might need to set an explicit storage version to get access to all compression algorithms. ### Compression Algorithms The compression algorithms supported by DuckDB include the following: * [Constant Encoding]({% post_url 2022-10-28-lightweight-compression %}#constant-encoding) * [Run-Length Encoding (RLE)]({% post_url 2022-10-28-lightweight-compression %}#run-length-encoding-rle) * [Bit Packing]({% post_url 2022-10-28-lightweight-compression %}#bit-packing) * [Frame of Reference (FOR)]({% post_url 2022-10-28-lightweight-compression %}#frame-of-reference) * [Dictionary Encoding]({% post_url 2022-10-28-lightweight-compression %}#dictionary-encoding) * [Fast Static Symbol Table (FSST)]({% post_url 2022-10-28-lightweight-compression %}#fsst) – [VLDB 2020 paper](https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf) * [Adaptive Lossless Floating-Point Compression (ALP)]({% post_url 2024-02-13-announcing-duckdb-0100 %}#adaptive-lossless-floating-point-compression-alp) – [SIGMOD 2024 paper](https://ir.cwi.nl/pub/33334/33334.pdf) * [Chimp]({% post_url 2022-10-28-lightweight-compression %}#chimp--patas) – [VLDB 2022 paper](https://www.vldb.org/pvldb/vol15/p3058-liakos.pdf) * [Patas]({% post_url 2022-11-14-announcing-duckdb-060 %}#compression-improvements) * [Zstd]({% post_url 2025-02-05-announcing-duckdb-120 %}#zstd-compression) ## Disk Usage The disk usage of DuckDB's format depends on a number of factors, including the data type and the data distribution, the compression methods used, etc. As a rough approximation, loading 100 GB of uncompressed CSV files into a DuckDB database file will require 25 GB of disk space, while loading 100 GB of Parquet files will require 120 GB of disk space. ## Row Groups DuckDB's storage format stores the data in _row groups,_ i.e., horizontal partitions of the data. This concept is equivalent to [Parquet's row groups](https://parquet.apache.org/docs/concepts/). Several features in DuckDB, including [parallelism]({% link docs/2.0/guides/performance/how_to_tune_workloads.md %}) and [compression]({% post_url 2022-10-28-lightweight-compression %}) are based on row groups. The row group size can be specified as an option of the `ATTACH` statement: ```sql ATTACH '/tmp/somefile.db' AS db (ROW_GROUP_SIZE 16384); ``` ## Troubleshooting ### Error Message When Opening a DuckDB 2.0 Database File with an Older Version DuckDB versions before 2.0 cannot read database files in the v2.0.0 storage format. Opening such a file with DuckDB 1.5.6 results in the following error: ```console IO Error: Trying to read a database file with version number 69, but we can only read versions between 64 and 68. The database file was created with a newer version of DuckDB. ``` DuckDB 1.5.5 and earlier versions report version number 999 instead of 69. To read the data with an older DuckDB version, either upgrade that process to DuckDB 2.0, or use DuckDB 2.0 to [copy the database into a file with an older storage version](#converting-between-storage-versions), e.g., `v1.5.0`. ### Error Message When Opening an Incompatible Database File When opening a database file that has been written by a different DuckDB version from the one you are using, the following error message may occur: ```console Error: unable to open database "...": Serialization Error: Failed to deserialize: ... ``` The message implies that the database file was created with a newer DuckDB version and uses features that the DuckDB version used to read the file does not support. There are two potential workarounds: 1. Update your DuckDB version to the latest stable version. 2. Open the database with the latest version of DuckDB, export it to a standard format (e.g., Parquet), then import it to any version of DuckDB. See the [`EXPORT/IMPORT DATABASE` statements]({% link docs/2.0/sql/statements/export.md %}) for details.