Table of Contents↗️

`filepack`

Filepack is a content-addressed package format with self-describing metadata. A package is a directory of files and a manifest containing their BLAKE3 hashes. File and directory hashes form a Merkle tree whose root hash, the package fingerprint, uniquely identifies the package. Packages can be signed with Ed25519 and verified against a manifest, fingerprint, or signature. Packages may contain machine-readable metadata describing their content, allowing programmatic search, preview, playback, and conversion. `filepack` is a command-line tool for creating, signing, and verifying packages, and includes an HTTP server for package upload, download, and display. A demo server with a selection of example packages is available at [filepack.com](https://filepack.com). Filepack is currently experimental. The `filepack` interface and package format may change at any time. Quickstart ---------- Optionally, add metadata describing your package: ```sh echo "title: Packaging Guide" > metadata.yaml ``` Create `manifest.filepack`: ```shell filepack create ``` Verify current directory against hashes in `manifest.filepack`: ```shell filepack verify ``` Print package fingerprint: ```shell filepack fingerprint ``` Print package metadata: ```shell filepack metadata ``` Installation ------------ `filepack` is written in [Rust](https://www.rust-lang.org/) and can be built from source and installed from a checked-out copy of this repo with: ```sh cargo install --locked --path . ``` Or from [crates.io](https://crates.io/crates/filepack) with: ```sh cargo install --locked filepack ``` See [rustup.rs](https://rustup.rs/) for installation instructions for Rust. ### Pre-Built Binaries Pre-built binaries for Linux, MacOS, and Windows can be found on [the releases page](https://github.com/casey/filepack/releases). You can use the following command on Linux, MacOS, or Windows to download the latest release, just replace `DEST` with the directory where you'd like to put `filepack`: ```sh curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to DEST ``` For example, to install `filepack` to `~/bin`: ```sh # create ~/bin mkdir -p ~/bin # download and extract filepack to ~/bin/filepack curl --proto '=https' --tlsv1.2 -sSf https://filepack.com/install.sh | bash -s -- --to ~/bin # add `~/bin` to the paths that your shell searches for executables # this line should be added to your shell's initialization file, # e.g. `~/.bashrc` or `~/.zshrc` export PATH="$PATH:$HOME/bin" # filepack should now be executable filepack --help ``` Note that `install.sh` may fail on GitHub Actions or in other environments where many machines share IP addresses. `install.sh` calls GitHub APIs in order to determine the latest version of `filepack` to install, and those API calls are rate-limited on a per-IP basis. To make `install.sh` more reliable in such circumstances, pass a specific tag to install with `--tag`. Usage ----- Filepack supports a number of subcommands, including `filepack create` to create a manifest, `filepack verify` to verify a manifest, and `filepack serve` to start an HTTP package server. See `filepack help` for supported subcommands and `filepack help SUBCOMMAND` for information about a particular subcommand. ### `filepack create` Create a manifest. Recommended lints can be enabled with: ```shell filepack create --deny distribution ``` ### `filepack verify` Verify the contents of a directory against a manifest. To verify the contents of `DIR` against `DIR/manifest.filepack`: ```shell filepack verify DIR ``` If the current directory contains `manifest.filepack`, `DIR` can be omitted: ```shell filepack verify ``` `filepack verify` takes an optional `--print` flag, which prints the manifest to standard output if verification succeeds. This can be used in a pipeline to ensure that you the manifest has been verified before proceeding: ```shell filepack verify --print | jq ``` ### `filepack serve` Start HTTP server. The filepack server has limited authentication and no disk usage quota and should be considered experimental. To serve HTTP on `0.0.0.0:80`: ```shell filepack serve ``` To serve HTTPS on `0.0.0.0:443` with an ACME TLS certificate: ```shell filepack serve --https --domain filepack.example ``` See `filepack serve --help` for more details. Files and packages can be uploaded with `filepack upload` and downloaded with `filepack download`. Packages can be deleted with `filepack delete`. Data Directory -------------- Filepack stores local data, including private keys, in the filepack data directory, located in `~/.filepack`, or `$XDG_DATA_HOME/filepack` if `$XDG_DATA_HOME` is set and non-empty. Filepack stores private keys in the `keychain` subdirectory of the filepack data directory, by default in `~/.filepack/keychain`. The location of the filepack data directory used by a command can be overridden with the `--data-dir` option or the `FILEPACK_DATA_DIR` environment variable. Manifest -------- `filepack` manifests are conventionally named `manifest.filepack` and are placed alongside the files they reference. Manifests are encoded with deco, a compact binary serialization format, and may be converted to JSON for inspection or manipulation with `filepack manifest`. Manifests, when converted to JSON, are an object with three mandatory keys, `embedded`, `package`, and `signatures`. ### `embedded` The value of the mandatory `embedded` key is an object mapping BLAKE3 hashes to hex-encoded file contents. The `package` object only includes file names, hashes, and sizes, but files may be embedded in the manifest archive in the `embedded` map. This is currently only used to embed `metadata.filemeta`. ### `package` The value of the mandatory `package` key is an object mapping path components to directory entries. Directory entries may be subdirectories or files. Files are objects with two fields: `hash`, the tagged hex BLAKE3 hash of the file, and `size`, the length of the file in bytes. Path components are UTF-8 and may not be `.` or `..`, contain the path separators `/` or `\`, contain control characters, or begin with a Windows drive prefix, such as `C:`. ### `signatures` The value of the mandatory `signatures` key is an array of signatures. Signatures are tagged hex strings that include an Ed25519 key, the package fingerprint the signature is made over, an optional timestamp, and the signature itself. Public keys are Curve25519 points and signatures are Ed25519 signatures made over the hash of a serialized deco byte string containing the package fingerprint, which commits to the content of `package`, and an optional timestamp. ### Example A manifest converted to JSON over a directory containing the files `README.md` and `src/main.rs`, with a signature: ```json { "embedded": {}, "package": { "README.md": { "hash": "hash1fc253b84551ce6b00e820a826ac18054dc7f63a318ce62f3175315f5c467a62a", "size": 11883 }, "src": { "main.rs": { "hash": "hash11fa48b95ed335369d45b91af8138bdccd1413364bcdbfa6e9034e8a2cfd6e17f", "size": 33 } } }, "signatures": ["…"] } ``` The signature is elided for brevity. Signatures are tagged hex strings containing a public key and signature. Tagged Hex Strings ------------------ Tagged hex strings begin with a tag identifying the type, followed by a `1`, followed by the the encoded hex payload. The tags are: | tag | type | |---|---| | `hash1…` | hash | | `package1…` | package fingerprint | | `private1…` | private key | | `public1…` | public key | | `revision1…` | revision hash | | `signature1…` | signature | | `token1…` | authentication token | Metadata -------- Filepack packages may contain a file named `metadata.filemeta` describing the package and its contents. If present, `metadata.filemeta` is also embedded in the `manifest.filepack` archive. Metadata is authored by creating a file named `metadata.yaml` in the root of a new package. `filepack create` then loads `metadata.yaml` if present, checks for validity and unknown fields, and writes the deco serialization to `metadata.filemeta` in the package root. `metadata.yaml` is retained as a human-readable reference and for amending metadata, but `metadata.filemeta` is the authoritative source of metadata. For consumption by scripts and tools, `filepack metadata` prints the contents of `metadata.filemeta` as JSON. Filepack metadata is intended to be a broadly useful machine and human readable description of the contents of a package, covering personal, distribution, and archival use-cases. Metadata follows a fixed schema and is not user-extensible. Future versions of `filepack` may define new metadata fields, causing verification errors if those fields are present and invalid according to the new schema. Please feel free to open an issue with ideas for new metadata fields. ### Schema This schema is for the YAML authoring format and the JSON output of `filepack metadata`. The deco schema is currently undocumented. Fields are given as `NAME: TYPE`. All fields are optional. Top-level fields: - `artwork: path.{jpg,png}`: The path of a JPEG or PNG file containing artwork for the content, for example, cover art for an album or key art for a movie. - `creator: text`: The person or group who created the content. - `description: text`: A description of the content. - `homepage: url`: Primary URL for the content. Should be the official homepage of the content, if any, and not, for example, a Wikipedia or media database link. - `language: language`: The primary language of the content. - `media: object`: The media metadata. - `package: object`: The package metadata. - `publisher: text`: The person or group who published the content. - `readme: path.md`: The path of the content readme. - `time: time`: The time the content was created or released. - `title: text`: The content's human-readable title. Fields of `media` containing media-specific metadata: - `type: {audio,image,video,web}`: The media type. If the media type is `audio`, `image`, or `video`, the media object contains a field named `items`, which is a list of objects containing metadata for individual items in the package. When authoring metadata YAML, each item is an object with a `path` field containing the path to the item. For example, for an `audio` package: ```yaml media: type: audio items: - path: foo.flac - path: bar.flac ``` Additionally, `video` items have an optional `placeholder` field with a path to an image that should be used as a stand-in before the video begins playing: ```yaml media: type: video items: - path: foo.webm placeholder: foo.jpg - path: bar.webm placeholder: bar.jpg ``` `filepack create` populates binary filepack metadata with metadata extracted from each item. Fields of `package` describing the package itself, as opposed its content: - `colophon: path.md`: The path of the package colophon. - `creator: text`: The person or group who created the package. - `description: text`: A description of the package. - `homepage: url`: Primary URL for the package. - `time: time`: The time the package was created. - `title: text`: The package's title. May contain details related to the source, packager, or encoding. Types: - `language`: A string containing an ISO 639-1 two-character language code. See `filepack languages` for valid language codes. - `path`: A string containing a relative path consisting of `/`-separated components. - `path.EXTENSION`: A path that must end with `.EXTENSION`. - `text`: A plain text string. May not contain control characters, including newlines and tabs. - `time`: A string containing a time with one of two precisions, a year only, or a year, month, and day, written `Y-MM-DD`. Times use the proleptic Gregorian calendar with astronomical year numbering. - `url`: A string containing a URL with scheme `http` or `https`. Example times: ```tsv 0 -44 1929 -13787000000 0-01-01 929-01-01 1970-01-01 -44-03-15 ``` ### Example ```yaml title: Tobin's Spirit Guide creator: John Horace Tobin publisher: Miskatonic University Press artwork: cover.png time: 1929 description: A compilation of supernatural occurrences, entities, and facts. homepage: https://tobin-society.org/spirit-guide language: en readme: README.md package: title: Tobin's Spirit Guide - First Edition creator: Egon Spengler time: 1984-07-08 description: >- First edition on loan from NYPL Main Branch research stacks. Captured via Microtek MS-300A flatbed scanner. homepage: https://ghost-busters.net/~egon colophon: COLOPHON.md ``` The `homepage` URLs are of course anachronistic, as the World Wide Web was created in 1989, some years after Egon first packaged Tobin's Spirit Guide. Lints ----- `filepack create` supports optional lints to check your package for issues: ```shell filepack create --deny distribution ``` The `distribution` lint group checks for issues which can cause problems if the package is intended for distribution, such as non-portable paths that are illegal on Windows, paths which would conflict on case-insensitive file systems, and inclusion of junk files such as `.DS_Store`. Lint groups and the lints they contain may be printed with: ```shell filepack lints ``` Individual lints may be allowed with `--allow`: ```shell filepack create --deny distribution --allow windows-reserved-filename ``` Keys and Signatures ------------------- `filepack` supports the generation of [Curve25519](https://en.wikipedia.org/wiki/Curve25519) public/private keypairs, and the creation and verification of [EdDSA](https://en.wikipedia.org/wiki/EdDSA) signatures over manifests. ### Keypair Generation Keypairs are generated with: ```shell filepack keygen ``` Which creates `master.public` and `master.private` files in the `keychain` subdirectory of the filepack data directory. ### Public Key Printing Generated public keys can be printed with: ```shell filepack key ``` ### Signing Signatures are created with: ```shell filepack sign ``` Which signs the manifest in the current directory with your master key and adds the signature to the manifest's `signatures` array. Signatures are made over a fingerprint hash, recursively calculated from the contents of the manifest. ### Signature Verification Signatures embedded in a manifest are verified whenever a manifest is verified. The presence of a signature by a particular public key can be asserted with: ```sh filepack verify --key PUBLIC_KEY ``` Which will fail if a valid signature for `PUBLIC_KEY` over the manifest contents is not present. Fingerprints ------------ Filepack signatures are made over the package fingerprint, which is the hash of a deco object which commits to the files and directories contained in the manifest. Fingerprints are BLAKE3 hashes, constructed such that it is impossible to produce packages which are different but which have the same fingerprint. Fingerprints may be used as a globally unique identifier. If two packages have the same fingerprint, they have the same content. Guide ----- The `[DIRECTORY]` argument in examples can be omitted, in which case it defaults to the current directory. ### Creating a Package To create a filepack manifest: ```shell filepack create [DIRECTORY] ``` This creates `manifest.filepack` containing hashes and file sizes of all files in the `DIRECTORY` and subdirectories. To enable linting, use the `--deny` flag with a lint or lint group: ```sh filepack create --deny [DIRECTORY] ``` To view all lints: ```shell filepack lints ``` Please feel free to open issues requesting new lints. ### Creating a Package with Metadata To create a package with metadata describing the package, create a file named `metadata.yaml` in the package directory, for example: ```yaml title: The Necronomicon creator: Abdul Alhazred description: >- The Old Ones, their history, and the rites by which they may be summoned. language: la ``` Then, to create the package: ```shell filepack create [DIRECTORY] ``` Which will validate metadata if present and write it to `metadata.filemeta`, as well as include it in `manifest.filepack`. See [metadata](#metadata) for the full schema. ### Signing a Package To sign a package when creating it, first create a new public and private key pair: ```shell filepack keygen ``` This creates a new public key, `master.public`, with corresponding private key, `master.private`, in the filepack keychain directory. The `filepack info` command will print the path to the filepack data directory, keychain directory, and any public keys in the keychain directory. Your public key can be printed with: ```shell filepack key ``` To sign a new package: ```shell filepack create --sign [DIRECTORY] ``` To add a signature to an existing package: ```shell filepack sign ``` ### Getting the Package Fingerprint To print the fingerprint of a package: ```shell filepack fingerprint [DIRECTORY] ``` ### Publishing a Manifest Manifests contain only hashes and not file content, and so can be published anywhere, including places where technical or legal limitations would preclude publication of the content itself. ### Verifying a Package Package content can be verified with: ```shell filepack verify [DIRECTORY] ``` Any extra files, missing files, or modified files will produce an error. This verifies files against hashes stored in the manifest. Running `filepack verify` on its own can detect accidental corruption, but not intentional modification, since an attacker could modify package content, and then modify the manifest to make it match the modified content and thus pass verification. However, if the manifest was obtained from a trusted source, verification will catch any modifications, intentional or otherwise, even if the package content was obtained from an untrusted source. To detect intentional modification, you can either verify a package against a known-good fingerprint: ```sh filepack verify --fingerprint [DIRECTORY] ``` This verifies both the contents of the package and that the manifest has the expected fingerprint. The former protects against accidental corruption, and the latter against intentional modification. Packages can also be verified by checking for a signature by a known public key. This also protects against intentional modification, as long as the private key corresponding to the public key is protected, and only known to you, or someone you trust. To verify that a package has a signature from a particular public key: ```sh filepack verify --key [DIRECTORY] ``` The `--key` option can be repeated to require signatures from multiple public keys. ### Detecting Accidental Corruption Create a filepack manifest with: ```shell filepack create ``` This will create `/manifest.filepack` To later verify the package against the manifest: ```shell filepack verify ``` Because the manifest contains cryptographic hashes, accidental corruption to the files or manifest will always be detected by `filepack verify`. This is *not* the case with intentional, malicious corruption, since an attacker could modify the files and replace the manifest hashes with the hashes of the modified files. ### Detecting Malicious Corruption Because an attacker could modify the files and replace the manifest hashes with the hashes of the modified files, you must ensure that the manifest has not been tampered with. This can be accomplished in a number of ways, either by saving the manifest to a secure location, saving the package fingerprint, or signing the package. #### Manifest To save the manifest in a secure location, use the `--manifest` option to save the manifest somewhere other than the package: ```shell filepack create --manifest ``` Then, verify the package against the saved manifest: ```shell filepack verify --manifest ``` Because the manifest was protected, any modification to the package will be detected. This has the advantage that not only will any modifications be detected, but which files were modified can also be detected. #### Fingerprint Create the manifest in the package root with: ```shell filepack create ``` Print the package fingerprint: ```shell filepack fingerprint ``` Save the fingerprint in a secure location. Then, verify the package against the saved fingerprint: ```sh filepack verify --fingerprint ``` Because the fingerprint was protected, any modification to the package will be detected. This has the advantage that you only have to save a small text string, but the disadvantage that while any modifications will be detected, you will not be able to determine which files have changed. #### Signature Create the manifest in the package root and sign it with your `master` key: ```shell filepack create --sign ``` Then, verify the package and its signature: ```shell filepack verify --key master ``` Any modification to the package or manifest will invalidate the signature, which will be detected. This has the advantage of not needing to save the manifest or fingerprint of packages you want to verify. However, you will need to generate and secure your private key. ### Determining Authenticity To check the authenticity of a package created by someone else, get their public key and verify that the package contains a signature by that key: ```sh filepack verify --key ``` Alternatives and Prior Art -------------------------- `filepack` serves the same purpose as programs like `shasum`, which hash files and output a text file containing file hashes and paths, which can later be used with the same program to verify that the files have not changed. They output hashes and paths one per line, separated by whitespace, and mainly differ in which hash function they use. Some examples, with links to implementations and the hash functions they use: | binary | hash function | |---|---| | [`b2sum`](https://github.com/BLAKE2/BLAKE2) | [BLAKE2](https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE2) | | [`b3sum`](https://github.com/BLAKE3-team/BLAKE3) | [BLAKE3](https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE3) | | [`cksfv`](https://zakalwe.fi/~shd/foss/cksfv/) | [CRC-32](https://en.wikipedia.org/wiki/Cyclic_redundancy_check) | | [`hashdeep`](https://github.com/jessek/hashdeep) | various | | [`hashdir`](https://github.com/ultimateanu/hashdir/) | various | | [`sha3sum`](https://codeberg.org/maandree/sha3sum) | [SHA-3](https://en.wikipedia.org/wiki/SHA-3) | | [`shasum`](https://metacpan.org/pod/Digest::SHA) | [SHA-1](https://en.wikipedia.org/wiki/SHA-1) and [SHA-2](https://en.wikipedia.org/wiki/SHA-2) | CRC-32 is not a cryptographic hash function and cannot be used to detect intentional modifications. Similarly, SHA-1 was thought to be a cryptographic hash function, but is now known to be insecure. `filepack` and `b3sum` both use BLAKE3, a fast, general-purpose cryptographic hash function. `filepack` can also create and verify signatures. Other signing and verification utilities include: | binary | about | |---|---| | [`gpg`](https://gnupg.org/) | general-purpose, [OpenPGP](https://www.openpgp.org/) implementation | | [`ssh-keygen`](https://man.openbsd.org/ssh-keygen.1) | general-purpose, shipped with [OpenSSH](https://www.openssh.com/) | | [`minisign`](https://github.com/jedisct1/minisign) | general-purpose | | [`signifiy`]( https://github.com/aperezdc/signify) | general-purpose | | [`SignTool`](https://learn.microsoft.com/en-us/windows/win32/seccrypto/signtool) | Windows code signing | | [`codesign`](https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/Procedures/Procedures.html) | macOS code signing | | [`jarsigner`](https://docs.oracle.com/javase/8/docs/technotes/tools/windows/jarsigner.html) | JDK code signing | Filepack for SFV Enjoyers ------------------------- If you package content for distribution, Filepack offers a number of benefits over simple file verification with `.sfv` files. - Filepack can detect both accidental corruption and intentional modification, whereas `.sfv` files can only detect accidental corruption. - Because `manifest.filepack` manifests contain file sizes, Filepack can tell the user not just whether a file has been modified, but also whether it is empty, truncated, or too long. - Filepack packages have a fingerprint, a short text string beginning with `package1…` which is guaranteed to be globally unique, allowing packages to be identified and referenced by fingerprint alone. - Fingerprints can be used to verify that a `manifest.filepack` manifest itself has not been tampered with, proving authenticity of a package regardless of its source. - Packages can be signed, allowing users to verify authenticity of any package from a packager by public key, a short string beginning with `public1…`. - Filepack can warn you if package filenames might cause issues on other operating systems or file systems. - Filepack packages are Merkle trees, both of files and within files. A user with a manifest, package fingerprint, or trusted public key can incrementally stream and verify files, or access and verify file content at random. Additionally, errors can be detected and recovered from by transmitting only those parts of the file which are corrupted. - Packages may contain machine readable metadata following a filepack-defined schema, allowing packages to be searched, indexed, and exposed through rich, featureful interfaces. Why is metadata important? -------------------------- Metadata is the difference between a good user experience and a bad user experience. On Netflix, movies are presented in an attractive fashion with artwork, titles, actors, and other information. All content is playable on whatever device you happen to be using, streaming starts instantly, search is useful, and a recommendation engine surfaces movies you might like. It is usable by anyone, of any age and with any degree of technical proficiency. On BitTorrent, movies are presented inscrutably in text listings with weirdly formatted titles, spotty or missing information, and no artwork. Content may or may not be playable on your device, streaming is impossible, search is unstructured, and recommendations are unavailable. It is only usable by a small and decreasing percentage of the population, and asking someone over to "BitTorrent and chill" will be met by blinking incomprehension, followed by derision as you fiddle with your torrent client like a digital steam engine operator, desperately trying to find seeds for the movie you want to watch. The difference between the two systems is not due to a difference in underlying data, every piece of content one could possibly imagine is available on BitTorrent. It is a difference in the availability of metadata. Netflix has a database of standardized, machine-readable metadata, supporting nearly every feature of the system. BitTorrent has a disparate collection of folders of files, with inconsistent and incomplete metadata, and the user experience and features of the system are a direct consequence. As polished services driven by metadata have become more popular, the user experience that systems based on folders of files can offer has become ever more unacceptable and alien, regardless of cost or underlying availability of content. Filepack seeks to rectify this by standardizing machine-readable metadata that can be included with folder-of-files content. Filepack metadata is stored a file, `metadata.filemeta`, in the root of a filepack package as well as in the `manifest.filepack` archive, and contains information related to what the package contains, which files in the package contain the package content and their file formats, and who created the package. This metadata can serve as a base for the creation of rich local and distributed applications and services, with user experiences that compete with and exceed those of closed centralized alternatives. Design ------ ### Manifest Format A filepack manifest contains all information needed to verify the contents of a directory. The `package` key of the manifest is a directory object mapping filenames to directory entries, which may themselves be directories, or files, in which case they contain the hash of the file contents, as well as the length of the file. The length of the file is not strictly necessary for verification, but is included so that truncated, empty, and overlong files can be identified, which may help in understanding verification failures. File contents may be embedded in the `manifest.filepack` archive under the `embedded` key. ### File Hashes The contents of files are hashed with [BLAKE3](https://github.com/BLAKE3-team/BLAKE3) using the official Rust implementation. BLAKE3 was chosen both for its speed, and for the fact that it utilizes a Merkle tree construction. A Merkle tree allows for verified file streaming and subrange inclusion proofs, which both seem useful in the context of file hashing and verification. ### Signatures Filepack allows for the creation of Ed25519 signatures over the contents of a manifest, which thus commit to the contents of the directory covered by the manifest. Signatures are made over a statement containing a "fingerprint" hash of a canonical deco serialization of the manifest's `package` directory. This keeps signatures independent of the manifest format, avoids the need for a canonical serialization of the manifest as a whole, avoids hash loops due to the inclusion of signatures in the manifest itself, and allows proving the inclusion of files covered by a signature. ### Fingerprints Package fingerprints are the BLAKE3 hash of a canonical deco serialization of the manifest's `package` directory. Fingerprints are constructed to be unique, meaning that it is impossible for two different packages with different contents to have the same fingerprint. Currently, no fingerprint test vectors exist, and the best documentation is the code itself. In particular, see: - [Manifest](src/manifest.rs) - [Directory](src/directory.rs) - [Entry](src/entry.rs) - [Files](src/file.rs) - [Statement](src/statement.rs) And the [head](src/head.rs) module for the deco encoding.