Standalone or Spinifex · Quick Start · S3 API Support · Architecture · Configuration · Spinifex Integration · Development · Roadmap

--- # Predastore: Distributed, S3-compatible object storage Predastore is a distributed object-storage system implementing commonly used Amazon S3 APIs. It combines Raft metadata, Reed–Solomon erasure coding, QUIC transport and append-only storage segments. Predastore can run independently and provides the default object-storage backend for Spinifex. ## Standalone or Spinifex? Predastore runs on its own or as the object store of a [Spinifex](https://github.com/mulgadc/spinifex) cluster, and one thing decides which: whether the config carries an `[iam]` table. Without it the gate resolves every access key from the `[[auth]]` entries in its own TOML, dials nothing, and depends on no other service. With it the gate also reads users, groups, roles, policies and session credentials out of Spinifex's NATS JetStream KV buckets, with the config-defined accounts still matched first. Standalone is enough when a fixed set of service accounts and buckets, written into the config and distributed with it, is the whole access model. Reach for Spinifex when credentials have to change without a restart, or when more than one tenant shares the cluster. | | Standalone | Under Spinifex | |---|---|---| | Credentials | `[[auth]]` entries in the TOML | IAM access keys in NATS KV, plus the config accounts | | Authorisation | none — every config account is fully trusted | IAM policies, evaluated per request | | Buckets | `[[bucket]]` entries, plus whatever clients create | created through the API, owned per account | | Multiple tenants | no — every key reaches every bucket | yes, enforced by account ownership | | Users, groups, roles | no | IAM users, groups and roles | | Temporary credentials | no | STS assumed-role sessions | | Rotating a key | edit the TOML, restart the host | `CreateAccessKey`, no restart | | Keys and certificates | `predastore-keygen`, distributed by you | issued on `spx admin init`, distributed on join | | Cluster topology | you write the TOML | generated from the joined node set | | External dependencies | none | NATS, and the Spinifex daemon that populates it | The second row is the one that decides most deployments. A config-defined account in `[[auth]]` bypasses both policy evaluation and the bucket-ownership check, so every key in the file reaches every bucket in the cluster no matter which account owns it. Standalone predastore cannot hand out a credential that is less than total, and that — rather than any missing S3 operation — is what a multi-tenant deployment needs Spinifex for. ## Quick Start ### Build ```bash make build ``` Build through the Makefile rather than calling `go build ./cmd/s3d` yourself. The binary links a FIPS boot check that panics on startup unless it was compiled with `GOFIPS140=v1.0.0`, which the Makefile exports and a bare `go build` does not. ### Run a Development Cluster `scripts/start.sh` launches a cluster from one of the profiles in `config/`, generating the TLS keypair and master key under `$PREDA_DIR` (default `/tmp/predastore`) on first run: ```bash ./scripts/start.sh -w 1host # one process, S3 on https://127.0.0.1:8443 ``` Four profiles ship with the repo. They differ only in how many hosts the nodes are spread over, which is what decides the transport between them: | Profile | Hosts | RS | Inter-node transport | Needs `sudo` | |---------|-------|----|----------------------|--------------| | `1host` | 1 | (1, 0) | in-process pipe only | no | | `3host` | 3 on `10.11.12.{1,2,3}` | (2, 1) | QUIC between hosts, pipe within | yes | | `4host` | 4 on `10.11.12.{1..4}` | (2, 1) | QUIC between hosts, pipe within | yes | | `5host` | 5 on `10.11.12.{1..5}` | (3, 2) | QUIC between hosts, pipe within | yes | Every host in every profile sets `admin_port = 9099`, so `/healthz` and `/readyz` answer on the host's own address. That is plain HTTP with no authentication, which is safe on a loopback dev cluster and is why the flag default is `0` rather than a port. The multi-host profiles put every host on one machine behind loopback aliases, so they need `sudo` to add those aliases and to install the generated certificate as a trust anchor. Each host answers S3 on its own address. Stop, reset or benchmark the cluster with: ```bash ./scripts/stop.sh # every cluster under $PREDA_DIR ./scripts/stop.sh 3host # only that one, leaving any others running ./scripts/clean.sh ./scripts/bench.sh 3host ./scripts/bench.sh disk ``` ### Run a Host `./bin/s3d` runs one host of a cluster: the nodes the config pins to it, the S3 gate among them. `-host` names the `[[host]]` to run, and everything else about the process — which nodes, which addresses, which ports — follows from that entry: ```bash ./bin/s3d \ -config cluster.toml \ -host 1 \ -data-dir /var/lib/predastore \ -tls-cert /etc/predastore/server.pem \ -tls-key /etc/predastore/server.key \ -encryption-key /etc/predastore/master.key ``` | Flag | Description | |------|-------------| | `-config` | Path to the configuration file (required) | | `-host` | ID of the `[[host]]` this process runs (required) | | `-data-dir` | On-disk root for this host's nodes; overrides `data_dir` | | `-encryption-key` | Path to the 32-byte AES-256 key protecting data at rest; overrides `encryption_key` | | `-tls-cert`, `-tls-key` | This host's TLS identity; overrides `tls_cert` / `tls_key` | | `-bind-addr` | Listen address for this host's cluster traffic, without a port; overrides `bind_addr` | | `-gate-bind-addr` | Listen address for the S3 API, without a port; overrides the gate node's `bind_addr` | | `-log-level` | `debug`, `info`, `warn` or `error` | Each of these host-local settings may come from either the file or a flag, so the same configuration file can be deployed unchanged to every machine and the paths supplied per process. The S3 port is not a flag: it is the gate node's `port`. A cluster whose `[[host.node]]` entries all sit under one `[[host]]` runs entirely in one process over the in-process pipe, with no inter-node socket and no certificate beyond the one the gate serves. That is a property of the config, not a launch mode. The encryption key file must be exactly 32 raw bytes (no base64, no header) with mode `0600`. Generate one with `( umask 0177 && openssl rand -out master.key 32 )`. The same key must be supplied to every host in a cluster; rotating it is not currently supported (see Roadmap → envelope encryption). ### Run as a systemd Service `make install` lays down the binary, a unit, and the drop-ins that create the service's user and its directories. It installs files and performs nothing else, so it is usable from a package build and leaves the sequencing to you: ```bash make build sudo make install # binary, unit, sysusers, tmpfiles, sysctl sudo systemd-sysusers && sudo systemd-tmpfiles --create sudo sysctl --system # required for multi-host sudo cp /etc/predastore/predastore.toml.example /etc/predastore/predastore.toml sudo $EDITOR /etc/predastore/predastore.toml # replace every CHANGEME sudo predastore-keygen /etc/predastore # the at-rest key and a TLS identity sudo systemctl daemon-reload && sudo systemctl enable --now predastore ``` `DESTDIR` and `PREFIX` are honoured, so a package build can stage the same target into a buildroot. An existing `./bin/s3d` is installed as it stands rather than rebuilt, which is what lets the install run under `sudo` without `go` on root's `PATH`, so run `make build` first to pick up any source change. The service runs as an unprivileged `predastore` user under a strict sandbox: `/etc` read-only, `/var/lib/predastore` the only writable path, no capabilities at all, and a syscall filter. Its host-local settings — which `[[host]]` it runs, where its data and keys live — come from `/etc/predastore/predastore.env`, which installs as `predastore.env.example` so an upgrade cannot overwrite an edited one. The shipped example config names those paths itself, which is what a single machine wants; a fleet distributing one identical TOML overrides them per host in the env file instead. The service logs at `warn`, rather than the `info` a hand-run `s3d` defaults to, on the grounds that a long-running service wants a quiet journal. `predastore-keygen` is idempotent, and it refuses to mint a TLS identity for a config naming more than one host. Peers verify each other against the system trust store, so hosts that each generated their own keypair never elect a leader rather than failing outright. Generate one keypair, distribute it to every host, and install its CA — see [Standalone TLS Trust](#standalone-tls-trust). ### Run in Docker The image needs no arguments and no configuration file. On first start it generates a TLS identity and an at-rest key into its volume and serves a single-host RS(1, 0) cluster: ```bash make docker-build docker run -d -p 8443:8443 -p 9099:9099 -v preda-data:/var/lib/predastore predastore:dev ``` Two compose topologies ship under `deploy/`. The cluster profile is four hosts at RS(2, 1), each with a gate, a meta replica and a blob node; every gate answers for the whole cluster, so `8443`–`8446` are four ways into the same thing: ```bash make compose-up # four hosts make compose-up COMPOSE_FILE=deploy/compose.single.yml make compose-down ``` `make docker-smoke` then checks the image itself — non-root, no private key in any layer — and drives aws-cli, rclone, restic and a `registry:3` on top of the running cluster. It asserts response content rather than exit codes, and gates on `scripts/smoke-baseline.txt` so it fails on a regression rather than on the checks predastore does not pass yet. Re-record that file with `make docker-smoke-baseline` in the same change as the fix. #### Environment `s3d` itself reads no environment variables, so the entrypoint renders a configuration file from these and passes it as `-config`. Every one is optional. | Variable | Default | Description | |---|---|---| | `PREDA_PEERS` | `127.0.0.1` | Comma-separated host addresses in host-id order. More than one entry makes it a cluster | | `PREDA_HOST_ID` | `1` | Which `[[host]]` this process runs | | `PREDA_HOST_ID_FROM_ORDINAL` | `0` | Set to `1` to derive the host id from a StatefulSet pod ordinal | | `PREDA_BIND_ADDR` | `0.0.0.0` | Cluster-plane listen address | | `PREDA_RS_DATA`, `PREDA_RS_PARITY` | `1`/`0` alone, `2`/`1` otherwise | Erasure-coding width | | `PREDA_BLOB_NODES` | `1` | Blob nodes per host | | `PREDA_REGION` | `ap-southeast-2` | Region | | `PREDA_S3_PORT`, `PREDA_META_PORT`, `PREDA_BLOB_PORT` | `8443`, `6660`, `9991` | Node ports | | `PREDA_ADMIN_PORT` | `9099` | `/healthz` and `/readyz`; `0` runs neither | | `PREDA_ACCESS_KEY_ID`, `PREDA_SECRET_ACCESS_KEY`, `PREDA_ACCOUNT_ID` | the AWS example pair | Config-defined service account | | `PREDA_DATA` | `/var/lib/predastore` | Data root | | `PREDA_TLS_CERT`, `PREDA_TLS_KEY` | under `$PREDA_DATA/tls/` | TLS identity, generated if absent | | `PREDA_KEY` | `$PREDA_DATA/master.key` | AES-256 key at rest, generated if absent | | `PREDA_CONFIG` | — | Use this profile verbatim and render nothing | | `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error` | Two of these have constraints the container cannot paper over: - A cluster is refused without explicit credentials. The AWS example pair is a single-host evaluation default, not something to run four hosts on. - A cluster needs **one TLS keypair shared by every host**, because peers verify each other against the system trust store. Hosts that each generated their own identity do not fail loudly, they simply never elect a leader, so the entrypoint refuses to generate one when `PREDA_PEERS` names more than one address. `deploy/docker/gen-certs.sh` mints a shared local-development keypair; `make compose-up` runs it for you. ## S3 API Support | Area | Operations | | --- | --- | | Buckets | `CreateBucket`, `DeleteBucket`, `ListBuckets`, `HeadBucket` | | Objects | `PutObject`, `GetObject`, `DeleteObject`, `HeadObject`, `ListObjects`, `ListObjectsV2` | | Multipart | `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload` | | Authentication | AWS Signature Version 4, presigned URLs | Multi-object delete (`DeleteObjects`) takes up to 1000 keys per request and reports the outcome of each one. How far that support goes is measured rather than claimed. [docs/S3-COMPATIBILITY.md](docs/S3-COMPATIBILITY.md) holds the result of running `ceph/s3-tests` — the suite Ceph RGW, MinIO and Garage are all validated against — with the per-case manifest in `scripts/s3-tests-baseline.txt`. Read it before assuming an operation behaves the way S3 documents it. ### AWS CLI Examples Against the `1host` dev profile. Its certificate is self-signed, so either install it as a trust anchor or pass `--no-verify-ssl`: ```bash export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export AWS_DEFAULT_REGION=ap-southeast-2 # Create a bucket aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 mb s3://my-bucket # Upload a file aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 cp ./file.txt s3://my-bucket/ # List bucket contents aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 ls s3://my-bucket/ # Download a file aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 cp s3://my-bucket/file.txt ./downloaded.txt ``` ## Architecture

Predastore: S3 applications and tooling on top, authenticated S3 services backed by Raft metadata and erasure-coded storage, distributed across encrypted storage nodes over QUIC.

A cluster is built from three node roles. `s3d` is one process running the roles a host is assigned, and any mix of them can share a host: - **gate** — serves the S3 HTTP interface and Signature Version 4 authentication. - **meta** — maintains strongly consistent metadata with HashiCorp Raft over a local embedded database. - **blob** — holds erasure-coded shards in append-only segment files, sealed with AES-256-GCM. Nodes sharing a host talk over an in-process pipe; nodes on different hosts talk over QUIC. Nothing in a deployment names a transport — it follows from where the config pins each node. See [`docs/DESIGN.md`](docs/DESIGN.md) for the design in full. ## Capabilities - Reed–Solomon erasure coding - Raft-based metadata - QUIC storage transport - Append-only segment storage - AES-256-GCM encryption at rest - Consistent-hash placement - Multipart uploads - Single-binary deployment ## Configuration A cluster configuration is a TOML file describing two levels: `[[host]]` entries, which are the s3d processes owning an address and a data directory, and the `[[host.node]]` entries nested under them, which are the roles that host runs. The same file also carries the Reed-Solomon parameters, the config-defined buckets and the S3 credentials. The file is meant to be identical on every machine, so the settings only the local machine cares about — its data directory, TLS identity and encryption key — may be left out and supplied as flags instead. ```toml version = 1 region = "ap-southeast-2" [rs] data = 2 # data shards parity = 1 # parity shards; data + parity must not exceed the blob node count # degraded_writes = false # accept a write once `data` shards are durable, so one # node down does not refuse writes; repair closes the gap # hinted_handoff = false # put a shard its owner refuses on the next node along the # ring, so the stripe is complete when the write is acked # [repair] # restore shards a node holds at a stale generation # enabled = false # interval_seconds = 0 # zero defaults to 300 # One host = one s3d process, launched with `-host `. [[host]] id = 1 addr = "10.11.12.1" # what other hosts dial; no port — nodes carry those # bind_addr = "10.11.12.1" # where raft and blob listen; defaults to addr # admin_port = 9099 # /healthz and /readyz on bind_addr; absent runs neither # data_dir = "/var/lib/predastore" # absolute; -data-dir supplies it otherwise # A role this host runs. One of "gate", "blob" or "meta". [[host.node]] id = 1 role = "gate" # the S3 endpoint; port is the S3 port port = 8443 # bind_addr = "0.0.0.0" # where S3 listens; gate only, defaults to the host's [[host.node]] id = 2 role = "meta" port = 6660 [[host.node]] id = 3 role = "blob" port = 9991 # A config-defined service account. Every request one signs is authorised, on # every bucket, so treat an entry here as a root credential. [[auth]] access_key_id = "AKIAIOSFODNN7EXAMPLE" secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" account_id = "123456789012" # required; stamped as owner on buckets this key creates # A bucket that exists from startup rather than being created through the API. [[bucket]] name = "artifacts" region = "ap-southeast-2" account_id = "123456789012" # required # public = true # serve GET and HEAD without a signature ``` Node ids are unique across the whole file; ports are unique within a host. A blob or meta node without its own `data_dir` derives one from the host's root and its node id, so separate disks are a per-node setting rather than a deployment layout. A freshly mounted drive is owned by root, and s3d refuses to start while any local node's directory is not writable by the user it runs as, naming each node and path it cannot write. Hand each drive's directory to the service user before starting it, e.g. `sudo chown predastore:predastore /var/lib/predastore/disk1 && sudo chmod 0700 /var/lib/predastore/disk1`. Under the systemd unit `/var/lib/predastore` is the only writable path, so mount the drives below it rather than elsewhere, or add their paths to `ReadWritePaths`. S3 is a public service and replication is not, so the two bind separately on a multi-homed machine: the host's `bind_addr` is where raft and blob traffic listens, and the gate's own `bind_addr` is where S3 does. Set the first to a private address and the second to `0.0.0.0` and the S3 endpoint answers on every interface while nothing reaches consensus or shard traffic from outside. Neither is required — a host that names only `addr` binds that one address for both, which is what a single-homed machine wants. `admin_port` puts `/healthz` and `/readyz` on the host's `bind_addr`, alongside the cluster plane and never on the S3 port. `/healthz` answers 200 once the process is listening. `/readyz` answers 200 when this process can serve: a meta replica observes a leader, the meta plane answers a read, and at least `data` blob nodes answer one — whichever of those the roles on the host can be asked. It answers 503 otherwise, with a body naming which check failed and nothing else. A host that names no `admin_port` runs no admin listener. A background sampler runs those checks on a fixed interval and `/readyz` only ever serves its last result, so answering a request costs nothing beyond reading it — hammering the endpoint does not add load to the meta plane or the blob nodes. The answer can lag reality by up to one sampling interval, and a request arriving before the first cycle completes gets 503 rather than an unfounded 200. Write every node under one `[[host]]` and the cluster runs in one process over the in-process pipe. Spread them across hosts and each process is launched separately with its own `-host` id. Nothing else changes. `config/` holds the ready-made profiles — see [Run a Development Cluster](#run-a-development-cluster). ### Accounts and Buckets `[[auth]]` is the whole of standalone access control. An entry's `account_id` is mandatory and becomes the owner of any bucket that key creates, but it does not confine the key: a config-defined account skips both policy evaluation and the bucket-ownership check, so it reads and writes every bucket in the cluster whoever owns it. Declare as few as the deployment needs, and use Spinifex if credentials have to be scoped — see [Standalone or Spinifex?](#standalone-or-spinifex). `[[bucket]]` declares a bucket that exists from startup without a `CreateBucket` call, which is what a deployment with a fixed set of them wants. `account_id` is required and an entry missing it refuses the whole config, while an entry with an invalid name is dropped with a warning rather than being fatal. `public = true` serves GET and HEAD without a signature, and is worth setting only on a bucket genuinely meant to be world-readable. `ListBuckets` answers from the metadata plane alone, so a bucket declared in the config does not appear in `aws s3 ls` until something creates it through the API. `HeadBucket`, GET and PUT all work against it in the meantime. ### Standalone TLS Trust When predastore is deployed by Spinifex, the cluster CA is installed into the host trust store automatically as part of node bootstrap — no manual action is required. Standalone operators must install the cluster CA into the host trust store before launching `s3d`, otherwise nodes cannot dial each other: ```bash # Debian / Ubuntu sudo cp cluster-ca.pem /usr/local/share/ca-certificates/predastore-cluster-ca.crt sudo update-ca-certificates # RHEL / Fedora / Amazon Linux sudo cp cluster-ca.pem /etc/pki/ca-trust/source/anchors/predastore-cluster-ca.pem sudo update-ca-trust ``` ### Socket Buffers for QUIC A multi-host cluster needs the kernel's socket buffer limit raised, and this is a correctness requirement rather than tuning. Blob, meta and raft traffic all ride QUIC, quic-go asks the kernel for a 7 MiB receive buffer, and at the default of 208 KiB that request is silently clamped and the socket drops datagrams under load. The symptom is failed shard writes, with nothing anywhere reporting a misconfigured limit. `make install` ships the setting as a drop-in, so a systemd deployment only has to apply it: ```bash sudo sysctl --system # applies deploy/systemd/99-predastore-net.conf ``` Set it directly otherwise, and persist it under `/etc/sysctl.d` so it survives a reboot: ```bash sudo sysctl -w net.core.rmem_max=16777216 sudo sysctl -w net.core.wmem_max=16777216 ``` A single-host cluster runs every node over the in-process pipe and opens no QUIC socket at all, so none of this applies to it. ## Storage Backend Distributed storage with erasure coding, Raft-consensus metadata, and QUIC transport. The data model: | Unit | Size | Description | |------|------|-------------| | Object | arbitrary | RS-encoded end-to-end into K data + M parity shards | | Shard | `⌈object_size / K⌉` | Per-node RS slice; occupies a contiguous extent | | Fragment | 32 B header + 8 KiB body + 16 B GCM tag = 8240 B | On-disk unit; AES-256-GCM seals body with AAD bound to `(objectHash, shardIndex, shardNum, fragNum)` | | Segment file | up to 4 GiB | Append-only container holding extents from one or more shards | See [DESIGN.md](docs/DESIGN.md) §6 for the on-disk format: the segment layout, the fragment header field by field, and the AAD the GCM seal is bound to. ## Spinifex Integration Predastore is the default S3 storage provider for [Spinifex](https://github.com/mulgadc/spinifex), which turns the integration on by rendering one table into predastore's config: ```toml [iam] nats_url = "nats://127.0.0.1:4222" nats_token = "..." master_key_path = "/etc/spinifex/master.key" # absolute; decrypts the stored secrets access_keys_bucket = "spinifex-iam-access-keys" ``` That table is the whole of it. With it present the gate opens Spinifex's JetStream KV buckets directly — there is no RPC and no callback into the control plane — and resolves an access key to the user, groups, roles and policies attached to it, decrypting each stored secret with the shared master key. A key prefixed `ASIA` is resolved as an STS session instead. Config-defined `[[auth]]` accounts are still matched first, so the service credentials in the file keep working whatever IAM holds. The connection retries indefinitely and the KV buckets are opened lazily, so predastore starts cleanly before the Spinifex daemon has created them and recovers on its own when NATS restarts. What the table buys is per-request authorisation: IAM users and roles, policy evaluation, temporary session credentials, and account ownership that genuinely separates tenants. Without it every credential is a root credential — see [Standalone or Spinifex?](#standalone-or-spinifex). Spinifex also leans on predastore as the store behind several of its own services, all of it ordinary S3 traffic through the same gate: - **EC2 AMI images** — machine images for VM launches - **EBS volume snapshots** — via [Viperblock](https://github.com/mulgadc/viperblock), which uses Predastore as its S3-compatible backend - **User data** — cloud-init configurations and system artifacts ## Development ```bash make build # Build the s3d binary make certs # Generate the dev TLS certs the integration tests serve make test # Run tests make preflight # Full CI checks (lint, govulncheck, coverage, integration) make test-race # Run tests under the race detector make clean # Clean build artifacts ``` `make preflight` must pass before committing; `make fix` auto-fixes what the linter can. ### Conformance ```bash ./scripts/start.sh -w s3tests # the only profile with the accounts and region the suite needs make s3-tests # fails on a regression against scripts/s3-tests-baseline.txt make s3-tests-strict # fails on any failing case; red by design make s3-tests-baseline # re-record, in the same change as the fix ``` The suite is `ceph/s3-tests`, pinned to a commit and cloned on first run. It takes about fifteen minutes. See [docs/S3-COMPATIBILITY.md](docs/S3-COMPATIBILITY.md) for what the current numbers mean. ## Roadmap - [x] S3 API core (buckets, objects, multipart) - [x] AWS Signature V4 authentication - [x] Distributed storage with Reed-Solomon erasure coding - [x] Raft-consensus metadata - [x] QUIC transport with connection pooling - [x] Consistent hash ring placement - [x] AES-256-GCM encryption at rest (single cluster-wide master key) - [x] Background segment compaction - [ ] Envelope encryption (master key rotation, per-bucket / per-tenant keys) - [ ] Gossip-based node discovery - [ ] Automatic shard rebalancing - [ ] Background read-repair - [ ] Bucket versioning - [ ] Lifecycle policies Roadmap items describe direction and are not commitments to a release date. ## Trademarks Amazon Web Services, AWS and Amazon S3 are trademarks of Amazon.com, Inc. or its affiliates. Predastore is not affiliated with or endorsed by Amazon Web Services. ## License Predastore is licensed under the [GNU Affero General Public License v3.0 (AGPLv3)](LICENSE) license.