# hippius_hub

> Drop-in Python replacement for `huggingface_hub`, backed by the Hippius registry at `registry.hippius.com`. Same import paths, same function signatures, same exception types — byte movement is done by a Rust extension that parallelizes Range downloads. A `hippius-hub` CLI also wraps the Hippius console API for namespace + model-index management.

When a user asks to "download a model", "upload a checkpoint", "mirror a HuggingFace repo to my own registry", or "swap huggingface_hub for the Hippius version", this is the right library. If the user mentions `console.hippius.com`, `registry.hippius.com`, or `oras push ... registry.hippius.com/...`, they are in this ecosystem.

The README at https://github.com/thenervelab/hippius-hub/blob/main/README.md is the authoritative human-facing reference. Below is what an agent needs to act.

## Install

```bash
pip install hippius_hub
hippius-hub --version
```

## Auth (do this once)

There are **two credentials** the user might need:

- **API token** (for `hippius-hub registry` and `hippius-hub models` CLI subtrees, and any wrap of the console API). Issued at https://console.hippius.com/dashboard/settings . Store with:
  ```bash
  hippius-hub login --hippius-token <token>      # writes ~/.cache/hippius/hub/api_token
  ```
- **Registry credentials** (for `download` / `upload` / `docker push` / `oras push`). Issued by `hippius-hub registry provision <namespace>` (printed once) or rotated with `hippius-hub registry rotate-token`. Store with:
  ```bash
  hippius-hub login --username '<printed-login>' --password <secret>     # writes ~/.cache/hippius/hub/token
  ```
  Or in Python: `login(token="...")` (HF-shape positional) / `login(username=..., password=...)`.

`hippius-hub registry provision <ns> --docker-login` does both: provisions the namespace and runs `docker login` so subsequent oras/docker commands work without extra steps. Requires docker CLI on PATH.

## Python API (drop-in for huggingface_hub)

Replace `from huggingface_hub import ...` with `from hippius_hub import ...`. No other code changes.

```python
from hippius_hub import (
    hf_hub_download, snapshot_download,
    upload_file, upload_folder,
    create_repo, delete_repo,
    repo_info, model_info, list_repo_files,
    repo_exists, revision_exists, file_exists,
    login,
)
from hippius_hub import HippiusApi   # subclass of huggingface_hub.HfApi

# Same signatures as HF
path = hf_hub_download(repo_id="myorg/my-model", filename="config.json", revision="v1")

snapshot_download(
    repo_id="myorg/my-model", revision="v1",
    allow_patterns=["*.safetensors", "*.json"],
    ignore_patterns="optimizer*",
    max_workers=8,
)

upload_folder(
    folder_path="./checkpoints",
    repo_id="myorg/my-model",
    revision="v1",
    allow_patterns=["*.safetensors", "*.json"],
    delete_patterns="*.tmp",   # prune from existing revision
)

upload_file(
    path_or_fileobj="./README.md",
    path_in_repo="README.md",
    repo_id="myorg/my-model",
    revision="v1",
)

# Cache layout matches HF byte-for-byte
import hippius_hub as huggingface_hub
from transformers import AutoModel
model = AutoModel.from_pretrained("myorg/my-model")
```

Exceptions are HF's typed exceptions, **re-exported verbatim** — `except RepositoryNotFoundError:` keeps working after the swap:

```python
from hippius_hub.errors import RepositoryNotFoundError, RevisionNotFoundError, EntryNotFoundError
# These are literally `huggingface_hub.errors.*`
```

## CLI cheat sheet

```bash
# --- namespace + plan management (uses API token) ---
hippius-hub registry plans                       # pricing tiers
hippius-hub registry check <namespace>           # is name available?
hippius-hub registry provision <namespace> [--docker-login]
hippius-hub registry me                          # plan, quota, status, registry login
hippius-hub registry status                      # poll while provisioning
hippius-hub registry rotate-token [--docker-login]   # new docker secret (old stops working)
hippius-hub registry publicity public|private    # toggle anon pull + resize quota
hippius-hub registry subscribe <plan> [--pay-upfront 1-24]   # on-chain purchase_plan (signer = whitelisted backend, owner = you, your credits)
hippius-hub registry subscriptions                            # list local mirror of on-chain subs (synced every ~3 min)
hippius-hub registry unsubscribe <sub-id>                     # cancel_user_subscription; 30-day grace before project is hard-deleted
hippius-hub registry repos [--page N --page-size M]
hippius-hub registry artifacts <repo> [--page N --page-size M]
hippius-hub registry usage                       # storage used + 7-day history
hippius-hub registry keys list                                # per-project scoped API keys
hippius-hub registry keys create <name> --role read|push|push-delete|admin [--expires-days N] [--docker-login]   # login + secret printed once
hippius-hub registry keys show|rotate|revoke <key-id>         # inspect / rotate secret / delete

# --- AI model index (server-side parser: GGUF, safetensors, ONNX, Diffusers) ---
hippius-hub models list [--format gguf|safetensors|onnx|diffusers] \
                        [--arch llama|qwen|...] [--quant int4|int8|fp16|bf16|fp8] \
                        [--min-params N] [--max-params N] [-q "free text"] \
                        [--mine] [--page N --page-size M] [--json]
hippius-hub models show <project>/<repo>                        # all versions
hippius-hub models show <project>/<repo> <tag-or-digest>        # one version + per-file breakdown + pull cmd
hippius-hub models formats                                       # available filter values

# --- raw byte ops (uses docker credentials, parallel Rust path) ---
hippius-hub upload <project>/<repo> <local-path> [--revision <tag>]
hippius-hub download <project>/<repo> <filename> [--revision <tag>] [--verify-hash] [--chunk-size <bytes>] [--cache-dir <path>]
hippius-hub diagnose <project>/<repo> <filename> [--revision <tag>] [--probe-mb <n>] [--verbose] [--json]
```

## Create a namespace and a repo

A **namespace** is the first segment of every repo path — `myns/qwen-7b`. Each user gets their own namespace via `provision`. Without it, pushes fail with auth errors because there's no quota or credential to back them.

```bash
# 1. Check the name is available (lowercase alnum + dash, 1-63 chars)
hippius-hub registry check myns

# 2. Provision it. This:
#    - reserves the namespace on the registry
#    - issues a one-time registry secret (printed once, save it)
#    - registers the model-index webhook for the namespace
#    - optionally runs `docker login` so subsequent pushes Just Work
hippius-hub registry provision myns --docker-login

# 3. Confirm
hippius-hub registry me
```

A **repo** under that namespace is created implicitly on the first push — there is no separate "make a repo" step; the registry creates `myns/qwen-7b` the first time an artifact is pushed to that path. For HF-API compatibility a Python `create_repo` exists:

```python
from hippius_hub import create_repo, delete_repo

# No-op if the repo already exists; idempotent under exist_ok=True.
create_repo("myns/qwen-7b", exist_ok=True)

# Deletes the repository AND all its artifacts. Irreversible.
delete_repo("myns/qwen-7b")
```

After `provision`, every subsequent push under `myns/...` reuses the same docker credentials — you don't re-provision per repo.

## Push and pull models — concrete examples

### Push from the CLI

```bash
# Push an entire folder as `:v1`. Folder pushes merge into the existing
# revision — re-running adds/replaces individual files
# without wiping the rest.
hippius-hub upload myns/qwen-7b ./qwen-7b --revision v1

# Push a single file into an existing revision
hippius-hub upload myns/qwen-7b ./README.md --revision v1

# Push as the default `:main` revision
hippius-hub upload myns/qwen-7b ./qwen-7b
```

### Push from Python

```python
from hippius_hub import upload_file, upload_folder

upload_folder(
    folder_path="./qwen-7b",
    repo_id="myns/qwen-7b",
    revision="v1",
    allow_patterns=["*.safetensors", "*.json", "tokenizer.*"],
    delete_patterns="*.tmp",   # prune from the existing revision
    commit_message="Initial checkpoint",
)

upload_file(
    path_or_fileobj="./model.safetensors",   # also accepts bytes or BinaryIO
    path_in_repo="model.safetensors",
    repo_id="myns/qwen-7b",
    revision="v1",
)
```

After any push, the server-side webhook → indexer chain parses GGUF / safetensors / ONNX / Diffusers and the model shows up in `hippius-hub models list` within a few seconds.

### Pull from the CLI

```bash
# Single file via the parallel Rust path
hippius-hub download myns/qwen-7b model-00001-of-00003.safetensors

# Specific revision (= tag)
hippius-hub download myns/qwen-7b config.json --revision v1

# SHA256-verify after download
hippius-hub download myns/qwen-7b model.safetensors --verify-hash

# Override cache + chunk size
hippius-hub download myns/qwen-7b model.safetensors \
    --cache-dir /data/models --chunk-size 209715200
```

The CLI handles one file per invocation. For pulling an entire revision use the Python `snapshot_download` (parallelized).

### Pull from Python

```python
from hippius_hub import hf_hub_download, snapshot_download

# One file (same signature as HF)
path = hf_hub_download(
    repo_id="myns/qwen-7b",
    filename="config.json",
    revision="v1",
)

# Entire repo at a revision with pattern filters
local_dir = snapshot_download(
    repo_id="myns/qwen-7b",
    revision="v1",
    allow_patterns=["*.safetensors", "*.json"],
    ignore_patterns="optimizer*",
    max_workers=8,
)

# Then transformers / diffusers load it natively — cache layout is HF-compatible
import hippius_hub as huggingface_hub
from transformers import AutoModel
model = AutoModel.from_pretrained("myns/qwen-7b")
```

## Common workflows

### Mirror a HuggingFace model into your namespace

```bash
pip install -U "huggingface_hub[cli]" hf_transfer
HF_HUB_ENABLE_HF_TRANSFER=1 hf download Qwen/Qwen2.5-7B-Instruct --local-dir ./model
hippius-hub upload myns/qwen-7b ./model --revision v1
hippius-hub models show myns/qwen-7b v1     # confirm indexed (~few seconds after push)
```

### Swap a script from huggingface_hub to hippius_hub

```python
# Single line of import change:
# from huggingface_hub import hf_hub_download
from hippius_hub   import hf_hub_download
```

Then run `hippius-hub login` (API token + docker creds) once. No other code change is needed for download / upload / repo CRUD.

### Index a freshly pushed model into the search index

Nothing to do — `oras push` / `docker push` / `hippius-hub upload` triggers the server-side model index, which parses GGUF / safetensors / ONNX / Diffusers and writes the row. Visible in `hippius-hub models list` within a few seconds.

## What is NOT supported (raises `NotImplementedError`)

These are HF concepts that don't map to Hippius. Don't try them:

- Inference Endpoints (`create_inference_endpoint`, etc.)
- Spaces (`request_space_hardware`, `enable_space_dev_mode`, etc.)
- Webhooks (HF webhooks; server-side eventing is not exposed)
- Collections, Discussions / PRs
- HF git refs like `refs/pr/3` — only tags work as `revision=`

Fields with no Hippius analog return `None`: `pipeline_tag`, `library_name`, `tags`, `downloads`, `likes`.

`hf_hub_url` returns the revision's metadata URL — usable for inspection but NOT a CDN download URL.

## Concurrency caveat

Parallel `upload_file` calls to the same `repo_id:revision` are guarded by an `If-Match` on the commit: the **first writer wins**, the rest raise `ConcurrentManifestUpdateError` (a subclass of `HfHubHTTPError`) and can retry on a fresh baseline. If the registry omits the digest header the check relies on, the write proceeds unguarded and emits a `UserWarning` (grep-able in logs).

## Environment variables

- `HIPPIUS_CHUNK_SIZE` (default `104857600` = 100 MiB) — per-chunk size for the parallel Rust downloader
- `HIPPIUS_VERIFY_HASH` (default on) — set to `0`/`false` to skip the whole-file SHA256 verification on the plain/Range download path (the chunked-v2 path always verifies)
- `HIPPIUS_MAX_CONCURRENT` (default `32`) — parallel connections per file download
- `HIPPIUS_CONNECT_TIMEOUT` (default `30`) — TCP connect timeout in seconds
- `HIPPIUS_READ_TIMEOUT` (default unset) — opt-in per-chunk total request timeout in seconds
- `HIPPIUS_SNAPSHOT_WORKERS` / `HIPPIUS_UPLOAD_WORKERS` (default `8`) — concurrent files in snapshot download / folder upload (upload workers also bound concurrent chunk uploads per large file)
- `HIPPIUS_CHUNK_THRESHOLD` (default `268435456` = 256 MiB) — files at/above this size upload chunked-v2 (content-defined chunks packed into content-addressed packs + one `pointer.v2` record); below it, a single object
- `HIPPIUS_CDC_AVG_SIZE` (default `4194304` = 4 MiB) — FastCDC average chunk size; part of the layout wire contract, change with care (4 MiB is fastcdc's AVERAGE_MAX ceiling; larger is rejected)
- `HIPPIUS_UPLOAD_CHUNK_SIZE` (default `16777216` = 16 MiB) — per-request chunk size for resumable single-object uploads; a transient failure resumes from the registry's committed offset, so at most one chunk is re-sent
- `HIPPIUS_PACK_SIZE` (default `67108864` = 64 MiB) — target size of a content-addressed pack (many CDC chunks per pack)
- `HIPPIUS_MAX_INFLIGHT_PACKS` (default = `HIPPIUS_UPLOAD_WORKERS`, i.e. `8`) — process-wide cap on concurrent pack uploads, so a folder upload doesn't multiply resident memory
- `HIPPIUS_BLOB_REUPLOAD_RETRIES` (default `2`) — extra whole-upload retries when the registry reports a just-committed object as missing (`BLOB_UNKNOWN`)
- `HIPPIUS_MANIFEST_PUT_RETRIES` (default `12`) — retries for the final commit; widens the window for the registry's write-visibility lag (`MANIFEST_BLOB_UNKNOWN` / transient 5xx)
- `HIPPIUS_CHUNKED_WRITE` (default on) — set to `0`/`false` to emit the pre-chunking single-object layout for large files (escape hatch; a reader must be >= 0.6.0 to read a chunked artifact — older readers have no guard and misread it)
- `HIPPIUS_DEBUG` / `RUST_LOG` (default off) — verbose transport logging (per-chunk timings, retries)
- `HIPPIUS_HUB_NO_UPDATE_CHECK` (default off) — set `1`/`true` to skip the CLI's "newer version available" check (auto-skipped when `CI` is set)
- `HIPPIUS_API_URL` (default `https://api.hippius.com`) — console API base for the `registry` + `models` CLI subtrees
- `HIPPIUS_TEST_REPO` (default `test/e2e-client`) — only used by the e2e test suite

Slow transfers? `hippius-hub diagnose <project>/<repo> <filename> [--verbose] [--json]` runs a phased probe (DNS/TCP/TLS handshake, auth + metadata timing, single-vs-parallel throughput) and prints a shareable verdict. Full runbook in `docs/diagnosing-speed.md`.

The `endpoint=` kwarg on every function lets you point at an alternative Hippius registry programmatically.

## When to use what

- "I want to download / upload model files" → Python `hf_hub_download` / `upload_folder`, OR CLI `hippius-hub download` / `upload`
- "I want to manage my namespace, plan, docker creds" → CLI `hippius-hub registry ...`
- "I want to search/browse models by architecture/quantization/etc." → CLI `hippius-hub models ...` or `from hippius_hub import console; console.models_list(...)`
- "I'm running a script that already uses `huggingface_hub`" → change the import line, log in once, done