IceGuard

# IceGuard **A modern web console to manage, inspect and maintain Apache Iceberg™ tables across multiple catalogs.** **🌐 [adelfardi.github.io/iceguard-site](https://adelfardi.github.io/iceguard-site/)** [![Website](https://img.shields.io/badge/website-iceguard--site-06b6d4.svg)](https://adelfardi.github.io/iceguard-site/) [![CI](https://github.com/adelfardi/iceguard/actions/workflows/ci.yml/badge.svg)](https://github.com/adelfardi/iceguard/actions/workflows/ci.yml) [![Release](https://img.shields.io/github/v/release/adelfardi/iceguard?sort=semver)](https://github.com/adelfardi/iceguard/releases/latest) [![GHCR images](https://img.shields.io/badge/ghcr.io-images-2496ED.svg?logo=docker&logoColor=white)](https://github.com/adelfardi/iceguard/pkgs/container/iceguard-backend) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE) [![Backend](https://img.shields.io/badge/backend-Java%2021%20%C2%B7%20Quarkus-4695EB.svg)](backend) [![Frontend](https://img.shields.io/badge/frontend-React%20%C2%B7%20TypeScript%20%C2%B7%20Vite-61DAFB.svg)](frontend) [![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
IceGuard demo: dashboard, versioning graph, storage health, Spark maintenance

IceGuard demo: dashboard, versioning graph, storage health, Spark maintenance

_{Dashboard → a table's branches & tags graph, storage health down to each partition, timeline and schema evolution → a Spark maintenance action with its result + logs → catalogs, wizards and pipelines. ▶ MP4 version · live demo}

> ⚠️ **Independent community project — not affiliated with or endorsed by the Apache Software Foundation.** > "Apache", "Apache Iceberg", "Iceberg" and "Apache Polaris" are trademarks of the ASF. --- ## What is IceGuard? IceGuard is an admin & maintenance console for the **Apache Iceberg** lakehouse. Point it at one or more catalogs (REST, Nessie, Polaris) and get a single UI to browse namespaces and tables, evolve schemas and partitions, edit properties, sample data, inspect storage layout, view lineage, and run (or schedule) maintenance like snapshot expiration and data-file compaction. It is **not** a catalog — it sits on top of your existing Iceberg catalogs. ### What IceGuard actually is IceGuard is **two components**: 1. **Frontend** — the React/TypeScript web UI. 2. **Backend** — the Quarkus REST API. It has **one required dependency**: a **PostgreSQL** database (you provide it), where the backend stores IceGuard's own state — registered catalogs, pipelines, schedules, alerts and execution history. Postgres is not part of IceGuard; it's infrastructure IceGuard needs to run, like a JVM. IceGuard then **connects to your own** Iceberg catalog(s) and object store. > The `docker-compose.yml` in this repo (MinIO, an Iceberg REST Catalog, Nessie, Apache Polaris, > optionally Spark) is **only a local test sandbox** so you can try IceGuard end-to-end without > bringing your own infrastructure. **It is not part of the product** — in production you point > IceGuard at your existing catalogs and storage. ## Features - **Multi-catalog** — register and switch between REST, Nessie, Polaris and Unity Catalog (None/Bearer/OAuth2 supported). Catalogs can be **tagged** and filtered, and edited on a dedicated page (credentials, arbitrary properties, S3 access modes). - **Dashboard** — configurable, live widgets (storage, maintenance reliability, recently active partitions, file sizes, catalogs, executions): add them from a gallery, pin them from a table, drag to reorder. - **Namespaces & tables** — browse the tree, create namespaces, create/drop/rename tables, insert sample rows. - **Schema editor** — add / rename / retype / re-doc / drop multiple columns, applied in a **single commit**. - **Properties editor** — add / update / remove table properties in a **single commit**. - **Partition evolution** — add or drop partition fields (identity, bucket, truncate, year/month/day/hour). - **Versioning** — a git-style graph of the snapshot history with one lane per branch and tag; branch, tag (with retention policy) and snapshot lists, snapshot details and rollback. Nessie history is rebuilt from its commit log. - **Storage & health** — point-in-time totals, file-size histogram and per-partition aggregates (server-side paginated, typed filters), with a **health grade** for the table and for every partition (file size vs target, small files, delete files; configurable thresholds). Drill down to a partition's files and to a file's rows, data and delete files alike. See **[docs/STORAGE_HEALTH.md](docs/STORAGE_HEALTH.md)**. - **Evolution** — schema-version history with column diffs, and a visual snapshot-to-snapshot diff. - **Timeline** — snapshots + executions on one timeline, click any item for its output/logs. - **Maintenance** — expire snapshots, rewrite data files (strategy, sort order, `where`, options; whole table or selected partitions), rewrite manifests, rewrite position / equality delete files, remove orphan files, rollback. Pluggable executors: a **Java API** executor (real compaction and orphan removal on small, append-only tables) and a **Spark** executor (local `local[*]` with tunable resources, or a remote Spark cluster). - **Commit activity** — the quietest window of the day, detected from snapshot timestamps, to schedule maintenance off-peak. - **Pipelines** — chain maintenance actions with per-task parameters, retries and delay, and a cron schedule built graphically; Airflow-style run view with rerun and retry of a failed task. - **Alerts** — threshold rules on table metrics with optional SMTP email notifications. - **Public demo mode** — `ICEGUARD_DEMO_MODE=true` on the frontend serves a read-only UI (banner, writes refused with a clear message), as on [demo.iceguard.cloud](https://demo.iceguard.cloud). ## Supported catalogs Any Iceberg REST-compatible catalog works. The **Add Catalog** wizard shows a card per engine, and the type is auto-detected from the catalog name/URI: | | Catalog | Typical URI | Auth | |:---:|---|---|---| |

Iceberg REST

| **Iceberg REST** | `http://host:8181` | None / Bearer | | Nessie

Nessie

| **Nessie** | `http://host:19120/iceberg` | None | |

Apache Polaris

| **Apache Polaris** | `http://host:8182/api/catalog` | OAuth2 | |

Unity Catalog

| **Unity Catalog** | `https:///api/2.1/unity-catalog/iceberg-rest` | Bearer (PAT) / OAuth2 | _{Unity Catalog (Databricks or the OSS `unitycatalog.io`) is reached through its Iceberg REST
Catalog API — browse works today; writes depend on the endpoint (Databricks managed-Iceberg
read/write is in preview, requires `EXTERNAL USE SCHEMA` + a PAT/OAuth token).} Nessie has a few specifics (snapshot history rebuilt from its commit log, S3 region requirement, flaky-remote handling) — see **[docs/NESSIE.md](docs/NESSIE.md)**. ## Stack | Layer | Tech | |-------|------| | Frontend (product) | React, TypeScript, Vite, Tailwind CSS, shadcn/ui, TanStack Query, Zustand, Recharts | | Backend (product) | Java 21, Quarkus 3.17, RESTEasy Reactive, Hibernate Panache, Flyway, Apache Iceberg 1.11 | | Required dependency | PostgreSQL (backend state) | | Test sandbox only | Docker Compose: MinIO (S3), Iceberg REST Catalog, Nessie, Apache Polaris, (optional) Spark | ## Quick start Requirements: Docker + Docker Compose. (For hot-reload development: JDK 21 & Maven, Node 20+.) The app is two services — **frontend** + **backend** — plus a required **Postgres**. `docker-compose.yml` bundles them in three tiers via Compose profiles: | Level | Command | Services | When | |------|---------|----------|------| | 1 — app (official) | `docker compose up -d --build` | frontend + backend | Bring your own Postgres (`ICEGUARD_DB_URL` in `.env`) | | 2 — + database | `docker compose --profile db up -d --build` | + Postgres | Self-contained app + its required DB | | 3 — + test sandbox | `docker compose --profile db --profile sandbox up -d --build` | + MinIO + REST catalog | Try everything locally | > **No local build?** Use the **published images** from GHCR instead — same levels, just point at > the images file (no `--build`): > ```bash > docker compose -f docker-compose.images.yml --profile db --profile sandbox up -d > ``` > Defaults to `:latest`; pin a release with `TAG=0.1.0 docker compose -f docker-compose.images.yml …`. For the quickest tour, use **Level 3** and open **http://localhost:8090**. #### Add your first catalog (sandbox) In the UI, go to **Catalogs → Add Catalog** and register the bundled Iceberg REST catalog: | Field | Value | |---|---| | Catalog type | **Iceberg REST** | | URI | `http://rest-catalog:8181` *(Docker service name — not `localhost`; the backend reaches it over the Compose network)* | | Warehouse | `s3://warehouse/rest/` | | Authentication | **None** | | S3 / Storage access | **Static keys** | | • Endpoint | `http://minio:9000` | | • Access key / Secret key | `minioadmin` / `minioadmin` | | • Region | `us-east-1` | | • Path-style access | enabled ✅ | The S3 settings let the backend read/write data files on the sandbox MinIO. Once added, create a namespace and a table to see browsing, schema, storage and maintenance in action. Tear down with `docker compose --profile db --profile sandbox down` (add `-v` to wipe data) — use `-f docker-compose.images.yml` in the same command if you started from the images file. | Service | URL | Notes | |---------|-----|-------| | Frontend (UI) | http://localhost:8090 | nginx; proxies `/api` to the backend | | Backend API | http://localhost:8080 | Swagger at `/q/swagger-ui` | | MinIO console | http://localhost:9001 | `minioadmin` / `minioadmin` (sandbox profile) | | Iceberg REST catalog | http://localhost:8181 | sandbox profile; register it as `http://rest-catalog:8181` | ### Develop (hot reload) Run the dependencies in Docker and the app from source: ```bash # deps only (Postgres :5433, REST catalog :8181, Nessie :19120, MinIO) — not the backend container docker compose -f docker-compose.dev.yml up -d postgres rest-catalog nessie minio minio-init cd backend && mvn quarkus:dev -Dquarkus.profile=docker # :8080 (Swagger: /q/swagger-ui) cd frontend && npm install && npm run dev # :5173 (proxies /api to :8080) ``` ### Advanced: full multi-catalog sandbox For every catalog type at once (REST **+ Nessie + Polaris**) and real Spark compaction: ```bash docker compose -f docker-compose.dev.yml up -d ./scripts/seed-catalog.sh # after the backend is up ``` The **REST Catalog** and **Nessie** demo catalogs work out of the box on MinIO. **Polaris writes require real AWS S3** — copy `.env.example` to `.env`. ### Kubernetes (Helm) A chart lives in [`charts/iceguard`](charts/iceguard). Like Level 1 above it deploys the two IceGuard services only — **frontend + backend** — against a **PostgreSQL you provide** (images default to the published `ghcr.io/adelfardi/iceguard-{frontend,backend}:0.3.0`): ```bash helm install iceguard ./charts/iceguard -n iceguard --create-namespace \ --set database.host=pg.internal \ --set database.existingSecret=iceguard-db-credentials # key: password kubectl port-forward -n iceguard svc/iceguard-frontend 8090:80 # then http://localhost:8090 ``` The schema must be empty on first install — Flyway owns it and migrates at startup. A managed database usually wants the full URL instead: `--set database.jdbcUrl='jdbc:postgresql://…?sslmode=require'`. Prefer plain `kubectl`? Render the manifests and apply them — Helm is only needed to produce them: ```bash helm template iceguard ./charts/iceguard -n iceguard --skip-tests \ --set database.host=pg.internal --set database.existingSecret=iceguard-db-credentials \ > iceguard.yaml kubectl create namespace iceguard && kubectl apply -n iceguard -f iceguard.yaml ``` Ingress, network policies, OIDC, replicas/PDBs and the full values reference are documented in [`charts/iceguard/README.md`](charts/iceguard/README.md). ### Using IceGuard for real Deploy the **frontend** and **backend**, point the backend at a **PostgreSQL** you provide (`QUARKUS_DATASOURCE_JDBC_URL` / username / password), then add your own catalog(s) from the UI (**Catalogs → Add Catalog**): REST / Nessie / Polaris URI, warehouse, auth (None / Bearer / OAuth2) and, if needed, S3 credentials. No MinIO/Spark/sandbox containers required. ## Architecture ``` IceGuard (the product) required dep your existing systems ┌───────────────────────────────┐ │ React + Vite (UI, :5173) │ │ │ │ │ Quarkus REST API (:8080) ──────┼──> PostgreSQL ─────────┐ └───────────────────────────────┘ (IceGuard state) │ │ │ └──────────────────> Iceberg catalogs (REST / Nessie / Polaris) └──> S3 / object store (data & metadata) ``` IceGuard's own state (registered catalogs, pipelines, schedules, alerts, execution history) lives in the PostgreSQL you provide; table data & metadata live in **your** object store via **your** Iceberg catalogs. ## Project layout ``` backend/ Quarkus REST API (com.iceguard.*) frontend/ React + TypeScript SPA charts/iceguard/ Helm chart (Kubernetes deployment) scripts/ seed + helper scripts docker-compose.yml app stack (frontend + backend; profiles: db / sandbox) docker-compose.dev.yml advanced multi-catalog sandbox (Nessie, Polaris, Spark) ``` ## Known limitations - The **Java** executor really runs compaction & orphan-removal, but only on **small, append-only** tables (hard size/file-count limits; refuses above them) — use the **Spark** executor for large or merge-on-read tables. - **Polaris + MinIO**: browsing works, but writes fail (Polaris ignores the S3-compatible endpoint) — use real AWS S3. - Catalog credentials are stored in PostgreSQL (not encrypted at rest) — treat the DB as sensitive. - Test coverage is early-stage. Contributions very welcome 🙂 ## Contributing Contributions, issues and ideas are welcome! Please read **[CONTRIBUTING.md](CONTRIBUTING.md)** and our **[Code of Conduct](CODE_OF_CONDUCT.md)**. Good first issues are labelled `good first issue`. How the pipelines and automation work (CI, releases, Dependabot, CodeQL, Trivy) is documented in **[docs/CI.md](docs/CI.md)**. ## License [Apache License 2.0](LICENSE).