# RAGFlow Instructions Use this file as the local operating guide for the current codebase. Prefer the code and the current AGENTS.md over any older convention or remembered project shape. ## Core Stance - Treat legacy code as liability, not as a compatibility target. - Prefer deletion over shims, deprecated branches, wrapper APIs, and dual-track migration notes. - If old and new implementations coexist, converge to one path unless an external contract forces compatibility. - Remove dead tests, commented-out code, stale docs, and "move later" notes instead of preserving them. - Reduce public surface area when a helper can be made private or internal. - Keep refactors centered on the owning abstraction, not on adjacent compatibility layers. ## Current stack - Backend: Go implements the API, admin, ingestor, and syncer modes in `cmd/ragflow_server.go`; the CLI runs from `cmd/ragflow-cli.go`. The Docker entrypoint starts the Go services according to its command-line flags, and the frontend development server uses fixed Go backend proxy routes. The Python API, admin, and workers are still present pending removal. - Frontend: React + TypeScript + Vite in `web/`. When working under `web/`, read and follow `web/AGENTS.md` for frontend conventions. - The Go module contains the server, ingestion, parsing, agent runtime, CLI, and supporting services. - Runtime services commonly include MySQL/PostgreSQL, Kvrocks, NATS JetStream, MinIO, ClickHouse, and Elasticsearch/Infinity/OpenSearch depending on configuration. ## Go-only target - Remove all Python code and Python-specific dependencies, scripts, containers, CI jobs, documentation, SDKs, and tests from the repository. Do not add new Python code while this removal is in progress. Replace Python-based preparation of Go native libraries, DeepDoc models, and the BPE table before removing the existing download scripts. - Move any still-required behavior and test coverage into the Go backend or the frontend before deleting the old implementation. Do not preserve a Python server, compatibility layer, fallback, or dual-backend path. - Before deleting `rag/`, move the Go DeepDoc `.ort` models and `ocr.res` out of `internal/rag/res/deepdoc/`, then update model discovery, dependency preparation, Docker packaging, and tests together. - Remove the frontend's Go/Python backend variants and update `web/CLAUDE.md` when the frontend no longer needs them. Until then, treat its dual-backend instructions as migration-only guidance; new frontend work should target the Go API. - Update Docker's default entrypoint, dependency image, and CI workflows when removing their remaining Python paths. Keep the frontend development proxy routes aligned with the Go API and Admin ports in `web/vite.config.ts`. - Keep this guide aligned with the checked-in tree. Once the Python directories are deleted, remove their migration inventory below. ## Code Layout to Expect - `cmd/ragflow_server.go`: Go entrypoint for the API, admin, ingestor, and syncer modes (`--api`, `--admin`, `--ingestor`, `--syncer`). - `cmd/ragflow-cli.go`: Go CLI entrypoint. - `api/`, `rag/`, `deepdoc/`, and `agent/`: legacy Python implementation pending deletion. Do not add features or extend dependencies there. - `internal/`: main Go application code. Important subtrees: - `internal/agent/`: Go agent runtime, canvas execution, components, tool bindings, workflow helpers. - `internal/admin/`: Go admin routes, handlers, and services. - `internal/binding/cpp/`: C++ tokenizer binding built by `build.sh` for the Go server. - `internal/cli/`: CLI parsing, HTTP transport, command execution, response formatting. - `internal/dao/`: Go data-access layer and persistence-facing helpers. - `internal/deepdoc/`: Go DeepDOC integrations, especially native-backed PDF/DOCX parsing. - `internal/engine/`: search/index backends such as Elasticsearch and Infinity. - `internal/entity/`: shared Go entities and model definitions. - `internal/handler/`: HTTP handlers and route-facing request logic. - `internal/ingestion/`: Go ingestion pipeline, canvas adapter, components, wiring, service orchestration. - `internal/ingestion/component/`: stage implementations such as file/parser/chunker/tokenizer/extractor. - `internal/ingestion/pipeline/`: DSL translation, canvas-driven execution, checkpoints, resume/run logic. - `internal/parser/`: parser and chunk libraries used by ingestion and other Go paths. - `internal/parser/parser/`: typed parse-result parsers for markdown/html/pdf/docx/xlsx/text and related families. - `internal/parser/chunk/`: chunk operator library and DSL/typed execution helpers. - `internal/service/`: higher-level business services used by handlers and server flows. - `internal/storage/`: storage backends and in-memory test doubles. - `internal/router/`: HTTP route registration. - `internal/server/`: server bootstrap/config wiring. - `internal/syncer/`: Go file synchronization service and connectors. - `internal/rag/`: Go retrieval and RAG logic. - `internal/harness/`: Go runtime harness and supporting execution code. - `web/`: frontend application. - `docker/`: local and production compose files. - `sdk/python/`: legacy Python client SDK pending deletion. - `tools/scripts/download_dev_deps.py` and `deps/download_deps.py`: Python downloaders for the Go build's dependencies. `download_dev_deps.py` pulls the native static libraries (pdfium, pdf_oxide, office_oxide, onnxruntime) that `build.sh` links; `download_deps.py` pulls the runtime resources (cl100k BPE table, DeepDoc models, embedding tokenizer assets) and installs the stagehand server into the SDK cache. Versions and URLs are defined inline in each downloader (`tools/scripts/download_dev_deps.py` for native static libraries, `deps/download_deps.py` for runtime resources); the tokenizer assets are inlined as `TOKENIZER_ASSETS` in each downloader. Replace these with non-Python preparation before removing Python dependencies. - `test/`: contains Python tests, including tests that import the legacy Python implementation directly. Migrate needed coverage to Go tests or frontend tests, then remove the Python tests; keep only tests for supported code. ## Go-Specific Rules - Treat `internal/ingestion`, `internal/parser`, and `internal/deepdoc` as actively refactored code. Prefer collapsing duplicate paths over preserving transitional wrappers. - Do not add or preserve deprecated Go APIs just to ease migration inside the repo. - Remove commented-out Go code instead of leaving recovery notes in place. - Keep package comments and doc comments aligned with the current runtime path, not with migration history. ## GORM query traps - **Never pass a bare string ID as the second argument to `Take`/`First`** (e.g. `Take(&x, id)`). Under GORM v1.25.7 a non-numeric string is treated as a raw SQL condition and concatenated into the `WHERE` clause: `Take(&x, "88b17c7a...")` emits `WHERE 88b17c7a... LIMIT 1`, which MySQL rejects with `Error 1054 (42S22): Unknown column '...' in 'where clause'`. Only strings that parse via `strconv.Atoi` are treated as primary-key values, so this bug is invisible for integer IDs and catastrophic for UUID primary keys. Always bind explicitly: `Take(&x, "id = ?", id)` or `Where("id = ?", id).Take(&x)`. - When you change a query form across many files, verify the generated SQL with a `Session(&gorm.Session{DryRun: true})` probe before relying on it. A passing unit test with no rows and a broken `WHERE` clause look identical unless the SQL is inspected. ## Shared database schema (Go + Python) The Go services and the Python API write to the same MySQL/PostgreSQL schema, and both own parts of it. Neither is authoritative over the whole. Each ORM derives index names differently: - Go (`internal/entity/`): GORM derives `idx__` for indexes and `uni_
_` for unique constraints. Tags in this repo name unique indexes explicitly, e.g. `uniqueIndex:idx_commit_file`. - Python (`api/db/db_models.py`): Peewee derives `__` (truncated with an md5 suffix past 64 chars, see `playhouse.migrate.make_index_name`). There is no `idx_` prefix. Both sides reconcile by **column set, not by name**: Go's `hasUniqueIndex` (`internal/dao/migration.go`) accepts an existing index only when its full column set equals the requested one, and Python's `ensure_model_indexes` (`api/db/db_models.py`) keys `DB.get_indexes()` on the column tuple. Consequences: - Divergent names covering the same columns are expected, not a defect. Whichever runtime reaches the database first decides the name. Do not "fix" this by renaming inside a migration; it rewrites live databases for no functional gain. - Add a shared unique index to **both** sides (Go tag + Peewee `Meta.indexes` or `unique=True`), so each runtime recreates it when absent. Do not add a second index over a column set an existing one already covers. - In Go, declare single-column uniqueness with a named `uniqueIndex:`, not a bare `unique` tag. A bare `unique` becomes a table-level constraint in DDL, which SQLite materializes as an implicit index that cannot be dropped by name, and which GORM's `MigrateColumnUnique` then tries to drop under its own `uni_
_` name — MySQL answers 1091 every startup. `namedIndexMigrator` (`internal/dao/database.go`) suppresses exactly that phantom drop; see `TestNamedIndexMigratorSkipsPhantomUniqueDrop`. - A Python model existing does not mean Python owns that table's constraints. `pipeline_operation_log` has no `run_count` column and no unique index in `db_models.py`; both are added only by `internal/dao/migration.go`. ## Go Test Tiers Go tests are classified by build tag. Tag a test file with `//go:build ` placed before the `package` clause. Run them through `build.sh` so the required CGO configuration and native libraries are available. | Tier | Build tag | Runs by default? | Needs | |---|---|---|---| | Unit | (none) | Yes (`bash build.sh --test`) | Native CGO static libs; no external services — uses in-memory SQLite, miniredis, or `httptest` stubs. | | Integration | `integration` | No (`-tags integration`) | A real service: MySQL/MinIO/Elasticsearch/Infinity/LLM. Single component, reasonably fast. | | E2E | `e2e` | No (`-tags e2e`) | Full cross-component pipeline (ingest → index → retrieve) against real services; heavy/slow. | | Manual | `manual` | No (`-tags manual`) | Very slow/expensive (deepdoc render/parity/snapshot/bench). **Local opt-in ONLY — never run in CI.** | | Native (orthogonal) | `cgo` / `!cgo` | `cgo` auto-satisfies under CGO_ENABLED=1 | Native static libs (`office_oxide`/`pdfium`/`pdf_oxide`). Combine with tiers, e.g. `//go:build cgo && integration`. | Run tiers locally via `build.sh`: ```bash bash build.sh --test # unit tier (no tags) bash build.sh --test-integration ./... # integration tier bash build.sh --test-e2e # e2e tier bash build.sh --test-manual # manual tier (very slow) bash build.sh --test-all # integration + e2e (never includes manual) ``` Rules: - New tests that touch a real external service MUST carry `integration`/`e2e`/`manual` — do not rely on `t.Skip` + env vars to soft-isolate them in the default unit run. Keep an env guard as a harmless secondary safety net if desired. - `manual` is never wired into CI or any automated pipeline. - `unit` (no tag) must stay free of external-service dependencies so `bash build.sh --test` runs without MySQL/MinIO/ES/Infinity/LLM. The native CGO static libraries (`office_oxide`/`pdfium`/`pdf_oxide`) are still required at build time and are wired by `build.sh --test`; that is expected, not an external service. ## Working Rules - When reviewing documentation or code, inspect the full affected path and report all verifiable findings in one review; do not return after only a few findings and expose further issues in later rounds. - When handling review comments, independently verify each substantive claim against the current code or tests before accepting, rejecting, or acting on it. - Before editing, inspect the nearest code path that actually owns the behavior. - For server changes, work in the Go implementation. If a Python behavior is still needed, port it to Go and test the Go path before removing the Python code. - Keep changes small and local unless the task is explicitly a broader refactor. - Prefer one implementation path instead of preserving old and new versions side by side. - Preserve behavior with focused tests when the behavior is still valid; do not keep tests that protect obsolete behavior. - If a surface is only there for compatibility, remove it unless the user asks to keep it. - Do not add new compatibility wording in comments or docs. - When a maintainer takes over a community PR, a new commit generated by rewriting history (e.g. `merge`, `rebase -i`) must preserve the original author and add the maintainer as co-author (via a `Co-authored-by:` trailer) instead of overwriting the author with the maintainer alone. ## Commands ### Go backend ```bash bash build.sh --all # build native code and Go server bash build.sh --go # build Go server with native library already built bash build.sh --test ./internal/handler/... # targeted unit tests bash build.sh --test # all unit tests bash build.sh --test-integration ./... # integration tests; requires real services bash build.sh --test-e2e # full-pipeline tests; requires real services bash build.sh --test-manual # local opt-in only bash build.sh --test-all # integration + e2e; excludes manual ``` Native libraries, Go DeepDoc models, tokenizer assets, and `deps/cl100k_base.tiktoken` come from `tools/scripts/download_dev_deps.py` (native static libs) and `deps/download_deps.py` (models, cl100k, tokenizer assets); run them with `uv run`. Build the Go resource image with `cd deps && docker build -f Dockerfile -t infiniflow/ragflow_deps:latest .` after downloading the resources. ### Frontend ```bash cd web npm install npm run dev npm run build npm run lint npm run test npm run type-check ``` ## Validation Preference - Run the narrowest relevant test, lint, or build command after a change. - For backend changes, first run package-scoped `bash build.sh --test ./path/to/package/...`. Run the integration or e2e tier when the changed behavior needs real services or a full pipeline. - For frontend changes, prefer the touched-package lint, type-check, or test command. - Do not default to raw `go test`, `go build`, or IDE Run/Debug for Go in this repo. They often miss the required CGO flags and native static libraries (`office_oxide`, `pdfium-static`, `pdf_oxide`) that `build.sh` wires correctly. - After a Go build, execute the resulting binary's help command for each changed server mode; a successful link alone does not prove the binary starts. On this host, the original `build.sh` selected default LLD 18 and produced binaries that crashed before Go `main`; linking with LLD 20 started successfully. Verify the linker actually selected on Linux before trusting a new binary. - If Go native builds fail, inspect `build.sh` and `internal/development.md` before changing code. Common environment issues are missing downloaded native dependencies or an incompatible `ld.lld` on Linux. ## Default review checklist - Remove instead of retaining `deprecated`, `legacy`, or compatibility-only code. - Collapse duplicate implementations to one path. - Drop stale comments and documentation that describe a superseded design. - Keep exported APIs only when the current code actually needs them.