English · 简体中文
## Import once. Retrieve with confidence.
`dsh-kb-manager` is a local knowledge-base lifecycle plugin for [DeepSeek Harness (dsh)](https://github.com/deepseek-ai/deepseek-harness): multi-format import → smart chunking → pure-TypeScript vector indexes → hybrid search (vector + BM25 → RRF → optional rerank) → citation tracing—plus snapshots/rollback, directory-watch sync, and portable `.kbpack` archives.
Ask in natural language. The plugin exposes **16 Agent tools** and an optional **Web panel** (KB list, import wizard, search bench, source locator)—no separate RAG server required.
## Why dsh-kb-manager?
| Capability | What it changes |
| --- | --- |
| **Multi-format import** | PDF (page-aware; scan pages flagged), DOCX, Markdown, HTML (Readability), CSV, JSON, TXT, plus URL fetch. |
| **Smart chunking** | `fixed` / `recursive` (default) / `semantic`; heading breadcrumbs, table-as-chunk, page/paragraph provenance. |
| **Pure-TS indexes** | Flat (exact) + HNSW (approx)—no native addons. |
| **Hybrid retrieval** | Dual-path recall → RRF (`k=60`) → optional rerank; `debug: true` returns per-stage scores. |
| **Offline fallback chain** | Local / OpenAI-compatible embedding → built-in HashEmbedder; rerank failure falls back to RRF order. |
| **Snapshots & rollback** | Manifest + index backup; embedding-model mismatch refuses unsafe restore. |
| **`.kbpack` portability** | tar.gz of source + chunks + index + metadata; `create_new` / `merge` import. |
| **Directory sync** | chokidar + SHA-256 change detection + catch-up scan; deleted watch dirs pause without cascading deletes. |
| **Read-only mode** | `read_only: true` makes write tools return `read_only_mode` immediately. |
| **Agent-friendly tools** | Few-shot examples embedded in every tool description. |
## Architecture
```mermaid
flowchart LR
A[Documents / URLs] --> B[Parse]
B --> C[Chunk]
C --> D[Embed]
D --> E[Vector index
Flat / HNSW]
C --> F[BM25]
E --> G[Hybrid search]
F --> G
G --> H[RRF]
H --> I[Optional rerank]
I --> J[Citations + debug scores]
E --> K[Snapshots / .kbpack]
C --> K
```
## Install
> [!NOTE]
> Requires an existing [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) installation.
### From GitHub
```sh
dsh plugin --profile web add github:xiaoshi7915/dsh-kb-manager
```
On git install, the `prepare` script runs `tsdown` for a self-contained build (no monorepo context needed).
### Build from source
```sh
git clone https://github.com/xiaoshi7915/dsh-kb-manager.git
cd dsh-kb-manager
npm install
npm run build
dsh plugin --profile web add .
```
Validate the composed profile, restart DSH, and refresh the Web UI:
```sh
dsh --profile web --dump-config
dsh web
```
Then try:
> Create a knowledge base named “project-docs”, import this PDF, and search for how authentication works. Show me the source chunk.
## How it works
1. Create a KB (`create_kb`) with an embedding model (default offline `hash-embed-v1`, or any OpenAI-compatible name when `embedding_api_base` is set).
2. Import local files or URLs (`import_document`) → parse → chunk → embed → index.
3. Search with `search_kb` / `multi_kb_search`: vector + BM25 → RRF → optional rerank; use `debug: true` to inspect stage scores.
4. Trace hits with `get_chunk` (surrounding context + provenance metadata).
5. Optionally snapshot before risky edits, restore later, or ship a `.kbpack` to another machine.
6. Optional `auto_sync_dir` watches a folder and incrementally updates the target KB.
Data lives under `storage_path` (default `~/.dsh/kb-manager/`). The Web client reads the same service surface for overview, import, search, and source location.
## Agent tools
| Tool | Description | Key params |
| --- | --- | --- |
| `create_kb` | Create a knowledge base | `name*`, `description*`, `embedding_model?`, `tags?` |
| `list_kbs` | List all knowledge bases | — |
| `get_kb` | KB details (model / docs / chunks / storage) | `kb_id*` |
| `delete_kb` | Delete an entire knowledge base | `kb_id*` |
| `import_document` | Import a local file or URL | `kb_id*`, `source*`, `metadata?` |
| `list_documents` | List documents (optional status filter) | `kb_id*`, `status_filter?` |
| `delete_document` | Soft-delete chunks; mark index dirty | `kb_id*`, `doc_id*` |
| `search_kb` | Hybrid search; optional debug scores | `kb_id*`, `query*`, `top_k?`, `filters?`, `rerank?`, `debug?` |
| `multi_kb_search` | Cross-KB search with `source_kb` | `kb_ids*`, `query*`, `top_k?` |
| `get_chunk` | Chunk text + neighbors for tracing | `kb_id*`, `chunk_id*` |
| `rebuild_index` | Full rebuild via shadow index swap | `kb_id*` |
| `create_snapshot` | Create a version snapshot | `kb_id*`, `note?` |
| `restore_snapshot` | Roll back to a snapshot | `kb_id*`, `snapshot_id*` |
| `export_kb` | Export `.kbpack` or JSON | `kb_id*`, `format?`, `output_path*` |
| `import_kb` | Import a pack (`create_new` / `merge`) | `file_path*`, `merge_strategy?` |
| `get_kb_stats` | Doc / chunk / index / avg-length stats | `kb_id*` |
Write tools (`create_kb`, `delete_kb`, `import_document`, `delete_document`, `rebuild_index`, `create_snapshot`, `restore_snapshot`, `import_kb`) return `{ success: false, error: 'read_only_mode' }` when read-only. Domain errors are returned as `{ success: false, error: , message }` (not thrown).
## When the Agent uses it
1. “Put this PDF / doc / page into the knowledge base” → `import_document`
2. “Search the KB for X” → `search_kb`
3. “Find Y across all knowledge bases” → `multi_kb_search`
4. “Where did this citation come from?” → `get_chunk`
5. “Retrieval feels wrong—show stage scores” → `search_kb({ debug: true })`
6. “Back up before a big change” → `create_snapshot`
7. “Roll back to before the change” → `restore_snapshot`
8. “Pack this KB for another machine” → `export_kb` + `import_kb`
9. “Will local file edits sync?” → directory watch incremental sync
10. “How big is this KB?” → `list_kbs` / `get_kb_stats`
## Configuration
Defaults work offline. Override in a trusted profile (`id: kb-manager`):
| Field | Default | Notes |
| --- | --- | --- |
| `storage_path` | `~/.dsh/kb-manager/` | Storage root; `~` expands to the home directory |
| `default_embedding_model` | `hash-embed-v1` | Offline default; or an OpenAI-compatible model name |
| `embedding_api_base` | `''` | OpenAI-compatible endpoint; empty → HashEmbedder |
| `embedding_api_key` | `''` | Embedding API key |
| `chunk_size` | `512` | Chunk size (characters) |
| `chunk_overlap` | `50` | Overlap length |
| `chunk_strategy` | `recursive` | `fixed` / `recursive` / `semantic` |
| `top_k` | `5` | Default hit count |
| `enable_rerank` | `true` | Enable post-RRF rerank |
| `rerank_endpoint` | `''` | Remote rerank; empty → built-in rule reranker |
| `index_type` | `hnsw` | `hnsw` / `flat` |
| `auto_sync_dir` | `''` | Watch path; empty disables |
| `auto_sync_kb_id` | `''` | Target KB for sync |
| `auto_sync_interval` | `300` | Catch-up interval (seconds) |
| `max_file_size_mb` | `100` | Per-file size cap (MB) |
| `read_only` | `false` | Disable write tools |
Example:
```yaml
- id: kb-manager
config:
storage_path: ~/.dsh/kb-manager/
default_embedding_model: text-embedding-3-small
embedding_api_base: https://api.openai.com/v1
chunk_strategy: recursive
index_type: hnsw
enable_rerank: true
read_only: false
```
## Boundaries
- Reads/writes under `storage_path`; `import_document` / `import_kb` also read user-specified paths or URLs; `export_kb` writes to `output_path`.
- Unsupported formats return `unsupported_format`; oversized files are rejected by `max_file_size_mb`.
- Fully local by default; network only when `embedding_api_base` / `rerank_endpoint` or URL import is configured.
- `export_kb` does not mutate KB data and remains available in read-only mode.
- Pure-TS indexes: no native vector DB process to operate.
## Differentiating vs typical RAG stacks
| Capability | This plugin | Common RAGFlow / Dify / kotaemon / pdfkb-mcp setups |
| --- | --- | --- |
| KB snapshots & rollback | ✅ | Usually missing |
| `.kbpack` (source + chunks + index + metadata) | ✅ | Usually missing |
| Directory watch + catch-up scan | ✅ chokidar + SHA-256 | Partial / missing |
| Per-stage retrieval debug scores | ✅ vector / BM25 / RRF / rerank | Often opaque |
| Full offline embed fallback | ✅ HashEmbedder | Often needs external services |
| Read-only mode | ✅ | Rare |
## Layout
```
dsh-kb-manager/
├── package.json cordis.patch.yml tsconfig.json tsdown.config.ts vitest.config.ts
├── awesome-entry.yml README.md README_ZH.md
├── assets/readme/ # hero.png (paste generated banner here)
├── src/
│ ├── index.ts # Plugin entry (name / inject / Config / apply)
│ ├── config.ts # Schemastery config schema
│ ├── core/ parse/ chunk/
│ ├── embed/ index/ search/
│ ├── kb/ # KBService, snapshots, sync, kbpack
│ ├── tools/ # 16 Agent tools
│ └── client/ # Web panel
└── tests/
```
## Development
```sh
npm install # install deps (prepare builds automatically)
npm run build # tsdown dual entry → lib/index.js + lib/client.js
npm test # vitest
npm run typecheck # tsc --noEmit
```
## License
[MIT](./LICENSE) © 2026 xiaoshi7915