# OKF4net.Catalog

A catalog of local [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
knowledge bundles for .NET: a hot-reloadable `catalog.json` manifest naming
one or more bundles as *sources*, and a resolver that searches every enabled
source and returns results grouped by source.

**OKF** is the Open Knowledge Format — a directory tree of markdown files with
YAML frontmatter (see [OKF4net](https://www.nuget.org/packages/OKF4net) for
the format core). `catalog.json` is **not** an OKF concept and is not part of
the OKF spec: it is an **OKF4net manifest**, a small piece of catalog
configuration that points at OKF bundles from the outside. Nothing about the
manifest format is portable to another OKF implementation.

This package references only `OKF4net` (the format core) — no third-party
runtime dependencies.

## What it does

- **Loads and validates `catalog.json`.** `FileKnowledgeCatalog` parses the
  manifest, validates every enabled source's `path` against the catalog root
  (rejecting absolute paths, paths that escape the root, missing directories,
  and reparse points in between), and publishes the result as an immutable,
  versioned `KnowledgeCatalogSnapshot`. An invalid *initial* manifest throws
  `CatalogException` (fail-fast at startup).
- **Hot-reloads, best-effort.** A debounced `FileSystemWatcher` on the
  manifest file triggers automatic reloads. This is best-effort only — OS/
  filesystem/container layers can miss or duplicate watcher events. Call
  `IKnowledgeCatalog.ReloadAsync()` directly whenever you need a reliable,
  synchronous guarantee that an edit has been picked up. A reload is
  errors-as-data: a malformed or invalid replacement manifest leaves the
  current snapshot untouched and records the reject reasons in
  `LastReloadDiagnostics` instead of throwing.
- **Searches every enabled source, with a selectable ranking strategy.**
  `IKnowledgeResolver` fans a query out across every enabled
  `KnowledgeCatalogSource` (using the same `ConceptSearch` scorer the
  `OKF4net.Agents` tools use). How the results come back is your choice:
  - `GroupedBySource` (default) — each source's own ranked results
    concatenated, source by source, in priority order. No fusion or dedup.
  - `Merged` — one cross-source ranking by descending score, with priority
    as a tie-break only.
  - `PriorityWeighted` — one cross-source ranking by source priority first,
    score only within a priority tier.

  Both merged strategies also collapse two manifest entries that resolve to
  the same directory (searching that bundle once, not twice) and accept an
  optional fairness quota that interleaves sources so one prolific source
  cannot crowd out the rest of a budget-truncated result.

## Minimal `catalog.json`

```json
{
  "version": 1,
  "sources": [
    { "id": "products", "path": "./bundles/products", "priority": 10, "enabled": true },
    { "id": "support", "path": "./bundles/support", "priority": 0, "enabled": true }
  ]
}
```

- `id` — unique within the manifest, a single OKF concept-id segment.
- `path` — relative to the manifest's own directory; must resolve inside the
  catalog root.
- `priority` (optional, default `0`) — higher priority sources are searched
  first and their passages appear first in a grouped result.
- `enabled` (optional, default `true`).
- `role` (optional, default `"knowledge"`) — `"knowledge"` (read-only, searched
  by the resolver) or `"memory"` (writable, scoped by tier — see below); any
  other string is rejected.
- `tier` — required when `role` is `"memory"`, one of `"session"`, `"user"`,
  or `"tenant"`; not allowed otherwise.

## Quick start

```csharp
using OKF4net.Catalog;

var options = new KnowledgeCatalogOptions
{
    CatalogFilePath = "./config/catalog.json",
    CatalogRoot = "./config",
};

using var catalog = new FileKnowledgeCatalog(options);
IKnowledgeResolver resolver = new GroupedKnowledgeResolver(catalog);

KnowledgeContext result = await resolver.SearchAsync(new KnowledgeQuery("refund policy"));

foreach (var passage in result.Passages)
{
    Console.WriteLine($"[{passage.SourceId}] {passage.ConceptId} ({passage.Score}): {passage.Excerpt}");
}

foreach (var diagnostic in result.Diagnostics)
{
    Console.WriteLine($"[{diagnostic.Code}] {diagnostic.Message}");
}
```

`CatalogRoot` is a strict containment boundary. When constructing
`KnowledgeCatalogOptions` directly, its shared path components with
`CatalogFilePath` must have identical spelling, including case. A
case-variant root is rejected as `OutsideRoot` even on a case-insensitive
filesystem. `AddKnowledge(o => o.AddCatalogFile(...))` derives both values
from one resolved path and therefore satisfies this requirement automatically.

`KnowledgeContext` is deliberately never a bare string: `Passages` (grouped by
source, in source-priority then per-source descending-score order) and
`Diagnostics` (e.g. `NoEnabledSources`, `SourceUnavailable`, `NoMatches`) let a
caller distinguish "no results" from "a source failed" from "no source is
enabled" without parsing text.

## Scoped memory (`role: "memory"`)

A `role: "memory"` source is written by capture (e.g.
`OkfContextProviderOptions.CaptureTier` in `OKF4net.Agents`), not searched by
`IKnowledgeResolver` — it feeds an `IMemoryStore` instead. Configure one
source per tier you need:

```json
{
  "version": 1,
  "sources": [
    { "id": "kb", "path": "./bundles/products", "role": "knowledge" },
    { "id": "mem-user", "path": "./memory/user", "role": "memory", "tier": "user" },
    { "id": "mem-tenant", "path": "./memory/tenant", "role": "memory", "tier": "tenant" },
    { "id": "mem-session", "path": "./memory/session", "role": "memory", "tier": "session" }
  ]
}
```

```csharp
using OKF4net.Catalog;
using OKF4net.Catalog.Hosting;

services.AddKnowledge(o => o.AddCatalogFile("./config/catalog.json"));
services.AddMemory();

// Elsewhere:
IMemoryStore memory = provider.GetRequiredService<IMemoryStore>();
await memory.DeleteScopeAsync(scope, MemoryTier.Session); // e.g. when a conversation ends
```

**There is no code-level distinction between ephemeral and persistent
tiers** — every `role:"memory"` source's `path`, like every other source's,
must be relative to the manifest directory and resolve inside the catalog
root (`CatalogPathResolver.TryResolve` rejects absolute paths, paths that
escape the root, and reparse points anywhere along the way), so a source
cannot point directly at an OS temp directory or a symlink into one.
"Ephemeral" therefore isn't a per-source path trick; it's one of two real
choices:

- Run the **whole catalog root** on ephemeral storage (e.g. a container's
  tmpfs mount or ephemeral volume) — every source under it, including a
  session-tier source like `mem-session` above at its own ordinary relative
  `path`, is then ephemeral by construction. The catalog root itself is
  exempt from the reparse-point walk, so this is the one place a mount
  point is fine.
- Treat any tier's subtree as revocable at will via
  `IMemoryStore.DeleteScopeAsync` — nothing purges automatically, but
  nothing stops a host from calling it the moment a conversation ends.

**V1 limitation:** `OKF4net.Catalog.Hosting`'s `AddMemory()` resolves the set
of `role:memory` sources once, at first `IMemoryStore` resolution from the
container, and does not pick up a source added/removed/edited afterward
(including via `IKnowledgeCatalog.ReloadAsync()`) — see `AddMemory`'s own XML
doc for the full explanation. Per-scope path resolution (the tenant/user/session
segments) stays fully live on every call; only the fixed set of configured
tiers is frozen.

## Choosing a ranking strategy

Set a default for the whole host, and override it per query where needed:

```csharp
using OKF4net.Catalog;
using OKF4net.Catalog.Hosting;

services.AddKnowledge(o =>
{
    o.AddCatalogFile("./config/catalog.json");
    o.DefaultResolverStrategy = KnowledgeResolverStrategy.Merged;
    o.DefaultFairnessQuota = 2;   // optional; null (the default) disables it
});

// Per-query override, through the same injected IKnowledgeResolver:
var context = await resolver.SearchAsync(new KnowledgeQuery("refund policy")
{
    ResolverStrategy = KnowledgeResolverStrategy.PriorityWeighted,
});
```

`DefaultResolverStrategy` defaults to `GroupedBySource`, so upgrading changes
no existing deployment's result ordering until you opt in.

A fairness quota caps how many *consecutive* passages one source may
contribute before another source's next-best passage is pulled ahead. It
reorders and never drops, so it matters only to consumers that truncate
early — an agent context provider spending a token budget top-down, for
instance, which would otherwise let one source's whole run consume the
budget.

## Choosing source visibility

Restrict which sources a caller may see, per host default or per query:

```csharp
services.AddKnowledge(o =>
{
    o.AddCatalogFile("./config/catalog.json");
    o.DefaultSourceVisibilityPolicy = (scope, source) =>
        // Fails CLOSED, not open: a caller with no TenantId (KnowledgeAccessScope.Local,
        // the default when a host resolves no scope at all) sees NOTHING here -- an empty
        // "" ?? fallback would make StartsWith("") true for every source, silently exposing
        // every tenant's catalog to an unauthenticated/unscoped caller. Matches a source
        // named EXACTLY as the tenant, or "tenantId-" followed by anything -- the "-" is an
        // explicit segment separator, not just a prefix, so a short tenant id (e.g. "acme")
        // can never accidentally match an unrelated one (e.g. "acmeland-kb"). Do not drop
        // the "-" to make a bare-named source match "more easily": that reopens exactly
        // that collision.
        scope.TenantId is { Length: > 0 } tenantId
        && (source.Id == tenantId || source.Id.StartsWith(tenantId + "-", StringComparison.Ordinal));
});

// Per-query override, through the same injected IKnowledgeResolver:
var context = await resolver.SearchAsync(new KnowledgeQuery("refund policy")
{
    Scope = new KnowledgeAccessScope(tenantId: "acme"),
    PermittedSourceIds = new HashSet<string> { "acme-support", "acme-billing" },
});
```

Two mutually exclusive mechanisms — setting both on the same query throws:

- `PermittedSourceIds` — a host-precomputed set of source IDs, the
  recommended default. A host does whatever lookup it needs (tenant,
  application, or both) and hands the resulting set to the query; `OKF4net.Catalog`
  never needs to know how it was computed. Always wins over any host-level
  default policy for that one call.
- `SourceVisibilityPolicy` — a function evaluated per source, for rules a
  flat ID list can't express conveniently. Configurable once per host
  (`DefaultSourceVisibilityPolicy`) and overridable per query, mirroring
  `DefaultResolverStrategy`.

Neither has any effect on a query that sets neither field and a host that
configures no default: every enabled source stays visible to every caller,
exactly as before this feature existed.

`KnowledgeAccessScope` has no value-equality override (reference equality
only) -- a `SourceVisibilityPolicy` function should compare `TenantId`/
`UserId`/`SessionId` individually, not compare two `KnowledgeAccessScope`
instances with `==`/`Equals`.

## V1 limits

- Local filesystem bundles only — no remote/HTTP sources, no external
  connectors.
- One shared catalog per `FileKnowledgeCatalog` instance.
- No semantic/fuzzy deduplication — two concepts with similar content in
  genuinely different bundles are both returned. Only under the two merged
  strategies are two manifest entries resolving to the *same directory*
  collapsed; the default `GroupedBySource` strategy does not dedup.
- No tenant-aware authorization of any kind.

See [the project README](https://github.com/jchable/okf4net) for the full
documentation, and NOTICE/LICENSE.Apache-2.0 for the attribution chain of the
underlying OKF implementation.

Licensed LGPL-3.0-or-later.