{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca217cf5",
   "metadata": {},
   "source": [
    "# 03 — Concept Ontology\n",
    "\n",
    "The `Concept` side-collection is the lightweight ontology layer that the\n",
    "ingest pass uses to tag every artifact. Each Concept carries:\n",
    "\n",
    "- a canonical `name` (e.g. `Hausdorff Dimension`)\n",
    "- `aliases` for synonym matching\n",
    "- a one-paragraph `definition`\n",
    "- `domain` tags (e.g. `dimension-theory`, `classics`)\n",
    "- an optional `hausdorff_dim_range` for classical fractals\n",
    "\n",
    "Every `FractalArtifact` has a `concepts` cross-reference to whichever\n",
    "concepts it mentions. This notebook walks the ontology and shows how to\n",
    "follow those cross-refs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f577bd83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fractal_rag.client import client_session\n",
    "from fractal_rag.config import get_settings\n",
    "from weaviate.classes.query import Filter\n",
    "\n",
    "settings = get_settings(refresh=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc6230f",
   "metadata": {},
   "source": [
    "## List concepts by domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe1a6b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "with client_session() as client:\n",
    "    coll = client.collections.get(settings.concept_collection)\n",
    "\n",
    "    # All concepts in the 'dimension-theory' domain.\n",
    "    response = coll.query.fetch_objects(\n",
    "        filters=Filter.by_property('domain').contains_any(['dimension-theory']),\n",
    "        limit=50,\n",
    "    )\n",
    "\n",
    "print(f'Found {len(response.objects)} dimension-theory concepts:')\n",
    "for o in response.objects:\n",
    "    p = o.properties\n",
    "    rng = p.get('hausdorff_dim_range') or []\n",
    "    rng_s = f\"  [D_H={rng[0]}]\" if len(rng) == 1 else (f\"  [D_H in {rng}]\" if rng else \"\")\n",
    "    print(f'  - {p[\"name\"]}{rng_s}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7aadb17",
   "metadata": {},
   "source": [
    "## Semantic search across definitions\n",
    "\n",
    "If you don't know the canonical name, `search_concepts` on `definition_vec` works as a free-text lookup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f97df48",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUERY = 'measure of long-range dependence in a time series'\n",
    "\n",
    "with client_session() as client:\n",
    "    coll = client.collections.get(settings.concept_collection)\n",
    "    response = coll.query.hybrid(\n",
    "        query=QUERY,\n",
    "        limit=5,\n",
    "        target_vector='definition_vec',\n",
    "    )\n",
    "\n",
    "for o in response.objects:\n",
    "    p = o.properties\n",
    "    print(f'- {p[\"name\"]}: {p[\"definition\"][:120]}...')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e9825fc",
   "metadata": {},
   "source": [
    "## Find all artifacts referencing a concept\n",
    "\n",
    "The cross-reference is `FractalArtifact.concepts -> Concept[]`. Filter using `Filter.by_ref('concepts').by_id().equal(<concept_uuid>)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce86e2c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "CONCEPT_NAME = 'Mandelbrot Set'\n",
    "\n",
    "with client_session() as client:\n",
    "    concepts = client.collections.get(settings.concept_collection)\n",
    "    hits = concepts.query.fetch_objects(\n",
    "        filters=Filter.by_property('name').equal(CONCEPT_NAME),\n",
    "        limit=1,\n",
    "    )\n",
    "    if not hits.objects:\n",
    "        raise SystemExit(f'concept {CONCEPT_NAME!r} not found')\n",
    "    cuid = str(hits.objects[0].uuid)\n",
    "    print(f'Concept UUID: {cuid}')\n",
    "\n",
    "    art = client.collections.get(settings.artifact_collection)\n",
    "    refs = art.query.fetch_objects(\n",
    "        filters=Filter.by_ref('concepts').by_id().equal(cuid),\n",
    "        limit=10,\n",
    "    )\n",
    "\n",
    "for o in refs.objects:\n",
    "    p = o.properties\n",
    "    print(f\"  [{p.get('source_type'):>10}] {p.get('title')[:70]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6558fab",
   "metadata": {},
   "source": [
    "## Sanity-check the tagging quality\n",
    "\n",
    "A quick eyeball check: list the 10 most-tagged concepts (i.e. those referenced by the most artifacts). Concepts that no artifact ever cites are candidates for removal from `seed.yaml`; concepts that only ever appear in 1-2 artifacts may be too narrow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1c01e81",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "with client_session() as client:\n",
    "    concepts = client.collections.get(settings.concept_collection)\n",
    "    all_concepts = concepts.query.fetch_objects(limit=200).objects\n",
    "\n",
    "    art = client.collections.get(settings.artifact_collection)\n",
    "    counts = Counter()\n",
    "    for cobj in all_concepts:\n",
    "        n = art.aggregate.over_all(\n",
    "            total_count=True,\n",
    "            filters=Filter.by_ref('concepts').by_id().equal(str(cobj.uuid)),\n",
    "        ).total_count\n",
    "        counts[cobj.properties['name']] = n\n",
    "\n",
    "print('Top 10 most-tagged concepts:')\n",
    "for name, n in counts.most_common(10):\n",
    "    print(f'  {n:>6}  {name}')\n",
    "print()\n",
    "print('Untagged concepts:')\n",
    "for name, n in counts.items():\n",
    "    if n == 0:\n",
    "        print(f'  - {name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc489a94",
   "metadata": {},
   "source": [
    "## Where to go from here\n",
    "\n",
    "- The `concepts/seed.yaml` file is the source of truth for this ontology.\n",
    "  Add new entries (with aliases) and rerun `fractal-rag ingest --source all`\n",
    "  to retag every artifact.\n",
    "- The `fractal-explorer` subagent uses these tools (`list_concepts`,\n",
    "  `get_concept`, `artifacts_by_concept`, `search_concepts`) under the hood\n",
    "  for any concept-focused question — see [Using the agent](../using-the-agent.md).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}