K12-KGraph construction pipeline

K12-KGraph

A Curriculum-Aligned Knowledge Graph for Benchmarking & Training Educational LLMs

Dataset on Hugging Face Models on Hugging Face Project Page NeurIPS 2026 Evaluations & Datasets Track β€” Accepted License

Nodes Edges Bench Train Subjects Stage

Built from the official People's Education Press (PEP) Chinese K–12 textbooks, K12-KGraph aligns the same scientific concept across definition, formula, experiment, exercise, structural location, and relational neighborhood.

🌐 Explore the interactive project page β†’

--- ## πŸ† News - **September 2026** β€” K12-KGraph was **accepted to the NeurIPS 2026 Evaluations & Datasets Track**! πŸŽ‰ - **July 2026** β€” K12-KGraph, developed by the **PKU-DCAI team**, received **Second Prize** in the **AGI4S Frontier Corpus Track** of the **MinerU Frontier Corpus and Data Intelligence Challenge**. [Award certificate](docs/awards/k12-kgraph-mineru-2026-second-prize.pdf). --- ## 🌟 Why K12-KGraph? Modern LLMs can answer "what is the Pythagorean theorem?" but struggle with **curriculum cognition** β€” the structured understanding of: - 🧭 *What are the prerequisites of a concept?* - πŸ”¬ *Which experiment verifies it?* - πŸ“ *Which exercises test it?* - πŸ“š *Where does it live in the textbook?* - πŸ•Έ *What are its taxonomic and relational neighbors?* K12-KGraph is the first open, multi-subject, official-textbook-grounded knowledge graph that explicitly aligns all five dimensions around each STEM concept, yielding two ready-to-use AI assets: | | [K12-Bench](https://huggingface.co/datasets/lhpku20010120/K12-KGraph) | [K12-Train](https://huggingface.co/datasets/lhpku20010120/K12-KGraph) | |---|---|---| | **Size** | 23,640 multi-select questions | 2,267 instruction–response pairs | | **Purpose** | Evaluate structural curriculum cognition | Teach it via KG-guided SFT | | **Task families / sources** | Ground Β· Prereq Β· Neighbor Β· Evidence Β· Locate | Node-grounded + Edge-grounded + Deterministic templates | | **Headline result** | Gemini-3-Flash reaches only **57.1%** EM | Beats 8 mainstream SFT corpora on GaokaoBench & EduEval under a strict 2,300-sample budget | --- ## πŸ“Š Leaderboard Snapshot (K12-Bench, zero-shot) Instance-level macro F1 and exact match, in %. | Model | Overall EM | Overall F1 | |---|---|---| | *Random guess baseline* | 6.7 | 36.4 | | Meta-LLaMA-3-8B-Instruct | 7.2 | 52.6 | | GLM-4.7-Flash | 31.7 | 63.9 | | GPT-4o | 31.1 | 65.9 | | Qwen3-32B | 42.6 | 69.5 | | Gemma-4-31B-IT | 46.4 | 69.5 | | GPT-5.2 | 42.8 | 68.0 | | Gemini-2.5-Flash | 48.3 | 66.7 | | **Gemini-3-Flash** | **57.1** | **73.0** | > Even the strongest proprietary model leaves > 40% of items unsolved on **Prereq** and **Neighbor** β€” the tasks requiring directed, structural reasoning. See the [project page](https://haolpku.github.io/K12-KGraph-page/) for the full 5-task breakdown. --- ## πŸ—ΊοΈ What's in this Repository? ``` K12-KGraph/ β”œβ”€β”€ src/ β”‚ β”œβ”€β”€ kg/ # Knowledge-graph construction pipeline β”‚ β”œβ”€β”€ benchmark/ # K12-Bench generation from graph queries β”‚ β”œβ”€β”€ sft_qa/ # K12-Train synthesis (node & edge grounded) β”‚ └── utils/ # Shared config / LLM client / IO β”œβ”€β”€ eval/ # Multiple-choice evaluation runner (OpenAI / vLLM) β”œβ”€β”€ config/ # Default pipeline configuration β”œβ”€β”€ demo/ # Trimmed JSON/JSONL samples β”œβ”€β”€ books.yaml # Book registry β”œβ”€β”€ docs/img/ # README figures └── requirements.txt ``` Pipeline flow: ``` PDF textbooks ─► MinerU parsing ─► Section split ─► GPT-5.2 schema-constrained extraction ─► Hierarchical merge (book β†’ subject β†’ global) ─► DAG validation + expert review ─► K12-KGraph ─► K12-Bench (queries) + K12-Train (QA synthesis) ``` --- ## πŸš€ Quick Start ### 1. Install ```bash git clone https://github.com/haolpku/K12-KGraph.git cd K12-KGraph pip install -r requirements.txt ``` > If you will run the graph pipeline from PDFs, also install [**MinerU**](https://github.com/opendatalab/MinerU) and make `magic-pdf` callable from the shell (command name configurable via `config/default.yaml`). ### 2. Load the released dataset ```python from datasets import load_dataset kg = load_dataset("lhpku20010120/K12-KGraph", split="train") bench = load_dataset("lhpku20010120/K12-KGraph", name="bench", split="test") train = load_dataset("lhpku20010120/K12-KGraph", name="train", split="train") ``` ### 3. Build the graph from scratch ```bash cp config/.env.example config/.env # add your OPENAI_API_KEY etc. python src/kg/run_pipeline.py \ --config config/default.yaml \ --filter-prefix # e.g. math_7a_rjb ``` ### 4. Derive Bench and SFT data ```bash python src/benchmark/run_pipeline.py --help python src/sft_qa/run_pipeline.py --help ``` ### 5. Evaluate a model on K12-Bench ```bash cp eval/configs/.env.example eval/configs/.env chmod +x eval/run.sh ./eval/run.sh # eval/configs/models/.yaml ``` --- ## πŸ€— Released Checkpoints Three standalone checkpoints trained on **K12-Train** are available on Hugging Face. Each repository contains the tokenizer, chat template, configuration, and merged/full weights; no separate base-model download is required. | Checkpoint | Parameters | Fine-tuning | Weight size | License | |---|---:|---|---:|---| | [Llama3.1-8B-K12KGraph](https://huggingface.co/lhpku20010120/llama3.1-8b-k12kgraph) | 8B | Full SFT | 16.1 GB | Llama 3.1 | | [Qwen3.5-2B-K12KGraph](https://huggingface.co/lhpku20010120/qwen3.5-2b-k12kgraph) | 2B | LoRA SFT, merged | 4.4 GB | Apache-2.0 | | [Qwen3-4B-K12KGraph](https://huggingface.co/lhpku20010120/qwen3-4b-k12kgraph) | 4B | Full SFT | 8.0 GB | Apache-2.0 | Download a checkpoint with the Hugging Face CLI: ```bash pip install -U huggingface_hub hf download lhpku20010120/qwen3.5-2b-k12kgraph \ --local-dir checkpoints/qwen3.5-2b-k12kgraph ``` Or serve one directly from the Hub and evaluate it on K12-Bench: ```bash # Replace qwen3_5_2b with llama3_1_8b or qwen3_4b as needed. bash eval/vllm_scripts/qwen3_5_2b_k12kgraph.sh ./eval/run.sh qwen3_5_2b_k12kgraph_vllm ``` The launchers accept the standard `VLLM_HOST`, `VLLM_PORT`, `VLLM_TP_SIZE`, `VLLM_MAX_MODEL_LEN`, `VLLM_GPU_MEM`, and `VLLM_MODEL_ID` overrides. --- ## 🧱 Schema at a Glance **7 node types** β€” `Book` Β· `Chapter` Β· `Section` Β· `Concept` Β· `Skill` Β· `Experiment` Β· `Exercise` **9 edge types** β€” `is_a` Β· `prerequisites_for` Β· `relates_to` Β· `verifies` Β· `tests_concept` Β· `tests_skill` Β· `appears_in` Β· `leads_to` Β· `is_part_of` Every `Concept` carries `name`, `definition`, `importance`, and optional `formula`, `aliases`, `examples`. Every `Experiment` carries `instruments`, `is_student`, `process`, `phenomena`, `conclusion`. Full schema and attribute specification in [`docs/schema.md`](docs/schema.md) (coming soon) or on the [project page](https://haolpku.github.io/K12-KGraph-page/).
A concrete example: how the same prerequisites_for subgraph yields a K12-Bench item (A) and a K12-Train QA pair (B)
--- ## πŸ“š Dataset Composition
K12-Bench distribution across subjects, task families, and difficulty.
| Subject | Books | Concepts | Skills | Experiments | Exercises | |---|---:|---:|---:|---:|---:| | Mathematics | 23 | 1,475 | 428 | 0 | 471 | | Physics | 9 | 1,154 | 197 | 220 | 186 | | Chemistry | 7 | 2,302 | 451 | 309 | 270 | | Biology | 9 | 1,648 | 288 | 123 | 244 | | **Total** | **48** | **6,579** | **1,364** | **652** | **1,171** | --- ## πŸ§ͺ Quality Assurance - **Fleiss' ΞΊ = 0.84** overall, from 12 subject-qualified expert annotators (ΞΊ by relation: `is_a` 0.91, `prerequisites_for` 0.82, `relates_to` 0.69, `verifies` 0.88) - **Automatic DAG validation** on `is_a` and `prerequisites_for` subgraphs - **Per-edge `evidence` field** linking back to textbook source text for auditability - **98.4%** stratified K12-Bench items verified as "fully correct" in a 3-expert spot-check --- ## 🌈 Explore Interactively Want to browse nodes, sample bench items, or inspect the training data without cloning the repo? The companion project page offers a rich interactive view:

--- ## 🀝 Contribute Contributions are welcome! We particularly appreciate: - 🏫 Adding support for other textbook publishers (BNU, Jiangsu, etc.) - πŸ§ͺ New task families that extend beyond the current 5 - πŸ› Bug reports and quality issues on existing graph edges (please cite the specific edge ID) - 🌍 Translation of the schema/documentation into additional languages Open an issue or pull request β€” GitHub Issues are monitored within 48 hours. --- ## πŸ“– Citation If you find K12-KGraph useful in your research, please cite: ```bibtex @misc{k12kgraph2026, title = {K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs}, author = {Hao Liang and others}, year = {2026}, howpublished = {NeurIPS 2026 Evaluations and Datasets Track}, note = {Accepted}, url = {https://github.com/haolpku/K12-KGraph} } ``` --- ## πŸ“„ License - **Dataset** (graph, benchmark, training data): [CC BY-NC-SA 4.0](LICENSE) - **Code** (this repository): MIT ---
Made with care by the K12-KGraph team Β· Project Page Β· Dataset Β· δΈ­ζ–‡ README