How Is the Model Trained?

1

Massive Pretraining

Web Code Math

Docs Languages Agents

Long Context

32T+ tokens
Web, code, math, docs, languages
Agent tasks and long-context data

2

Train Specialists

Math, code, agent, instruction experts
Focused domain training
Deepen specialist performance

3

Unified Distillation

Distill expert capabilities
Merge knowledge into one model
Stronger integrated performance

First train subject experts, then let one generalist model learn from all of them.