# SN38: ChronoLLM

**Bittensor Subnet 38**: Competitive training of chronologically consistent Large Language Models.

[![Discord](https://img.shields.io/discord/799672011265015819?label=Discord&logo=discord)](https://discord.com/channels/799672011265015819/1485634202895519844)

## The problem

Standard LLMs are trained on data from all time periods. When used for financial backtesting or historical analysis, they suffer from **lookahead bias**: the model "knows" things that weren't known at the time being analyzed. A model asked to predict 2015 market trends has already seen the 2020 crash.

## The solution

Miners train language models with strict temporal boundaries. Each model must be trained **only on data available up to its cutoff year**. A model for 2018 must know nothing about 2019 or beyond.

## Try it

Pick a model from the [leaderboard](https://leaderboard.chronollm.com/) and chat with it. Each model only knows events up to its vintage year.

These are completion models. Type the beginning of a sentence and the model will continue it.  
Question answering is not yet available but will be supported from round 10.

```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repo and install dependencies
git clone https://github.com/chronollm/sn38.git
cd sn38
uv sync

# Chat with a model
uv run python -m sn38.neurons.chat owner/repo
```

Example with a 2017 vintage model:
```
> The president of the United States in 2017 was ↵
Donald Trump, who took office on January 20, 2017.

> Bitcoin reached its all-time high in ↵
December 2017, when it briefly surpassed $19,000.
```

## How the subnet works

```
Miners                          Validators (TEE)                    Backend (private)
  |                                   |                                    |
  |  Train models per year            |                                    |
  |  Upload to HuggingFace            |                                    |
  |  Submit on-chain                  |                                    |
  |  ------------------------------>  |                                    |
  |                                   |  Download models                   |
  |                                   |  Fetch private evaluation data     |
  |                                   |  <---------------------------------|
  |                                   |                                    |
  |                                   |  Stage 0: Anti-copy                |
  |                                   |  Stage 1: Leak detection           |
  |                                   |  Stage 2: Qualification            |
  |                                   |  Stage 3: Quality duels            |
  |                                   |  Stage 4: Ranking                  |
  |                                   |  Stage 5: Reward distribution      |
  |                                   |                                    |
  |                                   |  Set weights on-chain              |
  |                                   |                                    |
```

## Evaluation pipeline

### Stage 0: Anti-copy

- **Weight checksum**: identical weights to a previously submitted model are rejected. First submitter has priority.

### Stage 1: Leak detection

- **Known score (pre-cutoff)**: the model must recognize at least 70% of items from its training period. Does the model know what it should know?
- **Unknown score (post-cutoff)**: the model must NOT recognize more than 10% of items after its cutoff year. This is the lookahead bias test.
- `leak_score = unknown_score - known_score`

### Stage 2: Dedup and qualification

- **SVD pairwise dedup**: all miners that passed Stage 1 are compared pairwise using SVD spectral distance. Near-duplicate submissions are rejected, earliest submitter wins. Invariant to rotation, permutation and uniform scaling attacks.
- **Qualification**: models with `leak_score > min_eval_score` qualify for quality duels. Currently set at -3.0.

### Stage 3: Quality duels

Top 30 miners by leak score are selected, then matched in round-robin 1v1 duels. Each model generates text completions for prompts that are **uniquely generated by the LLM judge at evaluation time**. These prompts are created fresh for each round: no one, has access to them in advance. This eliminates any possibility of overfitting on the evaluation dataset. Output quality is judged by GPT-5.4.

### Stage 4: Ranking

`final_score = quality_win_rate` (the leak test is a pass/fail gate: among models that pass, quality determines the ranking).

### Stage 5: Reward distribution

Top 10 receive emissions on an exponential decay. #1 gets roughly 52%, #2 roughly 23%, #3 roughly 12%, and so on.

> **Note**: All thresholds, weights, and constants mentioned above may change over time. Check the current values at [api.chronollm.com/config](https://api.chronollm.com/config).

## Rounds

Rounds last **one week**, starting every Monday at 12:00 UTC.

All previous submissions are cleared each round. Miners must resubmit every week: only submissions committed on-chain after the submission window opens are evaluated. This prevents outdated models from occupying slots and speeds up evaluation.

```
Monday 12:00 UTC                    Monday 12:00 UTC                    Monday 12:00 UTC
       |-------- Round N ------------------|-------- Round N+1 ------------------|
       |                                   |                                     |
       |  Miners: submit for Round N       |  Miners: submit for Round N+1       |
       |  Validators: evaluate Round N-1   |  Validators: evaluate Round N       |
```

## Why TEE

The evaluation dataset must stay private. If miners knew exactly what items are tested, they could overfit their models to pass without truly maintaining chronological consistency.

Validators run inside [Phala Cloud](https://phala.network/) using Intel TDX hardware. The TEE guarantees that:

1. The evaluation data is only accessible inside the secure enclave
2. The validator code is auditable via remote attestation
3. Neither the validator operator nor any miner can read the leak evaluation dataset

Anyone can verify the attestation to confirm the validator is running the correct, unmodified code.

## Model requirements

- Any architecture compatible with HuggingFace `AutoModelForCausalLM` (Llama, Qwen, Gemma, Mistral, etc.)
- Custom architectures can be submitted for review via PR (see [miner docs](docs/miner.md#submit-a-custom-architecture))
- Safetensors format (no pickle)
- Maximum 2.2B parameters
- `trust_remote_code=False`: no code from model repos is executed

## Self-test

Miners can verify their models pass the leak detection before submitting using the TEE-based self-test. It runs the exact same evaluation code against a separate dataset. Rate limited to 12 calls/day per hotkey. CPU only, no GPU needed. See [miner docs](docs/miner.md#self-test-your-model-leak-detection).

## Getting started

| | |
|--|--|
| **Mine** | [docs/miner.md](docs/miner.md) |
| **Validate** | [docs/validator.md](docs/validator.md) |
| ChronoGPT paper | [arXiv:2510.11677](https://arxiv.org/abs/2510.11677) |
| ChronoGPT models | [huggingface.co/manelalab](https://huggingface.co/manelalab) |
| Bittensor docs | [docs.learnbittensor.org](https://docs.learnbittensor.org/) |

## License

This repository is licensed under the MIT License. Full details are available in the [LICENSE](LICENSE) file.