# RTX 3090

The RTX 3090 runtime provides Bonsai-1.7B SFT AR and D4/D8 MTP generation. MTP drafting uses BF16 output projections; AR and verification use native T2. The transformer body uses packed T2-g128 weights.

## Environment

From the repository root, use Python 3.10, a CUDA development toolkit, and an isolated environment. The pinned runtime uses PyTorch 2.5.1+cu121, Transformers 4.49.0, and xFormers 0.0.28.post3.

```bash
python3.10 -m venv .venv
source .venv/bin/activate
```

On x86-64 Linux with glibc 2.28 or newer, install the pinned dependencies:

```bash
python -m pip install -r requirements/rtx3090.txt
```

On older glibc systems, the same xFormers version requires a source build. Install PyTorch first, then build xFormers with a CUDA-compatible GCC/G++ 9 or newer (GCC 11.4 was tested). Set the three toolchain paths below to your installation:

```bash
python -m pip install --extra-index-url https://download.pytorch.org/whl/cu121 \
  torch==2.5.1+cu121
python -m pip install setuptools==82.0.1 wheel==0.46.3 ninja==1.11.1.4 numpy==2.2.6
export CUDA_HOME=/path/to/cuda-12.4
export CC=/path/to/gcc CXX=/path/to/g++
export CUDAHOSTCXX="$CXX"
export PATH="$CUDA_HOME/bin:$PATH"
CUDA_VISIBLE_DEVICES='' TORCH_CUDA_ARCH_LIST=8.6 MAX_JOBS=2 NVCC_THREADS=2 \
  python -m pip install --no-deps --no-build-isolation --no-binary=xformers \
  xformers==0.0.28.post3
python -m pip install -r requirements/rtx3090.txt
```

After either dependency installation, install BitMTP and build its native kernel. Set `NVCC` to the development toolkit's compiler:

```bash
python -m pip install --no-deps -e .
python -m pip check
SM_ARCH=86 NVCC=/path/to/cuda-12.4/bin/nvcc \
  bash src/bonsai_sft/gpu/t2g128_kernels/compile.sh
```

The native build was checked with CUDA toolkit 12.4 and execution with NVIDIA driver 550.100. The PyTorch wheel's CUDA runtime and the separately installed development toolkit are different components.

Before running, check `nvidia-smi` and `torch.cuda.is_available()`.

## Generate

Download the D8 model and convert it using new output directories. See [model downloads](usage.md#download-weights) for access instructions.

```bash
python -m bonsai_sft.download_model \
  --model bonsai_1p7_d8 --output weights/BitMTP-Bonsai-1.7B-D8
python -m bonsai_sft.gpu.convert_bonsai_mtp_to_gpu \
  --src weights/BitMTP-Bonsai-1.7B-D8 \
  --out weights/BitMTP-Bonsai-1.7B-D8-packed --quant ternary
```

Then generate a response:

```bash
CUDA_VISIBLE_DEVICES=0 python -m bonsai_sft.gpu.run_generation \
  --ckpt weights/BitMTP-Bonsai-1.7B-D8-packed \
  --hf-ckpt weights/BitMTP-Bonsai-1.7B-D8 \
  --block-size 8 --mode hybrid --threshold 0.9 --min-chain 6 \
  --draft-precision bf16 --verify-precision native \
  --chat --prompt 'Write a short greeting.' --max-new-tokens 128
```

Use `--mode spec` without `--min-chain` for speculative-only decoding. See [decoding modes](usage.md#decode).

For D4 Hybrid, download `bonsai_1p7_d4`, convert it with the same command, and use its paths with `--block-size 4 --min-chain 3`. For SFT AR, download `bonsai_1p7_sft`, convert it, and use its paths with `--block-size 8 --mode ar --draft-precision native`, omitting `--min-chain`. SFT's block-size field is export metadata; AR still generates one token per step.

## Evaluate

Use the matching profile, model ID, and model/tokenizer paths in the [evaluation commands](evaluation.md):

| Checkpoint and route | `--profile` | `--model-id` |
| --- | --- | --- |
| SFT AR | `bonsai_1p7b_rtx3090_sft_ar` | `bonsai_1p7b_sft` |
| D4 Hybrid | `bonsai_1p7b_rtx3090_d4_hybrid` | `bonsai_1p7b_mtp_d4` |
| D8 Hybrid | `bonsai_1p7b_rtx3090_d8_hybrid` | `bonsai_1p7b_mtp_d8` |

RTX 3090 profiles use lm-eval 0.4.11. D8 also provides built-in confidence-only, speculative-only, and MTP-checkpoint AR routes. The SFT baseline uses separate SFT weights.

See [tests](testing.md) for software regression and runtime checks.