============================================
  MIDNIGHT 12L — Brotli + Mixed-Int, Full 600s, 8xH100
  Seed: 444
  Loader mode: coprime | no trigram | no n-gram eval
  SKIP_GPTQ=1 | COMPRESSOR=brotli | Parallel Muon | XSA-all-11
  Quant policy: attn=5 mlp=6 aux=6 embed=8 other=8
  NUM_LAYERS=12 | Bigram 2048 | RoPE 16
============================================
logs/e2e0c541-7eab-484c-b5bc-8896cec03796.txt
val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=./data/tokenizers/fineweb_1024_bpe.model
train_loader:dataset:fineweb10B_sp1024 train_shards:80
val_loader:shards pattern=./data/datasets/fineweb10B_sp1024/fineweb_val_*.bin tokens:62021632
model_params:29355620
mtp_num_heads:0 mtp_loss_weight:0.2 mtp_params:0
XSA:last_11 active_layers:[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
world_size:8 grad_accum_steps:1
sdp_backends:cudnn=False flash=True mem_efficient=False math=False
attention_mode:gqa num_heads:8 num_kv_heads:4
tie_embeddings:True embed_lr:0.035 head_lr:0.0 matrix_lr:0.025 scalar_lr:0.025
train_batch_tokens:786432 train_seq_len:2048 iterations:20000 warmup_steps:20 max_wallclock_seconds:600.000
compile:enabled=1 mode:default fullgraph=1
mlp_kernel_mode:eager
scale_init:attn=1.0000 mlp=1.0000 resid_mix=(1.0000,0.0000) ln_scale=1
quant_policy:attn=5 mlp=6 aux=6 embed=8 other=8 artifact=final_model.mixed.ptz
seed:444
loader:coprime shards:80 blocks:3906240 seq_len:2048 shards_per_batch:1 cache:4 batch_stride:47 hold_steps:64
warmup_step:1/20
warmup_step:20/20
loader_reset:loader:coprime shards:80 blocks:3906240 seq_len:2048 shards_per_batch:1 cache:4 batch_stride:47 hold_steps:64
step:0/20000 val_loss:6.9326 val_bpb:4.1058 train_time:0ms step_avg:0.01ms
step:1/20000 train_loss:6.9321 train_time:378ms step_avg:377.86ms
step:10/20000 train_loss:6.4548 train_time:1152ms step_avg:115.19ms
step:500/20000 train_loss:2.3466 train_time:48410ms step_avg:96.82ms
step:1000/20000 train_loss:2.2044 train_time:97225ms step_avg:97.22ms
step:1500/20000 train_loss:2.1790 train_time:145989ms step_avg:97.33ms
step:2000/20000 train_loss:2.0702 train_time:194714ms step_avg:97.36ms
step:2500/20000 train_loss:2.0641 train_time:243403ms step_avg:97.36ms
step:3000/20000 train_loss:2.1182 train_time:291827ms step_avg:97.28ms
step:3500/20000 train_loss:1.9980 train_time:340439ms step_avg:97.27ms
step:4000/20000 train_loss:1.9573 train_time:389056ms step_avg:97.26ms
step:4000/20000 val_loss:2.0044 val_bpb:1.1871 train_time:389112ms step_avg:97.28ms
step:4500/20000 train_loss:2.0028 train_time:437629ms step_avg:97.25ms
step:5000/20000 train_loss:1.9857 train_time:486232ms step_avg:97.25ms
swa:start step:5500
step:5500/20000 train_loss:1.9478 train_time:534589ms step_avg:97.20ms
late_qat:enabled step:5642 scale:0.1499
step:6000/20000 train_loss:1.9468 train_time:584095ms step_avg:97.35ms
step:6160/20000 val_loss:1.9079 val_bpb:1.1300 train_time:600077ms step_avg:97.42ms
stopping_early: wallclock_cap train_time:600077ms step:6160/20000
peak memory allocated: 24663 MiB reserved: 24814 MiB
gptq:SKIPPED (SKIP_GPTQ=1)
ema:applying EMA weights
DIAGNOSTIC post_ema val_loss:1.9064 val_bpb:1.1291 eval_time:2239ms
Serialized model: 115607162 bytes
Code size: 124698 bytes
mixed_quantize: 0 GPTQ tensors, 4 naive tensors int5:2 int6:2 int8:3
Serialized model mixed+brotli: 15506905 bytes
Total submission size mixed+brotli: 15631603 bytes
final_quant_roundtrip val_loss:1.9296 val_bpb:1.1428 eval_time:6605ms
final_quant_roundtrip_exact val_loss:1.92961624 val_bpb:1.14282742
final_sliding_window val_loss:1.8669 val_bpb:1.1057 stride:64 eval_time:107985ms
final_sliding_window_exact val_loss:1.86688857 val_bpb:1.10567949
============================================
  DONE
============================================