Controlled CivBench on Vox Populi 5.2.7, Civilization V

RatingsA controlled version of CivBench (Chen et al., 2026). Each LLM strategists rotate through 3 standard maps (8 players), each game features 2 of the same LLM strategists + 6 Vox Populi AI players. Experimented with 2 conditions: having LLMs make a decision every turn (base); having them make a decision every 5 turns, or when an important event comes up (per-5).

Compare strategists' relative skill across games, using ratings that summarize how consistently they outperform their opponents.

Pairwise skill ratingsEstimates each player type's relative skill from pairwise comparisons of model-adjusted strength within each game (Bradley-Terry Elo ratings). Module: ratings.bradley_terry group_by: player_type; strength_table: strength; adjust_stage: strength; strength_estimator: attention; estimator_model: attention_mlp; estimator_fit: pretrained; estimator_predict: in_sample; adjust_block: auto/start_cell; adjust_turn_progress_min: 0.2; adjust_weight: turn_progress; adjust_enforce_winner: True; adjust_civ_adjust: ols_logit; adjust_baseline_experiment: vanilla-standard-fixed; adjust_post_cell_normalize: none; adjust_cell_gain_bend: 0.5

GLM-5.3-Simple-Per-5 leads 37 identities at 1634 Elo; rating spread: 322 points.

bt_main: ratings

Open interactive figure (HTML)

Downloads and supporting files (1)

Pairwise strategy ratingsEstimates relative skill for each player type and strategy combination from pairwise comparisons of model-adjusted strength within each game (Bradley-Terry Elo ratings). Module: ratings.bradley_terry group_by: player_type, strategy; strength_table: strength; adjust_stage: strength; strength_estimator: attention; estimator_model: attention_mlp; estimator_fit: pretrained; estimator_predict: in_sample; adjust_block: auto/start_cell; adjust_turn_progress_min: 0.2; adjust_weight: turn_progress; adjust_enforce_winner: True; adjust_civ_adjust: ols_logit; adjust_baseline_experiment: vanilla-standard-fixed; adjust_post_cell_normalize: none; adjust_cell_gain_bend: 0.5

GPT-6.1-Sol-Simple-Domination leads 128 player and condition combinations at 1768 Elo across 4 groups.

ratings.bradley_terry strategy ratingsElo for each player type overall (General) and within each strategy, with Vanilla fixed at 1500. Colors run from red (below Vanilla) through yellow (level with Vanilla) to blue (above). Stars mark a z-test against Vanilla in the same column: * p<0.05, ** p<0.01, *** p<0.001. Hover a cell for the standard error, appearances, and the strategy's share of the row's games.
ratings.bradley_terry strategy ratings
Strategist | ConditionGeneral (all strategies)DominationCultureDiplomaticScience
Null1311***1366***1136***1394*
Vanilla15001500152515081466
GPT-OSS-120B-Simple | Every-turn1423***14451306***1449*1610**
GPT-OSS-120B-Simple | Per-51382***14771341***1290***1427
Opus-5.5-Simple | Per-51594***15211642***1533*
Sonnet-5.5-Simple | Per-51496145815361444
GLM-5.1-Simple | Every-turn148315091417***1092***1722***
GLM-5.2-Simple | Every-turn1536*1420*15461618***
GLM-5.2-Simple | Per-51495153515061140***1555**
GLM-5.3-Simple | Per-51634***1597***1656***15651661***
GLM-5.3-Flash-Simple | Every-turn14711445148914741477
GLM-5.3-Flash-Simple | Per-51435***14471386***1667***
MiniMax-M2.7-Simple | Every-turn1473148015071418
MiniMax-M2.7-Simple | Per-51446**14511434***1531
MiniMax-M3-Simple | Per-51447**1437*1467*
Kimi-K2.7-Simple | Every-turn1574***1598**1755***14731553***
Kimi-K2.7-Simple | Per-51517147715471391**1583***
Kimi-K2.6-Simple | Every-turn15041417*15931627**1477
DeepSeek-V4-Flash-Simple | Every-turn1431***1398***1414***963***1553**
DeepSeek-V4-Flash-Simple | Per-514791380***15191626*1502
DeepSeek-V4.1-Flash-Simple | Per-5149614531462*1431*1637***
Qwen-3.5-Simple | Every-turn1459*1435*1431***15061573*
Qwen-3.5-Simple | Per-51423***14671415***1249***1531
Qwen-3.6-27B-Simple | Every-turn1387***1395***1416***1356*
Qwen-3.6-27B-Simple | Per-51344***1423**1322***1318***
Qwen-3.8-27B-Simple | Every-turn151714431620**15531425
Qwen-3.8-27B-Simple | Per-51455**146514701247***1548**
Qwen-3.8-Flash-Next-Simple | Every-turn1449**15131439***1405*
Qwen-3.8-Flash-Next-Simple | Per-5149515581453***1623***
Gemma-4-Simple | Every-turn1377***1368***1342***1471
Gemma-4-Simple | Per-51458*15261353***1553**
GPT-6-Luna-Simple | Every-turn14841385**1506*
GPT-6-Luna-Simple | Per-51510155315411516*
GPT-6.1-Sol-Simple | Every-turn1545**1768***15151373***1594***
GPT-6.1-Sol-Simple | Per-51599***1608**15071665***
Nemotron-3-Super-Simple | Every-turn1439***1405**1404***14921546*
Nemotron-3-Super-Simple | Per-51434***1368***15271429
  • 963
  • 1500 (Vanilla)
  • 2037

full CSV

Downloads and supporting files (1)

Adjusted-strength matchupsCompares every pair of player types using model-adjusted strength, including mean differences and win rates. Module: ratings.matchups mode: both; strength_table: strength; adjust_stage: strength; strength_estimator: attention; estimator_model: attention_mlp; estimator_fit: pretrained; estimator_predict: in_sample; adjust_block: auto/start_cell; adjust_turn_progress_min: 0.2; adjust_weight: turn_progress; adjust_enforce_winner: True; adjust_civ_adjust: ols_logit; adjust_baseline_experiment: vanilla-standard-fixed; adjust_post_cell_normalize: none; adjust_cell_gain_bend: 0.5; display: vs_reference

GLM-5.3-Simple-Per-5 has the highest observed strength win rate: 66.7% against VPAI, across 288 player comparisons.

Win rate

matchup_strength: strength_winrate

Open interactive figure (HTML)

Mean difference

matchup_strength: strength_mean

Open interactive figure (HTML)

Downloads and supporting files (7)

Victory matchupsShows wins per player appearance in shared games and final-score margins. The equal-chance victory rate is 1 divided by the number of players in each game (12.5% for eight players). Module: ratings.outcome_matchups table: panel; include_score_ratio: True; victory_rate_unit: wins per player; expected_win_rate: Equal chance: 1 / full game player count, averaged over player appearances.; p_value_win_rate: Binomial test against equal chance, available for fixed game sizes with one appearance of the player type per game.; display: vs_reference; score_ratio_margin: row minus column

Opus-5.5-Simple-Per-5 has the highest per-player matchup victory rate: 27.1% (13/48; expected 12.5% in 8-player games.)

Victory rate

matchup_winrates: win_rate

Open interactive figure (HTML)

Score margin

matchup_winrates: score_ratio_margin

Open interactive figure (HTML)

Downloads and supporting files (5)