CivBench · Civilization V with Vox Populi 5.2.7

Which AI makes the best long-term strategist?

Civilization V is a strategy game where you lead a nation from its first village to the space age. A single game takes hundreds of turns, and the winner is often decided by choices made long before the end. That makes it a good test of whether an AI can plan ahead.

Latest news

Leading right nowGLM-5.3, deciding every 5 turns, rated 1634 ▶ Watch a game

Who plays best?

Each bar shows how far an AI setup's rating sits above or below the built-in AI, the dashed line.

1300 1350 1400 1450 1500 1550 1600 1650 1 GLM-5.3 every 5 turns 1634 2 GPT-6.1-Sol every 5 turns 1599 3 Opus-5.5 every 5 turns 1594 4 Kimi-K2.7 every turn 1574 5 GPT-6.1-Sol every turn 1545 6 GLM-5.2 every turn 1536 7 Kimi-K2.7 every 5 turns 1517 8 Qwen-3.8-27B every turn 1517 9 GPT-6-Luna every 5 turns 1510 10 Kimi-K2.6 every turn 1504
Show the other 27
11 Built-in AI 1500 12 DeepSeek-V4.1-Flash every 5 turns 1496 13 Sonnet-5.5 every 5 turns 1496 14 GLM-5.2 every 5 turns 1495 15 Qwen-3.8-Flash-Next every 5 turns 1495 16 GPT-6-Luna every turn 1484 17 GLM-5.1 every turn 1483 18 DeepSeek-V4-Flash every 5 turns 1479 19 MiniMax-M2.7 every turn 1473 20 GLM-5.3-Flash every turn 1471 21 Qwen-3.5 every turn 1459 22 Gemma-4 every 5 turns 1458 23 Qwen-3.8-27B every 5 turns 1455 24 Qwen-3.8-Flash-Next every turn 1449 25 MiniMax-M3 every 5 turns 1447 26 MiniMax-M2.7 every 5 turns 1446 27 Nemotron-3-Super every turn 1439 28 GLM-5.3-Flash every 5 turns 1435 29 Nemotron-3-Super every 5 turns 1434 30 DeepSeek-V4-Flash every turn 1431 31 Qwen-3.5 every 5 turns 1423 32 GPT-OSS-120B every turn 1423 33 Qwen-3.6-27B every turn 1387 34 GPT-OSS-120B every 5 turns 1382 35 Gemma-4 every turn 1377 36 Qwen-3.6-27B every 5 turns 1344 37 Do-nothing player 1311

Click a player to see its games · Full results →

What is this built on?

Three pieces of software make these games possible. You do not need to play any of them to read this report.

Civilization V

A 2010 strategy game by Firaxis. You lead a nation through history one turn at a time: building cities, trading, making friends and enemies.

Steam store page ↗

Vox Populi

A large community mod that rebalances Civilization V and makes its computer players much smarter. All games here use version 5.2.7.

Vox Populi on GitHub ↗

Vox Deorum

Our open-source bridge that lets an AI agent read the game and steer a nation (through the execution of the built-in AI).

Vox Deorum on GitHub ↗

Watch any game in the replay viewer ↗ · Read the CivBench paper ↗ · Read the Vox Deorum paper ↗

Is the pricier model worth it?

More compute usually buys a bit more skill. Setups above the dashed line play better than their price suggests.

  • every turn
  • every 5 turns
25¢ 50¢ $1 $2 $5 $10 $20 1350 1400 1450 1500 1550 1600 1650 Built-in AI Typical rating for the price Cost per player per game, US dollars (each step multiplies the cost) Rating GLM-5.3 GPT-6.1-Sol Opus-5.5 Qwen-3.6-27B Opus-5.5 Sonnet-5.5 DeepSeek-V4-Flash DeepSeek-V4-Flash DeepSeek-V4.1-Flash GLM-5.1 GLM-5.2 GLM-5.2 GLM-5.3-Flash GLM-5.3-Flash GLM-5.3 GPT-6-Luna GPT-6-Luna GPT-6.1-Sol GPT-6.1-Sol GPT-OSS-120B GPT-OSS-120B Gemma-4 Gemma-4 Kimi-K2.6 Kimi-K2.7 Kimi-K2.7 MiniMax-M2.7 MiniMax-M2.7 MiniMax-M3 Nemotron-3-Super Nemotron-3-Super Qwen-3.5 Qwen-3.5 Qwen-3.6-27B Qwen-3.6-27B Qwen-3.8-27B Qwen-3.8-27B Qwen-3.8-Flash-Next Qwen-3.8-Flash-Next

Full results →

How do they like to win?

Each bar shows how much of the game an AI spent aiming for each victory type.

GLM-5.3 · every 5 turns 33% 27% 16% 24% GPT-6.1-Sol · every 5 turns 11% 28% 17% 44% Opus-5.5 · every 5 turns 18% 46% 28% Kimi-K2.7 · every turn 21% 22% 11% 45% GPT-6.1-Sol · every turn 12% 27% 17% 43% GLM-5.2 · every turn 27% 32% 12% 29% Kimi-K2.7 · every 5 turns 24% 22% 17% 37% Qwen-3.8-27B · every turn 15% 36% 27% 22% GPT-6-Luna · every 5 turns 11% 76% Kimi-K2.6 · every turn 21% 19% 13% 47% Built-in AI 28% 15% 21% 36%

Click a player to see its games · Full results →

Key findings

One sentence each. The ? button shows the technical version, and the link opens the full results.

Strongest player

Pairwise skill ratings: GLM-5.3-Simple-Per-5 leads 37 identities at 1634 Elo; rating spread: 322 points. Estimates each player type's relative skill from pairwise comparisons of model-adjusted strength within each game (Bradley-Terry Elo ratings). Module: ratings.bradley_terry group_by: player_type; strength_table: strength; adjust_stage: strength; strength_estimator: attention; estimator_model: attention_mlp; estimator_fit: pretrained; estimator_predict: in_sample; adjust_block: auto/start_cell; adjust_turn_progress_min: 0.2; adjust_weight: turn_progress; adjust_enforce_winner: True; adjust_civ_adjust: ols_logit; adjust_baseline_experiment: vanilla-standard-fixed; adjust_post_cell_normalize: none; adjust_cell_gain_bend: 0.5

GLM-5.3 (every 5 turns) is the strongest so far. It beats the built-in AI 68% of the time head-to-head.

See details →

Most wins

Victory matchups: Opus-5.5-Simple-Per-5 has the highest per-player matchup victory rate: 27.1% (13/48; expected 12.5% in 8-player games.) Shows wins per player appearance in shared games and final-score margins. The equal-chance victory rate is 1 divided by the number of players in each game (12.5% for eight players). Module: ratings.outcome_matchups table: panel; include_score_ratio: True; victory_rate_unit: wins per player; expected_win_rate: Equal chance: 1 / full game player count, averaged over player appearances.; p_value_win_rate: Binomial test against equal chance, available for fixed game sizes with one appearance of the player type per game.; display: vs_reference; score_ratio_margin: row minus column

Opus-5.5 (every 5 turns) won 27% of its games. With 8 players, an even chance would be 1 in 8.

See details →

How reliable is this?

Prediction quality: attention performs best on roc auc at 0.8543 using 891 games; scores range from 0.8060 to 0.8543. Measures how well each estimator identifies likely winners and matches observed outcomes (discrimination and calibration). Module: prediction.evaluate metrics: roc_auc, brier_score, log_loss, balanced_accuracy; n_models: 3

Shown a winner and a loser from the same game, our best predictor picks the winner 85% of the time, well before the game ends.

See details →

Best value

Usage, cost, and skill: Most cost-efficient: GLM-5.3-Simple-Per-5 (+96 Elo vs the fitted curve). Least cost-efficient: Qwen-3.6-27B-Simple-Per-5 (-118 Elo vs the fitted curve). Compares cost and token use per player per game with skill, and measures Elo above or below the fitted usage-skill curve. Module: performance.usage_efficiency currency: usd; log_x: True; ratings_stage: bt_main; dropped_baselines: 1; unpriced_identities: 0; unrated_identities: 0; cost_basis: per player per complete game; cached_input_estimated: True; efficiency_metric: elo - expected_elo; usage_skill_equation: expected_elo = intercept + slope * log10(average_usage); usage_skill_fits: {'cost': {'intercept': 1443.1786316605792, 'slope': 72.12423921082431, 'n': 35, 'r_squared': 0.42845390034510067}, 'input': {'intercept': 893.0219589354363, 'slope': 84.44167917373612, 'n': 35, 'r_squared': 0.04538167687312056}, 'output': {'intercept': 1418.955740250512, 'slope': 10.215192676980365, 'n': 35, 'r_squared': 0.006202953555220936}}; baseline_elo: 1500.0; baseline_name: Vanilla; null_baseline_elo: 1311.4423726797052

GLM-5.3 (every 5 turns) gets the most skill for its price. Qwen-3.6-27B (every 5 turns) gets the least.

See details →

Changed habits

Strategic settings: Against the completed-experiment average, the largest departure is Qwen-3.6-27B-Simple | Every-turn on waterconnection (-31). Shows how each strategist sets the in-game AI's flavors (0 to 100, 50 is balanced), against the average of completed experiments on the same map and seat and in absolute terms. Module: behavior.flavors n_absolute_players: 1686; baseline: completed-experiment average; n_relative_players: 1686; n_unmatched_controlled_players: 0; n_baseline_players: 1686; n_baseline_experiments: 35; flavors: Offense, Defense, CityDefense, Mobilization, MilitaryTraining, Recon, Ranged, Mobile, Nuke, UseNuke, Naval, NavalRecon, Air, Antiair, AirCarrier, Airlift, Expansion, Growth, TileImprovement, Infrastructure, Production, Gold, Science, Culture, Happiness, NavalGrowth, NavalTileImprovement, WaterConnection, GreatPeople, Wonder, Religion, Diplomacy, Espionage, Spaceship

Against the completed-experiment average, the largest departure is Qwen-3.6-27B (every turn) on waterconnection (-31).

See details →

Diplomacy

Diplomatic behavior: Friendliest: Nemotron-3-Super-Simple | Every-turn (+207.4 net) · Least friendly: MiniMax-M3-Simple | Per-5 (-37.3 net) · Most masked: DeepSeek-V4-Flash-Simple | Every-turn (4.4%) Describes diplomatic persona traits and the public and private stances strategists set toward rivals, including how often the two conflict. Module: behavior.diplomacy n_absolute_players: 1686; baseline: matched in-game AI; baseline_experiments: vanilla-standard-fixed; n_relative_players: 1686; n_unmatched_controlled_players: 0; n_baseline_players: 192; traits: DiplomaticBalance, Friendliness, WorkWithWillingness, WorkAgainstWillingness, Loyalty, DenounceWillingness, Forgiveness, Meanness, Neediness, Chattiness, DeceptiveBias; rate: per_100_turns

Friendliest: Nemotron-3-Super (every turn) (+207.4 net) · Least friendly: MiniMax-M3 (every 5 turns) (-37.3 net) · Most masked: DeepSeek-V4-Flash (every turn) (4.4%)

See details →

Ways to win

Strategic commitment: Domination: Gemma-4-Simple | Every-turn (43%) · Culture: Sonnet-5.5-Simple | Per-5 (59%) · Diplomatic: GPT-OSS-120B-Simple | Every-turn (51%) · Science: GPT-6-Luna-Simple | Per-5 (76%) Shows how often strategists act and revise their settings, how large and how lasting their changes are, and which grand strategy they hold. Module: behavior.commitment n_absolute_players: 1686; baseline: completed-experiment average; n_relative_players: 1686; n_unmatched_controlled_players: 0; n_baseline_players: 1686; n_baseline_experiments: 35; grand_strategies: Conquest, Culture, UnitedNations, Spaceship; rate: per_100_turns

Who aims for each kind of win most often. Domination: Gemma-4 (every turn), 43%; Culture: Sonnet-5.5 (every 5 turns), 59%; Diplomacy: GPT-OSS-120B (every turn), 51%; Science: GPT-6-Luna (every 5 turns), 76%.

See details →

Politics

Policy paths: Freedom: Kimi-K2.7-Simple | Every-turn (29%) · Autocracy: Qwen-3.6-27B-Simple | Every-turn (33%) · Order: GPT-6-Luna-Simple | Every-turn (64%) Shows which policy branches and ideologies each player type adopts, which it picks first in each tier, and how early, against the in-game AI on the same map and seat. Module: behavior.policies n_absolute_players: 1734; baseline: matched in-game AI; baseline_experiments: vanilla-standard-fixed; n_relative_players: 1734; n_unmatched_controlled_players: 0; n_baseline_players: 192; branches: tradition, authority, progress, fealty, statecraft, artistry, industry, imperialism, rationalism, freedom, autocracy, order

Freedom: Kimi-K2.7 (every turn) (29%) · Autocracy: Qwen-3.6-27B (every turn) (33%) · Order: GPT-6-Luna (every turn) (64%)

See details →

Citation

Paper

@article{chen2026civbench,
  title={CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V},
  author={Chen, John and Cheng, Sihan and Gurkan, Can and Lin, Mingyi},
  journal={Conference on Language Modeling (COLM)},
  year={2026}
}

Benchmark results

@misc{civbench_results,
  title={Controlled CivBench on Vox Populi 5.2.7, Civilization V},
  url={https://github.com/vox-deorum/civ-bench}
}