# 评测明细 — DLR · birdminidev

> **说明**：question 驱动三级锚定——L1 dsh MCP 语义层（`dlr_semantic_query` 等 7 工具，含 `get_full_data_info` 下探）/ L2 共识（`dlr_search_consensus`）/ L3 SOP（`dlr_search_sop`），交叉验证后出 SQL。
> **判定**：gold SQL 在数据集 SQLite 上执行得期望值 ↔ agent 答案（**数据集原生，不修正**）；数值逐级容差、文本归一化包含，**都只看 `Final Answer:` 结论句区域**（正文里的备选读法不算命中）。
> **评定**：SOP 生效时按 SOP 裁定——与 gold 对不上但答法合 SOP 口径 = **翻盘**（计正确，但**单独标注、单独计数，不并入 PASS**）。
> **数据来源**：`results/<轮次>/{questions.csv, raw/*.ndjson}` ｜ 本文件由 `tsm stats` 自动重建（定性观察一节在跑批后按 SOP 案例补写）。
> **列义**：判定 PASS ｜ FAIL ｜ UNCERTAIN（抽不出可比对的值）｜ GOLD_ERR（gold 本身执行失败）；评定 ✅ 正确 ｜ 🔁 翻盘 ｜ ❌ 错误 ｜ ⚠️ 待仲裁；**备注** = 这一行的判定依据 + 裁定依据（人话一句）。
> **本文档 = 总账**：覆盖度 / 汇总 / 数据集缺陷与裁定；**逐题校验表与证据正文按库拆分**，见下方分库索引。

## 跑题覆盖度（跑过多少题）

> **跑题覆盖度 = 跑过的题（去重）÷ 数据集全量（mini_dev 原生题数）**——只看跑没跑过，与对了多少题无关（判定 / 评定见上表与汇总）。

| 数据库 | 全量 | 已跑 | 剩余 | 覆盖 |
|---|---|---|---|---|
| california_schools | 30 | 30 | 0 | 100.0% ✅ |
| card_games | 52 | 52 | 0 | 100.0% ✅ |
| codebase_community | 49 | 49 | 0 | 100.0% ✅ |
| debit_card_specializing | 30 | 30 | 0 | 100.0% ✅ |
| european_football_2 | 51 | 51 | 0 | 100.0% ✅ |
| financial | 32 | 32 | 0 | 100.0% ✅ |
| formula_1 | 66 | 66 | 0 | 100.0% ✅ |
| student_club | 48 | 48 | 0 | 100.0% ✅ |
| superhero | 52 | 52 | 0 | 100.0% ✅ |
| thrombosis_prediction | 50 | 50 | 0 | 100.0% ✅ |
| toxicology | 40 | 40 | 0 | 100.0% ✅ |
| **合计** | **500** | **500** | **0** | **100.0%** |

## 汇总

**评定**（按 SOP 裁定 · **主口径**；🔁 翻盘单独计，不并入 ✅ 正确——数据集错误不记在应用头上）

| 评定 | 值 |
|---|---|
| ✅ 正确（与 gold 一致） | 426 / 500（85.2%） |
| 🔁 翻盘（按 SOP 裁定为正确） | 74 |
| ❌ 错误 | 0 |
| ⚠️ 待仲裁 | 0 |
| **合计正确（正确 + 翻盘）** | **500 / 500（100.0%）** |

**判定**（与 gold 原始比对 · 留档；gold 数据集原生、不修正）

| 判定 | 值 |
|---|---|
| PASS（与 gold 一致） | 426 / 500（85.2%） |
| UNCERTAIN（抽不出可比对的值） | 15 |
| FAIL（与 gold 不符） | 59 |
| GOLD_ERR（gold 本身执行失败） | 0 |

**效率**

| 指标 | 值 |
|---|---|
| token 平均 / 中位 | 73,307 / 55,898 |
| token 最低 / 最高 | 26,925 / 689,012 |
| 步数均值 / 工具调用均值 | 6 / 10 |

> **口径**：本文档汇总按**去重题数**计（同题多轮取**最新一轮**的判定/评定）——与 [results/STATS.md](results/STATS.md) 的**按次数**分布会不同（重跑过或跑挂过的题，那边会多计一次）。仅覆盖已跑轮次，勿外推为全数据集结论。token = input + cache_read + output（不含 CoT 的 reasoning 分项由 harness 单独计）。

## 定性观察

> 全量 500 题跑完后按 SOP 案例撰写；下列数字与 `results/` 归档 CSV、本文件汇总表逐项一致（防止「先写结论后找证据」）。

### 1. 总账

- **500/500 跑完**：✅ 正确 **426** ｜ 🔁 翻盘 **74**（14.8%）｜ ❌ 0 ｜ ⚠️ 0。
- 翻盘 = **数据集缺陷认定**（gold 有毛病、按 L3 节口径判对）。按库：football 14 ｜ card_games 12 ｜ debit_card 12 ｜ codebase 8 ｜ formula_1 7 ｜ financial 6 ｜ thrombosis 6 ｜ toxicology 6 ｜ california 1 ｜ student_club 1 ｜ superhero 1。
- SOP（L3）**180 节**：74 处翻盘**全部**在节里钉了口径，可逐条复核（见「数据集缺陷与裁定」表）。

### 2. 缺陷的类型学（74 处归纳）

| 类型 | 例 | 说明 |
|---|---|---|
| **计数对象错** | q352 q416 q1037 q1080 q1490 q1505 q1525 q186 q1241 q951 q963 | 「多少卡/人/客户/车手」被算成"多少行/多少记录"——分子分母不在同一实体粒度 |
| **文本当数字** | q115 q847 q879 q1011 | 缺 CAST：`'9…' > '257'` 的字典序，或 NULL 排最前，选错行 |
| **SQL 优先级** | q1247 q1265 | 缺括号 → `OR` 分支绕过前面的过滤，人群/分母被放大 |
| **join 放大分母** | q263 q371 | 按 join 行计权而非按实体计 |
| **时间序列误聚合** | q1094 q1115 q1144 q1107 | 球员/FIFA 属性是"带日期的观测序列"，被求和或把某一天的点当成唯一值 |
| **分母基数歧义** | q94 q95 q152 q118 q1185 | 行 vs 去重实体；起始值 vs 结束值当基准 |
| **符号 / 方向** | q819 q829 q1152 | 减法方向与题面词序相反（结果应为负）；比值方向反过来 |
| **条件只做一半** | q137 q192 q416 | 「running contract」只取 C 漏 D；「without power」漏 `'*'` |
| **数据缺口 / 样本窗** | q906 q944 q1500 q1501 | 整个分站缺失（2007/2008 澳洲站）；逐笔记录只有 4 天窗口 |
| **布尔反义** | q1205 | gold 的布尔"true = 超阈值"，与题面"是否在正常范围"相反 |
| **evidence 与 gold 不一致** | q474（`<10` vs `<100`）等 | 证据文本自身的笔误，按题面/gold 钉 |
| **并列取值** | q1389 等 | 多实体并列极值，gold 按行序取一个 |

（类型可叠加；归纳不改单题裁定。）

### 3. 三层各自的表现

- **L1 `dlr`**：召回**只在 LE 面收口**——跑批中的"走错库/走错实体"都由 LE description 修复（英文 + 例名）；`PE = 视图`契约（`get_pe_mapping` 只回建模列、`get_full_data_info` 下探 + 反馈缺口）把结构税压住。
- **L2 `consensus`**：领域口径锚点。**瘦身有收益也有代价的实证**：football q1031 在 L2 瘦身后无节兜底 → agent 反复查 consensus 10+ 次、涨到 304k；补 L3 节后 63k（**−79%**）→ 结论：**evidence 的题级口径归 L3，L2 只留领域级**。
- **L3 `sop`**：`dlr_search_sop` 按题检索交付（不再整文件载入）。**节写完必须复跑**——节晚于跑批时，旧 ✅ 可能是假阳性撑着（q1136 / q1472 / q340 / q726 同型）。

### 4. 工程观察（会复现的坑）

- **病态 SQL 钉死服务**：agent 写的 `CASE WHEN EXISTS(...)` 打在大表（22.9 万行无索引）上 → 单线程 MCP 同步执行、无 interrupt → **整批作废**（09-29 两轮 card_games 共 26 题白跑）。已加**子进程 + 20s 硬超时**护栏，工具描述同时注明反模式（子进程超时中止、服务不受影响）。
- **judge 假阳性族**：期望值出现在"被否决的备选读法"里会被判 PASS（q371 `"10"`、q1058 `"Min"` ← SQL 里的 `MIN(`、q273 28.95）→ 加固后**只看 `Final Answer:` 结论句 + 词边界**，全量重判。
- **答案形态影响判定**：长列表题把结果 `group_concat` 压成单列 → 判定的结果集比对无法对齐（q340 / q726 / q1241）→ 逐题在节里钉"一行一条目"的答案 SQL 形态。
- **网络与超时**：瞬断（`TRANSPORT`）产出 0-token 空轮——**作废轮不计遍数，重跑即可**；单题 10 分钟 `timeout` 会砍掉跑飞的 agent（q186 / q1241 各一次，收紧节后通过）。
- **金缓存**：>2s 的 gold 查询落盘缓存（card_games q518 单条约 5 分钟 → 热缓存后 `tsm grade`/`stats` 秒级）。

### 5. 效率

| 库 | token 中位 | 备注 |
|---|---|---|
| california_schools | 66,005 | 迁新基线 **−38%** |
| card_games | 60,552 | 迁新基线 **−21%**（中位 −35%） |
| codebase_community | 47,901 | 迁新基线 **−43%**（最大降幅） |
| debit_card_specializing | 43,973 | 迁新基线 **−17%** |
| european_football_2 | 60,026 | 迁新基线 **−15%** |
| financial | 64,174 | 新库（无对照基线） |
| formula_1 | 61,576 | 新库 |
| student_club | 54,420 | 新库（基线最干净：90% 直通） |
| superhero | 45,861 | 新库（最便宜） |
| thrombosis_prediction | 88,615 | 新库（最难：基线 57% 直通、临床口径密集） |
| toxicology | 45,684 | 新库 |

- 五个迁新基线的库**全部下降 15%–43%**（PE=视图 + L3 检索化 + L2 瘦身 + judge 加固的合力）。
- 新库**裸跑基线**（无 SOP）均值显著高于补节之后；节生效的个案降幅 47%–90%（football q1031 −79%、california q72 −90%）。

## 分库明细（逐题校验表 + 证据正文）

> 每题一行台账（题号锚点跳到该题证据块）+ 每题一段正文（命中口径 / 执行 SQL / 结论）。

| 数据库 | 已跑 | ✅ 正确 | 🔁 翻盘 | ❌ 错误 | ⚠️ 待仲裁 | token 中位 |
|---|---|---|---|---|---|---|
| [california_schools](DETAIL/california_schools.md) | 30 | 29 | 1 | 0 | 0 | 66,005 |
| [card_games](DETAIL/card_games.md) | 52 | 40 | 12 | 0 | 0 | 60,552 |
| [codebase_community](DETAIL/codebase_community.md) | 49 | 41 | 8 | 0 | 0 | 47,901 |
| [debit_card_specializing](DETAIL/debit_card_specializing.md) | 30 | 18 | 12 | 0 | 0 | 43,973 |
| [european_football_2](DETAIL/european_football_2.md) | 51 | 37 | 14 | 0 | 0 | 60,026 |
| [financial](DETAIL/financial.md) | 32 | 26 | 6 | 0 | 0 | 64,174 |
| [formula_1](DETAIL/formula_1.md) | 66 | 59 | 7 | 0 | 0 | 61,576 |
| [student_club](DETAIL/student_club.md) | 48 | 47 | 1 | 0 | 0 | 54,420 |
| [superhero](DETAIL/superhero.md) | 52 | 51 | 1 | 0 | 0 | 45,861 |
| [thrombosis_prediction](DETAIL/thrombosis_prediction.md) | 50 | 44 | 6 | 0 | 0 | 88,615 |
| [toxicology](DETAIL/toxicology.md) | 40 | 34 | 6 | 0 | 0 | 45,684 |

## 数据集缺陷与裁定（SOP 条目缘由）

| 题号 | 库 | 判定 | 评定 | 类型 | 问题（截） | 裁定（sop.md 摘要） |
|---|---|---|---|---|---|---|
| q27 | california_schools | ⚠️ UNCERTAIN | 🔁 翻盘 | 难题 | What is the average score in writing for the schools that we | "Communication number" is the school's phone number -- there is no separate contact table. Date reading: "opened after 1991" means the openi |
| q341 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What are the borderless cards available without powerful foi | "Powerful foils" are the printings listed by the card marketplace **both** as a card and as a foil -- one of the two being present is not en |
| q349 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Name the card and artist with the most ruling information. A | "Ruling information" is the card's rulings: count the rulings attached to each card and take the largest -- **Teferi's Protection**, illustr |
| q352 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Calculate the percentage of the cards availabe in Chinese Si | "Percentage of the cards" puts **cards** on both sides of the fraction: the cards that have a Chinese Simplified printing, divided by all th |
| q366 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | What is the rule of playing card "Benalish Knight"? | "The rule of playing card X" asks for the card's **rules text** -- the abilities printed on it. For Benalish Knight those are **flash** (it |
| q371 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the percentage of cards whose language is French amo | "Percentage of cards" puts **cards** on both sides of the fraction: the Story Spotlight cards that have a French printing, over all Story Sp |
| q383 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | How many of the banned cards are white border? | "Banned cards" counts **cards**: a card banned in several formats is still one card, so count each card once -- 89 white-bordered cards are |
| q402 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the percentage of Story Spotlight cards that do not | A card "does not have a text box" when it is **textless**. Check the Story Spotlight cards for that: **none of them is textless**, so the pe |
| q407 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 · 难题 | Lists all types of cards in German. | "Types of cards **in German**" asks for the type names as they read in German -- the German-language type strings recorded for German printi |
| q416 | card_games | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | What percentage of cards without power are in French? | "Cards without power" = the cards whose power is missing or recorded as `*`. "In French" = the card has a French printing. The percentage pu |
| q483 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Please list the Italian text ruling of all the cards in the | "The Italian text of a card" is the card's **rules text as printed in Italian** -- one text per card (the text is a long block, so identical |
| q484 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Please list the Italian names of the cards in the set Coldsn | "Highest converted mana cost" in this set is 7, and **twelve cards share it** -- so the answer is twelve names, not one: Devastazione Solare |
| q529 | card_games | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Find and list the names of sets which doesn't have Japanese | A set "has a translation" in a language when that language's **text is actually present**: every set carries a row per language, and a row w |
| q557 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the posts with a score of over 5, what is the percenta | "Among the posts with a score of over 5" is the denominator: **all** posts with a score above 5 (11,465 of them), each counted once. Posts w |
| q584 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Write all the comments left by users who edited the post tit | "The comments left by users who edited the post" are the **notes the editing users left with their edits** -- the short note each edit of th |
| q639 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Based on posts posted by Community, calculate the percentage | The fraction puts **one set of posts** on both sides: the posts that use the R language, among the posts that account posted. That account o |
| q640 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Calculate the difference in view count from post posted by M | "The posts posted by an author" are the posts that author owns, and a post's view count is the count the post itself records -- counted once |
| q672 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the users located in United Kingdom, how many users wh | The question counts **users**, and "a favorite amount of 4 or more" is a post's own favorite count (not a sum across the user's posts). Of t |
| q682 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Which is the most valuable post in 2010? Please give its id | "Most valuable" is the post carrying the largest **FavoriteCount**, and "in 2010" is the **post's own** creation year (the evidence reads MA |
| q683 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the percentage of posts whose owners had a reputatio | "In 2011" scopes the whole question -- the posts of that year, and among them the share whose owner's reputation is over 1000. That year has |
| q710 | codebase_community | ❌ FAIL | 🔁 翻盘 | 数据集问题 | In posts with 1 comment, how many of the comments have 0 sco | "In posts with 1 comment" picks the posts whose **recorded comment count** is exactly 1 -- not the posts that merely happen to have one comm |
| q1473 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What was the average monthly consumption of customers in SME | Consumption is recorded customer-month by customer-month: every recorded figure is already one customer's consumption for one month. So "ave |
| q1481 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | What is the difference in the annual average consumption of | Per customer, total up their 2013 CZK consumption. Then, **per segment**, take the customer(s) with the lowest 2013 total. "Annual average c |
| q1482 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Which of the three segments—SME, LAM and KAM—has the biggest | The question names the currency, so the consumption must be filtered to customers whose billing currency is EUR (`Currency = 'EUR'` in the c |
| q1490 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | How many percent of LAM customer consumed more than 46.73? | "Percent of customers" is counted **per customer**, not per customer-month record. One customer = one unit in both the numerator and the den |
| q1498 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the highest monthly consumption in the year 2012? | Consumption is recorded customer-month by customer-month, and each recorded figure is already one customer's monthly consumption. So the yea |
| q1500 | debit_card_specializing | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Please list the product description of the products consumed | The individual-purchase records are only a **four-day sample**: they cover 2012-08-23 through 2012-08-26, and nothing else. Any month outsid |
| q1501 | debit_card_specializing | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Please list the countries of the gas stations with transacti | Same sample-window fact: the individual-purchase records cover only 2012-08-23~26, so no purchases took place in June 2013, and the truthful |
| q1505 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the customers who paid in euro, how many of them have | "Of them" means **customers**: count each Euro customer once, however many months cross the threshold. The condition is on a monthly figure |
| q1525 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the percentage of the customers who used EUR in 2012 | "Percentage of customers" = **customers**, not transactions: one customer counts once, in both the numerator and the denominator, and both a |
| q1526 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | For the customer who paid 634.8 in 2012/8/25, what was the c | "paid 634.8" identifies the customer through a single purchase of that amount on that date -- a purchase-level condition, not a monthly tota |
| q1529 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the amount spent by customer "38508" at the gas stat | "Amount spent by a customer" is that customer's total consumption across all gas stations -- a question about the customer's monthly figures |
| q1531 | debit_card_specializing | ❌ FAIL | 🔁 翻盘 | 数据集问题 · 难题 | Who is the top spending customer and how much is the average | "Top spending customer" is decided by the customer's total consumption across all gas stations (the month-by-month figures), not by adding u |
| q1028 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | In Scotland Premier League, which away team won the most dur | "Away team won the most" = per away team, count how many matches it won away in that league and season, and take the largest count. The "201 |
| q1029 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What are the speed in which attacks are put together of the | "Speed in which attacks are put together" and "build-up play speed" are the same team attribute -- the question names one quantity twice. "H |
| q1037 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Calculate the percentage of players who prefer left foot, wh | "Percentage of players" counts **players**, not rating records: one player counts once in both the numerator and the denominator, even thoug |
| q1058 | european_football_2 | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Who has the highest average finishing rate between the highe | The question compares exactly two players: the tallest and the shortest one. Compare their average finishing rates over their dated records |
| q1080 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the players whose preferred foot was the left foot whe | "Among the players ... how many of them" counts **players**: one player counts once, even though a player has many dated records (and his pr |
| q1094 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How much higher in percentage is Ariel Borysiuk's overall ra | A player's rating is a **dated series of observations**, not one number: the same player has many rating records over the years. A question |
| q1107 | european_football_2 | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | When was the first time did Kevin Constant have his highest | A player's scores are a dated series. "His highest crossing score" is the largest value in that series, and he can carry it on more than one |
| q1115 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What percentage is Landon Donovan's overall rating higher th | The question names a date, so take each player's rating record of that day, then express how much higher the first is as a percentage of the |
| q1124 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 难题 | Who are the players that tend to be attacking when their mat | "Tend to be attacking when their mates were doing attack moves" is the **high** attacking work rate; the answer is the list of players carry |
| q1133 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many football players born after the 1990s have the firs | "Born after the 1990s" here means born **after 1990** -- the players born from 1991 on whose name starts with Aaron. (Read literally as "aft |
| q1135 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Please provide top four football players' IDs who are among | Two readings decide this question, and both must be right: - **Right-footed**: only records whose preferred foot is the right one compete. - |
| q1136 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many players had the highest potential score for crossin | "The highest potential score for crossing" is the highest score in the **crossing** column (the best a player can reach at crossing), not th |
| q1144 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Please state the finishing rate and curve score of the playe | The heaviest player is the one with the largest weight; his attributes are a dated series, and with no date in the question take his **prese |
| q1148 | european_football_2 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the percentage of players that are under 180 cm who | Two readings decide this question: - "An overall strength of more than 70" is the player's **overall rating** above 70 -- the overall talent |
| q94 | financial | ❌ FAIL | 🔁 翻盘 | 数据集问题 | List out the account numbers of female clients who are oldes | Two conditions pick one person and one district: the oldest female client (`gender = 'F'`, smallest `birth_date`) **within the district whos |
| q95 | financial | ❌ FAIL | 🔁 翻盘 | 数据集问题 | List out the account numbers of clients who are youngest and | Both conditions apply to the same pick: the client with the largest `birth_date` (youngest) **inside the district whose average salary `A11` |
| q115 | financial | ❌ FAIL | 🔁 翻盘 | 数据集问题 | For the branch which located in the south Bohemia with bigge | "The branch in south Bohemia with the biggest number of inhabitants" is the district with `A3 = 'south Bohemia'` whose `A4` (inhabitants) is |
| q129 | financial | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Which are the top ten withdrawals (non-credit card) by distr | "Top ten withdrawals (non-credit card) by district" ranks the districts by the **total amount** of their non-card withdrawals (`type = 'VYDA |
| q152 | financial | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the average number of crimes committed in 1995 in re | The candidates are the **distinct districts** with `A15 > 4000` that hold at least one account opened in 1997 or later -- **26 regions** -- |
| q186 | financial | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What percentage of male clients request for weekly statement | A client "requests weekly statements" when they hold an account whose frequency is `'POPLATEK TYDNE'` (link clients to accounts through `dis |
| q847 | formula_1 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the surname of the driver with the best lap time in | The best lap in the second qualifying period is the smallest **non-empty** `q2` time of race 19: Kimi **Räikkönen**, `1:34.188`. The dataset |
| q879 | formula_1 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | For the driver who set the fastest lap speed, what is his na | "Fastest lap speed" is the largest **numeric** `fastestLapSpeed`: **257.320** km/h, whose driver is **Brazilian**. The dataset's own query o |
| q906 | formula_1 | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Which was Lewis Hamilton first race? What was his points rec | The dataset holds **no 2007 Australian Grand Prix** (his real first race is absent from `races`), so his first race in the data is the **Mal |
| q951 | formula_1 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many Japanese constructors have 0 points in 2 races? | **One** Japanese constructor carries zero points in exactly two standings rows: **Kojima**. The dataset's own query reports the row count (2 |
| q962 | formula_1 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | From 2000 to 2005, what percentage of drivers who were born | Counted over **drivers**: every one of the 52 drivers of the 2000-2005 seasons was born before 1985, and all 52 have a race with more than 5 |
| q963 | formula_1 | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many French drivers who obtain the laptime less than 02: | Count **drivers**, not lap records: **9** French drivers hold a lap under two minutes (compare the time numerically, e.g. `milliseconds < 12 |
| q1011 | formula_1 | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | Which top 20 driver created the shortest lap time ever recor | Rank the drivers by their **shortest lap** using the numeric `milliseconds` column and take the top twenty; the first eight are Lewis Hamilt |
| q1322 | student_club | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the events attended by more than 10 members of the Stu | A meeting is an event with `type = 'Meeting'`; "attended by more than 10 members" means more than ten attendance rows. **Four** meetings qua |
| q772 | superhero | ❌ FAIL | 🔁 翻盘 | 数据集问题 | List the eyes, hair and skin colour of all female superheroe | List the **colour names**, one row per hero: **Buffy** (eyes Green, hair Blond, skin No Colour), **Elastigirl** (Brown, Brown, No Colour), * |
| q1152 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the ratio of outpatient to inpatient followed up tre | The question asks the ratio of **outpatient to inpatient** among SLE patients, so the outpatient count sits on top: 84 / 110 = **0.763636363 |
| q1205 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Was the patient with the number 57266's uric acid within a n | Patient 57266 is male with UA readings 6.2 and 5.4 -- both **within** the normal male range (UA <= 8.0). The answer must carry the dataset's |
| q1241 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | For patients with abnormal platelet level, state the number | Abnormal platelet level is `PLT <= 100` or `PLT >= 400`. The number of **patients** below the range is **36** and above it is **82** (so mor |
| q1247 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Among the male patients who have a normal level of white blo | Normal WBC is between 3.5 and 9.0; abnormal fibrinogen is `FG <= 150` or `FG >= 450`. Apply **all** the conditions together -- male, normal |
| q1251 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many patients with an Ig G higher than normal? | Higher than normal is `IGG >= 2000`, counted over patients of the laboratory table: **136**. The dataset's own query additionally joins `Exa |
| q1265 | thrombosis_prediction | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many patients have a normal level of anti-ribonuclear pr | A normal anti-RNP reads `'negative'` or `'0'`, and the admission is `Admission = '+'`. Apply **both** conditions: **35** patients. The datas |
| q197 | toxicology | ❌ FAIL | 🔁 翻盘 | 数据集问题 | Calculate the average number of oxygen atoms in single-bonde | The average is taken over **all** single-bonded molecules -- including the ones with no oxygen atom (they count as zero): **2.16129032258064 |
| q198 | toxicology | ❌ FAIL | 🔁 翻盘 | 数据集问题 | On average how many carcinogenic molecules are single bonded | The average is taken over **all** carcinogenic molecules (`label = '+'`), counting each molecule's single bonds (molecules without any singl |
| q207 | toxicology | ⚠️ UNCERTAIN | 🔁 翻盘 | 数据集问题 | What elements are in a double type bond? | The elements are the atoms that actually sit in a double bond (join `atom` to `bond` through `connected`, `bond_type = '='`): **c, o, n, s a |
| q218 | toxicology | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What percentage of carcinogenic-type molecules does not cont | Of the **152** carcinogenic molecules only **one** contains fluorine, so **151/152 = 99.34210526315789%** do not. The dataset's own query co |
| q234 | toxicology | ❌ FAIL | 🔁 翻盘 | 数据集问题 | How many bonds which involved atom 12 does molecule TR009 ha | Molecule TR009 has **3** bonds involving its atom `TR009_12` (via `connected`). The dataset's own query tests `'_1'` and `'_2'` (a typo for |
| q263 | toxicology | ❌ FAIL | 🔁 翻盘 | 数据集问题 | What is the composition of element chlorine in percentage am | Among the single-bond molecules, count each molecule **once** and take chlorine atoms over all their atoms: **3.4823684499615513** (about 3. |