id,task,model,Best score (across scorers),Scores,log viewer,logs,job,started_at,Status,Manual evaluation,task version
V3YQtSEnQ3TPZEPPgkCKQM,GPQA diamond,gpt-5.1_medium,0.850,choice:0.85±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FV3YQtSEnQ3TPZEPPgkCKQM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/V3YQtSEnQ3TPZEPPgkCKQM.eval,https://github.com/epoch-research/benchmarks/actions/runs/19445092846/job/55637765314,17 November 2025 4:28pm,Success,,1.0.1
DKLrszPeTvFUGq39AaTniX,FrontierMath-2025-02-28-Private,o3-2025-04-16_low,0.097,verification_code:0.1±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/DKLrszPeTvFUGq39AaTniX.eval,https://github.com/epoch-research/benchmarks/actions/runs/19415312900/job/55542751205,16 November 2025 8:17pm,Success,,1.1.0
XQ46BgPhHaYZsrPQuTvz8J,FrontierMath-2025-02-28-Private,o3-2025-04-16_high,0.187,verification_code:0.19±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XQ46BgPhHaYZsrPQuTvz8J.eval,https://github.com/epoch-research/benchmarks/actions/runs/19411617222/job/55533858218,16 November 2025 3:35pm,Success,,1.1.0
XwNqEvxaaCQ8M2NpKJGoZG,FrontierMath-2025-02-28-Private,o3-2025-04-16_medium,0.169,verification_code:0.17±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XwNqEvxaaCQ8M2NpKJGoZG.eval,https://github.com/epoch-research/benchmarks/actions/runs/19411617222/job/55533858210,16 November 2025 3:35pm,Success,,1.1.0
cqUZ6gcq2HBnShFRWvKjGr,GPQA diamond,gemini-2.5-pro,0.853,choice:0.85±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcqUZ6gcq2HBnShFRWvKjGr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cqUZ6gcq2HBnShFRWvKjGr.eval,https://github.com/epoch-research/benchmarks/actions/runs/19410488871/job/55531171009,16 November 2025 2:00pm,Success,,1.0.1
iJp7PgYZa3PiXAV8UWd7MC,OTIS Mock AIME 2024-2025,gemini-2.5-pro,0.847,"model_extracted_exact_match:0.84±0.05,model_graded:0.85±0.05",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FiJp7PgYZa3PiXAV8UWd7MC.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/iJp7PgYZa3PiXAV8UWd7MC.eval,https://github.com/epoch-research/benchmarks/actions/runs/19410488871/job/55531171006,16 November 2025 2:00pm,Success,,1.0.1
n26C8XHigC9Xfsdb24JZtd,FrontierMath-2025-02-28-Private,claude-sonnet-4-5-20250929,0.093,verification_code:0.09±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/n26C8XHigC9Xfsdb24JZtd.eval,https://github.com/epoch-research/benchmarks/actions/runs/19405660542/job/55519940777,16 November 2025 7:36am,Success,,1.1.0
LUpSd7DzWf3Fw4iwXb3MPV,FrontierMath-2025-02-28-Private,claude-sonnet-4-5-20250929_32K,0.152,verification_code:0.15±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/LUpSd7DzWf3Fw4iwXb3MPV.eval,https://github.com/epoch-research/benchmarks/actions/runs/19405660542/job/55519940782,16 November 2025 7:36am,Success,,1.1.0
82dL9ukqav96KZ8aHfqPpq,FrontierMath-2025-02-28-Private,o4-mini-2025-04-16_low,0.107,verification_code:0.11±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/82dL9ukqav96KZ8aHfqPpq.eval,https://github.com/epoch-research/benchmarks/actions/runs/19399583634/job/55504887957,15 November 2025 10:21pm,Success,,1.1.0
W2YTmg94XqVbH7p86TAqJT,FrontierMath-2025-02-28-Private,o3-mini-2025-01-31_high,0.124,verification_code:0.12±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/W2YTmg94XqVbH7p86TAqJT.eval,https://github.com/epoch-research/benchmarks/actions/runs/19398612300/job/55502380218,15 November 2025 8:50pm,Success,,1.1.0
5PD6uczKMgqAUJfgbKQynj,FrontierMath-2025-02-28-Private,gemini-2.5-pro,0.121,verification_code:0.12±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5PD6uczKMgqAUJfgbKQynj.eval,https://github.com/epoch-research/benchmarks/actions/runs/19350773727/job/55361957832,13 November 2025 8:02pm,Success,,1.1.0
Zpbbguj8FXwr65ys2thEfM,FrontierMath-2025-02-28-Public,kimi-k2-thinking-turbo,0.200,verification_code:0.2±0.13,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZpbbguj8FXwr65ys2thEfM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Zpbbguj8FXwr65ys2thEfM.eval,https://github.com/epoch-research/benchmarks/actions/runs/19350111349/job/55359883974,13 November 2025 7:26pm,Success,,1.1.0
ee2YzVv39Tk6tmA7DNjTG5,FrontierMath-Tier-4-2025-07-01-Private,gpt-5.1_high,0.125,verification_code:0.13±0.05,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ee2YzVv39Tk6tmA7DNjTG5.eval,https://github.com/epoch-research/benchmarks/actions/runs/19345054772/job/55343235018,13 November 2025 3:36pm,Success,,1.1.0
nveMCdbvDijDTBN9JPWNve,FrontierMath-2025-02-28-Private,gpt-5.1_high,0.310,verification_code:0.31±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/nveMCdbvDijDTBN9JPWNve.eval,https://github.com/epoch-research/benchmarks/actions/runs/19345054772/job/55343235006,13 November 2025 3:36pm,Success,,1.1.0
mYKKTdV27TRJv9FpUg7rkn,OTIS Mock AIME 2024-2025,gpt-5.1_high,0.886,"model_extracted_exact_match:0.89±0.04,model_graded:0.89±0.04",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmYKKTdV27TRJv9FpUg7rkn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mYKKTdV27TRJv9FpUg7rkn.eval,https://github.com/epoch-research/benchmarks/actions/runs/19343186056/job/55339369861,13 November 2025 2:52pm,Success,,1.0.1
CSVtMksEVjBNsGJGjW4GcQ,GPQA diamond,gpt-5.1_high,0.876,choice:0.88±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FCSVtMksEVjBNsGJGjW4GcQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/CSVtMksEVjBNsGJGjW4GcQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/19343186056/job/55339369886,13 November 2025 2:52pm,Success,,1.0.1
JwfCZuni4TRRnA77UWwpyg,FrontierMath-2025-02-28-Private,gpt-5-2025-08-07_high,0.324,verification_code:0.32±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/JwfCZuni4TRRnA77UWwpyg.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302790,13 November 2025 2:38pm,Success,,1.1.0
3GzYKQW2K4iijm6a73rVeh,FrontierMath-2025-02-28-Private,claude-sonnet-4-5-20250929_59K,0.135,verification_code:0.13±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/3GzYKQW2K4iijm6a73rVeh.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302798,13 November 2025 10:54am,Success,,1.1.0
DkMXbLvTgWJDWnUQVuBA8J,FrontierMath-2025-02-28-Private,gpt-5-mini-2025-08-07_high,0.272,verification_code:0.27±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/DkMXbLvTgWJDWnUQVuBA8J.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302771,13 November 2025 10:54am,Success,,1.1.0
XPCVZtFEC6BYKAwhLauMSD,FrontierMath-2025-02-28-Private,o4-mini-2025-04-16_medium,0.190,verification_code:0.19±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XPCVZtFEC6BYKAwhLauMSD.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302794,13 November 2025 10:54am,Success,,1.1.0
Z7hiWJnEkkLmQXviEiTdHt,FrontierMath-2025-02-28-Private,o4-mini-2025-04-16_high,0.248,verification_code:0.25±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Z7hiWJnEkkLmQXviEiTdHt.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302779,13 November 2025 10:54am,Success,,1.1.0
VPHGtT9EdPS6YxbzDAt98q,FrontierMath-2025-02-28-Private,gpt-5-mini-2025-08-07_medium,0.203,verification_code:0.2±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/VPHGtT9EdPS6YxbzDAt98q.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302734,13 November 2025 10:54am,Success,,1.1.0
oN54TKXxcn9gofYBASTVCY,FrontierMath-2025-02-28-Private,grok-4-0709,0.197,verification_code:0.2±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/oN54TKXxcn9gofYBASTVCY.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302791,13 November 2025 10:54am,Success,,1.1.0
hrJBSLQUe5hrHECkuySSJr,FrontierMath-2025-02-28-Private,gpt-5-2025-08-07_medium,0.272,verification_code:0.27±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/hrJBSLQUe5hrHECkuySSJr.eval,https://github.com/epoch-research/benchmarks/actions/runs/19337405152/job/55316302732,13 November 2025 10:54am,Success,,1.1.0
8BM7dCHaUZgcYJwZWiBhiQ,OTIS Mock AIME 2024-2025,kimi-k2-thinking-turbo,0.831,model_extracted_exact_match:0.83±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F8BM7dCHaUZgcYJwZWiBhiQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/8BM7dCHaUZgcYJwZWiBhiQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/19334646026/job/55306349292,13 November 2025 9:21am,Success,,1.0.1
muGecFpAg9wHd6gexWJ99L,GPQA diamond,kimi-k2-thinking-turbo,0.842,choice:0.84±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmuGecFpAg9wHd6gexWJ99L.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/muGecFpAg9wHd6gexWJ99L.eval,https://github.com/epoch-research/benchmarks/actions/runs/19261060544/job/55065827388,11 November 2025 4:27am,Success,,1.0.1
7mwva8MznCtza9r8ML54zE,GPQA diamond,gpt-oss-120b_high,0.708,choice:0.71±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7mwva8MznCtza9r8ML54zE.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7mwva8MznCtza9r8ML54zE.eval,https://github.com/epoch-research/benchmarks/actions/runs/19173274474/job/54811375227,7 November 2025 10:41am,Success,,1.0.1
P4n7e8FZVpukTeCELirbmZ,OTIS Mock AIME 2024-2025,gpt-oss-120b_high,0.711,"model_extracted_exact_match:0.71±0.06,model_graded:0.71±0.06",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FP4n7e8FZVpukTeCELirbmZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/P4n7e8FZVpukTeCELirbmZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/19173274474/job/54811375226,7 November 2025 10:41am,Success,,1.0.1
Y9YnaJQWpPFAPhe6jtKMhH,OTIS Mock AIME 2024-2025,gpt-5-nano-2025-08-07_high,0.811,"model_extracted_exact_match:0.81±0.05,model_graded:0.81±0.05",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FY9YnaJQWpPFAPhe6jtKMhH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Y9YnaJQWpPFAPhe6jtKMhH.eval,https://github.com/epoch-research/benchmarks/actions/runs/18968978430/job/54172027766,31 October 2025 5:55am,Success,,1.0.1
eA9vyfaDkpqLyCkPESaXGu,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-mini-2025-08-07_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FeA9vyfaDkpqLyCkPESaXGu.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/eA9vyfaDkpqLyCkPESaXGu.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131041,30 October 2025 12:22pm,Success,,1.0.1
5rh6TyPLbkJmQZs8iY6msJ,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-mini-2025-08-07_high,0.063,verification_code:0.06±0.04,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5rh6TyPLbkJmQZs8iY6msJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131142,30 October 2025 12:22pm,Success,,1.0.1
knrPUMVyK4PQe3PsYDLp2L,FrontierMath-2025-02-28-Public,gpt-5-nano-2025-08-07_high,0.200,verification_code:0.2±0.13,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FknrPUMVyK4PQe3PsYDLp2L.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/knrPUMVyK4PQe3PsYDLp2L.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131133,30 October 2025 12:22pm,Success,,1.0.1
KWhvxKCHXs6ZRQ7YLEgXmD,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-nano-2025-08-07_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FKWhvxKCHXs6ZRQ7YLEgXmD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/KWhvxKCHXs6ZRQ7YLEgXmD.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131034,30 October 2025 12:22pm,Success,,1.0.1
3W8PgXVhFQ32N57rwsY6fw,FrontierMath-2025-02-28-Public,gpt-5-mini-2025-08-07_high,0.300,verification_code:0.3±0.15,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3W8PgXVhFQ32N57rwsY6fw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3W8PgXVhFQ32N57rwsY6fw.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131071,30 October 2025 12:22pm,Success,,1.0.1
CvK572G5SSd5Yk8ZGikDR6,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-nano-2025-08-07_high,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/CvK572G5SSd5Yk8ZGikDR6.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131046,30 October 2025 12:22pm,Success,,1.0.1
2pSDjMSGDh4HsAxkqUuNQP,GPQA diamond,gpt-5-nano-2025-08-07_high,0.694,choice:0.69±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2pSDjMSGDh4HsAxkqUuNQP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2pSDjMSGDh4HsAxkqUuNQP.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131079,30 October 2025 12:20pm,Success,,1.0.1
2eMHpXXR9h7P2oNPDQkTBY,FrontierMath-2025-02-28-Private,gpt-5-nano-2025-08-07_high,0.083,verification_code:0.08±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/2eMHpXXR9h7P2oNPDQkTBY.eval,https://github.com/epoch-research/benchmarks/actions/runs/18947536838/job/54103131060,30 October 2025 12:20pm,Success,,1.0.1
HGLcT8RytVK548aPQ73s7f,SWE-Bench verified,gpt-5-mini-2025-08-07_high,0.616,swe_bench_scorer:0.62±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FHGLcT8RytVK548aPQ73s7f.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/HGLcT8RytVK548aPQ73s7f.eval,https://github.com/epoch-research/benchmarks/actions/runs/18938214582/job/54070320820,30 October 2025 6:54am,Success,,1.0.1
5duoiNDccsgaW7UVt6feYX,MATH level 5,gpt-5-mini-2025-08-07_high,0.978,"normalized_string_match:0.89±0.01,sympy_equiv:0.92±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5duoiNDccsgaW7UVt6feYX.eval,https://github.com/epoch-research/benchmarks/actions/runs/18936756405/job/54065495295,30 October 2025 6:00am,Success,,1.0.1
jP8EyLtPFsakgnsHe2oJM6,GPQA diamond,gpt-5-mini-2025-08-07_high,0.750,choice:0.75±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjP8EyLtPFsakgnsHe2oJM6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jP8EyLtPFsakgnsHe2oJM6.eval,https://github.com/epoch-research/benchmarks/actions/runs/18936756405/job/54065495290,30 October 2025 5:59am,Success,,1.0.1
R82gycJfm5jaCdh32tFkH4,OTIS Mock AIME 2024-2025,gpt-5-mini-2025-08-07_high,0.867,"model_extracted_exact_match:0.87±0.04,model_graded:0.87±0.04",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FR82gycJfm5jaCdh32tFkH4.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/R82gycJfm5jaCdh32tFkH4.eval,https://github.com/epoch-research/benchmarks/actions/runs/18936756405/job/54065495310,30 October 2025 5:59am,Success,,1.0.1
dWLFQpAhLyBM7BXDZPWuQo,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-2025-08-07_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdWLFQpAhLyBM7BXDZPWuQo.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dWLFQpAhLyBM7BXDZPWuQo.eval,https://github.com/epoch-research/benchmarks/actions/runs/18927681144/job/54037947338,29 October 2025 10:14pm,Success,,1.0.1
6LwyJZsWEzuD8mhSRUF2xn,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-2025-08-07_high,0.125,verification_code:0.13±0.05,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6LwyJZsWEzuD8mhSRUF2xn.eval,https://github.com/epoch-research/benchmarks/actions/runs/18927681144/job/54037947334,29 October 2025 10:14pm,Success,,1.0.1
DAqDqDjNJNDt6GGQSq4BuF,FrontierMath-2025-02-28-Public,gpt-5-2025-08-07_high,0.600,verification_code:0.6±0.16,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FDAqDqDjNJNDt6GGQSq4BuF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/DAqDqDjNJNDt6GGQSq4BuF.eval,https://github.com/epoch-research/benchmarks/actions/runs/18926557503/job/54034761661,29 October 2025 9:04pm,Success,,1.0.1
4wN6FKe2db9CLQ3NUFfYge,MATH level 5,gpt-5-2025-08-07_high,0.981,"normalized_string_match:0.9±0.01,sympy_equiv:0.91±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/4wN6FKe2db9CLQ3NUFfYge.eval,https://github.com/epoch-research/benchmarks/actions/runs/18923863075/job/54026519325,29 October 2025 6:29pm,Success,,1.0.1
3D8aaFBLZwQcnWHYKwvnmq,OTIS Mock AIME 2024-2025,gpt-5-2025-08-07_high,0.914,"model_extracted_exact_match:0.91±0.04,model_graded:0.91±0.04",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3D8aaFBLZwQcnWHYKwvnmq.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3D8aaFBLZwQcnWHYKwvnmq.eval,https://github.com/epoch-research/benchmarks/actions/runs/18923863075/job/54026519328,29 October 2025 6:29pm,Success,,1.0.1
9nPyNrZxwtoT7eS4DZbQF6,GPQA diamond,gpt-5-2025-08-07_high,0.862,choice:0.86±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9nPyNrZxwtoT7eS4DZbQF6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9nPyNrZxwtoT7eS4DZbQF6.eval,https://github.com/epoch-research/benchmarks/actions/runs/18923863075/job/54026519323,29 October 2025 6:29pm,Success,,1.0.1
LkVuUxvsHNzpcg3sUiH7f3,GPQA diamond,claude-sonnet-4-5-20250929_16K,0.788,choice:0.79±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FLkVuUxvsHNzpcg3sUiH7f3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/LkVuUxvsHNzpcg3sUiH7f3.eval,https://github.com/epoch-research/benchmarks/actions/runs/18876881988/job/53869012990,28 October 2025 9:46am,Success,,1.0.0
XaGuWDxPtqVvyxqdMncKFj,GPQA diamond,claude-sonnet-4-5-20250929_59K,0.823,choice:0.82±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FXaGuWDxPtqVvyxqdMncKFj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/XaGuWDxPtqVvyxqdMncKFj.eval,https://github.com/epoch-research/benchmarks/actions/runs/18876881988/job/53869012992,28 October 2025 9:46am,Success,,1.0.0
k5PDL6MpCC6P5eLAhodJiS,OTIS Mock AIME 2024-2025,claude-sonnet-4-5-20250929_59K,0.778,"model_extracted_exact_match:0.78±0.06,model_graded:0.78±0.06",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fk5PDL6MpCC6P5eLAhodJiS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/k5PDL6MpCC6P5eLAhodJiS.eval,https://github.com/epoch-research/benchmarks/actions/runs/18871492679/job/53850886723,28 October 2025 6:21am,Success,,1.0.0
7QkPRbRPpRx8nJT2ZcqSx6,OTIS Mock AIME 2024-2025,claude-sonnet-4-5-20250929_16K,0.711,model_extracted_exact_match:0.71±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7QkPRbRPpRx8nJT2ZcqSx6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7QkPRbRPpRx8nJT2ZcqSx6.eval,https://github.com/epoch-research/benchmarks/actions/runs/18870867784/job/53848800832,28 October 2025 5:57am,Success,,1.0.0
hGVxg4EhSSJSCdH88m9soq,OTIS Mock AIME 2024-2025,gpt-4-0613,0.011,"model_extracted_exact_match:0.01±0.01,model_graded:0.01±0.01",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FhGVxg4EhSSJSCdH88m9soq.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/hGVxg4EhSSJSCdH88m9soq.eval,https://github.com/epoch-research/benchmarks/actions/runs/18754639699/job/53503388590,23 October 2025 12:09pm,Success,,1.0.0
FkbLkW5ed8rnZGFJMevDFn,GPQA diamond,gpt-4-0314,0.357,choice:0.36±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFkbLkW5ed8rnZGFJMevDFn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FkbLkW5ed8rnZGFJMevDFn.eval,https://github.com/epoch-research/benchmarks/actions/runs/18747945427/job/53479886643,23 October 2025 8:16am,Success,,1.0.0
cLTgetxaaTCZoA9vwY4SdL,OTIS Mock AIME 2024-2025,gpt-4-0314,0.006,"model_extracted_exact_match:0.01±0,model_graded:0.01±0",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcLTgetxaaTCZoA9vwY4SdL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cLTgetxaaTCZoA9vwY4SdL.eval,https://github.com/epoch-research/benchmarks/actions/runs/18723481283/job/53402069658,22 October 2025 12:49pm,Success,,1.0.0
YRVwm4B88qK6aUEFBdNQmP,FrontierMath-Tier-4-2025-07-01-Private,claude-haiku-4-5-20251001_32K,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YRVwm4B88qK6aUEFBdNQmP.eval,https://github.com/epoch-research/benchmarks/actions/runs/18721225744/job/53394037312,22 October 2025 11:24am,Success,,1.0.0
d8WWwzb6b57mWqddhBXuDi,FrontierMath-Tier-4-2025-07-01-Private,claude-sonnet-4-5-20250929_32K,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/d8WWwzb6b57mWqddhBXuDi.eval,https://github.com/epoch-research/benchmarks/actions/runs/18719252235/job/53386969258,22 October 2025 10:19am,Success,,1.0.0
Qwci6JAG4DUWqA9eSHThDJ,MATH level 5,claude-haiku-4-5-20251001_32K,0.964,"normalized_string_match:0.93±0.01,sympy_equiv:0.91±0.01,model_graded_equiv:0.96±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Qwci6JAG4DUWqA9eSHThDJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/18718003542/job/53382512207,22 October 2025 9:37am,Success,,1.0.0
fdNzD5mp8r2bHxy2PXx6Gh,FrontierMath-2025-02-28-Private,claude-haiku-4-5-20251001_32K,0.059,verification_code:0.06±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/fdNzD5mp8r2bHxy2PXx6Gh.eval,https://github.com/epoch-research/benchmarks/actions/runs/18716132658/job/53376028338,22 October 2025 8:29am,Success,,1.0.0
HGWgs7RBpJf4cgv2nixY2u,FrontierMath-2025-02-28-Public,claude-haiku-4-5-20251001_32K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FHGWgs7RBpJf4cgv2nixY2u.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/HGWgs7RBpJf4cgv2nixY2u.eval,https://github.com/epoch-research/benchmarks/actions/runs/18713563867/job/53367578332,22 October 2025 6:43am,Success,,1.0.0
neZvEzLi6yeL9z5KjGef7H,OTIS Mock AIME 2024-2025,claude-haiku-4-5-20251001_32K,0.667,model_extracted_exact_match:0.67±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FneZvEzLi6yeL9z5KjGef7H.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/neZvEzLi6yeL9z5KjGef7H.eval,https://github.com/epoch-research/benchmarks/actions/runs/18713261711/job/53366606425,22 October 2025 6:31am,Success,,1.0.0
GEaAdb9UmoUFbEtBx7WzAR,GPQA diamond,claude-haiku-4-5-20251001_32K,0.712,choice:0.71±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FGEaAdb9UmoUFbEtBx7WzAR.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/GEaAdb9UmoUFbEtBx7WzAR.eval,https://github.com/epoch-research/benchmarks/actions/runs/18712073223/job/53362769637,22 October 2025 5:43am,Success,,1.0.0
FqjAH25VNjgwQTbDJFfqAg,MATH level 5,claude-sonnet-4-5-20250929_32K,0.977,"normalized_string_match:0.95±0.01,sympy_equiv:0.92±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/FqjAH25VNjgwQTbDJFfqAg.eval,https://github.com/epoch-research/benchmarks/actions/runs/18690715743/job/53295579768,21 October 2025 12:28pm,Success,,1.0.0
ii6JE9j57eSbuswEsJArTa,GPQA diamond,claude-sonnet-4-5-20250929_32K,0.817,choice:0.82±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fii6JE9j57eSbuswEsJArTa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ii6JE9j57eSbuswEsJArTa.eval,https://github.com/epoch-research/benchmarks/actions/runs/18688633754/job/53288296635,21 October 2025 11:15am,Success,,1.0.0
kqTyhmHyT9irTAWk3xBt2x,OTIS Mock AIME 2024-2025,claude-sonnet-4-5-20250929_32K,0.778,"model_extracted_exact_match:0.78±0.06,model_graded:0.78±0.06",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FkqTyhmHyT9irTAWk3xBt2x.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/kqTyhmHyT9irTAWk3xBt2x.eval,https://github.com/epoch-research/benchmarks/actions/runs/18683887074/job/53271422688,21 October 2025 8:33am,Success,,1.0.0
Wf2kwcXu8YXNgzDCxChtvj,OTIS Mock AIME 2024-2025,claude-haiku-4-5-20251001,0.358,"model_extracted_exact_match:0.36±0.06,model_graded:0.36±0.06",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWf2kwcXu8YXNgzDCxChtvj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Wf2kwcXu8YXNgzDCxChtvj.eval,https://github.com/epoch-research/benchmarks/actions/runs/18560411409/job/52907838956,16 October 2025 7:58am,Success,,1.0.0
cgnopvpB6mzsfb9yGnYK2z,GPQA diamond,claude-haiku-4-5-20251001,0.605,choice:0.6±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcgnopvpB6mzsfb9yGnYK2z.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cgnopvpB6mzsfb9yGnYK2z.eval,https://github.com/epoch-research/benchmarks/actions/runs/18560411409/job/52907838946,16 October 2025 7:58am,Success,,1.0.0
3w2dD9rcMPHLYsEsJmYsCX,MATH level 5,claude-haiku-4-5-20251001,0.869,"normalized_string_match:0.84±0.01,sympy_equiv:0.82±0.01,model_graded_equiv:0.87±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/3w2dD9rcMPHLYsEsJmYsCX.eval,https://github.com/epoch-research/benchmarks/actions/runs/18560411409/job/52907838961,16 October 2025 7:58am,Success,,1.0.0
NNRfKgR23F8kXjFh8jin8g,SWE-Bench verified,claude-haiku-4-5-20251001,0.606,swe_bench_scorer:0.61±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNNRfKgR23F8kXjFh8jin8g.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NNRfKgR23F8kXjFh8jin8g.eval,https://github.com/epoch-research/benchmarks/actions/runs/18556527593/job/52896388419,16 October 2025 5:36am,Success,,1.0.0
8sTJmYZ7MSk8obK9vjUjHm,FrontierMath-2025-02-28-Private,claude-haiku-4-5-20251001,0.041,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/8sTJmYZ7MSk8obK9vjUjHm.eval,https://github.com/epoch-research/benchmarks/actions/runs/18556527593/job/52896388442,16 October 2025 5:34am,Success,,1.0.0
f2vxirYMpXwkaZxb4ywnYM,MATH level 5,qwen3-max-2025-09-23,0.971,"normalized_string_match:0.92±0.01,sympy_equiv:0.92±0.01,model_graded_equiv:0.97±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/f2vxirYMpXwkaZxb4ywnYM.eval,https://github.com/epoch-research/benchmarks/actions/runs/18380028225/job/52364172987,9 October 2025 11:30am,Success,,1.0.0
VshpC5TEp4F72cutXRRmhs,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-pro-2025-10-06_high,0.125,verification_code:0.13±0.05,,,,9 October 2025 9:40am,Success,,1.0.0
mH5viJmAywMMTHg4j4ehkZ,FrontierMath-2025-02-28-Public,gpt-5-pro-2025-10-06_high,0.600,verification_code:0.6±0.16,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmH5viJmAywMMTHg4j4ehkZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mH5viJmAywMMTHg4j4ehkZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/18341756653/job/52238805153,8 October 2025 6:32am,Success,,1.0.0
46RCssznDADYeg8MuSgvsj,GPQA diamond,qwen3-max-2025-09-23,0.726,choice:0.73±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F46RCssznDADYeg8MuSgvsj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/46RCssznDADYeg8MuSgvsj.eval,https://github.com/epoch-research/benchmarks/actions/runs/18284288413/job/52055144510,6 October 2025 12:27pm,Success,,1.0.0
ULeuPtwp3ZoyuXQYuuQ82i,OTIS Mock AIME 2024-2025,qwen3-max-2025-09-23,0.733,"model_extracted_exact_match:0.73±0.06,model_graded:0.73±0.06",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FULeuPtwp3ZoyuXQYuuQ82i.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ULeuPtwp3ZoyuXQYuuQ82i.eval,https://github.com/epoch-research/benchmarks/actions/runs/18284288413/job/52055144460,6 October 2025 10:32am,Success,,1.0.0
Kav47MyxqkJPddWwDDoC8J,FrontierMath-2025-02-28-Public,claude-sonnet-4-5-20250929,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FKav47MyxqkJPddWwDDoC8J.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Kav47MyxqkJPddWwDDoC8J.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147149,29 September 2025 5:03pm,Success,,1.0.0
iZoPmtRrd5k6qnrRraiLZo,SWE-Bench verified,claude-sonnet-4-5-20250929,0.648,swe_bench_scorer:0.65±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FiZoPmtRrd5k6qnrRraiLZo.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/iZoPmtRrd5k6qnrRraiLZo.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147117,29 September 2025 2:41pm,Success,,1.0.0
dEVgttSw8u6Tt2RsPWTc3C,OTIS Mock AIME 2024-2025,claude-sonnet-4-5-20250929,0.356,"model_extracted_exact_match:0.36±0.07,model_graded:0.36±0.07",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdEVgttSw8u6Tt2RsPWTc3C.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dEVgttSw8u6Tt2RsPWTc3C.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147127,29 September 2025 2:39pm,Success,,1.0.0
fsZmzw8jjyUgMQGwxodBwN,GPQA diamond,claude-sonnet-4-5-20250929,0.737,choice:0.74±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfsZmzw8jjyUgMQGwxodBwN.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fsZmzw8jjyUgMQGwxodBwN.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147139,29 September 2025 2:04pm,Success,,1.0.0
2o8XZ8V6W5jKD7TuY4frux,FrontierMath-Tier-4-2025-07-01-Private,claude-sonnet-4-5-20250929,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/2o8XZ8V6W5jKD7TuY4frux.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147128,29 September 2025 1:41pm,Success,,1.0.0
D7RACzWb6yDDgo4e4WBfpP,FrontierMath-Tier-4-2025-07-01-Public,claude-sonnet-4-5-20250929,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FD7RACzWb6yDDgo4e4WBfpP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/D7RACzWb6yDDgo4e4WBfpP.eval,https://github.com/epoch-research/benchmarks/actions/runs/18105243746/job/51518147113,29 September 2025 1:33pm,Success,,1.0.0
2VVQUtzK8sSDs4FAFZnWWB,SWE-Bench verified,DeepSeek-V3.1,0.521,swe_bench_scorer:0.52±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2VVQUtzK8sSDs4FAFZnWWB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2VVQUtzK8sSDs4FAFZnWWB.eval,https://github.com/epoch-research/benchmarks/actions/runs/17561256390/job/49878087464,8 September 2025 3:07pm,Success,,1.0.0
7T8enYY2e5AwhWLxFY7cP5,FrontierMath-Tier-4-2025-07-01-Public,glm-4.5,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7T8enYY2e5AwhWLxFY7cP5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7T8enYY2e5AwhWLxFY7cP5.eval,https://github.com/epoch-research/benchmarks/actions/runs/17561087786/job/49877498284,8 September 2025 3:00pm,Success,,1.0.0
ewXZ7Lgi46TVYbg6fibkrT,FrontierMath-2025-02-28-Public,glm-4.5,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FewXZ7Lgi46TVYbg6fibkrT.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ewXZ7Lgi46TVYbg6fibkrT.eval,https://github.com/epoch-research/benchmarks/actions/runs/17555065793/job/49857189251,8 September 2025 11:02am,Success,,1.0.0
eDzsqj2CnVxbkp5JRuULXF,MATH level 5,gpt-5-2025-08-07_medium,0.979,"normalized_string_match:0.89±0.01,sympy_equiv:0.91±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/eDzsqj2CnVxbkp5JRuULXF.eval,https://github.com/epoch-research/benchmarks/actions/runs/17097003179/job/48483763824,20 August 2025 7:28am,Success,,1.0.0
SSLfQCm7RvLhSv5aPKaLNk,MATH level 5,gpt-5-nano-2025-08-07_medium,0.952,"normalized_string_match:0.86±0.01,sympy_equiv:0.88±0.01,model_graded_equiv:0.95±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/SSLfQCm7RvLhSv5aPKaLNk.eval,https://github.com/epoch-research/benchmarks/actions/runs/17097003179/job/48483763823,20 August 2025 7:28am,Success,,1.0.0
TYQuXJw9H8NRjSovr5VoBn,MATH level 5,gpt-5-nano-2025-08-07_high,0.949,"normalized_string_match:0.86±0.01,sympy_equiv:0.88±0.01,model_graded_equiv:0.95±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/TYQuXJw9H8NRjSovr5VoBn.eval,https://github.com/epoch-research/benchmarks/actions/runs/17097003179/job/48483763817,20 August 2025 7:28am,Success,,1.0.0
3NYyCGnM6XhgytjyVGtwh8,MATH level 5,gpt-5-mini-2025-08-07_medium,0.968,"normalized_string_match:0.88±0.01,sympy_equiv:0.91±0.01,model_graded_equiv:0.97±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/3NYyCGnM6XhgytjyVGtwh8.eval,https://github.com/epoch-research/benchmarks/actions/runs/17097003179/job/48483763842,20 August 2025 7:28am,Success,,1.0.0
5bSiKbVpt5fDXcBEf7YS68,SWE-Bench verified,o3-2025-04-16_medium,0.437,swe_bench_scorer:0.44±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F5bSiKbVpt5fDXcBEf7YS68.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/5bSiKbVpt5fDXcBEf7YS68.eval,https://github.com/epoch-research/benchmarks/actions/runs/16920074356/job/47943801636,12 August 2025 4:28pm,Success,,1.0.0
dR2xMMN4mNQQeK9iutdRup,FrontierMath-Tier-4-2025-07-01-Public,grok-4-0709,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdR2xMMN4mNQQeK9iutdRup.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dR2xMMN4mNQQeK9iutdRup.eval,https://github.com/epoch-research/benchmarks/actions/runs/16880374180/job/47814596116,11 August 2025 8:47am,Success,,1.0.0
Cqdzvvh9tdrdmDjvu8bntM,FrontierMath-Tier-4-2025-07-01-Private,grok-4-0709,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Cqdzvvh9tdrdmDjvu8bntM.eval,https://github.com/epoch-research/benchmarks/actions/runs/16880374180/job/47814596105,11 August 2025 8:47am,Success,,1.0.0
ccQHpj8RWNTDXn5qG7azw9,FrontierMath-Tier-4-2025-07-01-Public,o4-mini-2025-04-16_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FccQHpj8RWNTDXn5qG7azw9.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ccQHpj8RWNTDXn5qG7azw9.eval,https://github.com/epoch-research/benchmarks/actions/runs/16816727327/job/47635266551,7 August 2025 6:00pm,Success,,1.0.0
oBjz3oZSvAkw5ArLtZMwWW,FrontierMath-Tier-4-2025-07-01-Private,o4-mini-2025-04-16_medium,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/oBjz3oZSvAkw5ArLtZMwWW.eval,https://github.com/epoch-research/benchmarks/actions/runs/16816727327/job/47635266534,7 August 2025 6:00pm,Success,,1.0.0
ctsGxm6WkzCGEqiAbJR3kk,OTIS Mock AIME 2024-2025,gpt-5-2025-08-07_medium,0.872,model_graded:0.87±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FctsGxm6WkzCGEqiAbJR3kk.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ctsGxm6WkzCGEqiAbJR3kk.eval,https://github.com/epoch-research/benchmarks/actions/runs/16816727327/job/47635266548,7 August 2025 5:58pm,Success,,1.0.0
PazJb6CuZfUTKf6dtz5Z45,SWE-Bench verified,gpt-4o-2024-11-20,0.254,swe_bench_scorer:0.25±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPazJb6CuZfUTKf6dtz5Z45.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PazJb6CuZfUTKf6dtz5Z45.eval,https://github.com/epoch-research/benchmarks/actions/runs/16814527661/job/47627969376,7 August 2025 4:02pm,Success,,1.0.0
RRbQkGCVhZCeRDt8kB27TB,SWE-Bench verified,gpt-5-mini-2025-08-07_medium,0.578,swe_bench_scorer:0.58±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRRbQkGCVhZCeRDt8kB27TB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/RRbQkGCVhZCeRDt8kB27TB.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165348,7 August 2025 3:36pm,Success,,1.0.0
65DJZN2XzPw4RGpa4TCfA5,OTIS Mock AIME 2024-2025,gpt-5-mini-2025-08-07_medium,0.783,model_graded:0.78±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F65DJZN2XzPw4RGpa4TCfA5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/65DJZN2XzPw4RGpa4TCfA5.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165337,7 August 2025 3:34pm,Success,,1.0.0
CFNzodoTSVVVgp9PUXftUP,GPQA diamond,gpt-5-mini-2025-08-07_medium,0.717,choice:0.72±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FCFNzodoTSVVVgp9PUXftUP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/CFNzodoTSVVVgp9PUXftUP.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165358,7 August 2025 3:34pm,Success,,1.0.0
YQCwJRoGbGAqNCQoaL6cjk,GPQA diamond,gpt-5-2025-08-07_medium,0.854,choice:0.85±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FYQCwJRoGbGAqNCQoaL6cjk.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/YQCwJRoGbGAqNCQoaL6cjk.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165333,7 August 2025 3:34pm,Success,,1.0.0
XRFS8R4WV7q6RMv5keEAdM,OTIS Mock AIME 2024-2025,gpt-5-nano-2025-08-07_medium,0.742,model_graded:0.74±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FXRFS8R4WV7q6RMv5keEAdM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/XRFS8R4WV7q6RMv5keEAdM.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165351,7 August 2025 3:34pm,Success,,1.0.0
ELAxkvAigvEVdMmhxiaMiy,GPQA diamond,gpt-5-nano-2025-08-07_medium,0.674,choice:0.67±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FELAxkvAigvEVdMmhxiaMiy.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ELAxkvAigvEVdMmhxiaMiy.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165327,7 August 2025 3:34pm,Success,,1.0.0
knygsTpEGHyBFW4iSAWzn6,SWE-Bench verified,gpt-5-2025-08-07_medium,0.588,swe_bench_scorer:0.59±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FknygsTpEGHyBFW4iSAWzn6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/knygsTpEGHyBFW4iSAWzn6.eval,https://github.com/epoch-research/benchmarks/actions/runs/16813997887/job/47626165318,7 August 2025 3:34pm,Success,,1.0.0
7TaaSrTfJMs9HhA9Co5E3n,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-mini-2025-08-07_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7TaaSrTfJMs9HhA9Co5E3n.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7TaaSrTfJMs9HhA9Co5E3n.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577862,7 August 2025 2:43pm,Success,,1.0.0
WLygZuaoy2rPgu5f2hxqMD,FrontierMath-2025-02-28-Public,gpt-5-nano-2025-08-07_medium,0.100,verification_code:0.1±0.1,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWLygZuaoy2rPgu5f2hxqMD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WLygZuaoy2rPgu5f2hxqMD.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577852,7 August 2025 2:43pm,Success,,1.0.0
RrmHCttQmYWNaADyThwFwf,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-2025-08-07_medium,0.063,verification_code:0.06±0.04,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/RrmHCttQmYWNaADyThwFwf.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577802,7 August 2025 2:43pm,Success,,1.0.0
EAqUKKCdvdkYTGegACdean,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-mini-2025-08-07_medium,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/EAqUKKCdvdkYTGegACdean.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577806,7 August 2025 2:43pm,Success,,1.0.0
3Nan43on9XRt8wUbs3iEEh,FrontierMath-2025-02-28-Public,gpt-5-mini-2025-08-07_medium,0.400,verification_code:0.4±0.16,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3Nan43on9XRt8wUbs3iEEh.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3Nan43on9XRt8wUbs3iEEh.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577834,7 August 2025 2:43pm,Success,,1.0.0
F6Tw2b9fDKde6N9eBu6ory,FrontierMath-Tier-4-2025-07-01-Private,gpt-5-nano-2025-08-07_medium,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/F6Tw2b9fDKde6N9eBu6ory.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577812,7 August 2025 2:43pm,Success,,1.0.0
62EDqhG4Nw8xe5GTTnbLcF,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-nano-2025-08-07_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F62EDqhG4Nw8xe5GTTnbLcF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/62EDqhG4Nw8xe5GTTnbLcF.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577803,7 August 2025 2:43pm,Success,,1.0.0
e5BDYMfFF2SGHirAWRsCYV,FrontierMath-2025-02-28-Public,gpt-5-2025-08-07_medium,0.700,verification_code:0.7±0.15,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fe5BDYMfFF2SGHirAWRsCYV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/e5BDYMfFF2SGHirAWRsCYV.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577879,7 August 2025 2:43pm,Success,,1.0.0
JXb2vjUSfytHjZXZcgGrxe,FrontierMath-Tier-4-2025-07-01-Public,gpt-5-2025-08-07_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJXb2vjUSfytHjZXZcgGrxe.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JXb2vjUSfytHjZXZcgGrxe.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577829,7 August 2025 2:43pm,Success,,1.0.0
dHQzrNGjuR5kiHKtNxG62X,FrontierMath-2025-02-28-Private,gpt-5-nano-2025-08-07_medium,0.072,verification_code:0.07±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/dHQzrNGjuR5kiHKtNxG62X.eval,https://github.com/epoch-research/benchmarks/actions/runs/16812922265/job/47622577823,7 August 2025 2:43pm,Success,,1.0.0
ZKroUnGN6qoNssoSQbtzzz,FrontierMath-2025-02-28-Private,claude-opus-4-20250514_27K,0.041,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZKroUnGN6qoNssoSQbtzzz.eval,https://github.com/epoch-research/benchmarks/actions/runs/16763372200/job/47463603822,5 August 2025 7:13pm,Success,,1.0.0
a6VpLAmwU23BmxsqrZTGZg,FrontierMath-Tier-4-2025-07-01-Private,claude-opus-4-1-20250805_27K,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/a6VpLAmwU23BmxsqrZTGZg.eval,https://github.com/epoch-research/benchmarks/actions/runs/16762492946/job/47460893359,5 August 2025 6:14pm,Success,,1.0.0
ezJ5C38svWJkQMjxTcrgg7,FrontierMath-2025-02-28-Private,claude-opus-4-1-20250805_27K,0.072,verification_code:0.07±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ezJ5C38svWJkQMjxTcrgg7.eval,https://github.com/epoch-research/benchmarks/actions/runs/16759902132/job/47452259708,5 August 2025 3:56pm,Success,,1.0.0
TQQnqmugDk3GX5hkr8HMWz,FrontierMath-2025-02-28-Private,claude-opus-4-1-20250805,0.059,verification_code:0.06±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/TQQnqmugDk3GX5hkr8HMWz.eval,https://github.com/epoch-research/benchmarks/actions/runs/16759902132/job/47452259709,5 August 2025 3:56pm,Success,,1.0.0
J86NTPTbMV6yt2Ezp8yB9n,FrontierMath-2025-02-28-Public,claude-opus-4-1-20250805_16K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJ86NTPTbMV6yt2Ezp8yB9n.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/J86NTPTbMV6yt2Ezp8yB9n.eval,https://github.com/epoch-research/benchmarks/actions/runs/16759845864/job/47452068031,5 August 2025 3:55pm,Success,,1.0.0
Bhwtuh9ZRhWn2aDbq2juLU,FrontierMath-2025-02-28-Public,claude-opus-4-1-20250805_27K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBhwtuh9ZRhWn2aDbq2juLU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Bhwtuh9ZRhWn2aDbq2juLU.eval,https://github.com/epoch-research/benchmarks/actions/runs/16759845864/job/47452068035,5 August 2025 3:53pm,Success,,1.0.0
UH9RY8MCAotDgFocHKA4j6,OTIS Mock AIME 2024-2025,claude-opus-4-1-20250805_27K,0.689,model_graded:0.69±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FUH9RY8MCAotDgFocHKA4j6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/UH9RY8MCAotDgFocHKA4j6.eval,https://github.com/epoch-research/benchmarks/actions/runs/16756085155/job/47439039061,5 August 2025 12:53pm,Success,,1.0.0
3obcsGCfuwYeEm9wRyqzPg,GPQA diamond,claude-opus-4-1-20250805_27K,0.768,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3obcsGCfuwYeEm9wRyqzPg.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3obcsGCfuwYeEm9wRyqzPg.eval,https://github.com/epoch-research/benchmarks/actions/runs/16756085155/job/47439039070,5 August 2025 12:53pm,Success,,1.0.0
erLgyKQs2mg2memE9SJvK5,GPQA diamond,claude-opus-4-1-20250805_16K,0.773,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FerLgyKQs2mg2memE9SJvK5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/erLgyKQs2mg2memE9SJvK5.eval,https://github.com/epoch-research/benchmarks/actions/runs/16756046108/job/47438910815,5 August 2025 12:51pm,Success,,1.0.0
kf9vy9sMjMBBswrX6NMzce,OTIS Mock AIME 2024-2025,claude-opus-4-1-20250805_16K,0.644,model_graded:0.64±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fkf9vy9sMjMBBswrX6NMzce.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/kf9vy9sMjMBBswrX6NMzce.eval,https://github.com/epoch-research/benchmarks/actions/runs/16756046108/job/47438910805,5 August 2025 12:51pm,Success,,1.0.0
9CTibDqrsrcQzMuFTaCZem,SWE-Bench verified,claude-opus-4-1-20250805,0.632,swe_bench_scorer:0.63±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9CTibDqrsrcQzMuFTaCZem.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9CTibDqrsrcQzMuFTaCZem.eval,https://github.com/epoch-research/benchmarks/actions/runs/16755796636/job/47438446413,5 August 2025 12:47pm,Success,,1.0.0
e9KxzZaLsArQ3cEiHuv2T6,FrontierMath-2025-02-28-Public,claude-opus-4-1-20250805,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fe9KxzZaLsArQ3cEiHuv2T6.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/e9KxzZaLsArQ3cEiHuv2T6.eval,https://github.com/epoch-research/benchmarks/actions/runs/16755796636/job/47438446415,5 August 2025 12:47pm,Success,,1.0.0
RaiKuDSzUJYDVMxn4jfzNC,OTIS Mock AIME 2024-2025,claude-opus-4-1-20250805,0.400,model_graded:0.4±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRaiKuDSzUJYDVMxn4jfzNC.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/RaiKuDSzUJYDVMxn4jfzNC.eval,https://github.com/epoch-research/benchmarks/actions/runs/16755796636/job/47438446398,5 August 2025 12:45pm,Success,,1.0.0
j62bkFDjBSeFyVpnGzuzg4,GPQA diamond,claude-opus-4-1-20250805,0.732,choice:0.73±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fj62bkFDjBSeFyVpnGzuzg4.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/j62bkFDjBSeFyVpnGzuzg4.eval,https://github.com/epoch-research/benchmarks/actions/runs/16755796636/job/47438446419,5 August 2025 12:45pm,Success,,1.0.0
MPJkYPEMsQrowqtecK3mmz,OTIS Mock AIME 2024-2025,gpt-4o-mini-2024-07-18,0.069,model_graded:0.07±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMPJkYPEMsQrowqtecK3mmz.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MPJkYPEMsQrowqtecK3mmz.eval,https://github.com/epoch-research/benchmarks/actions/runs/16627279155/job/47047216166,30 July 2025 11:47am,Success,,1.0.0
GKD2ELzaXHx3giARevEetr,FrontierMath-2025-02-28-Private,claude-opus-4-20250514,0.045,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/GKD2ELzaXHx3giARevEetr.eval,https://github.com/epoch-research/benchmarks/actions/runs/16071901181/job/45358309095,4 July 2025 6:42am,Success,,1.0.0
UcfxxJmo4ftRzPEnXrRdSw,FrontierMath-2025-02-28-Private,claude-sonnet-4-20250514,0.041,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/UcfxxJmo4ftRzPEnXrRdSw.eval,https://github.com/epoch-research/benchmarks/actions/runs/16071901181/job/45358309089,4 July 2025 6:41am,Success,,1.0.0
d8ktep2eycxDywk4JjNXeA,FrontierMath-Tier-4-2025-07-01-Public,gemini-2.5-pro,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fd8ktep2eycxDywk4JjNXeA.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/d8ktep2eycxDywk4JjNXeA.eval,https://github.com/epoch-research/benchmarks/actions/runs/16054824276/job/45306539766,3 July 2025 11:44am,Success,,1.0.0
g7m7nM2iRMskxAUbyrV766,FrontierMath-2025-02-28-Public,gemini-2.5-pro,0.400,verification_code:0.4±0.16,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fg7m7nM2iRMskxAUbyrV766.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/g7m7nM2iRMskxAUbyrV766.eval,https://github.com/epoch-research/benchmarks/actions/runs/16054824276/job/45306539757,3 July 2025 11:44am,Success,,1.0.0
XtEW42hsVgroBKHK465wqA,FrontierMath-Tier-4-2025-07-01-Private,gemini-2.5-pro,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XtEW42hsVgroBKHK465wqA.eval,https://github.com/epoch-research/benchmarks/actions/runs/16054824276/job/45306539762,3 July 2025 11:44am,Success,,1.0.0
HN7YDEAzGvPRMBDo7mGuXG,FrontierMath-2025-02-28-Private,gemini-2.5-pro,0.110,verification_code:0.11±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/HN7YDEAzGvPRMBDo7mGuXG.eval,https://github.com/epoch-research/benchmarks/actions/runs/16054824276/job/45306539774,3 July 2025 11:43am,Success,,1.0.0
hNLQNC6TASXNNzat8kVS77,FrontierMath-2025-02-28-Private,gemini-2.5-pro-preview-06-05,0.103,verification_code:0.1±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/hNLQNC6TASXNNzat8kVS77.eval,https://github.com/epoch-research/benchmarks/actions/runs/16052348226/job/45298044671,3 July 2025 9:57am,Success,,1.0.0
iwLqUvjgt378uQoKiCdSXG,FrontierMath-Tier-4-2025-07-01-Private,gemini-2.5-pro-preview-06-05,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/iwLqUvjgt378uQoKiCdSXG.eval,https://github.com/epoch-research/benchmarks/actions/runs/16052348226/job/45298044785,3 July 2025 9:56am,Success,,1.0.0
DVgoky7pTLzriyfcj3oiTC,FrontierMath-2025-02-28-Public,gemini-2.5-pro-preview-06-05,0.300,verification_code:0.3±0.15,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FDVgoky7pTLzriyfcj3oiTC.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/DVgoky7pTLzriyfcj3oiTC.eval,https://github.com/epoch-research/benchmarks/actions/runs/16052348226/job/45298044696,3 July 2025 9:56am,Success,,1.0.0
3DgJz7t3iDQFuXMPrC3Jdc,FrontierMath-Tier-4-2025-07-01-Public,gemini-2.5-pro-preview-06-05,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3DgJz7t3iDQFuXMPrC3Jdc.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3DgJz7t3iDQFuXMPrC3Jdc.eval,https://github.com/epoch-research/benchmarks/actions/runs/16052348226/job/45298044663,3 July 2025 9:55am,Success,,1.0.0
oDrEMnoqz3AapWeyrxZd6C,FrontierMath-Tier-4-2025-07-01-Private,gpt-4.1-2025-04-14,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/oDrEMnoqz3AapWeyrxZd6C.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437549,1 July 2025 1:05pm,Success,,1.0.0
MUYxn78tdpBMLETSQg5rgN,FrontierMath-Tier-4-2025-07-01-Private,claude-opus-4-20250514,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/MUYxn78tdpBMLETSQg5rgN.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437473,1 July 2025 1:05pm,Success,,1.0.0
WfXvqfvSMcUkzC3Zuuw4vc,FrontierMath-Tier-4-2025-07-01-Public,claude-sonnet-4-20250514,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWfXvqfvSMcUkzC3Zuuw4vc.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WfXvqfvSMcUkzC3Zuuw4vc.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437504,1 July 2025 1:05pm,Success,,1.0.0
a9fSMpBem4uhc6pxuEiQqM,FrontierMath-Tier-4-2025-07-01-Private,o3-2025-04-16_high,0.021,verification_code:0.02±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/a9fSMpBem4uhc6pxuEiQqM.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437454,1 July 2025 1:05pm,Success,,1.0.0
hsc7UqWcaQWLPYMpiE4oPP,FrontierMath-Tier-4-2025-07-01-Public,grok-3-mini-beta_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fhsc7UqWcaQWLPYMpiE4oPP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/hsc7UqWcaQWLPYMpiE4oPP.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437519,1 July 2025 1:05pm,Success,,1.0.0
LJJ5HXgW6FD8EzRLUuXpnD,FrontierMath-Tier-4-2025-07-01-Public,claude-3-5-sonnet-20241022,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FLJJ5HXgW6FD8EzRLUuXpnD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/LJJ5HXgW6FD8EzRLUuXpnD.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437572,1 July 2025 1:05pm,Success,,1.0.0
ELsuPViQBj3UPywy6WyB5H,FrontierMath-Tier-4-2025-07-01-Private,claude-3-5-sonnet-20240620,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ELsuPViQBj3UPywy6WyB5H.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437551,1 July 2025 1:05pm,Success,,1.0.0
7Ws2sNVbNKhjB2a2G3rtkK,FrontierMath-Tier-4-2025-07-01-Public,claude-opus-4-20250514_27K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7Ws2sNVbNKhjB2a2G3rtkK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7Ws2sNVbNKhjB2a2G3rtkK.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437503,1 July 2025 1:05pm,Success,,1.0.0
YZ6Ts7tSVrvzjSRGxJowdn,FrontierMath-Tier-4-2025-07-01-Private,o3-mini-2025-01-31_high,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YZ6Ts7tSVrvzjSRGxJowdn.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437501,1 July 2025 1:05pm,Success,,1.0.0
Ee6YnzHKyAUxmYeL23TL9s,FrontierMath-Tier-4-2025-07-01-Private,claude-opus-4-20250514_27K,0.042,verification_code:0.04±0.03,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Ee6YnzHKyAUxmYeL23TL9s.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437507,1 July 2025 1:05pm,Success,,1.0.0
dnaCP4is3bimt8yUUwffH3,FrontierMath-Tier-4-2025-07-01-Public,claude-3-5-sonnet-20240620,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdnaCP4is3bimt8yUUwffH3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dnaCP4is3bimt8yUUwffH3.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437446,1 July 2025 1:05pm,Success,,1.0.0
MgFPM3V8RwNxYnJdCZ8v3k,FrontierMath-Tier-4-2025-07-01-Public,claude-sonnet-4-20250514_59K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMgFPM3V8RwNxYnJdCZ8v3k.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MgFPM3V8RwNxYnJdCZ8v3k.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437525,1 July 2025 1:05pm,Success,,1.0.0
8mhf6cwsHtrRkAwxZSBgfV,FrontierMath-Tier-4-2025-07-01-Private,claude-3-5-sonnet-20241022,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/8mhf6cwsHtrRkAwxZSBgfV.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437466,1 July 2025 1:05pm,Success,,1.0.0
jHzq79NimF4kneemSx5aAb,FrontierMath-Tier-4-2025-07-01-Public,claude-3-7-sonnet-20250219_64K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjHzq79NimF4kneemSx5aAb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jHzq79NimF4kneemSx5aAb.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437479,1 July 2025 1:05pm,Success,,1.0.0
BfYjXxmPCAHe5bCSaoSTrf,FrontierMath-Tier-4-2025-07-01-Public,gpt-4.1-2025-04-14,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBfYjXxmPCAHe5bCSaoSTrf.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/BfYjXxmPCAHe5bCSaoSTrf.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437517,1 July 2025 1:05pm,Success,,1.0.0
WgEkfyLvjHYXfPje5Z2CPX,FrontierMath-Tier-4-2025-07-01-Private,o4-mini-2025-04-16_high,0.063,verification_code:0.06±0.04,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/WgEkfyLvjHYXfPje5Z2CPX.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437448,1 July 2025 1:05pm,Success,,1.0.0
f3pUdKEoUSRsrGBbHNMnis,FrontierMath-Tier-4-2025-07-01-Public,o3-mini-2025-01-31_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Ff3pUdKEoUSRsrGBbHNMnis.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/f3pUdKEoUSRsrGBbHNMnis.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437468,1 July 2025 1:05pm,Success,,1.0.0
d8zmtr2kCW7EdrgrsX7hAj,FrontierMath-Tier-4-2025-07-01-Public,DeepSeek-R1-0528,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fd8zmtr2kCW7EdrgrsX7hAj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/d8zmtr2kCW7EdrgrsX7hAj.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437480,1 July 2025 1:05pm,Success,,1.0.0
ZyHLq7azxVkZJZRLTj3kMB,FrontierMath-Tier-4-2025-07-01-Private,grok-3-beta,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZyHLq7azxVkZJZRLTj3kMB.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437442,1 July 2025 1:05pm,Success,,1.0.0
7FCAS2h9fcpPZenwBZBs2X,FrontierMath-Tier-4-2025-07-01-Public,grok-3-beta,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7FCAS2h9fcpPZenwBZBs2X.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7FCAS2h9fcpPZenwBZBs2X.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437451,1 July 2025 1:05pm,Success,,1.0.0
XMYUtk8tkqDFgohY8iYfmy,FrontierMath-Tier-4-2025-07-01-Private,claude-sonnet-4-20250514,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XMYUtk8tkqDFgohY8iYfmy.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437542,1 July 2025 1:04pm,Success,,1.0.0
m2HBT8LbzJ3jVsjY7vEkiY,FrontierMath-Tier-4-2025-07-01-Public,claude-opus-4-20250514,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fm2HBT8LbzJ3jVsjY7vEkiY.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/m2HBT8LbzJ3jVsjY7vEkiY.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437440,1 July 2025 1:04pm,Success,,1.0.0
AZPLK4bMcWu4nwtVJrc2NQ,FrontierMath-Tier-4-2025-07-01-Public,o4-mini-2025-04-16_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FAZPLK4bMcWu4nwtVJrc2NQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/AZPLK4bMcWu4nwtVJrc2NQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437554,1 July 2025 1:04pm,Success,,1.0.0
WRNa5qHKxVsn9tEmPssZiP,FrontierMath-Tier-4-2025-07-01-Public,o3-2025-04-16_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWRNa5qHKxVsn9tEmPssZiP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WRNa5qHKxVsn9tEmPssZiP.eval,https://github.com/epoch-research/benchmarks/actions/runs/16005581896/job/45151437434,1 July 2025 1:04pm,Success,,1.0.0
mJsWT3i6UHaxDmyy8vEDqq,OTIS Mock AIME 2024-2025,magistral-small-2506,0.300,model_graded:0.3±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmJsWT3i6UHaxDmyy8vEDqq.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mJsWT3i6UHaxDmyy8vEDqq.eval,https://github.com/epoch-research/benchmarks/actions/runs/15562834298/job/43819432438,10 June 2025 10:53am,Success,,1.0.0
5hAnRpZh5ZZubbPgrrcm6D,GPQA diamond,magistral-small-2506,0.484,choice:0.48±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F5hAnRpZh5ZZubbPgrrcm6D.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/5hAnRpZh5ZZubbPgrrcm6D.eval,https://github.com/epoch-research/benchmarks/actions/runs/15562834298/job/43819432453,10 June 2025 10:53am,Success,,1.0.0
76hqeNUL93NiB5SHQfFwrw,FrontierMath-2025-02-28-Public,magistral-small-2506,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F76hqeNUL93NiB5SHQfFwrw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/76hqeNUL93NiB5SHQfFwrw.eval,https://github.com/epoch-research/benchmarks/actions/runs/15562834298/job/43819432429,10 June 2025 10:53am,Success,,1.0.0
Q4qTzHnjLyiCCRvNpXioUq,GPQA diamond,gemini-2.5-pro-preview-06-05,0.848,choice:0.85±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQ4qTzHnjLyiCCRvNpXioUq.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Q4qTzHnjLyiCCRvNpXioUq.eval,https://github.com/epoch-research/benchmarks/actions/runs/15471871305/job/43558158589,5 June 2025 12:07pm,Success,,1.0.0
BTDXT8mHQu4je5jysTDuXV,MATH level 5,qwen3-235b-a22b,0.689,"normalized_string_match:0.66±0.01,sympy_equiv:0.64±0.01,model_graded_equiv:0.69±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/BTDXT8mHQu4je5jysTDuXV.eval,https://github.com/epoch-research/benchmarks/actions/runs/15421560516/job/43397824372,3 June 2025 6:33pm,Success,,1.0.0
FnqDHcmXj3RWTTpoQJuXNq,FrontierMath-2025-02-28-Public,qwen3-235b-a22b,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFnqDHcmXj3RWTTpoQJuXNq.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FnqDHcmXj3RWTTpoQJuXNq.eval,https://github.com/epoch-research/benchmarks/actions/runs/15421139320/job/43396372014,3 June 2025 11:16am,Success,,1.0.0
agebmxGseTvmvN4AHN8i4m,GPQA diamond,qwen3-235b-a22b,0.707,choice:0.71±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FagebmxGseTvmvN4AHN8i4m.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/agebmxGseTvmvN4AHN8i4m.eval,https://github.com/epoch-research/benchmarks/actions/runs/15416521652/job/43380560268,3 June 2025 7:52am,Success,,1.0.0
TT66gGiVoeYLGQMCESPrEG,GPQA diamond,gemini-2.5-pro-preview-05-06,0.667,choice:0.67±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTT66gGiVoeYLGQMCESPrEG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TT66gGiVoeYLGQMCESPrEG.eval,https://github.com/epoch-research/benchmarks/actions/runs/15413225529/job/43369994537,3 June 2025 5:10am,Success,,1.0.0
H4X3AsrkcXMTQNKdrAgJe7,SWE-Bench verified,DeepSeek-R1-0528,0.333,swe_bench_scorer:0.33±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FH4X3AsrkcXMTQNKdrAgJe7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/H4X3AsrkcXMTQNKdrAgJe7.eval,https://github.com/epoch-research/benchmarks/actions/runs/15331211840/job/43138274443,29 May 2025 2:54pm,Success,,1.0.0
2wd9khVKiBzsAMUquz2R5s,MATH level 5,DeepSeek-R1-0528,0.966,"normalized_string_match:0.92±0.01,sympy_equiv:0.92±0.01,model_graded_equiv:0.97±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/2wd9khVKiBzsAMUquz2R5s.eval,https://github.com/epoch-research/benchmarks/actions/runs/15322126965/job/43108090267,29 May 2025 10:09am,Success,,1.0.0
LiRiryJsGb3h5UuxX8bmJv,GPQA diamond,DeepSeek-R1-0528,0.763,choice:0.76±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FLiRiryJsGb3h5UuxX8bmJv.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/LiRiryJsGb3h5UuxX8bmJv.eval,https://github.com/epoch-research/benchmarks/actions/runs/15322126965/job/43108090260,29 May 2025 9:30am,Success,,1.0.0
bBV5wCrrZtihrgxEsg9jPD,OTIS Mock AIME 2024-2025,DeepSeek-R1-0528,0.664,model_graded:0.66±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbBV5wCrrZtihrgxEsg9jPD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/bBV5wCrrZtihrgxEsg9jPD.eval,https://github.com/epoch-research/benchmarks/actions/runs/15322126965/job/43108090269,29 May 2025 6:59am,Success,,1.0.0
VrRccga4uRVuCjNp637MeY,OTIS Mock AIME 2024-2025,claude-opus-4-20250514_27K,0.644,model_graded:0.64±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVrRccga4uRVuCjNp637MeY.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/VrRccga4uRVuCjNp637MeY.eval,https://github.com/epoch-research/benchmarks/actions/runs/15307994505/job/43065511718,28 May 2025 2:42pm,Success,,1.0.0
6fbE9sokzKiRhNPfRnVnZA,SWE-Bench verified,claude-opus-4-20250514,0.622,swe_bench_scorer:0.62±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F6fbE9sokzKiRhNPfRnVnZA.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/6fbE9sokzKiRhNPfRnVnZA.eval,https://github.com/epoch-research/benchmarks/actions/runs/15283754649/job/42988924577,27 May 2025 3:16pm,Success,,1.0.0
2t26VSNdsAE5pefQEMP25x,GPQA diamond,claude-3-7-sonnet-20250219_64K,0.797,"choice:0.77±0.03,choice:0.8±0.02",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2t26VSNdsAE5pefQEMP25x.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2t26VSNdsAE5pefQEMP25x.eval,https://github.com/epoch-research/benchmarks/actions/runs/15263076164/job/42924152756,26 May 2025 6:53pm,Success,,1.0.0
JFZMWFVGNi5R5TdZrvQUFw,GPQA diamond,grok-3-mini-beta_high,0.755,"choice:0.74±0.03,choice:0.76±0.02",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJFZMWFVGNi5R5TdZrvQUFw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JFZMWFVGNi5R5TdZrvQUFw.eval,https://github.com/epoch-research/benchmarks/actions/runs/15260145619/job/42916363880,26 May 2025 2:38pm,Success,,1.0.0
fFGyQR2mfDufvrSV2rzWyV,GPQA diamond,DeepSeek-R1,0.717,"choice:0.72±0.03,choice:0.67±0.03",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfFGyQR2mfDufvrSV2rzWyV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fFGyQR2mfDufvrSV2rzWyV.eval,https://github.com/epoch-research/benchmarks/actions/runs/15260145619/job/42916363888,26 May 2025 2:38pm,Success,,1.0.0
J9wjcQqKJ7hCSYnnoa5E9i,GPQA diamond,claude-sonnet-4-20250514_59K,0.792,"choice:0.79±0.03,choice:0.76±0.02",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJ9wjcQqKJ7hCSYnnoa5E9i.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/J9wjcQqKJ7hCSYnnoa5E9i.eval,https://github.com/epoch-research/benchmarks/actions/runs/15260145619/job/42916363885,26 May 2025 2:38pm,Success,,1.0.0
FQ8Ykq4JWRSXnyF9rbmLc7,GPQA diamond,grok-3-beta,0.758,"choice:0.76±0.03,choice:0.59±0.02",https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFQ8Ykq4JWRSXnyF9rbmLc7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FQ8Ykq4JWRSXnyF9rbmLc7.eval,https://github.com/epoch-research/benchmarks/actions/runs/15259443091/job/42914349427,26 May 2025 1:42pm,Success,,1.0.0
g8qaq3s2TJqfvrskHYmapM,OTIS Mock AIME 2024-2025,claude-sonnet-4-20250514_59K,0.689,model_graded:0.69±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fg8qaq3s2TJqfvrskHYmapM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/g8qaq3s2TJqfvrskHYmapM.eval,https://github.com/epoch-research/benchmarks/actions/runs/15219129503/job/42811296962,23 May 2025 5:01pm,Success,,1.0.0
8hGbWtbpUjAQpebkKQfSi7,GPQA diamond,claude-sonnet-4-20250514_32K,0.783,choice:0.78±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F8hGbWtbpUjAQpebkKQfSi7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/8hGbWtbpUjAQpebkKQfSi7.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198491248/job/42748050182,22 May 2025 7:06pm,Success,,1.0.0
FZHzYjwAQexVykNQTnRtwT,OTIS Mock AIME 2024-2025,claude-sonnet-4-20250514_32K,0.711,model_graded:0.71±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFZHzYjwAQexVykNQTnRtwT.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FZHzYjwAQexVykNQTnRtwT.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198491248/job/42748050166,22 May 2025 7:06pm,Success,,1.0.0
BL9e5eBaVdxVcws6qPwrdh,SWE-Bench verified,claude-sonnet-4-20250514,0.606,swe_bench_scorer:0.61±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBL9e5eBaVdxVcws6qPwrdh.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/BL9e5eBaVdxVcws6qPwrdh.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198491248/job/42748050159,22 May 2025 7:06pm,Success,,1.0.0
Xz2F28L5k8HFNfY7Lo8Ggw,OTIS Mock AIME 2024-2025,claude-opus-4-20250514_16K,0.600,model_graded:0.6±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FXz2F28L5k8HFNfY7Lo8Ggw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Xz2F28L5k8HFNfY7Lo8Ggw.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198346360/job/42747605974,22 May 2025 6:54pm,Success,,1.0.0
CtxFusPxCraL3EeRJu2c5k,GPQA diamond,claude-opus-4-20250514_16K,0.763,choice:0.76±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FCtxFusPxCraL3EeRJu2c5k.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/CtxFusPxCraL3EeRJu2c5k.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198346360/job/42747605983,22 May 2025 6:54pm,Success,,1.0.0
nDrnRwsADSyj8U6BD4in4r,OTIS Mock AIME 2024-2025,claude-sonnet-4-20250514_16K,0.533,model_graded:0.53±0.08,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FnDrnRwsADSyj8U6BD4in4r.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/nDrnRwsADSyj8U6BD4in4r.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198346360/job/42747605972,22 May 2025 6:54pm,Success,,1.0.0
PPLoBnbKQ9CEYZNSk3E7LE,GPQA diamond,claude-sonnet-4-20250514_16K,0.758,choice:0.76±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPPLoBnbKQ9CEYZNSk3E7LE.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PPLoBnbKQ9CEYZNSk3E7LE.eval,https://github.com/epoch-research/benchmarks/actions/runs/15198346360/job/42747605973,22 May 2025 6:54pm,Success,,1.0.0
FhYfB6gn2MEompxg7WoKcX,FrontierMath-2025-02-28-Public,claude-sonnet-4-20250514,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFhYfB6gn2MEompxg7WoKcX.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FhYfB6gn2MEompxg7WoKcX.eval,https://github.com/epoch-research/benchmarks/actions/runs/15193173733/job/42730896995,22 May 2025 1:40pm,Success,,1.0.0
dZLNhNKQVUgnB5ygAEUCcT,MATH level 5,claude-sonnet-4-20250514,0.844,"normalized_string_match:0.79±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.84±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/dZLNhNKQVUgnB5ygAEUCcT.eval,https://github.com/epoch-research/benchmarks/actions/runs/15193173733/job/42730896989,22 May 2025 1:39pm,Success,,1.0.0
YbrpPbWuYAXNqbSfU4so7f,MATH level 5,claude-opus-4-20250514,0.850,"normalized_string_match:0.81±0.01,sympy_equiv:0.79±0.01,model_graded_equiv:0.85±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YbrpPbWuYAXNqbSfU4so7f.eval,https://github.com/epoch-research/benchmarks/actions/runs/15193173733/job/42730897000,22 May 2025 1:39pm,Success,,1.0.0
8tJAHDQwvGBc5ZxJiug3dh,FrontierMath-2025-02-28-Public,claude-opus-4-20250514,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F8tJAHDQwvGBc5ZxJiug3dh.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/8tJAHDQwvGBc5ZxJiug3dh.eval,https://github.com/epoch-research/benchmarks/actions/runs/15193173733/job/42730897004,22 May 2025 1:39pm,Success,,1.0.0
aCbfcxGpLpPc9XkZNcuF4o,OTIS Mock AIME 2024-2025,claude-opus-4-20250514,0.422,model_graded:0.42±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FaCbfcxGpLpPc9XkZNcuF4o.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/aCbfcxGpLpPc9XkZNcuF4o.eval,https://github.com/epoch-research/benchmarks/actions/runs/15192940720/job/42730135327,22 May 2025 1:26pm,Success,,1.0.0
ZF4eejw7J2KrcoNWd6hUbU,GPQA diamond,claude-sonnet-4-20250514,0.667,choice:0.67±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZF4eejw7J2KrcoNWd6hUbU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZF4eejw7J2KrcoNWd6hUbU.eval,https://github.com/epoch-research/benchmarks/actions/runs/15192940720/job/42730135257,22 May 2025 1:26pm,Success,,1.0.0
EnjJ8bGi9vRNNQFzfkLCkQ,GPQA diamond,claude-opus-4-20250514,0.692,choice:0.69±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FEnjJ8bGi9vRNNQFzfkLCkQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/EnjJ8bGi9vRNNQFzfkLCkQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/15192940720/job/42730135251,22 May 2025 1:26pm,Success,,1.0.0
Ua8sCkk3R3mBLWeSnnPG7a,OTIS Mock AIME 2024-2025,claude-sonnet-4-20250514,0.289,model_graded:0.29±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FUa8sCkk3R3mBLWeSnnPG7a.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Ua8sCkk3R3mBLWeSnnPG7a.eval,https://github.com/epoch-research/benchmarks/actions/runs/15192940720/job/42730135246,22 May 2025 1:26pm,Success,,1.0.0
9e62UjoHQbcZD2xjgfRkdK,OTIS Mock AIME 2024-2025,gemini-2.5-flash-preview-05-20,0.708,model_graded:0.71±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9e62UjoHQbcZD2xjgfRkdK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9e62UjoHQbcZD2xjgfRkdK.eval,https://github.com/epoch-research/benchmarks/actions/runs/15148114822/job/42588666620,20 May 2025 5:22pm,Success,,1.0.0
H2sQhavY5BPYxyYHva8mgS,SWE-Bench verified,claude-3-5-sonnet-20240620,0.320,swe_bench_scorer:0.32±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FH2sQhavY5BPYxyYHva8mgS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/H2sQhavY5BPYxyYHva8mgS.eval,https://github.com/epoch-research/benchmarks/actions/runs/15022674725/job/42215630735,14 May 2025 10:00am,Success,,1.0.0
dcjgZVG35bHr3iEdUoGgL3,FrontierMath-2025-02-28-Public,qwen-plus-2025-04-28,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdcjgZVG35bHr3iEdUoGgL3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dcjgZVG35bHr3iEdUoGgL3.eval,https://github.com/epoch-research/benchmarks/actions/runs/15021515258/job/42211691113,14 May 2025 9:11am,Success,,1.0.0
bVFEkuHiUSxjoh8SQm74jo,FrontierMath-2025-02-28-Private,qwen-plus-2025-04-28,0.017,verification_code:0.02±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/bVFEkuHiUSxjoh8SQm74jo.eval,https://github.com/epoch-research/benchmarks/actions/runs/15021515258/job/42211691111,14 May 2025 9:10am,Success,,1.0.0
SCpoD9h4p73bd6SwebPop4,SWE-Bench verified,qwen-plus-2025-04-28,0.280,swe_bench_scorer:0.28±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FSCpoD9h4p73bd6SwebPop4.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/SCpoD9h4p73bd6SwebPop4.eval,https://github.com/epoch-research/benchmarks/actions/runs/14905371281/job/41866458985,8 May 2025 7:28am,Success,,1.0.0
kvVyU6crkxAnG3DAUciMGc,SWE-Bench verified,o3-mini-2025-01-31_medium,0.378,swe_bench_scorer:0.38±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FkvVyU6crkxAnG3DAUciMGc.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/kvVyU6crkxAnG3DAUciMGc.eval,https://github.com/epoch-research/benchmarks/actions/runs/14903875650/job/41861809396,8 May 2025 6:07am,Success,,1.0.0
fpjeeuxXr5G2EdUXAgtynx,FrontierMath-2025-02-28-Public,DeepSeek-V3-0324,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfpjeeuxXr5G2EdUXAgtynx.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fpjeeuxXr5G2EdUXAgtynx.eval,https://github.com/epoch-research/benchmarks/actions/runs/14903875650/job/41861807851,8 May 2025 6:04am,Success,,1.0.0
Mi8LCxmduKukxazAtmuhqg,MATH level 5,gemini-2.5-pro-preview-05-06,0.959,"normalized_string_match:0.9±0.01,sympy_equiv:0.9±0.01,model_graded_equiv:0.96±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Mi8LCxmduKukxazAtmuhqg.eval,https://github.com/epoch-research/benchmarks/actions/runs/14887836248/job/41812192194,8 May 2025 1:28am,Success,,1.0.0
XRkZCzSoQC5yYiQ3xitNdC,MATH level 5,gemini-2.5-pro-preview-03-25,0.956,"normalized_string_match:0.91±0.01,sympy_equiv:0.9±0.01,model_graded_equiv:0.96±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XRkZCzSoQC5yYiQ3xitNdC.eval,https://github.com/epoch-research/benchmarks/actions/runs/14887836248/job/41812190932,7 May 2025 5:59pm,Success,,1.0.0
K86QrHt55VSsziWQMEz4jS,FrontierMath-2025-02-28-Public,mistral-medium-2505,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FK86QrHt55VSsziWQMEz4jS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/K86QrHt55VSsziWQMEz4jS.eval,https://github.com/epoch-research/benchmarks/actions/runs/14889565283/job/41818092001,7 May 2025 1:34pm,Success,,1.0.0
PZRX6ME25bn26B6BTRWx6W,FrontierMath-2025-02-28-Private,mistral-medium-2505,0.003,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PZRX6ME25bn26B6BTRWx6W.eval,https://github.com/epoch-research/benchmarks/actions/runs/14889565283/job/41818092447,7 May 2025 1:33pm,Success,,1.0.0
FNMbfvEEm9PRnrJnrqk2PH,GPQA diamond,mistral-medium-2505,0.595,choice:0.6±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFNMbfvEEm9PRnrJnrqk2PH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FNMbfvEEm9PRnrJnrqk2PH.eval,https://github.com/epoch-research/benchmarks/actions/runs/14889565283/job/41818090989,7 May 2025 1:32pm,Success,,1.0.0
QWw65xeTn8x42SymMsVREN,OTIS Mock AIME 2024-2025,mistral-medium-2505,0.322,model_graded:0.32±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQWw65xeTn8x42SymMsVREN.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QWw65xeTn8x42SymMsVREN.eval,https://github.com/epoch-research/benchmarks/actions/runs/14889565283/job/41818091563,7 May 2025 1:32pm,Success,,1.0.0
LNUnvvrDra89Vhdc9bzdCy,MATH level 5,mistral-medium-2505,0.816,"normalized_string_match:0.77±0.01,sympy_equiv:0.76±0.01,model_graded_equiv:0.82±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/LNUnvvrDra89Vhdc9bzdCy.eval,https://github.com/epoch-research/benchmarks/actions/runs/14889565283/job/41818092840,7 May 2025 1:32pm,Success,,1.0.0
GUbGX7GABSbhxMXiYC5NFr,SWE-Bench verified,claude-3-5-sonnet-20241022,0.406,swe_bench_scorer:0.41±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FGUbGX7GABSbhxMXiYC5NFr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/GUbGX7GABSbhxMXiYC5NFr.eval,https://github.com/epoch-research/benchmarks/actions/runs/14800198436/job/41556994294,2 May 2025 1:35pm,Success,,1.0.0
dD8n3SgLkV3Xse6bbG8mMB,SWE-Bench verified,gpt-4.1-mini-2025-04-14,0.328,swe_bench_scorer:0.33±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdD8n3SgLkV3Xse6bbG8mMB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dD8n3SgLkV3Xse6bbG8mMB.eval,https://github.com/epoch-research/benchmarks/actions/runs/14753190375/job/41415252104,30 April 2025 7:13am,Success,,1.0.0
Q6UB2pBAwruHWXCNQvKGVv,SWE-Bench verified,gpt-4.1-2025-04-14,0.410,swe_bench_scorer:0.41±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQ6UB2pBAwruHWXCNQvKGVv.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Q6UB2pBAwruHWXCNQvKGVv.eval,https://github.com/epoch-research/benchmarks/actions/runs/14753190375/job/41415253223,30 April 2025 7:13am,Success,,1.0.0
Sp6nNR4N7zRGrqPCkgJ4Mn,SWE-Bench verified,grok-3-beta,0.386,swe_bench_scorer:0.39±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FSp6nNR4N7zRGrqPCkgJ4Mn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Sp6nNR4N7zRGrqPCkgJ4Mn.eval,https://github.com/epoch-research/benchmarks/actions/runs/14751561376/job/41410088563,30 April 2025 5:43am,Success,,1.0.0
ne2Anciuxh2k3VXGYG4Nqy,SWE-Bench verified,gemini-2.0-flash-001,0.220,swe_bench_scorer:0.22±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fne2Anciuxh2k3VXGYG4Nqy.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ne2Anciuxh2k3VXGYG4Nqy.eval,https://github.com/epoch-research/benchmarks/actions/runs/14657846053/job/41135958214,25 April 2025 3:22am,Success,,1.0.0
MgQteQgHEpQAAQw7DFbh2u,SWE-Bench verified,grok-3-mini-beta_low,0.152,swe_bench_scorer:0.15±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMgQteQgHEpQAAQw7DFbh2u.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MgQteQgHEpQAAQw7DFbh2u.eval,https://github.com/epoch-research/benchmarks/actions/runs/14657846053/job/41135958002,25 April 2025 2:42am,Success,,1.0.0
MCv2wDLoTzqMRrN8dATPEv,SWE-Bench verified,claude-3-7-sonnet-20250219,0.522,swe_bench_scorer:0.52±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMCv2wDLoTzqMRrN8dATPEv.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MCv2wDLoTzqMRrN8dATPEv.eval,https://github.com/epoch-research/benchmarks/actions/runs/14649537961/job/41111840770,24 April 2025 6:49pm,Success,,1.0.0
eTCmM2smJzrY4xjJnrboT2,SWE-Bench verified,o4-mini-2025-04-16_medium,0.346,swe_bench_scorer:0.35±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FeTCmM2smJzrY4xjJnrboT2.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/eTCmM2smJzrY4xjJnrboT2.eval,https://github.com/epoch-research/benchmarks/actions/runs/14649537961/job/41111838698,24 April 2025 3:01pm,Success,,1.0.0
V7txn6pocsNsZSzt5dkZkJ,FrontierMath-2025-02-28-Public,o4-mini-2025-04-16_medium,0.200,verification_code:0.2±0.13,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FV7txn6pocsNsZSzt5dkZkJ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/V7txn6pocsNsZSzt5dkZkJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/14596722844/job/40944529392,22 April 2025 10:05am,Success,,1.0.0
BMEJvvJKbhHa5q7YTuCeAi,FrontierMath-2025-02-28-Public,o4-mini-2025-04-16_low,0.300,verification_code:0.3±0.15,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBMEJvvJKbhHa5q7YTuCeAi.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/BMEJvvJKbhHa5q7YTuCeAi.eval,https://github.com/epoch-research/benchmarks/actions/runs/14596722844/job/40944531439,22 April 2025 10:03am,Success,,1.0.0
KpZYG5nvdJ6QYweLz6dn5a,OTIS Mock AIME 2024-2025,gemini-2.5-flash-preview-04-17,0.731,model_graded:0.73±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FKpZYG5nvdJ6QYweLz6dn5a.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/KpZYG5nvdJ6QYweLz6dn5a.eval,https://github.com/epoch-research/benchmarks/actions/runs/14592056445/job/40929445765,22 April 2025 6:03am,Success,,1.0.0
47yfmJgzAJq5DfUcBGVebV,FrontierMath-2025-02-28-Public,o3-2025-04-16_low,0.100,verification_code:0.1±0.1,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F47yfmJgzAJq5DfUcBGVebV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/47yfmJgzAJq5DfUcBGVebV.eval,https://github.com/epoch-research/benchmarks/actions/runs/14584666885/job/40908014350,21 April 2025 9:30pm,Success,,1.0.0
gdKs4ZBYcVz8oUvzu5VkDB,FrontierMath-2025-02-28-Public,o3-2025-04-16_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FgdKs4ZBYcVz8oUvzu5VkDB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/gdKs4ZBYcVz8oUvzu5VkDB.eval,https://github.com/epoch-research/benchmarks/actions/runs/14584666885/job/40908014628,21 April 2025 9:30pm,Success,,1.0.0
gXQ6PHjaqbjcBj6KV9UKSK,FrontierMath-2025-02-28-Public,o3-2025-04-16_high,0.100,verification_code:0.1±0.1,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FgXQ6PHjaqbjcBj6KV9UKSK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/gXQ6PHjaqbjcBj6KV9UKSK.eval,https://github.com/epoch-research/benchmarks/actions/runs/14517716758/job/40730829720,17 April 2025 10:17am,Success,,1.0.0
E9AfnKpQrDsFWKmXMhBbJS,FrontierMath-2025-02-28-Public,o4-mini-2025-04-16_high,0.300,verification_code:0.3±0.15,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FE9AfnKpQrDsFWKmXMhBbJS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/E9AfnKpQrDsFWKmXMhBbJS.eval,https://github.com/epoch-research/benchmarks/actions/runs/14517716758/job/40730828653,17 April 2025 10:15am,Success,,1.0.0
9Sgsz5XCjscuZkexf7MX9c,GPQA diamond,o4-mini-2025-04-16_high,0.796,choice:0.8±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9Sgsz5XCjscuZkexf7MX9c.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9Sgsz5XCjscuZkexf7MX9c.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676045590,16 April 2025 2:17pm,Success,,1.0.0
evANZQ9oQTdGYbDERw7VrD,GPQA diamond,o3-2025-04-16_high,0.818,choice:0.82±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FevANZQ9oQTdGYbDERw7VrD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/evANZQ9oQTdGYbDERw7VrD.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676048513,16 April 2025 2:17pm,Success,,1.0.0
727ixGKZ7A7G8LmavE2ScL,MATH level 5,o4-mini-2025-04-16_high,0.978,"normalized_string_match:0.84±0.01,sympy_equiv:0.88±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/727ixGKZ7A7G8LmavE2ScL.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676044412,16 April 2025 2:17pm,Success,,1.0.0
TVsxkJ8ZyfMVmVx88Qq3KN,OTIS Mock AIME 2024-2025,o4-mini-2025-04-16_high,0.817,model_graded:0.82±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTVsxkJ8ZyfMVmVx88Qq3KN.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TVsxkJ8ZyfMVmVx88Qq3KN.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676042155,16 April 2025 2:17pm,Success,,1.0.0
6oW3GceNQznewQJRwLigth,MATH level 5,o3-2025-04-16_high,0.978,"normalized_string_match:0.86±0.01,sympy_equiv:0.87±0.01,model_graded_equiv:0.98±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6oW3GceNQznewQJRwLigth.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676046334,16 April 2025 2:17pm,Success,,1.0.0
6anLU9CfBBNQkWyJG2GDCa,OTIS Mock AIME 2024-2025,o3-2025-04-16_high,0.839,model_graded:0.84±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F6anLU9CfBBNQkWyJG2GDCa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/6anLU9CfBBNQkWyJG2GDCa.eval,https://github.com/epoch-research/benchmarks/actions/runs/14499624581/job/40676047604,16 April 2025 2:17pm,Success,,1.0.0
R4GLbPVPjbATgWQJFjF2EQ,FrontierMath-2025-02-28-Public,gpt-4.1-nano-2025-04-14,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FR4GLbPVPjbATgWQJFjF2EQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/R4GLbPVPjbATgWQJFjF2EQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543125879,14 April 2025 6:51pm,Success,,1.0.0
chtQvebcoaE33KkhAjbCnw,FrontierMath-2025-02-28-Private,gpt-4.1-nano-2025-04-14,0.010,verification_code:0.01±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/chtQvebcoaE33KkhAjbCnw.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543126661,14 April 2025 6:51pm,Success,,1.0.0
6ybVweWdTfiCuirnS2BKXD,MATH level 5,gpt-4.1-mini-2025-04-14,0.873,"normalized_string_match:0.81±0.01,sympy_equiv:0.82±0.01,model_graded_equiv:0.87±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6ybVweWdTfiCuirnS2BKXD.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543126112,14 April 2025 6:50pm,Success,,1.0.0
3qLHwibDhgtWEbKGwxUYVa,OTIS Mock AIME 2024-2025,gpt-4.1-nano-2025-04-14,0.289,model_graded:0.29±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3qLHwibDhgtWEbKGwxUYVa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3qLHwibDhgtWEbKGwxUYVa.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543124728,14 April 2025 6:50pm,Success,,1.0.0
dXHddeGRX7JRVa5UNfad5C,GPQA diamond,gpt-4.1-mini-2025-04-14,0.658,choice:0.66±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdXHddeGRX7JRVa5UNfad5C.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dXHddeGRX7JRVa5UNfad5C.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543124172,14 April 2025 6:49pm,Success,,1.0.0
cXbbdQDW2jeYassmRuZYpU,OTIS Mock AIME 2024-2025,gpt-4.1-mini-2025-04-14,0.447,model_graded:0.45±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcXbbdQDW2jeYassmRuZYpU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cXbbdQDW2jeYassmRuZYpU.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543126938,14 April 2025 6:49pm,Success,,1.0.0
K4TfMXsaPd4Z46zFmiV4Rf,MATH level 5,gpt-4.1-nano-2025-04-14,0.700,"normalized_string_match:0.64±0.01,sympy_equiv:0.65±0.01,model_graded_equiv:0.7±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/K4TfMXsaPd4Z46zFmiV4Rf.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543126377,14 April 2025 6:49pm,Success,,1.0.0
azWfeFPP6zSocuQX3JYBU5,GPQA diamond,gpt-4.1-nano-2025-04-14,0.489,choice:0.49±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FazWfeFPP6zSocuQX3JYBU5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/azWfeFPP6zSocuQX3JYBU5.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543125592,14 April 2025 6:49pm,Success,,1.0.0
jM7D9QpAZVXuP93HM6ST5w,FrontierMath-2025-02-28-Public,gpt-4.1-mini-2025-04-14,0.100,verification_code:0.1±0.1,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjM7D9QpAZVXuP93HM6ST5w.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jM7D9QpAZVXuP93HM6ST5w.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543125317,14 April 2025 6:49pm,Success,,1.0.0
ephMMUGDANc9iVTNNuuoZx,FrontierMath-2025-02-28-Private,gpt-4.1-mini-2025-04-14,0.045,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ephMMUGDANc9iVTNNuuoZx.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457273474/job/40543125037,14 April 2025 6:49pm,Success,,1.0.0
QjrnNoyDCzDYmFxgfYCuTm,FrontierMath-2025-02-28-Public,gpt-4.1-2025-04-14,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQjrnNoyDCzDYmFxgfYCuTm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QjrnNoyDCzDYmFxgfYCuTm.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457143876/job/40542757871,14 April 2025 6:40pm,Success,,1.0.0
NPeiYpjbscreYnFkHPvcNB,FrontierMath-2025-02-28-Private,gpt-4.1-2025-04-14,0.055,verification_code:0.06±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/NPeiYpjbscreYnFkHPvcNB.eval,https://github.com/epoch-research/benchmarks/actions/runs/14457143876/job/40542757442,14 April 2025 6:40pm,Success,,1.0.0
YyJXPVXJjN5EnLfYjfCTrQ,MATH level 5,gpt-4.1-2025-04-14,0.830,"normalized_string_match:0.78±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.83±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YyJXPVXJjN5EnLfYjfCTrQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/14456892319/job/40542044498,14 April 2025 6:22pm,Success,,1.0.0
Gj8pd5gU9r4LqwDK8XZ4mL,GPQA diamond,gpt-4.1-2025-04-14,0.669,choice:0.67±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FGj8pd5gU9r4LqwDK8XZ4mL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Gj8pd5gU9r4LqwDK8XZ4mL.eval,https://github.com/epoch-research/benchmarks/actions/runs/14453036499/job/40530055041,14 April 2025 2:34pm,Success,,1.0.0
VAQtARXUPTpdxGKMekUSbX,OTIS Mock AIME 2024-2025,gpt-4.1-2025-04-14,0.383,model_graded:0.38±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVAQtARXUPTpdxGKMekUSbX.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/VAQtARXUPTpdxGKMekUSbX.eval,https://github.com/epoch-research/benchmarks/actions/runs/14453036499/job/40530054531,14 April 2025 2:34pm,Success,,1.0.0
PtxHF8BH8WuNdNfDnutGBb,GPQA diamond,qwq-plus,0.654,choice:0.65±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPtxHF8BH8WuNdNfDnutGBb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PtxHF8BH8WuNdNfDnutGBb.eval,https://github.com/epoch-research/benchmarks/actions/runs/14405108880/job/40399768027,11 April 2025 11:34am,Success,,1.0.0
McxfQdC5aypnD3STwZUBjH,GPQA diamond,grok-3-mini-beta_low,0.763,choice:0.76±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMcxfQdC5aypnD3STwZUBjH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/McxfQdC5aypnD3STwZUBjH.eval,https://github.com/epoch-research/benchmarks/actions/runs/14389691016/job/40353684670,10 April 2025 4:30pm,Success,,1.0.0
TTmhagb2ovbi7GJnDxEfrj,OTIS Mock AIME 2024-2025,grok-3-mini-beta_low,0.622,model_graded:0.62±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTTmhagb2ovbi7GJnDxEfrj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TTmhagb2ovbi7GJnDxEfrj.eval,https://github.com/epoch-research/benchmarks/actions/runs/14389691016/job/40353685434,10 April 2025 4:30pm,Success,,1.0.0
5QyCm49Lce2PpGVcZLomCz,MATH level 5,grok-3-mini-beta_low,0.909,"normalized_string_match:0.85±0.01,sympy_equiv:0.84±0.01,model_graded_equiv:0.91±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5QyCm49Lce2PpGVcZLomCz.eval,https://github.com/epoch-research/benchmarks/actions/runs/14389691016/job/40353686999,10 April 2025 4:30pm,Success,,1.0.0
mvPsa56tAProncoFm6Y7aW,FrontierMath-2025-02-28-Public,grok-3-mini-beta_low,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmvPsa56tAProncoFm6Y7aW.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mvPsa56tAProncoFm6Y7aW.eval,https://github.com/epoch-research/benchmarks/actions/runs/14389691016/job/40353686219,10 April 2025 4:30pm,Success,,1.0.0
9FTMS2t6Ksm43CduHeqtJr,FrontierMath-2025-02-28-Private,grok-3-mini-beta_low,0.028,verification_code:0.03±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/9FTMS2t6Ksm43CduHeqtJr.eval,https://github.com/epoch-research/benchmarks/actions/runs/14389691016/job/40353687529,10 April 2025 4:30pm,Success,,1.0.0
HMm8XaDBr4CQdMFddasEWb,MATH level 5,grok-3-mini-beta_high,0.881,"normalized_string_match:0.82±0.01,sympy_equiv:0.82±0.01,model_graded_equiv:0.88±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/HMm8XaDBr4CQdMFddasEWb.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387155727/job/40345430009,10 April 2025 2:02pm,Success,,1.0.0
fPtize44vDPcDAzMFuAurw,OTIS Mock AIME 2024-2025,grok-3-mini-beta_high,0.778,model_graded:0.78±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfPtize44vDPcDAzMFuAurw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fPtize44vDPcDAzMFuAurw.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387155727/job/40345429412,10 April 2025 2:02pm,Success,,1.0.0
HQ3GnLdmf8Cy6dceb8d7cL,FrontierMath-2025-02-28-Public,grok-3-mini-beta_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FHQ3GnLdmf8Cy6dceb8d7cL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/HQ3GnLdmf8Cy6dceb8d7cL.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387155727/job/40345428244,10 April 2025 2:02pm,Success,,1.0.0
Z83F6bJ7Ne5rXaXwgR2xWT,FrontierMath-2025-02-28-Private,grok-3-mini-beta_high,0.059,verification_code:0.06±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Z83F6bJ7Ne5rXaXwgR2xWT.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387155727/job/40345430673,10 April 2025 2:02pm,Success,,1.0.0
DzjFQArKFTZHioQVwKYVtX,FrontierMath-2025-02-28-Private,grok-3-beta,0.038,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/DzjFQArKFTZHioQVwKYVtX.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387047740/job/40345014123,10 April 2025 1:57pm,Success,,1.0.0
i9eqMbaVTDDuCxWxzGHxyU,MATH level 5,grok-3-beta,0.887,"normalized_string_match:0.78±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.89±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/i9eqMbaVTDDuCxWxzGHxyU.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387047740/job/40345017007,10 April 2025 1:55pm,Success,,1.0.0
DKTzh9kSoipZGkDsC4NGf3,OTIS Mock AIME 2024-2025,grok-3-beta,0.556,model_graded:0.56±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FDKTzh9kSoipZGkDsC4NGf3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/DKTzh9kSoipZGkDsC4NGf3.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387047740/job/40345019746,10 April 2025 1:55pm,Success,,1.0.0
WBwJHBpCqKQvZBnoZ4Y8H9,FrontierMath-2025-02-28-Public,grok-3-beta,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWBwJHBpCqKQvZBnoZ4Y8H9.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WBwJHBpCqKQvZBnoZ4Y8H9.eval,https://github.com/epoch-research/benchmarks/actions/runs/14387047740/job/40345015330,10 April 2025 1:55pm,Success,,1.0.0
QjvBS34fsngfBBMZUirSyh,FrontierMath-2025-02-28-Private,Llama-4-Maverick-17B-128E-Instruct-FP8,0.007,verification_code:0.01±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/QjvBS34fsngfBBMZUirSyh.eval,https://github.com/epoch-research/benchmarks/actions/runs/14330947489/job/40166712424,8 April 2025 6:17am,Success,,1.0.0
M2qDSno6GVHdWjcdGRGy9D,FrontierMath-2025-02-28-Public,Llama-4-Maverick-17B-128E-Instruct-FP8,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FM2qDSno6GVHdWjcdGRGy9D.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/M2qDSno6GVHdWjcdGRGy9D.eval,https://github.com/epoch-research/benchmarks/actions/runs/14330947489/job/40166712919,8 April 2025 6:17am,Success,,1.0.0
bhjLDA5A7gddsX5cAYKma8,FrontierMath-2025-02-28-Private,Llama-4-Scout-17B-16E-Instruct,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/bhjLDA5A7gddsX5cAYKma8.eval,https://github.com/epoch-research/benchmarks/actions/runs/14330947489/job/40166711772,8 April 2025 6:14am,Success,,1.0.0
m96PYCivjSmTVDd27zw84m,FrontierMath-2025-02-28-Public,Llama-4-Scout-17B-16E-Instruct,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fm96PYCivjSmTVDd27zw84m.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/m96PYCivjSmTVDd27zw84m.eval,https://github.com/epoch-research/benchmarks/actions/runs/14329838859/job/40163205227,8 April 2025 6:03am,Success,,1.0.0
GJEpPThqUxXJBSbvkuXVnw,OTIS Mock AIME 2024-2025,Llama-4-Maverick-17B-128E-Instruct-FP8,0.206,model_graded:0.21±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FGJEpPThqUxXJBSbvkuXVnw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/GJEpPThqUxXJBSbvkuXVnw.eval,https://github.com/epoch-research/benchmarks/actions/runs/14329838859/job/40163204194,8 April 2025 5:58am,Success,,1.0.0
HkjCosgMFq5QXB2CyTyU7r,MATH level 5,Llama-4-Maverick-17B-128E-Instruct-FP8,0.730,"normalized_string_match:0.7±0.01,sympy_equiv:0.7±0.01,model_graded_equiv:0.73±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/HkjCosgMFq5QXB2CyTyU7r.eval,https://github.com/epoch-research/benchmarks/actions/runs/14329838859/job/40163203072,8 April 2025 5:46am,Success,,1.0.0
T3X5DeC47iqCaoRYXu5k8L,OTIS Mock AIME 2024-2025,Llama-4-Scout-17B-16E-Instruct,0.078,model_graded:0.08±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FT3X5DeC47iqCaoRYXu5k8L.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/T3X5DeC47iqCaoRYXu5k8L.eval,https://github.com/epoch-research/benchmarks/actions/runs/14329838859/job/40163202510,8 April 2025 5:43am,Success,,1.0.0
c6rTyTFCeoHMBzC2QwjxFL,MATH level 5,Llama-4-Scout-17B-16E-Instruct,0.623,"normalized_string_match:0.6±0.01,sympy_equiv:0.6±0.01,model_graded_equiv:0.62±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/c6rTyTFCeoHMBzC2QwjxFL.eval,https://github.com/epoch-research/benchmarks/actions/runs/14329838859/job/40163202053,8 April 2025 5:22am,Success,,1.0.0
MsXAXDEKP3xBKtiA7aTBBG,GPQA diamond,Llama-4-Scout-17B-16E-Instruct,0.518,choice:0.52±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMsXAXDEKP3xBKtiA7aTBBG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MsXAXDEKP3xBKtiA7aTBBG.eval,https://github.com/epoch-research/benchmarks/actions/runs/14328907350/job/40160223780,8 April 2025 4:34am,Success,,1.0.0
LyHy99ubGCqBhoaksQjkiF,GPQA diamond,Llama-4-Maverick-17B-128E-Instruct-FP8,0.670,choice:0.67±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FLyHy99ubGCqBhoaksQjkiF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/LyHy99ubGCqBhoaksQjkiF.eval,https://github.com/epoch-research/benchmarks/actions/runs/14328907350/job/40160224514,8 April 2025 4:34am,Success,,1.0.0
9Ke5At9bYuf56SmKJJ7Sz7,GPQA diamond,qwen-turbo-2024-11-01,0.418,choice:0.42±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9Ke5At9bYuf56SmKJJ7Sz7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9Ke5At9bYuf56SmKJJ7Sz7.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121460779,7 April 2025 5:38pm,Success,,1.0.0
jWV936hYKg6rYcPEynzEEt,MATH level 5,qwen-turbo-2024-11-01,0.562,"normalized_string_match:0.52±0.01,sympy_equiv:0.51±0.01,model_graded_equiv:0.56±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/jWV936hYKg6rYcPEynzEEt.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121460248,7 April 2025 3:55pm,Success,,1.0.0
QtraMoKiQRRpZc9txpFAYF,GPQA diamond,qwen-plus-2025-01-25,0.481,choice:0.48±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQtraMoKiQRRpZc9txpFAYF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QtraMoKiQRRpZc9txpFAYF.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121459815,7 April 2025 3:28pm,Success,,1.0.0
G5gxbTH33S7LXA29o5fgmv,OTIS Mock AIME 2024-2025,qwen-turbo-2024-11-01,0.061,model_graded:0.06±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FG5gxbTH33S7LXA29o5fgmv.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/G5gxbTH33S7LXA29o5fgmv.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121459305,7 April 2025 3:18pm,Success,,1.0.0
AULNJG2C75JBP6uaiAgj3P,OTIS Mock AIME 2024-2025,qwen-plus-2025-01-25,0.178,model_graded:0.18±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FAULNJG2C75JBP6uaiAgj3P.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/AULNJG2C75JBP6uaiAgj3P.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121458807,7 April 2025 3:09pm,Success,,1.0.0
J3WoEpLKAJZWGGpyBVCCCn,MATH level 5,qwen-plus-2025-01-25,0.653,"normalized_string_match:0.62±0.01,sympy_equiv:0.62±0.01,model_graded_equiv:0.65±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/J3WoEpLKAJZWGGpyBVCCCn.eval,https://github.com/epoch-research/benchmarks/actions/runs/14315764573/job/40121458196,7 April 2025 1:38pm,Success,,1.0.0
F77vDPynTKJbz7m2bt7Py8,FrontierMath-2025-02-28-Private,qwen-max-2025-01-25,0.010,verification_code:0.01±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/F77vDPynTKJbz7m2bt7Py8.eval,https://github.com/epoch-research/benchmarks/actions/runs/14225170294/job/39862937048,2 April 2025 12:55pm,Success,,1.0.0
GBf7rqHkGGzfX87V37F9mH,GPQA diamond,qwen-max-2025-01-25,0.561,choice:0.56±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FGBf7rqHkGGzfX87V37F9mH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/GBf7rqHkGGzfX87V37F9mH.eval,https://github.com/epoch-research/benchmarks/actions/runs/14202506670/job/39792564560,1 April 2025 3:22pm,Success,,1.0.0
GFZDNH95TnVegNrEtmUT2Q,MATH level 5,qwen-max-2025-01-25,0.672,"normalized_string_match:0.64±0.01,sympy_equiv:0.64±0.01,model_graded_equiv:0.67±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/GFZDNH95TnVegNrEtmUT2Q.eval,https://github.com/epoch-research/benchmarks/actions/runs/14202506670/job/39792564065,1 April 2025 1:50pm,Success,,1.0.0
PZt4Yb9bpXVPDoPi4uTxwK,OTIS Mock AIME 2024-2025,qwen-max-2025-01-25,0.161,model_graded:0.16±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPZt4Yb9bpXVPDoPi4uTxwK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PZt4Yb9bpXVPDoPi4uTxwK.eval,https://github.com/epoch-research/benchmarks/actions/runs/14201678798/job/39789965625,1 April 2025 1:11pm,Success,,1.0.0
hWj4Vg5DgUWxhZrT4LWKeU,FrontierMath-2025-02-28-Public,qwen-max-2025-01-25,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FhWj4Vg5DgUWxhZrT4LWKeU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/hWj4Vg5DgUWxhZrT4LWKeU.eval,https://github.com/epoch-research/benchmarks/actions/runs/14201678798/job/39789965056,1 April 2025 1:06pm,Success,,1.0.0
63MnE4e4AXvbQ585MaWTFv,OTIS Mock AIME 2024-2025,Hermes-2-Theta-Llama-3-70B,0.025,model_graded:0.03±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F63MnE4e4AXvbQ585MaWTFv.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/63MnE4e4AXvbQ585MaWTFv.eval,https://github.com/epoch-research/benchmarks/actions/runs/14199067634/job/39781395329,1 April 2025 11:03am,Success,,1.0.0
YFyn28VJ6m2hfop3KGRkBs,MATH level 5,DeepSeek-V3-0324,0.755,"normalized_string_match:0.7±0.01,sympy_equiv:0.69±0.01,model_graded_equiv:0.76±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YFyn28VJ6m2hfop3KGRkBs.eval,https://github.com/epoch-research/benchmarks/actions/runs/14198378060/job/39779077596,1 April 2025 10:52am,Success,,1.0.0
VjfLwKbT6kTprYsMGcz2YT,GPQA diamond,DeepSeek-V3-0324,0.676,choice:0.68±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVjfLwKbT6kTprYsMGcz2YT.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/VjfLwKbT6kTprYsMGcz2YT.eval,https://github.com/epoch-research/benchmarks/actions/runs/14198378060/job/39779076853,1 April 2025 10:40am,Success,,1.0.0
VpsMexBEhXkPYJBe7v6Fk9,OTIS Mock AIME 2024-2025,DeepSeek-V3-0324,0.378,model_graded:0.38±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVpsMexBEhXkPYJBe7v6Fk9.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/VpsMexBEhXkPYJBe7v6Fk9.eval,https://github.com/epoch-research/benchmarks/actions/runs/14198378060/job/39779075767,1 April 2025 10:33am,Success,,1.0.0
WKjzEQdfnBHnYeZoKeCWnB,GPQA diamond,gemini-2.5-pro-exp-03-25,0.838,choice:0.84±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWKjzEQdfnBHnYeZoKeCWnB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WKjzEQdfnBHnYeZoKeCWnB.eval,https://github.com/epoch-research/benchmarks/actions/runs/14176046374/job/39711170292,31 March 2025 12:06pm,Success,,1.0.0
b4vEwq8UVmGmAKh4eqp6G3,OTIS Mock AIME 2024-2025,mistral-small-2501,0.053,model_graded:0.05±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fb4vEwq8UVmGmAKh4eqp6G3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/b4vEwq8UVmGmAKh4eqp6G3.eval,https://github.com/epoch-research/benchmarks/actions/runs/13928623226/job/38979799060,18 March 2025 12:39pm,Success,,1.0.0
S8bBDAbucsFceirHxeEDvu,GPQA diamond,mistral-small-2503,0.475,choice:0.47±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FS8bBDAbucsFceirHxeEDvu.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/S8bBDAbucsFceirHxeEDvu.eval,https://github.com/epoch-research/benchmarks/actions/runs/13927960346/job/38977500424,18 March 2025 12:08pm,Success,,1.0.0
Kas7Y5PBsT7Li5Wm6Y8mj8,MATH level 5,mistral-small-2503,0.468,"normalized_string_match:0.45±0.01,sympy_equiv:0.44±0.01,model_graded_equiv:0.47±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Kas7Y5PBsT7Li5Wm6Y8mj8.eval,https://github.com/epoch-research/benchmarks/actions/runs/13927960346/job/38977499602,18 March 2025 12:08pm,Success,,1.0.0
JxqtHWrwCtYoQYBB3jVqB9,OTIS Mock AIME 2024-2025,mistral-small-2503,0.058,model_graded:0.06±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJxqtHWrwCtYoQYBB3jVqB9.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JxqtHWrwCtYoQYBB3jVqB9.eval,https://github.com/epoch-research/benchmarks/actions/runs/13927960346/job/38977501094,18 March 2025 12:08pm,Success,,1.0.0
epCyFNTf6vyUi5Dby7GeTq,MATH level 5,claude-3-7-sonnet-20250219_64K,0.912,"normalized_string_match:0.83±0.01,sympy_equiv:0.83±0.01,model_graded_equiv:0.91±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/epCyFNTf6vyUi5Dby7GeTq.eval,https://github.com/epoch-research/benchmarks/actions/runs/13839730258/job/38723940201,13 March 2025 5:33pm,Success,,1.0.0
QW3FUBn5qtAFEX4oQuzEAV,OTIS Mock AIME 2024-2025,claude-3-7-sonnet-20250219_64K,0.578,model_graded:0.58±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQW3FUBn5qtAFEX4oQuzEAV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QW3FUBn5qtAFEX4oQuzEAV.eval,https://github.com/epoch-research/benchmarks/actions/runs/13839730258/job/38723939661,13 March 2025 5:20pm,Success,,1.0.0
DoU54Xy93J3MxztC4uxwYy,FrontierMath-2025-02-28-Private,claude-3-7-sonnet-20250219_64K,0.031,verification_code:0.03±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/DoU54Xy93J3MxztC4uxwYy.eval,https://github.com/epoch-research/benchmarks/actions/runs/13839730258/job/38723938319,13 March 2025 12:49pm,Success,,1.0.0
Rfy7A5TbRK343mTyQkeGt7,FrontierMath-2025-02-28-Public,claude-3-7-sonnet-20250219_64K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRfy7A5TbRK343mTyQkeGt7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Rfy7A5TbRK343mTyQkeGt7.eval,https://github.com/epoch-research/benchmarks/actions/runs/13839730258/job/38723937655,13 March 2025 12:31pm,Success,,1.0.0
FtxiPnS7ZH5qLrmFgp4ca2,GPQA diamond,gemma-3-27b-it,0.489,choice:0.49±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFtxiPnS7ZH5qLrmFgp4ca2.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FtxiPnS7ZH5qLrmFgp4ca2.eval,https://github.com/epoch-research/benchmarks/actions/runs/13839179509/job/38722033048,13 March 2025 12:05pm,Success,,1.0.0
A3ZGeCNhHMp55ZjtRtCGVB,OTIS Mock AIME 2024-2025,gemma-3-27b-it,0.197,model_graded:0.2±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FA3ZGeCNhHMp55ZjtRtCGVB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/A3ZGeCNhHMp55ZjtRtCGVB.eval,https://github.com/epoch-research/benchmarks/actions/runs/13827990820/job/38686346612,13 March 2025 8:46am,Success,,1.0.0
aZqVpPa4SWHapN9QHEoEm3,FrontierMath-2025-02-28-Private,claude-3-7-sonnet-20250219_32K,0.034,verification_code:0.03±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/aZqVpPa4SWHapN9QHEoEm3.eval,https://github.com/epoch-research/benchmarks/actions/runs/13827990820/job/38686346426,13 March 2025 5:34am,Success,,1.0.0
amrtN7UjAtuyEmv5DaGshP,MATH level 5,gemma-3-27b-it,0.740,"normalized_string_match:0.71±0.01,sympy_equiv:0.71±0.01,model_graded_equiv:0.74±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/amrtN7UjAtuyEmv5DaGshP.eval,https://github.com/epoch-research/benchmarks/actions/runs/13827990820/job/38686346066,13 March 2025 1:52am,Success,,1.0.0
VzgR6YSTMWX3dFDwQgmFpM,FrontierMath-2025-02-28-Public,claude-3-7-sonnet-20250219_32K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVzgR6YSTMWX3dFDwQgmFpM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/VzgR6YSTMWX3dFDwQgmFpM.eval,https://github.com/epoch-research/benchmarks/actions/runs/13824354071/job/38676522216,12 March 2025 10:24pm,Success,,1.0.0
GSvi9Zf5aK3t5TfCPqNRHy,MATH level 5,claude-3-5-haiku-20241022,0.464,"normalized_string_match:0.42±0.01,sympy_equiv:0.43±0.01,model_graded_equiv:0.46±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/GSvi9Zf5aK3t5TfCPqNRHy.eval,https://github.com/epoch-research/benchmarks/actions/runs/13823756383/job/38674752662,12 March 2025 7:51pm,Success,,1.0.0
JpLxrpvthx7A75sbaHZ9Et,GPQA diamond,claude-3-5-haiku-20241022,0.381,choice:0.38±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJpLxrpvthx7A75sbaHZ9Et.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JpLxrpvthx7A75sbaHZ9Et.eval,https://github.com/epoch-research/benchmarks/actions/runs/13823756383/job/38674752092,12 March 2025 7:51pm,Success,,1.0.0
gFT36W4KU3wggLQ9nNCctg,GPQA diamond,gemini-1.5-flash-8b-001,0.330,choice:0.33±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FgFT36W4KU3wggLQ9nNCctg.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/gFT36W4KU3wggLQ9nNCctg.eval,https://github.com/epoch-research/benchmarks/actions/runs/13821788357/job/38668706790,12 March 2025 5:23pm,Success,,1.0.0
WHGMHrneqJs4nefAh4TjZR,MATH level 5,claude-3-7-sonnet-20250219_32K,0.900,"normalized_string_match:0.83±0.01,sympy_equiv:0.84±0.01,model_graded_equiv:0.9±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/WHGMHrneqJs4nefAh4TjZR.eval,https://github.com/epoch-research/benchmarks/actions/runs/13820136778/job/38663279324,12 March 2025 3:45pm,Success,,1.0.0
dSuGgEbnQErengjEboPn62,OTIS Mock AIME 2024-2025,claude-3-7-sonnet-20250219_32K,0.533,model_graded:0.53±0.08,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdSuGgEbnQErengjEboPn62.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dSuGgEbnQErengjEboPn62.eval,https://github.com/epoch-research/benchmarks/actions/runs/13820004193/job/38662824993,12 March 2025 3:36pm,Success,,1.0.0
R8XGt5ScHpmTbD5DnXR59V,MATH level 5,DeepSeek-R1-Distill-Qwen-14B,0.871,"normalized_string_match:0.81±0.01,sympy_equiv:0.81±0.01,model_graded_equiv:0.87±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/R8XGt5ScHpmTbD5DnXR59V.eval,https://github.com/epoch-research/benchmarks/actions/runs/13777862846/job/38530617978,10 March 2025 8:50pm,Success,,1.0.0
8UqaLobhmNJLuPGTTpeaCw,GPQA diamond,DeepSeek-R1-Distill-Qwen-14B,0.447,choice:0.45±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F8UqaLobhmNJLuPGTTpeaCw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/8UqaLobhmNJLuPGTTpeaCw.eval,https://github.com/epoch-research/benchmarks/actions/runs/13776472544/job/38526698171,10 March 2025 7:03pm,Success,,1.0.0
oPDoTJFMhMNvU4sybYu2jQ,MATH level 5,DeepSeek-R1-Distill-Llama-70B,0.899,"normalized_string_match:0.83±0.01,sympy_equiv:0.83±0.01,model_graded_equiv:0.9±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/oPDoTJFMhMNvU4sybYu2jQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/13776343626/job/38526294199,10 March 2025 6:53pm,Success,,1.0.0
iv9yfwZ9vTznxQfvJF6J7p,GPQA diamond,claude-3-7-sonnet-20250219_32K,0.768,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fiv9yfwZ9vTznxQfvJF6J7p.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/iv9yfwZ9vTznxQfvJF6J7p.eval,https://github.com/epoch-research/benchmarks/actions/runs/13775660516/job/38524291923,10 March 2025 6:05pm,Success,,1.0.0
f6pnAATjxmrFJS353EjQKi,GPQA diamond,DeepSeek-R1-Distill-Llama-70B,0.557,choice:0.56±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Ff6pnAATjxmrFJS353EjQKi.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/f6pnAATjxmrFJS353EjQKi.eval,https://github.com/epoch-research/benchmarks/actions/runs/13773765489/job/38518313802,10 March 2025 4:08pm,Success,,1.0.0
5RpSr8gJZfcZn586aV2hDW,FrontierMath-2025-02-28-Private,gemini-2.0-flash-001,0.017,verification_code:0.02±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5RpSr8gJZfcZn586aV2hDW.eval,https://github.com/epoch-research/benchmarks/actions/runs/13750745859/job/38451431308,9 March 2025 1:18pm,Success,,1.0.0
aXcfCytWFYNk7E5AZGMzLb,FrontierMath-2025-02-28-Private,DeepSeek-V3,0.017,verification_code:0.02±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/aXcfCytWFYNk7E5AZGMzLb.eval,https://github.com/epoch-research/benchmarks/actions/runs/13731253161/job/38408623942,7 March 2025 6:36pm,Success,,1.0.0
iKvrBWEvZT26XhFBRcavtf,FrontierMath-2025-02-28-Public,DeepSeek-V3,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FiKvrBWEvZT26XhFBRcavtf.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/iKvrBWEvZT26XhFBRcavtf.eval,https://github.com/epoch-research/benchmarks/actions/runs/13731168595/job/38408387213,7 March 2025 6:28pm,Success,,1.0.0
FGcofc566CradiNmHUGqVB,OTIS Mock AIME 2024-2025,Llama-3.1-Tulu-3-70B-DPO,0.044,model_graded:0.04±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FFGcofc566CradiNmHUGqVB.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/FGcofc566CradiNmHUGqVB.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730976224/job/38407891177,7 March 2025 6:13pm,Success,,1.0.0
jD6rJdvfNPErKEz9qkpssj,OTIS Mock AIME 2024-2025,gemma-2-27b-it,0.014,model_graded:0.01±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjD6rJdvfNPErKEz9qkpssj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jD6rJdvfNPErKEz9qkpssj.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730955015/job/38407832125,7 March 2025 6:11pm,Success,,1.0.0
WNE7cveVN8rsi7DmsFsFVy,OTIS Mock AIME 2024-2025,gemma-2-9b-it,0.006,model_graded:0.01±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWNE7cveVN8rsi7DmsFsFVy.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WNE7cveVN8rsi7DmsFsFVy.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730955015/job/38407832334,7 March 2025 6:11pm,Success,,1.0.0
89zcekqEYnE8zhwG4pR5Nu,OTIS Mock AIME 2024-2025,DeepSeek-R1-Distill-Llama-70B,0.514,model_graded:0.51±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F89zcekqEYnE8zhwG4pR5Nu.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/89zcekqEYnE8zhwG4pR5Nu.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730850652/job/38407547143,7 March 2025 6:03pm,Success,,1.0.0
2vg9qw9pd8zSyRMu9YNiVm,OTIS Mock AIME 2024-2025,claude-2.1,0.019,model_graded:0.02±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2vg9qw9pd8zSyRMu9YNiVm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2vg9qw9pd8zSyRMu9YNiVm.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730304272/job/38405930933,7 March 2025 5:18pm,Success,,1.0.0
7LLTmqs4LC5GLVfJEqJWSC,OTIS Mock AIME 2024-2025,claude-2.0,0.025,model_graded:0.03±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7LLTmqs4LC5GLVfJEqJWSC.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7LLTmqs4LC5GLVfJEqJWSC.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730304272/job/38405931814,7 March 2025 5:18pm,Success,,1.0.0
cXz2gsCmGYSvQ6Ey4LDiN7,OTIS Mock AIME 2024-2025,gemini-2.0-flash-thinking-exp-01-21,0.578,model_graded:0.58±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcXz2gsCmGYSvQ6Ey4LDiN7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cXz2gsCmGYSvQ6Ey4LDiN7.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730304272/job/38405931586,7 March 2025 5:18pm,Success,,1.0.0
Cprjaj9JQ9MY5NCXqeQR5H,OTIS Mock AIME 2024-2025,o1-preview-2024-09-12,0.311,model_graded:0.31±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FCprjaj9JQ9MY5NCXqeQR5H.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Cprjaj9JQ9MY5NCXqeQR5H.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730235594/job/38405733874,7 March 2025 5:14pm,Success,,1.0.0
RaMuNb5qvzGzMveuZuhEAZ,FrontierMath-2025-02-28-Public,gemini-2.0-pro-exp-02-05,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRaMuNb5qvzGzMveuZuhEAZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/RaMuNb5qvzGzMveuZuhEAZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/13730099354/job/38405338553,7 March 2025 5:04pm,Success,,1.0.0
bynpttv6jLvbXTY7ktnBpk,FrontierMath-2025-02-28-Private,o1-2024-12-17_high,0.093,verification_code:0.09±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/bynpttv6jLvbXTY7ktnBpk.eval,https://github.com/epoch-research/benchmarks/actions/runs/13728106196/job/38399160352,7 March 2025 2:46pm,Success,,1.0.0
BhT6UhKLcghqng7J8NbPWH,FrontierMath-2025-02-28-Public,o1-2024-12-17_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBhT6UhKLcghqng7J8NbPWH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/BhT6UhKLcghqng7J8NbPWH.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727847249/job/38398311497,7 March 2025 2:27pm,Success,,1.0.0
SmT4ukjkxsXZvFExMMVFgW,FrontierMath-2025-02-28-Private,gpt-4o-2024-08-06,0.003,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/SmT4ukjkxsXZvFExMMVFgW.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727320119/job/38396682866,7 March 2025 1:56pm,Success,,1.0.0
PDACS493wNy85eyMC9NULs,FrontierMath-2025-02-28-Public,gemini-2.0-flash-001,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPDACS493wNy85eyMC9NULs.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PDACS493wNy85eyMC9NULs.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727320119/job/38396684236,7 March 2025 1:56pm,Success,,1.0.0
i5xfb9Bk8TaiuQ6WccCAzu,FrontierMath-2025-02-28-Public,claude-3-5-sonnet-20240620,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fi5xfb9Bk8TaiuQ6WccCAzu.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/i5xfb9Bk8TaiuQ6WccCAzu.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727320119/job/38396682449,7 March 2025 1:56pm,Success,,1.0.0
BhMT5JGDGaz4FvSyNLYmok,FrontierMath-2025-02-28-Public,gpt-4o-2024-08-06,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FBhMT5JGDGaz4FvSyNLYmok.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/BhMT5JGDGaz4FvSyNLYmok.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727320119/job/38396683865,7 March 2025 1:56pm,Success,,1.0.0
h7f9jZVWzZNDHNxP5eewNH,FrontierMath-2025-02-28-Private,claude-3-5-sonnet-20240620,0.010,verification_code:0.01±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/h7f9jZVWzZNDHNxP5eewNH.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727320119/job/38396683195,7 March 2025 1:54pm,Success,,1.0.0
8FtfWCm8DdAqF9rnMVwv8h,FrontierMath-2025-02-28-Private,gemini-1.5-flash-002,0.000,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/8FtfWCm8DdAqF9rnMVwv8h.eval,https://github.com/epoch-research/benchmarks/actions/runs/13727049622/job/38396038368,7 March 2025 1:40pm,Success,,1.0.0
Hs7dxLcim5bsmuG2DqnnsA,FrontierMath-2025-02-28-Public,gemini-1.5-flash-002,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FHs7dxLcim5bsmuG2DqnnsA.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Hs7dxLcim5bsmuG2DqnnsA.eval,https://github.com/epoch-research/benchmarks/actions/runs/13726553729/job/38394455299,7 March 2025 1:11pm,Success,,1.0.0
JQcqpW9xPfJRKJkHne6DpK,FrontierMath-2025-02-28-Private,claude-3-5-haiku-20241022,0.003,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/JQcqpW9xPfJRKJkHne6DpK.eval,https://github.com/epoch-research/benchmarks/actions/runs/13726553729/job/38394454565,7 March 2025 1:11pm,Success,,1.0.0
Qe4yHtQm26TQHFCAjhKBaA,FrontierMath-2025-02-28-Public,claude-3-5-haiku-20241022,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQe4yHtQm26TQHFCAjhKBaA.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Qe4yHtQm26TQHFCAjhKBaA.eval,https://github.com/epoch-research/benchmarks/actions/runs/13726553729/job/38394454919,7 March 2025 1:09pm,Success,,1.0.0
ADAsCz7ZtJ8E4inNusQebu,FrontierMath-2025-02-28-Public,claude-3-5-sonnet-20241022,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FADAsCz7ZtJ8E4inNusQebu.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ADAsCz7ZtJ8E4inNusQebu.eval,https://github.com/epoch-research/benchmarks/actions/runs/13710554210/job/38345987158,6 March 2025 6:56pm,Success,,1.0.0
MVnDrt94wyokzx3a7ZJkph,FrontierMath-2025-02-28-Private,claude-3-5-sonnet-20241022,0.021,verification_code:0.02±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/MVnDrt94wyokzx3a7ZJkph.eval,https://github.com/epoch-research/benchmarks/actions/runs/13710554210/job/38345986937,6 March 2025 6:54pm,Success,,1.0.0
JPadBpaxxGpdvTXtfHuvPV,FrontierMath-2025-02-28-Public,o1-mini-2024-09-12_high,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJPadBpaxxGpdvTXtfHuvPV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JPadBpaxxGpdvTXtfHuvPV.eval,https://github.com/epoch-research/benchmarks/actions/runs/13709452985/job/38342620506,6 March 2025 5:32pm,Success,,1.0.0
QKY2tL2pcQ33bDkdMCQzdr,FrontierMath-2025-02-28-Public,o1-mini-2024-09-12_medium,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQKY2tL2pcQ33bDkdMCQzdr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QKY2tL2pcQ33bDkdMCQzdr.eval,https://github.com/epoch-research/benchmarks/actions/runs/13709452985/job/38342621691,6 March 2025 5:32pm,Success,,1.0.0
hC2FY4WuKm6BLgPhbWeZDY,FrontierMath-2025-02-28-Private,o1-mini-2024-09-12_medium,0.017,verification_code:0.02±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/hC2FY4WuKm6BLgPhbWeZDY.eval,https://github.com/epoch-research/benchmarks/actions/runs/13709452985/job/38342620930,6 March 2025 5:32pm,Success,,1.0.0
2ekT2UzaebvmGprKvmeeEa,FrontierMath-2025-02-28-Private,o1-mini-2024-09-12_high,0.014,verification_code:0.01±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/2ekT2UzaebvmGprKvmeeEa.eval,https://github.com/epoch-research/benchmarks/actions/runs/13709452985/job/38342621351,6 March 2025 5:30pm,Success,,1.0.0
9zhPZLvNeUPQ7JBSpTwKPp,OTIS Mock AIME 2024-2025,o1-mini-2024-09-12_high,0.469,model_graded:0.47±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9zhPZLvNeUPQ7JBSpTwKPp.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9zhPZLvNeUPQ7JBSpTwKPp.eval,https://github.com/epoch-research/benchmarks/actions/runs/13708800572/job/38340538445,6 March 2025 4:47pm,Success,,1.0.0
QUTfpiKjphedrKg8H6fGzF,OTIS Mock AIME 2024-2025,o1-mini-2024-09-12_medium,0.447,model_graded:0.45±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQUTfpiKjphedrKg8H6fGzF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QUTfpiKjphedrKg8H6fGzF.eval,https://github.com/epoch-research/benchmarks/actions/runs/13708800572/job/38340538088,6 March 2025 4:47pm,Success,,1.0.0
PU7LXbdNB8Unfvu4qCeJZF,FrontierMath-2025-02-28-Private,claude-3-7-sonnet-20250219_16K,0.041,verification_code:0.04±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PU7LXbdNB8Unfvu4qCeJZF.eval,https://github.com/epoch-research/benchmarks/actions/runs/13706355671/job/38332537271,6 March 2025 2:16pm,Success,,1.0.0
8wm9yBTJHoY6ui45QTpjn2,FrontierMath-2025-02-28-Public,claude-3-7-sonnet-20250219_16K,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F8wm9yBTJHoY6ui45QTpjn2.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/8wm9yBTJHoY6ui45QTpjn2.eval,https://github.com/epoch-research/benchmarks/actions/runs/13706355671/job/38332536680,6 March 2025 2:14pm,Success,,1.0.0
JJqAHUs8LTLCKhnTWQUVTi,FrontierMath-2025-02-28-Public,o3-mini-2025-01-31_high,0.400,verification_code:0.4±0.16,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJJqAHUs8LTLCKhnTWQUVTi.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JJqAHUs8LTLCKhnTWQUVTi.eval,https://github.com/epoch-research/benchmarks/actions/runs/13706297481/job/38332347812,6 March 2025 2:13pm,Success,,1.0.0
GZeqEeaiEhnLJSzVogT9N4,FrontierMath-2025-02-28-Private,o3-mini-2025-01-31_high,0.110,verification_code:0.11±0.02,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/GZeqEeaiEhnLJSzVogT9N4.eval,https://github.com/epoch-research/benchmarks/actions/runs/13706297481/job/38332347290,6 March 2025 2:10pm,Success,,1.0.0
UjGSDrmEPtVoVGdVSbUMsb,FrontierMath-2025-02-28-Public,o3-mini-2025-01-31_medium,0.113,verification_code:0.11±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FUjGSDrmEPtVoVGdVSbUMsb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/UjGSDrmEPtVoVGdVSbUMsb.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328610835,6 March 2025 1:06pm,Success,,1.0.0
kinyvFjqSqUiY5FyYv7tsA,FrontierMath-2025-02-28-Private,grok-2-1212,0.007,verification_code:0.01±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/kinyvFjqSqUiY5FyYv7tsA.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328615602,6 March 2025 1:06pm,Success,,1.0.0
SNDo6TsviYYTiH2kCmf7Pr,FrontierMath-2025-02-28-Public,gpt-4o-2024-11-20,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FSNDo6TsviYYTiH2kCmf7Pr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/SNDo6TsviYYTiH2kCmf7Pr.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328613417,6 March 2025 1:06pm,Success,,1.0.0
kPCieUA4pKDto2xEfvQqSL,FrontierMath-2025-02-28-Public,mistral-large-2411,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FkPCieUA4pKDto2xEfvQqSL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/kPCieUA4pKDto2xEfvQqSL.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328615113,6 March 2025 1:06pm,Success,,1.0.0
dvMWxBZMaxp6adP9zGszYG,FrontierMath-2025-02-28-Public,claude-3-7-sonnet-20250219,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdvMWxBZMaxp6adP9zGszYG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dvMWxBZMaxp6adP9zGszYG.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328614602,6 March 2025 1:06pm,Success,,1.0.0
U8CjtcxudQeP2cnMNixfJH,FrontierMath-2025-02-28-Private,o3-mini-2025-01-31_medium,0.081,verification_code:0.08±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/U8CjtcxudQeP2cnMNixfJH.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328614012,6 March 2025 1:06pm,Success,,1.0.0
7rbXSQ8hFiXCCFv2vtqiiH,FrontierMath-2025-02-28-Public,grok-2-1212,0.000,verification_code:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7rbXSQ8hFiXCCFv2vtqiiH.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7rbXSQ8hFiXCCFv2vtqiiH.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328612891,6 March 2025 1:06pm,Success,,1.0.0
m7X2ppkiEkweCtFppiRW8S,FrontierMath-2025-02-28-Private,claude-3-7-sonnet-20250219,0.031,verification_code:0.03±0.01,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/m7X2ppkiEkweCtFppiRW8S.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328611661,6 March 2025 1:05pm,Success,,1.0.0
cEvTUVyxAqq5UDgsGQbvLM,FrontierMath-2025-02-28-Private,mistral-large-2411,0.003,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/cEvTUVyxAqq5UDgsGQbvLM.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328616213,6 March 2025 1:05pm,Success,,1.0.0
Z7HU95T8eema56EpLtECvG,FrontierMath-2025-02-28-Private,gpt-4o-2024-11-20,0.003,verification_code:0±0,,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Z7HU95T8eema56EpLtECvG.eval,https://github.com/epoch-research/benchmarks/actions/runs/13705146309/job/38328612389,6 March 2025 1:03pm,Success,,1.0.0
japqZFi3FS7um32EHG6wwW,MATH level 5,gpt-4.5-preview-2025-02-27,0.786,"normalized_string_match:0.72±0.01,sympy_equiv:0.73±0.01,model_graded_equiv:0.79±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/japqZFi3FS7um32EHG6wwW.eval,https://github.com/epoch-research/benchmarks/actions/runs/13578638511/job/38001563075,28 February 2025 11:37am,Success,,1.0.0
dodibSv7pk8GtqeMy4YLQg,GPQA diamond,gpt-4.5-preview-2025-02-27,0.687,choice:0.69±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdodibSv7pk8GtqeMy4YLQg.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dodibSv7pk8GtqeMy4YLQg.eval,https://github.com/epoch-research/benchmarks/actions/runs/13584996181/job/37978008808,28 February 2025 4:17am,Success,,1.0.0
9DCJMt7YsTMDenTdX4Db6V,OTIS Mock AIME 2024-2025,gpt-4.5-preview-2025-02-27,0.378,model_graded:0.38±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9DCJMt7YsTMDenTdX4Db6V.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9DCJMt7YsTMDenTdX4Db6V.eval,https://github.com/epoch-research/benchmarks/actions/runs/13584996181/job/37978009207,28 February 2025 4:17am,Success,,1.0.0
AZAJomaSEfZtqYqXUre9fS,MATH level 5,gpt-4-turbo-2024-04-09,0.467,"normalized_string_match:0.44±0.01,sympy_equiv:0.44±0.01,model_graded_equiv:0.47±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/AZAJomaSEfZtqYqXUre9fS.eval,https://github.com/epoch-research/benchmarks/actions/runs/13568698132/job/37927894790,27 February 2025 9:29am,Success,,1.0.0
TaCoN7eyPQEfDWXqs2MpcF,GPQA diamond,gpt-4-turbo-2024-04-09,0.466,choice:0.47±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTaCoN7eyPQEfDWXqs2MpcF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TaCoN7eyPQEfDWXqs2MpcF.eval,https://github.com/epoch-research/benchmarks/actions/runs/13568698132/job/37927895461,27 February 2025 9:29am,Success,,1.0.0
Mjx6vYbgAb2DHXK8xcupX8,OTIS Mock AIME 2024-2025,o3-mini-2025-01-31_high,0.769,model_graded:0.77±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMjx6vYbgAb2DHXK8xcupX8.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Mjx6vYbgAb2DHXK8xcupX8.eval,https://github.com/epoch-research/benchmarks/actions/runs/13568698132/job/37927894174,27 February 2025 9:29am,Success,,1.0.0
MZABVPEY8VCPAyCZ4iir68,OTIS Mock AIME 2024-2025,gpt-4-turbo-2024-04-09,0.067,model_graded:0.07±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMZABVPEY8VCPAyCZ4iir68.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MZABVPEY8VCPAyCZ4iir68.eval,https://github.com/epoch-research/benchmarks/actions/runs/13568698132/job/37927893372,27 February 2025 9:29am,Success,,1.0.0
KvcqycaJpCQMBigiEosyzC,OTIS Mock AIME 2024-2025,o1-2024-12-17_medium,0.733,model_graded:0.73±0.07,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FKvcqycaJpCQMBigiEosyzC.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/KvcqycaJpCQMBigiEosyzC.eval,https://github.com/epoch-research/benchmarks/actions/runs/13566550765/job/37926959990,27 February 2025 9:16am,Success,,1.0.0
64LAJymM6eXDkRZb92Mdg8,OTIS Mock AIME 2024-2025,claude-3-7-sonnet-20250219_16K,0.467,model_graded:0.47±0.08,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F64LAJymM6eXDkRZb92Mdg8.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/64LAJymM6eXDkRZb92Mdg8.eval,https://github.com/epoch-research/benchmarks/actions/runs/13549180001/job/37868413630,26 February 2025 11:56am,Success,,1.0.0
bBY5VZzpDkmTBdSSvu5Ups,OTIS Mock AIME 2024-2025,DeepSeek-R1,0.533,model_graded:0.53±0.08,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbBY5VZzpDkmTBdSSvu5Ups.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/bBY5VZzpDkmTBdSSvu5Ups.eval,https://github.com/epoch-research/benchmarks/actions/runs/13549180001/job/37868414666,26 February 2025 11:56am,Success,,1.0.0
Vn3Ug88CRbxEaJhLbRzAWD,GPQA diamond,claude-3-7-sonnet-20250219_16K,0.768,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVn3Ug88CRbxEaJhLbRzAWD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Vn3Ug88CRbxEaJhLbRzAWD.eval,https://github.com/epoch-research/benchmarks/actions/runs/13549180001/job/37868413159,26 February 2025 11:56am,Success,,1.0.0
hWeVyRo3hN5H4AspNc9cuf,MATH level 5,claude-3-7-sonnet-20250219_16K,0.863,"normalized_string_match:0.78±0.01,sympy_equiv:0.79±0.01,model_graded_equiv:0.86±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/hWeVyRo3hN5H4AspNc9cuf.eval,https://github.com/epoch-research/benchmarks/actions/runs/13549180001/job/37868414136,26 February 2025 11:56am,Success,,1.0.0
6RD3A6J432TZzxrKYRxph2,OTIS Mock AIME 2024-2025,Meta-Llama-3-70B-Instruct,0.043,model_graded:0.04±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F6RD3A6J432TZzxrKYRxph2.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/6RD3A6J432TZzxrKYRxph2.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523750560/job/37789035761,25 February 2025 9:38am,Success,,1.0.0
3S6fjGAMKWKK3zcdxkmh9H,OTIS Mock AIME 2024-2025,gemini-1.5-pro-001,0.068,model_graded:0.07±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F3S6fjGAMKWKK3zcdxkmh9H.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/3S6fjGAMKWKK3zcdxkmh9H.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523750560/job/37789034170,25 February 2025 9:38am,Success,,1.0.0
PpSVDb67Ucbxpcd8JQv5ii,OTIS Mock AIME 2024-2025,grok-2-1212,0.115,model_graded:0.12±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPpSVDb67Ucbxpcd8JQv5ii.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PpSVDb67Ucbxpcd8JQv5ii.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523750560/job/37789035004,25 February 2025 9:38am,Success,,1.0.0
ZjsXQpZ7SsejXDo52aetZZ,OTIS Mock AIME 2024-2025,Llama-3.3-70B-Instruct,0.051,model_graded:0.05±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZjsXQpZ7SsejXDo52aetZZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZjsXQpZ7SsejXDo52aetZZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786682785,25 February 2025 9:05am,Success,,1.0.0
LHRL6nQJZXGJmYppR7kbGw,OTIS Mock AIME 2024-2025,Llama-3.2-90B-Vision-Instruct,0.026,model_graded:0.03±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FLHRL6nQJZXGJmYppR7kbGw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/LHRL6nQJZXGJmYppR7kbGw.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786680824,25 February 2025 9:05am,Success,,1.0.0
NLk2oE4rzSFihPzF95peHr,OTIS Mock AIME 2024-2025,Llama-2-70b-chat-hf,0.000,model_graded:0±0,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNLk2oE4rzSFihPzF95peHr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NLk2oE4rzSFihPzF95peHr.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786666579,25 February 2025 9:05am,Success,,1.0.0
2xNPhKLsiWLHQBLLDGU4VJ,OTIS Mock AIME 2024-2025,claude-3-5-sonnet-20240620,0.065,model_graded:0.07±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2xNPhKLsiWLHQBLLDGU4VJ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2xNPhKLsiWLHQBLLDGU4VJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786679119,25 February 2025 9:05am,Success,,1.0.0
fhCdyX9uUS4WPtjLiVdBdK,OTIS Mock AIME 2024-2025,gemini-1.5-pro-002,0.231,model_graded:0.23±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfhCdyX9uUS4WPtjLiVdBdK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fhCdyX9uUS4WPtjLiVdBdK.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786675559,25 February 2025 9:05am,Success,,1.0.0
mBUKqiVQsqXd3qmdtKP5UP,OTIS Mock AIME 2024-2025,gpt-4o-2024-08-06,0.064,model_graded:0.06±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmBUKqiVQsqXd3qmdtKP5UP.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mBUKqiVQsqXd3qmdtKP5UP.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786676771,25 February 2025 9:05am,Success,,1.0.0
QdGZ2m52PWt3EkLPbcoRwz,OTIS Mock AIME 2024-2025,gemini-1.5-flash-002,0.163,model_graded:0.16±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQdGZ2m52PWt3EkLPbcoRwz.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QdGZ2m52PWt3EkLPbcoRwz.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786682233,25 February 2025 9:05am,Success,,1.0.0
NRCNnCCL5aQKsZf6cKC69Z,OTIS Mock AIME 2024-2025,qwen2.5-32b-instruct,0.074,model_graded:0.07±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNRCNnCCL5aQKsZf6cKC69Z.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NRCNnCCL5aQKsZf6cKC69Z.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786667860,25 February 2025 9:05am,Success,,1.0.0
ZVHvk3GTPUNTgjc5w3vkhW,OTIS Mock AIME 2024-2025,mistral-large-2402,0.019,model_graded:0.02±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZVHvk3GTPUNTgjc5w3vkhW.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZVHvk3GTPUNTgjc5w3vkhW.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786674216,25 February 2025 9:05am,Success,,1.0.0
jGY5Ee5P9dyStA4Fw5Pa6q,OTIS Mock AIME 2024-2025,gemini-1.5-flash-8b-001,0.046,model_graded:0.05±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjGY5Ee5P9dyStA4Fw5Pa6q.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jGY5Ee5P9dyStA4Fw5Pa6q.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786667165,25 February 2025 9:05am,Success,,1.0.0
beRGgJXTJggTi8PPEZNoZ5,OTIS Mock AIME 2024-2025,mistral-large-2411,0.078,model_graded:0.08±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbeRGgJXTJggTi8PPEZNoZ5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/beRGgJXTJggTi8PPEZNoZ5.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786677386,25 February 2025 9:05am,Success,,1.0.0
AeRCEiG87bMFpxbxbmjCyn,OTIS Mock AIME 2024-2025,claude-3-sonnet-20240229,0.025,model_graded:0.03±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FAeRCEiG87bMFpxbxbmjCyn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/AeRCEiG87bMFpxbxbmjCyn.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786662575,25 February 2025 9:05am,Success,,1.0.0
7saSEhfVvXuz4yTcGy5u2y,OTIS Mock AIME 2024-2025,DeepSeek-V3,0.158,model_graded:0.16±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7saSEhfVvXuz4yTcGy5u2y.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7saSEhfVvXuz4yTcGy5u2y.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786668976,25 February 2025 9:05am,Success,,1.0.0
NpJvYQDcmwbGgS5MzZUPUg,OTIS Mock AIME 2024-2025,gemini-1.0-pro-001,0.011,model_graded:0.01±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNpJvYQDcmwbGgS5MzZUPUg.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NpJvYQDcmwbGgS5MzZUPUg.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786670829,25 February 2025 9:05am,Success,,1.0.0
6SY8iSf5uKn7SrCzzrabPb,OTIS Mock AIME 2024-2025,Llama-3.1-8B-Instruct,0.025,model_graded:0.03±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F6SY8iSf5uKn7SrCzzrabPb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/6SY8iSf5uKn7SrCzzrabPb.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786674838,25 February 2025 9:05am,Success,,1.0.0
2fQJP3brRApfBdQq7Zy3bm,OTIS Mock AIME 2024-2025,Llama-3.1-405B-Instruct,0.097,model_graded:0.1±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F2fQJP3brRApfBdQq7Zy3bm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/2fQJP3brRApfBdQq7Zy3bm.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786665338,25 February 2025 9:05am,Success,,1.0.0
N2NcmgmHyGJ9HNmQe5yvcS,OTIS Mock AIME 2024-2025,claude-3-7-sonnet-20250219,0.219,model_graded:0.22±0.05,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FN2NcmgmHyGJ9HNmQe5yvcS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/N2NcmgmHyGJ9HNmQe5yvcS.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786679617,25 February 2025 9:05am,Success,,1.0.0
geVVaRWxuprKUqnW2VzvCa,OTIS Mock AIME 2024-2025,claude-3-5-sonnet-20241022,0.085,model_graded:0.08±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FgeVVaRWxuprKUqnW2VzvCa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/geVVaRWxuprKUqnW2VzvCa.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786681577,25 February 2025 9:05am,Success,,1.0.0
67YbVQfJh7sd77KH3zFgTn,OTIS Mock AIME 2024-2025,qwen2.5-72b-instruct,0.081,model_graded:0.08±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F67YbVQfJh7sd77KH3zFgTn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/67YbVQfJh7sd77KH3zFgTn.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786669493,25 February 2025 9:05am,Success,,1.0.0
9cCn5aW522cTRxJSE69crm,OTIS Mock AIME 2024-2025,gpt-4o-2024-05-13,0.063,model_graded:0.06±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9cCn5aW522cTRxJSE69crm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9cCn5aW522cTRxJSE69crm.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786676142,25 February 2025 9:05am,Success,,1.0.0
X2tQXGELXtEbc2VX6gARd7,OTIS Mock AIME 2024-2025,Llama-3.1-70B-Instruct,0.036,model_graded:0.04±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FX2tQXGELXtEbc2VX6gARd7.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/X2tQXGELXtEbc2VX6gARd7.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786665990,25 February 2025 9:05am,Success,,1.0.0
cqhrR5KnwpcuAVpHkZGQFd,OTIS Mock AIME 2024-2025,claude-3-5-haiku-20241022,0.043,model_graded:0.04±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FcqhrR5KnwpcuAVpHkZGQFd.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/cqhrR5KnwpcuAVpHkZGQFd.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786664746,25 February 2025 9:05am,Success,,1.0.0
U9GVBhYjRzU55YVjna5mMT,OTIS Mock AIME 2024-2025,gemini-1.5-flash-001,0.039,model_graded:0.04±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FU9GVBhYjRzU55YVjna5mMT.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/U9GVBhYjRzU55YVjna5mMT.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786680220,25 February 2025 9:05am,Success,,1.0.0
ThMd6DNDTtyWxhPcb9Kq4y,OTIS Mock AIME 2024-2025,claude-3-haiku-20240307,0.018,model_graded:0.02±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FThMd6DNDTtyWxhPcb9Kq4y.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ThMd6DNDTtyWxhPcb9Kq4y.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786678562,25 February 2025 9:05am,Success,,1.0.0
7P5HLaaD6DBjNYFx7o6Jma,OTIS Mock AIME 2024-2025,Meta-Llama-3-8B-Instruct,0.008,model_graded:0.01±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7P5HLaaD6DBjNYFx7o6Jma.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7P5HLaaD6DBjNYFx7o6Jma.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786673512,25 February 2025 9:05am,Success,,1.0.0
mYb9MGQD39RS8VdDyMqujr,OTIS Mock AIME 2024-2025,claude-3-opus-20240229,0.047,model_graded:0.05±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmYb9MGQD39RS8VdDyMqujr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mYb9MGQD39RS8VdDyMqujr.eval,https://github.com/epoch-research/benchmarks/actions/runs/13523028515/job/37786670160,25 February 2025 9:05am,Success,,1.0.0
ajytUBY9bDPNNxuBfi5UBS,OTIS Mock AIME 2024-2025,o3-mini-2025-01-31_medium,0.639,model_graded:0.64±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FajytUBY9bDPNNxuBfi5UBS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ajytUBY9bDPNNxuBfi5UBS.eval,https://github.com/epoch-research/benchmarks/actions/runs/13522428324/job/37784639644,25 February 2025 8:33am,Success,,1.0.0
5mmrBSQb7hVSJ9VYtNm46i,OTIS Mock AIME 2024-2025,phi-4,0.138,model_graded:0.14±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F5mmrBSQb7hVSJ9VYtNm46i.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/5mmrBSQb7hVSJ9VYtNm46i.eval,https://github.com/epoch-research/benchmarks/actions/runs/13522165879/job/37783783008,25 February 2025 8:19am,Success,,1.0.0
buNVL2TWcdVgxAcMfVE8xJ,OTIS Mock AIME 2024-2025,gpt-4o-2024-11-20,0.063,model_graded:0.06±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbuNVL2TWcdVgxAcMfVE8xJ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/buNVL2TWcdVgxAcMfVE8xJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/13521972102/job/37783169553,25 February 2025 8:10am,Success,,1.0.0
MMqL6fQSwsBKAdBSfxrYpU,MATH level 5,mistral-large-2411,0.503,"normalized_string_match:0.47±0.01,sympy_equiv:0.47±0.01,model_graded_equiv:0.5±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/MMqL6fQSwsBKAdBSfxrYpU.eval,https://github.com/epoch-research/benchmarks/actions/runs/13521243052/job/37780892700,25 February 2025 7:31am,Success,,1.0.0
n9gpXfUzo8SbWqvrG2q4xM,GPQA diamond,mistral-large-2411,0.513,choice:0.51±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fn9gpXfUzo8SbWqvrG2q4xM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/n9gpXfUzo8SbWqvrG2q4xM.eval,https://github.com/epoch-research/benchmarks/actions/runs/13521243052/job/37780893125,25 February 2025 7:31am,Success,,1.0.0
ZnGSRrKbiEnuju4kk4sgVb,OTIS Mock AIME 2024-2025,gemini-2.0-flash-001,0.311,model_graded:0.31±0.06,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZnGSRrKbiEnuju4kk4sgVb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZnGSRrKbiEnuju4kk4sgVb.eval,https://github.com/epoch-research/benchmarks/actions/runs/13521129822/job/37780559091,25 February 2025 7:25am,Success,,1.0.0
bHLfPfZj8HgD9Gef75NtrM,OTIS Mock AIME 2024-2025,mistral-large-2407,0.085,model_graded:0.08±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbHLfPfZj8HgD9Gef75NtrM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/bHLfPfZj8HgD9Gef75NtrM.eval,https://github.com/epoch-research/benchmarks/actions/runs/13520643249/job/37779047265,25 February 2025 6:59am,Success,,1.0.0
PyNEkFCfMEReodeE2oiPhh,GPQA diamond,claude-3-7-sonnet-20250219,0.660,choice:0.66±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPyNEkFCfMEReodeE2oiPhh.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PyNEkFCfMEReodeE2oiPhh.eval,https://github.com/epoch-research/benchmarks/actions/runs/13505916497/job/37735366866,24 February 2025 2:14pm,Success,,1.0.0
6uSKG2QxR8atueXmDWAPXn,MATH level 5,claude-3-7-sonnet-20250219,0.682,"normalized_string_match:0.61±0.01,sympy_equiv:0.63±0.01,model_graded_equiv:0.68±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6uSKG2QxR8atueXmDWAPXn.eval,https://github.com/epoch-research/benchmarks/actions/runs/13505916497/job/37735367417,24 February 2025 2:14pm,Success,,1.0.0
Kpck3pLw3veM5Mt7WZAb8G,MATH level 5,grok-2-1212,0.635,"normalized_string_match:0.6±0.01,sympy_equiv:0.6±0.01,model_graded_equiv:0.64±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Kpck3pLw3veM5Mt7WZAb8G.eval,https://github.com/epoch-research/benchmarks/actions/runs/13378944447/job/37363904801,17 February 2025 4:39pm,Success,,1.0.0
NAkCFiFiDSN3NXMj7HMvMS,GPQA diamond,o1-2024-12-17_high,0.768,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNAkCFiFiDSN3NXMj7HMvMS.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NAkCFiFiDSN3NXMj7HMvMS.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313179292/job/37180750157,13 February 2025 12:12pm,Success,,1.0.0
jVXtRkTBDnqH8p4sjVUbgN,MATH level 5,o1-2024-12-17_high,0.947,"normalized_string_match:0.78±0.01,sympy_equiv:0.8±0.01,model_graded_equiv:0.95±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/jVXtRkTBDnqH8p4sjVUbgN.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313179292/job/37180750684,13 February 2025 12:12pm,Success,,1.0.0
4DWnpJowAfeQDHc2xDNy2C,MATH level 5,o1-mini-2024-09-12_high,0.892,"normalized_string_match:0.82±0.01,sympy_equiv:0.82±0.01,model_graded_equiv:0.89±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/4DWnpJowAfeQDHc2xDNy2C.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313098994/job/37180469593,13 February 2025 12:07pm,Success,,1.0.0
X2gKewiGPZx5DLFEddUUqW,GPQA diamond,o1-mini-2024-09-12_high,0.624,choice:0.62±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FX2gKewiGPZx5DLFEddUUqW.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/X2gKewiGPZx5DLFEddUUqW.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313098994/job/37180470703,13 February 2025 12:07pm,Success,,1.0.0
MzsTvCEEfEpzn4WvxYdJkX,GPQA diamond,o3-mini-2025-01-31_high,0.770,choice:0.77±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMzsTvCEEfEpzn4WvxYdJkX.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MzsTvCEEfEpzn4WvxYdJkX.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313098994/job/37180470142,13 February 2025 12:07pm,Success,,1.0.0
3VDhPT5WvzjJZkRc36Kuvi,MATH level 5,o3-mini-2025-01-31_high,0.965,"normalized_string_match:0.8±0.01,sympy_equiv:0.82±0.01,model_graded_equiv:0.96±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/3VDhPT5WvzjJZkRc36Kuvi.eval,https://github.com/epoch-research/benchmarks/actions/runs/13313098994/job/37180471237,13 February 2025 12:07pm,Success,,1.0.0
g5YE7G6DXb45EH4HxEdiht,GPQA diamond,gemini-2.0-pro-exp-02-05,0.657,choice:0.66±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fg5YE7G6DXb45EH4HxEdiht.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/g5YE7G6DXb45EH4HxEdiht.eval,https://github.com/epoch-research/benchmarks/actions/runs/13195201579/job/36835159846,7 February 2025 2:28am,Success,,1.0.0
ag9w2w4xa47ghZaWt5oZey,GPQA diamond,gemini-2.0-flash-thinking-exp-01-21,0.571,choice:0.57±0.04,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fag9w2w4xa47ghZaWt5oZey.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ag9w2w4xa47ghZaWt5oZey.eval,https://github.com/epoch-research/benchmarks/actions/runs/13189243444/job/36818973402,6 February 2025 5:40pm,Success,,1.0.0
fvDt6HaHV5d3gSEuavixsv,MATH level 5,gemini-2.0-pro-exp-02-05,0.835,"normalized_string_match:0.79±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.83±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/fvDt6HaHV5d3gSEuavixsv.eval,https://github.com/epoch-research/benchmarks/actions/runs/13184052596/job/36801881863,6 February 2025 12:07pm,Success,,1.0.0
fbszwTApm3f28z5VA6oXBK,GPQA diamond,gemini-2.0-flash-001,0.641,choice:0.64±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfbszwTApm3f28z5VA6oXBK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fbszwTApm3f28z5VA6oXBK.eval,https://github.com/epoch-research/benchmarks/actions/runs/13182616465/job/36797026298,6 February 2025 10:51am,Success,,1.0.0
SqwGZz2UfTmbvPiw2rq3LH,MATH level 5,gemini-2.0-flash-001,0.822,"normalized_string_match:0.79±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.82±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/SqwGZz2UfTmbvPiw2rq3LH.eval,https://github.com/epoch-research/benchmarks/actions/runs/13182616465/job/36797025731,6 February 2025 10:51am,Success,,1.0.0
D3G2BKGnh6YSvw4bEYSdiY,MATH level 5,gpt-4o-2024-11-20,0.498,"normalized_string_match:0.45±0.01,sympy_equiv:0.47±0.01,model_graded_equiv:0.5±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/D3G2BKGnh6YSvw4bEYSdiY.eval,https://github.com/epoch-research/benchmarks/actions/runs/13153219447/job/36707709678,5 February 2025 3:31am,Success,,1.0.0
gULKGqAe9nG9syxmYMtkhi,GPQA diamond,gpt-4o-2024-11-20,0.479,choice:0.48±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FgULKGqAe9nG9syxmYMtkhi.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/gULKGqAe9nG9syxmYMtkhi.eval,https://github.com/epoch-research/benchmarks/actions/runs/13153219447/job/36704554109,5 February 2025 3:31am,Success,,1.0.0
njA86GC9DzSXyiZWtUxXiD,MATH level 5,Phi-3-medium-128k-instruct,0.176,"normalized_string_match:0.07±0,sympy_equiv:0.08±0,model_graded_equiv:0.18±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/njA86GC9DzSXyiZWtUxXiD.eval,https://github.com/epoch-research/benchmarks/actions/runs/13082664506/job/36509174015,31 January 2025 6:48pm,Success,,1.0.0
XVqLs2khGFSNwEbHJTEgRU,MATH level 5,DeepSeek-R1,0.931,"normalized_string_match:0.78±0.01,sympy_equiv:0.78±0.01,model_graded_equiv:0.93±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/XVqLs2khGFSNwEbHJTEgRU.eval,https://github.com/epoch-research/benchmarks/actions/runs/13082664506/job/36509173805,31 January 2025 6:48pm,Success,,1.0.0
PnvmfAHVxH9oXfcdcUeUZ3,MATH level 5,phi-4,0.649,"normalized_string_match:0.04±0,sympy_equiv:0.05±0,model_graded_equiv:0.65±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PnvmfAHVxH9oXfcdcUeUZ3.eval,https://github.com/epoch-research/benchmarks/actions/runs/13082664506/job/36509173585,31 January 2025 6:48pm,Success,,1.0.0
RAwuPRwJedy7P2KeQhEmr8,GPQA diamond,phi-4,0.561,choice:0.56±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRAwuPRwJedy7P2KeQhEmr8.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/RAwuPRwJedy7P2KeQhEmr8.eval,https://github.com/epoch-research/benchmarks/actions/runs/13082203539/job/36507954998,31 January 2025 6:03pm,Success,,1.0.0
bDS2AGEf3AywbVcdDyAKZK,GPQA diamond,Phi-3-medium-128k-instruct,0.276,choice:0.28±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbDS2AGEf3AywbVcdDyAKZK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/bDS2AGEf3AywbVcdDyAKZK.eval,https://github.com/epoch-research/benchmarks/actions/runs/13081982125/job/36507279664,31 January 2025 5:41pm,Success,,1.0.0
2RiSB2ii5kf2Q6fDBv2P3E,MATH level 5,o3-mini-2025-01-31_medium,0.952,"normalized_string_match:0.78±0.01,sympy_equiv:0.81±0.01,model_graded_equiv:0.95±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/2RiSB2ii5kf2Q6fDBv2P3E.eval,https://github.com/epoch-research/benchmarks/actions/runs/13079508039/job/36499657510,31 January 2025 2:32pm,Success,,1.0.0
ehvA5nisC7GMbgbyy3Z4Et,GPQA diamond,o3-mini-2025-01-31_medium,0.743,choice:0.74±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FehvA5nisC7GMbgbyy3Z4Et.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ehvA5nisC7GMbgbyy3Z4Et.eval,https://github.com/epoch-research/benchmarks/actions/runs/13079365912/job/36499187420,31 January 2025 2:21pm,Success,,1.0.0
oDq358GfyV5Au65scERdVo,GPQA diamond,mistral-small-2501,0.453,choice:0.45±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FoDq358GfyV5Au65scERdVo.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/oDq358GfyV5Au65scERdVo.eval,https://github.com/epoch-research/benchmarks/actions/runs/13061281198/job/36444695915,30 January 2025 4:10pm,Success,,1.0.0
Nk4mmCXFH3mEyCYnQPdPx5,MATH level 5,qwen2.5-32b-instruct,0.561,"normalized_string_match:0.51±0.01,sympy_equiv:0.54±0.01,model_graded_equiv:0.56±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Nk4mmCXFH3mEyCYnQPdPx5.eval,https://github.com/epoch-research/benchmarks/actions/runs/13059332878/job/36438250255,30 January 2025 2:02pm,Success,,1.0.0
d6JihAFVqUy3u9suNt4P3L,GPQA diamond,qwen2.5-32b-instruct,0.461,choice:0.46±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Fd6JihAFVqUy3u9suNt4P3L.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/d6JihAFVqUy3u9suNt4P3L.eval,https://github.com/epoch-research/benchmarks/actions/runs/13058835364/job/36436614709,30 January 2025 1:31pm,Success,,1.0.0
NbsnvBsMoMizozbPZY8LLb,MATH level 5,mistral-small-2501,0.448,"normalized_string_match:0.43±0.01,sympy_equiv:0.42±0.01,model_graded_equiv:0.45±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/NbsnvBsMoMizozbPZY8LLb.eval,https://github.com/epoch-research/benchmarks/actions/runs/13057762671/job/36433055052,30 January 2025 12:26pm,Success,,1.0.0
Hc2ycMP6DyCZxCPjKbZFQU,GPQA diamond,gemini-1.5-flash-001,0.404,choice:0.4±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FHc2ycMP6DyCZxCPjKbZFQU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Hc2ycMP6DyCZxCPjKbZFQU.eval,https://github.com/epoch-research/benchmarks/actions/runs/12990127182/job/36224710675,26 January 2025 7:00pm,Success,,1.0.0
mTa4f52zsEuxdrPy7WfQvX,GPQA diamond,claude-2.0,0.347,choice:0.35±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FmTa4f52zsEuxdrPy7WfQvX.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/mTa4f52zsEuxdrPy7WfQvX.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803505131,26 January 2025 7:00pm,Success,,1.0.0
hTnDa4gn8RYduCEYqoNywe,GPQA diamond,claude-3-opus-20240229,0.472,choice:0.47±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FhTnDa4gn8RYduCEYqoNywe.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/hTnDa4gn8RYduCEYqoNywe.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803505680,26 January 2025 7:00pm,Success,,1.0.0
4NiUpCrLfz3oFaerkrMeBt,GPQA diamond,o1-mini-2024-09-12_medium,0.595,choice:0.6±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F4NiUpCrLfz3oFaerkrMeBt.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/4NiUpCrLfz3oFaerkrMeBt.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803501056,26 January 2025 7:00pm,Success,,1.0.0
fairJpL9VekDBKxYHG9ZoG,GPQA diamond,gpt-4o-2024-08-06,0.492,choice:0.49±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfairJpL9VekDBKxYHG9ZoG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fairJpL9VekDBKxYHG9ZoG.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803500569,26 January 2025 7:00pm,Success,,1.0.0
7CkADTsoNbDd88uqYbqfwj,GPQA diamond,o1-preview-2024-09-12,0.503,choice:0.5±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F7CkADTsoNbDd88uqYbqfwj.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/7CkADTsoNbDd88uqYbqfwj.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803500809,26 January 2025 7:00pm,Success,,1.0.0
QLejPZXhjX6SQCQgzjNvCG,GPQA diamond,claude-3-sonnet-20240229,0.406,choice:0.41±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FQLejPZXhjX6SQCQgzjNvCG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/QLejPZXhjX6SQCQgzjNvCG.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803503809,26 January 2025 7:00pm,Success,,1.0.0
JCCVcVLz3J4TQXSU67hMJZ,GPQA diamond,claude-3-5-sonnet-20241022,0.553,choice:0.55±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJCCVcVLz3J4TQXSU67hMJZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/JCCVcVLz3J4TQXSU67hMJZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803504396,26 January 2025 7:00pm,Success,,1.0.0
4j3WRhhruXrfaq9Jki3NDc,GPQA diamond,gpt-4-0125-preview,0.423,choice:0.42±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F4j3WRhhruXrfaq9Jki3NDc.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/4j3WRhhruXrfaq9Jki3NDc.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803504548,26 January 2025 7:00pm,Success,,1.0.0
eMAisrJd7fAUHPfcPWXeXo,GPQA diamond,claude-2.1,0.330,choice:0.33±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FeMAisrJd7fAUHPfcPWXeXo.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/eMAisrJd7fAUHPfcPWXeXo.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804478709,26 January 2025 7:00pm,Success,,1.0.0
ZUJxsJDpHnLXxNiDcvj22W,GPQA diamond,gemma-2-27b-it,0.365,choice:0.36±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZUJxsJDpHnLXxNiDcvj22W.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZUJxsJDpHnLXxNiDcvj22W.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803504925,26 January 2025 7:00pm,Success,,1.0.0
T3FWdCCWBryVjxwTcoY5bK,GPQA diamond,claude-3-haiku-20240307,0.363,choice:0.36±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FT3FWdCCWBryVjxwTcoY5bK.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/T3FWdCCWBryVjxwTcoY5bK.eval,https://github.com/epoch-research/benchmarks/actions/runs/12835464845/job/35795014852,26 January 2025 7:00pm,Success,,1.0.0
78Wpvp8WDByAcQRBas72cs,GPQA diamond,claude-3-5-sonnet-20240620,0.540,choice:0.54±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F78Wpvp8WDByAcQRBas72cs.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/78Wpvp8WDByAcQRBas72cs.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803505307,26 January 2025 7:00pm,Success,,1.0.0
Y59q8WbEDp2HH8oovyJCj9,GPQA diamond,gpt-4o-2024-05-13,0.489,choice:0.49±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FY59q8WbEDp2HH8oovyJCj9.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Y59q8WbEDp2HH8oovyJCj9.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803501304,26 January 2025 7:00pm,Success,,1.0.0
PCLvrs4TEBtqQqcYeU8pAG,GPQA diamond,gemini-1.5-flash-002,0.473,choice:0.47±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPCLvrs4TEBtqQqcYeU8pAG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PCLvrs4TEBtqQqcYeU8pAG.eval,https://github.com/epoch-research/benchmarks/actions/runs/12835464845/job/35795006637,26 January 2025 7:00pm,Success,,1.0.0
fu7pWqbSC4MUBdwv3GFNa5,GPQA diamond,gpt-4-0613,0.307,choice:0.31±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2Ffu7pWqbSC4MUBdwv3GFNa5.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fu7pWqbSC4MUBdwv3GFNa5.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803501743,26 January 2025 7:00pm,Success,,1.0.0
fQjoHKyUEPAuwdrBGfL8SG,GPQA diamond,gemma-2-9b-it,0.275,choice:0.27±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FfQjoHKyUEPAuwdrBGfL8SG.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/fQjoHKyUEPAuwdrBGfL8SG.eval,https://github.com/epoch-research/benchmarks/actions/runs/12835464845/job/35795008115,26 January 2025 7:00pm,Success,,1.0.0
enhRcQmfN6RFB2zu7R3Uxr,GPQA diamond,gpt-3.5-turbo-1106,0.280,choice:0.28±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FenhRcQmfN6RFB2zu7R3Uxr.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/enhRcQmfN6RFB2zu7R3Uxr.eval,https://github.com/epoch-research/benchmarks/actions/runs/12835464845/job/35795007717,26 January 2025 7:00pm,Success,,1.0.0
PKdqpKZ9tqDWyWJ7GksBWz,GPQA diamond,gemini-1.5-pro-001,0.459,choice:0.46±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPKdqpKZ9tqDWyWJ7GksBWz.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/PKdqpKZ9tqDWyWJ7GksBWz.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803502379,26 January 2025 7:00pm,Success,,1.0.0
AWewipyfSQb8hXtS8d4fNL,GPQA diamond,gpt-4-1106-preview,0.424,choice:0.42±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FAWewipyfSQb8hXtS8d4fNL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/AWewipyfSQb8hXtS8d4fNL.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803502827,26 January 2025 7:00pm,Success,,1.0.0
L9yfwZC3snsKZshBXTaXjc,GPQA diamond,gpt-4o-mini-2024-07-18,0.377,choice:0.38±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FL9yfwZC3snsKZshBXTaXjc.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/L9yfwZC3snsKZshBXTaXjc.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803503027,26 January 2025 7:00pm,Success,,1.0.0
9TrLEEw8TvsyugyCjwvUvD,GPQA diamond,gemini-1.5-pro-002,0.572,choice:0.57±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9TrLEEw8TvsyugyCjwvUvD.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9TrLEEw8TvsyugyCjwvUvD.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803503410,26 January 2025 7:00pm,Success,,1.0.0
eZaQhpbhbzbo3sbKGS5Efa,GPQA diamond,gpt-3.5-turbo-0125,0.272,choice:0.27±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FeZaQhpbhbzbo3sbKGS5Efa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/eZaQhpbhbzbo3sbKGS5Efa.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838231295/job/35803503606,26 January 2025 7:00pm,Success,,1.0.0
WsSrH8oHBm7m7vq9yTRqSC,MATH level 5,gpt-4-0613,0.230,"normalized_string_match:0.2±0.01,sympy_equiv:0.2±0.01,model_graded_equiv:0.23±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/WsSrH8oHBm7m7vq9yTRqSC.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804475254,26 January 2025 7:00pm,Success,,1.0.0
CPpd4hqQHtZ5pUejBELwKb,MATH level 5,gemini-1.5-flash-001,0.251,"normalized_string_match:0.24±0.01,sympy_equiv:0.23±0.01,model_graded_equiv:0.25±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/CPpd4hqQHtZ5pUejBELwKb.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804476432,26 January 2025 7:00pm,Success,,1.0.0
QyqLmgcJpeDMsnQyaaJVkR,MATH level 5,gemini-1.5-pro-002,0.704,"normalized_string_match:0.67±0.01,sympy_equiv:0.67±0.01,model_graded_equiv:0.7±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/QyqLmgcJpeDMsnQyaaJVkR.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804476588,26 January 2025 7:00pm,Success,,1.0.0
9tSgxGV3nzhTBpq7GcX7yx,MATH level 5,gpt-3.5-turbo-0125,0.116,"normalized_string_match:0.1±0,sympy_equiv:0.11±0.01,model_graded_equiv:0.12±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/9tSgxGV3nzhTBpq7GcX7yx.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804477923,26 January 2025 7:00pm,Success,,1.0.0
PxLmyq7PFfARAuqxg5oPtE,MATH level 5,gemini-1.5-pro-001,0.407,"normalized_string_match:0.38±0.01,sympy_equiv:0.37±0.01,model_graded_equiv:0.41±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PxLmyq7PFfARAuqxg5oPtE.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804478213,26 January 2025 7:00pm,Success,,1.0.0
VBStRVGFi47EVz2pn33xu4,MATH level 5,gpt-4-1106-preview,0.400,"normalized_string_match:0.37±0.01,sympy_equiv:0.38±0.01,model_graded_equiv:0.4±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/VBStRVGFi47EVz2pn33xu4.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804478542,26 January 2025 7:00pm,Success,,1.0.0
cJWh887zSiTtnxm9XAJVkr,MATH level 5,gpt-3.5-turbo-1106,0.159,"normalized_string_match:0.14±0.01,sympy_equiv:0.15±0.01,model_graded_equiv:0.16±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/cJWh887zSiTtnxm9XAJVkr.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804477260,26 January 2025 7:00pm,Success,,1.0.0
kDnfjYKQpnLGEjuhBc9AvM,MATH level 5,gemini-1.5-flash-002,0.619,"normalized_string_match:0.6±0.01,sympy_equiv:0.59±0.01,model_graded_equiv:0.62±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/kDnfjYKQpnLGEjuhBc9AvM.eval,https://github.com/epoch-research/benchmarks/actions/runs/12838592251/job/35804477421,26 January 2025 7:00pm,Success,,1.0.0
WWS3aotkrymLmBBDhhtvWQ,GPQA diamond,Llama-3.1-8B-Instruct,0.259,choice:0.26±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWWS3aotkrymLmBBDhhtvWQ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WWS3aotkrymLmBBDhhtvWQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12872689305/job/35888776536,26 January 2025 7:00pm,Success,,1.0.0
MEwMcmSV32JHrsNoAq4BFL,GPQA diamond,Llama-3.1-70B-Instruct,0.442,choice:0.44±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMEwMcmSV32JHrsNoAq4BFL.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MEwMcmSV32JHrsNoAq4BFL.eval,https://github.com/epoch-research/benchmarks/actions/runs/12872689305/job/35888775730,26 January 2025 7:00pm,Success,,1.0.0
S5QYXSvQBRSbUbXSnAGbMm,GPQA diamond,Llama-3.1-405B-Instruct,0.509,choice:0.51±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FS5QYXSvQBRSbUbXSnAGbMm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/S5QYXSvQBRSbUbXSnAGbMm.eval,https://github.com/epoch-research/benchmarks/actions/runs/12872689305/job/35888776141,26 January 2025 7:00pm,Success,,1.0.0
Gs4vxvDD5wmrSvyTkqx2YV,MATH level 5,claude-2.0,0.117,"normalized_string_match:0.1±0.01,sympy_equiv:0.1±0.01,model_graded_equiv:0.12±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Gs4vxvDD5wmrSvyTkqx2YV.eval,https://github.com/epoch-research/benchmarks/actions/runs/12875936772/job/35898128525,26 January 2025 7:00pm,Success,,1.0.0
AjroyuZECXLtMTwnzaMHtX,MATH level 5,claude-3-5-sonnet-20241022,0.569,"normalized_string_match:0.49±0.01,sympy_equiv:0.52±0.01,model_graded_equiv:0.57±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/AjroyuZECXLtMTwnzaMHtX.eval,https://github.com/epoch-research/benchmarks/actions/runs/12875936772/job/35898129075,26 January 2025 7:00pm,Success,,1.0.0
H42HJG29bwqzaMewTMsmky,MATH level 5,claude-3-5-sonnet-20240620,0.517,"normalized_string_match:0.44±0.01,sympy_equiv:0.47±0.01,model_graded_equiv:0.52±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/H42HJG29bwqzaMewTMsmky.eval,https://github.com/epoch-research/benchmarks/actions/runs/12873680967/job/35891748261,26 January 2025 7:00pm,Success,,1.0.0
6SNzvpd6kXXxqpJoaHsSDd,MATH level 5,claude-3-opus-20240229,0.375,"normalized_string_match:0.35±0.01,sympy_equiv:0.36±0.01,model_graded_equiv:0.37±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6SNzvpd6kXXxqpJoaHsSDd.eval,https://github.com/epoch-research/benchmarks/actions/runs/12877381513/job/35901762988,26 January 2025 7:00pm,Success,,1.0.0
QSnWVKUVj9six4jYj4HJD9,MATH level 5,claude-3-haiku-20240307,0.149,"normalized_string_match:0.13±0.01,sympy_equiv:0.13±0.01,model_graded_equiv:0.15±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/QSnWVKUVj9six4jYj4HJD9.eval,https://github.com/epoch-research/benchmarks/actions/runs/12873680967/job/35891747774,26 January 2025 7:00pm,Success,,1.0.0
FD2ztfjFUeqGyrtF8yuCUB,MATH level 5,claude-3-sonnet-20240229,0.182,"normalized_string_match:0.15±0.01,sympy_equiv:0.16±0.01,model_graded_equiv:0.18±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/FD2ztfjFUeqGyrtF8yuCUB.eval,https://github.com/epoch-research/benchmarks/actions/runs/12875936772/job/35898128000,26 January 2025 7:00pm,Success,,1.0.0
Jwi8JTeeTxJuJY8qidxngw,GPQA diamond,Yi-1.5-34B-Chat,0.320,choice:0.32±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FJwi8JTeeTxJuJY8qidxngw.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Jwi8JTeeTxJuJY8qidxngw.eval,https://github.com/epoch-research/benchmarks/actions/runs/12877381513/job/35901762861,26 January 2025 7:00pm,Success,,1.0.0
WxMpEBcTf8qPFPz4BrF86B,GPQA diamond,Yi-34B-Chat,0.147,choice:0.15±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWxMpEBcTf8qPFPz4BrF86B.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WxMpEBcTf8qPFPz4BrF86B.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874298061/job/35893510079,26 January 2025 7:00pm,Success,,1.0.0
gyxR3YTcmh4UcKZkET5pPX,MATH level 5,Llama-3.1-8B-Instruct,0.229,"normalized_string_match:0.21±0.01,sympy_equiv:0.21±0.01,model_graded_equiv:0.23±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/gyxR3YTcmh4UcKZkET5pPX.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874334167/job/35893609888,26 January 2025 7:00pm,Success,,1.0.0
ZR7amdUBRr5Cke8Aoz4snr,MATH level 5,Llama-3.1-405B-Instruct,0.498,"normalized_string_match:0.47±0.01,sympy_equiv:0.47±0.01,model_graded_equiv:0.5±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZR7amdUBRr5Cke8Aoz4snr.eval,https://github.com/epoch-research/benchmarks/actions/runs/12881809083/job/35913026410,26 January 2025 7:00pm,Success,,1.0.0
6JiXiQ67iyiC7cFFRiogvs,MATH level 5,Llama-3.1-70B-Instruct,0.367,"normalized_string_match:0.34±0.01,sympy_equiv:0.34±0.01,model_graded_equiv:0.37±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6JiXiQ67iyiC7cFFRiogvs.eval,https://github.com/epoch-research/benchmarks/actions/runs/12877977969/job/35903276325,26 January 2025 7:00pm,Success,,1.0.0
83DRJD4FvjwqU9eStRaNiU,GPQA diamond,grok-2-1212,0.538,choice:0.54±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F83DRJD4FvjwqU9eStRaNiU.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/83DRJD4FvjwqU9eStRaNiU.eval,https://github.com/epoch-research/benchmarks/actions/runs/12892934827/job/35948335118,26 January 2025 7:00pm,Success,,1.0.0
Uqsvq9TAGoqD2xpKfhHBmS,MATH level 5,gpt-4o-2024-05-13,0.510,"normalized_string_match:0.47±0.01,sympy_equiv:0.47±0.01,model_graded_equiv:0.51±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Uqsvq9TAGoqD2xpKfhHBmS.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874621939/job/35894428536,26 January 2025 7:00pm,Success,,1.0.0
5iyKgWVhAmNwrA5rNL7MbW,MATH level 5,gpt-4o-mini-2024-07-18,0.526,"normalized_string_match:0.48±0.01,sympy_equiv:0.48±0.01,model_graded_equiv:0.53±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/5iyKgWVhAmNwrA5rNL7MbW.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874621939/job/35894428770,26 January 2025 7:00pm,Success,,1.0.0
ZQgz8AGm4KHin9MFupLu58,MATH level 5,gpt-4o-2024-08-06,0.533,"normalized_string_match:0.49±0.01,sympy_equiv:0.5±0.01,model_graded_equiv:0.53±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZQgz8AGm4KHin9MFupLu58.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874621939/job/35894428195,26 January 2025 7:00pm,Success,,1.0.0
Pex6SpYJX8ZPySncfNNy7H,GPQA diamond,qwen2-72b-instruct,0.408,choice:0.41±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPex6SpYJX8ZPySncfNNy7H.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Pex6SpYJX8ZPySncfNNy7H.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874766668/job/35894859667,26 January 2025 7:00pm,Success,,1.0.0
NeqeQSxHmdUNLk9Ytpo6qo,GPQA diamond,qwen1.5-72b-chat,0.288,choice:0.29±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FNeqeQSxHmdUNLk9Ytpo6qo.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/NeqeQSxHmdUNLk9Ytpo6qo.eval,https://github.com/epoch-research/benchmarks/actions/runs/12875936772/job/35898128253,26 January 2025 7:00pm,Success,,1.0.0
9K6c9b6v9zQrQYtw827BDn,GPQA diamond,qwen1.5-32b-chat,0.307,choice:0.31±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9K6c9b6v9zQrQYtw827BDn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9K6c9b6v9zQrQYtw827BDn.eval,https://github.com/epoch-research/benchmarks/actions/runs/12874766668/job/35894859467,26 January 2025 7:00pm,Success,,1.0.0
FaRZDKSriUbS3Brt6qprP9,MATH level 5,o1-mini-2024-09-12_medium,0.843,"normalized_string_match:0.78±0.01,sympy_equiv:0.77±0.01,model_graded_equiv:0.84±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/FaRZDKSriUbS3Brt6qprP9.eval,https://github.com/epoch-research/benchmarks/actions/runs/12877554122/job/35902213067,26 January 2025 7:00pm,Success,,1.0.0
DHdeZNynu4VtRp2VGkCGvd,GPQA diamond,Hermes-2-Theta-Llama-3-70B,0.375,choice:0.37±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FDHdeZNynu4VtRp2VGkCGvd.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/DHdeZNynu4VtRp2VGkCGvd.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878296724/job/35904053376,26 January 2025 7:00pm,Success,,1.0.0
evPjJw9SKqq3TieSdxrSQn,GPQA diamond,Llama-2-70b-chat-hf,0.263,choice:0.26±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FevPjJw9SKqq3TieSdxrSQn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/evPjJw9SKqq3TieSdxrSQn.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878296724/job/35904053220,26 January 2025 7:00pm,Success,,1.0.0
jrvutCoFFG4KxYtU7REdpe,GPQA diamond,Meta-Llama-3-70B-Instruct,0.406,choice:0.41±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FjrvutCoFFG4KxYtU7REdpe.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/jrvutCoFFG4KxYtU7REdpe.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878427328/job/35904381329,26 January 2025 7:00pm,Success,,1.0.0
iT3BBFWjsZk2rKHKeJDsM8,GPQA diamond,Meta-Llama-3-8B-Instruct,0.261,choice:0.26±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FiT3BBFWjsZk2rKHKeJDsM8.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/iT3BBFWjsZk2rKHKeJDsM8.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878427328/job/35904381151,26 January 2025 7:00pm,Success,,1.0.0
ZAVTwsDqrvbWbqSgeNdXFd,MATH level 5,Hermes-2-Theta-Llama-3-70B,0.227,"normalized_string_match:0.2±0.01,sympy_equiv:0.2±0.01,model_graded_equiv:0.23±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZAVTwsDqrvbWbqSgeNdXFd.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880697099/job/35910116987,26 January 2025 7:00pm,Success,,1.0.0
BURdKU7mhUX4YMPUDoFsed,MATH level 5,Meta-Llama-3-8B-Instruct,0.061,"normalized_string_match:0.04±0,sympy_equiv:0.05±0,model_graded_equiv:0.06±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/BURdKU7mhUX4YMPUDoFsed.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878797017/job/35905315467,26 January 2025 7:00pm,Success,,1.0.0
dekbZbU3BtiQZNKG8Txaax,MATH level 5,Meta-Llama-3-70B-Instruct,0.226,"normalized_string_match:0.21±0.01,sympy_equiv:0.21±0.01,model_graded_equiv:0.23±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/dekbZbU3BtiQZNKG8Txaax.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878797017/job/35905316300,26 January 2025 7:00pm,Success,,1.0.0
4bkXhqCNoe6RpTe9taJ7kR,MATH level 5,Llama-2-70b-chat-hf,0.033,"normalized_string_match:0.02±0,sympy_equiv:0.03±0,model_graded_equiv:0.03±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/4bkXhqCNoe6RpTe9taJ7kR.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878797017/job/35905315921,26 January 2025 7:00pm,Success,,1.0.0
bH5kVfLE53CxY3qWmB9C5K,MATH level 5,Yi-1.5-34B-Chat,0.255,"normalized_string_match:0.24±0.01,sympy_equiv:0.24±0.01,model_graded_equiv:0.25±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/bH5kVfLE53CxY3qWmB9C5K.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880697099/job/35910117168,26 January 2025 7:00pm,Success,,1.0.0
nzy4SGSeKGzQSKJLWExKpV,MATH level 5,Yi-34B-Chat,0.051,"normalized_string_match:0.04±0,sympy_equiv:0.04±0,model_graded_equiv:0.05±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/nzy4SGSeKGzQSKJLWExKpV.eval,https://github.com/epoch-research/benchmarks/actions/runs/12878797017/job/35905315229,26 January 2025 7:00pm,Success,,1.0.0
c4V7sc8TUNJ8wQKRhuA2Rv,MATH level 5,gpt-4-0125-preview,0.354,"normalized_string_match:0.33±0.01,sympy_equiv:0.33±0.01,model_graded_equiv:0.35±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/c4V7sc8TUNJ8wQKRhuA2Rv.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880865454/job/35910536124,26 January 2025 7:00pm,Success,,1.0.0
YbQkMPC2GtNLhxqTEhhVsa,GPQA diamond,Mistral-7B-Instruct-v0.3,0.152,choice:0.15±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FYbQkMPC2GtNLhxqTEhhVsa.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/YbQkMPC2GtNLhxqTEhhVsa.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880865454/job/35910535947,26 January 2025 7:00pm,Success,,1.0.0
82RTgqgiC2UuM5M37L3WWM,GPQA diamond,deepseek-llm-67b-chat,0.246,choice:0.25±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F82RTgqgiC2UuM5M37L3WWM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/82RTgqgiC2UuM5M37L3WWM.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880958496/job/35910781233,26 January 2025 7:00pm,Success,,1.0.0
Sn2qbB2HiySe7NcG28jnyM,GPQA diamond,Mixtral-8x7B-Instruct-v0.1,0.306,choice:0.31±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FSn2qbB2HiySe7NcG28jnyM.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Sn2qbB2HiySe7NcG28jnyM.eval,https://github.com/epoch-research/benchmarks/actions/runs/12880958496/job/35910781437,26 January 2025 7:00pm,Success,,1.0.0
Hdcqs42VnPNVBquXttexhq,MATH level 5,gemma-2-9b-it,0.210,"normalized_string_match:0.18±0.01,sympy_equiv:0.18±0.01,model_graded_equiv:0.21±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Hdcqs42VnPNVBquXttexhq.eval,https://github.com/epoch-research/benchmarks/actions/runs/12881005726/job/35910915124,26 January 2025 7:00pm,Success,,1.0.0
PM3GCQcVK7t2MujNp4F4kH,MATH level 5,gemma-2-27b-it,0.279,"normalized_string_match:0.24±0.01,sympy_equiv:0.24±0.01,model_graded_equiv:0.28±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PM3GCQcVK7t2MujNp4F4kH.eval,https://github.com/epoch-research/benchmarks/actions/runs/12881005726/job/35910915421,26 January 2025 7:00pm,Success,,1.0.0
WtALgW6VSFdihDZWXRm2AZ,GPQA diamond,qwen2.5-72b-instruct,0.491,choice:0.49±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FWtALgW6VSFdihDZWXRm2AZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/WtALgW6VSFdihDZWXRm2AZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12911606506/job/36004525889,26 January 2025 7:00pm,Success,,1.0.0
9mWoY9fM2ifKvDg3xaug6h,MATH level 5,qwen2-72b-instruct,0.391,"normalized_string_match:0.37±0.01,sympy_equiv:0.36±0.01,model_graded_equiv:0.39±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/9mWoY9fM2ifKvDg3xaug6h.eval,https://github.com/epoch-research/benchmarks/actions/runs/12882002438/job/35913534653,26 January 2025 7:00pm,Success,,1.0.0
6xHApTdQAWM9e5oCvKYS3W,MATH level 5,Mistral-7B-Instruct-v0.3,0.036,"normalized_string_match:0.02±0,sympy_equiv:0.02±0,model_graded_equiv:0.04±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/6xHApTdQAWM9e5oCvKYS3W.eval,https://github.com/epoch-research/benchmarks/actions/runs/12885319997/job/35923562232,26 January 2025 7:00pm,Success,,1.0.0
4PTATufH2ra3guNeLwT9p9,MATH level 5,Mixtral-8x7B-Instruct-v0.1,0.093,"normalized_string_match:0.07±0,sympy_equiv:0.08±0,model_graded_equiv:0.09±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/4PTATufH2ra3guNeLwT9p9.eval,https://github.com/epoch-research/benchmarks/actions/runs/12885319997/job/35923561447,26 January 2025 7:00pm,Success,,1.0.0
Cm4UkgA5TtmzQegrq3rbvm,MATH level 5,deepseek-llm-67b-chat,0.064,"normalized_string_match:0.05±0,sympy_equiv:0.05±0,model_graded_equiv:0.06±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Cm4UkgA5TtmzQegrq3rbvm.eval,https://github.com/epoch-research/benchmarks/actions/runs/12885319997/job/35923560582,26 January 2025 7:00pm,Success,,1.0.0
MQJA3RUjsQzSubGJiukZ7A,GPQA diamond,WizardLM-2-8x22B,0.434,choice:0.43±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FMQJA3RUjsQzSubGJiukZ7A.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/MQJA3RUjsQzSubGJiukZ7A.eval,https://github.com/epoch-research/benchmarks/actions/runs/12882982775/job/35916323566,26 January 2025 7:00pm,Success,,1.0.0
TTa3hu7PgRxHgizVreZsdJ,GPQA diamond,dbrx-instruct,0.329,choice:0.33±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTTa3hu7PgRxHgizVreZsdJ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TTa3hu7PgRxHgizVreZsdJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12883275599/job/35917223724,26 January 2025 7:00pm,Success,,1.0.0
4bAYQuh4heJoTNhapVZLNB,MATH level 5,WizardLM-2-8x22B,0.257,"normalized_string_match:0.22±0.01,sympy_equiv:0.23±0.01,model_graded_equiv:0.26±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/4bAYQuh4heJoTNhapVZLNB.eval,https://github.com/epoch-research/benchmarks/actions/runs/12882982775/job/35916323194,26 January 2025 7:00pm,Success,,1.0.0
PMbefrjykQmTBpABMa7FJi,MATH level 5,dbrx-instruct,0.117,"normalized_string_match:0.09±0.01,sympy_equiv:0.09±0.01,model_graded_equiv:0.12±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/PMbefrjykQmTBpABMa7FJi.eval,https://github.com/epoch-research/benchmarks/actions/runs/12883275599/job/35917223016,26 January 2025 7:00pm,Success,,1.0.0
ibpS8bs4z3wD3Hb6F3kpdV,GPQA diamond,gemini-1.0-pro-001,0.340,choice:0.34±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FibpS8bs4z3wD3Hb6F3kpdV.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ibpS8bs4z3wD3Hb6F3kpdV.eval,https://github.com/epoch-research/benchmarks/actions/runs/12892896126/job/35948192089,26 January 2025 7:00pm,Success,,1.0.0
iUdecHq9vU5utytzgTXzqY,MATH level 5,gemini-1.0-pro-001,0.112,"normalized_string_match:0.1±0.01,sympy_equiv:0.1±0.01,model_graded_equiv:0.11±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/iUdecHq9vU5utytzgTXzqY.eval,https://github.com/epoch-research/benchmarks/actions/runs/12892896126/job/35948191734,26 January 2025 7:00pm,Success,,1.0.0
bJV4D8SEo9zMJPEYTTXJNZ,GPQA diamond,ministral-3b-2410,0.253,choice:0.25±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FbJV4D8SEo9zMJPEYTTXJNZ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/bJV4D8SEo9zMJPEYTTXJNZ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12907407727/job/35990819691,26 January 2025 7:00pm,Success,,1.0.0
dDeREdDJ5bVyci3APZxzWb,GPQA diamond,mistral-large-2402,0.388,choice:0.39±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FdDeREdDJ5bVyci3APZxzWb.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/dDeREdDJ5bVyci3APZxzWb.eval,https://github.com/epoch-research/benchmarks/actions/runs/12907820278/job/35992136615,26 January 2025 7:00pm,Success,,1.0.0
ifAM3DJBdm5eU9Y2DjaEvW,GPQA diamond,open-mixtral-8x22b,0.341,choice:0.34±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FifAM3DJBdm5eU9Y2DjaEvW.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ifAM3DJBdm5eU9Y2DjaEvW.eval,https://github.com/epoch-research/benchmarks/actions/runs/12911185685/job/36003093207,26 January 2025 7:00pm,Success,,1.0.0
DGEzZA6nvTne4MEGSuo3Tm,GPQA diamond,ministral-8b-2410,0.271,choice:0.27±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FDGEzZA6nvTne4MEGSuo3Tm.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/DGEzZA6nvTne4MEGSuo3Tm.eval,https://github.com/epoch-research/benchmarks/actions/runs/12907407727/job/35990819282,26 January 2025 7:00pm,Success,,1.0.0
ZQPNnicTuhuscJvni85PFY,GPQA diamond,mistral-large-2407,0.490,choice:0.49±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZQPNnicTuhuscJvni85PFY.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/ZQPNnicTuhuscJvni85PFY.eval,https://github.com/epoch-research/benchmarks/actions/runs/12907820278/job/35992137109,26 January 2025 7:00pm,Success,,1.0.0
Y6KPSujFepdwqinKLf9mDF,GPQA diamond,open-mixtral-8x7b,0.298,choice:0.3±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FY6KPSujFepdwqinKLf9mDF.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Y6KPSujFepdwqinKLf9mDF.eval,https://github.com/epoch-research/benchmarks/actions/runs/12911185685/job/36003094176,26 January 2025 7:00pm,Success,,1.0.0
5XuVuHeUDekHqjUy2P8gww,GPQA diamond,open-mistral-7b,0.132,choice:0.13±0.01,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F5XuVuHeUDekHqjUy2P8gww.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/5XuVuHeUDekHqjUy2P8gww.eval,https://github.com/epoch-research/benchmarks/actions/runs/12907820278/job/35992137610,26 January 2025 7:00pm,Success,,1.0.0
XCSWtbDquA7FUiWfd7kh9q,GPQA diamond,open-mistral-nemo-2407,0.299,choice:0.3±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FXCSWtbDquA7FUiWfd7kh9q.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/XCSWtbDquA7FUiWfd7kh9q.eval,https://github.com/epoch-research/benchmarks/actions/runs/12911185685/job/36003093667,26 January 2025 7:00pm,Success,,1.0.0
YNhHxRT9Ee2uEWWJh9RXuP,MATH level 5,ministral-3b-2410,0.144,"normalized_string_match:0.13±0.01,sympy_equiv:0.13±0.01,model_graded_equiv:0.14±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/YNhHxRT9Ee2uEWWJh9RXuP.eval,https://github.com/epoch-research/benchmarks/actions/runs/12913124170/job/36009676520,26 January 2025 7:00pm,Success,,1.0.0
9ZQaypWLwLHbQrqTWfoEor,MATH level 5,open-mistral-7b,0.037,"normalized_string_match:0.02±0,sympy_equiv:0.03±0,model_graded_equiv:0.04±0",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/9ZQaypWLwLHbQrqTWfoEor.eval,https://github.com/epoch-research/benchmarks/actions/runs/12915566284/job/36017771232,26 January 2025 7:00pm,Success,,1.0.0
hqdiKcvEPSRyvenUVB9qxM,MATH level 5,ministral-8b-2410,0.149,"normalized_string_match:0.14±0.01,sympy_equiv:0.13±0.01,model_graded_equiv:0.15±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/hqdiKcvEPSRyvenUVB9qxM.eval,https://github.com/epoch-research/benchmarks/actions/runs/12913124170/job/36009675016,26 January 2025 7:00pm,Success,,1.0.0
div9uE9eovqykqx883kSpE,MATH level 5,mistral-large-2407,0.448,"normalized_string_match:0.41±0.01,sympy_equiv:0.41±0.01,model_graded_equiv:0.45±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/div9uE9eovqykqx883kSpE.eval,https://github.com/epoch-research/benchmarks/actions/runs/12917780478/job/36024967539,26 January 2025 7:00pm,Success,,1.0.0
9qCrggpoq8FjQWVQhHaGBT,MATH level 5,open-mistral-nemo-2407,0.108,"normalized_string_match:0.1±0.01,sympy_equiv:0.1±0.01,model_graded_equiv:0.11±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/9qCrggpoq8FjQWVQhHaGBT.eval,https://github.com/epoch-research/benchmarks/actions/runs/12914195025/job/36013283350,26 January 2025 7:00pm,Success,,1.0.0
GN3Hez798byishFw3U6cpp,MATH level 5,open-mixtral-8x7b,0.100,"normalized_string_match:0.08±0,sympy_equiv:0.08±0,model_graded_equiv:0.1±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/GN3Hez798byishFw3U6cpp.eval,https://github.com/epoch-research/benchmarks/actions/runs/12915050764/job/36016055413,26 January 2025 7:00pm,Success,,1.0.0
cGMb2vs9NhfkwZJsQXzLmQ,MATH level 5,open-mixtral-8x22b,0.242,"normalized_string_match:0.22±0.01,sympy_equiv:0.22±0.01,model_graded_equiv:0.24±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/cGMb2vs9NhfkwZJsQXzLmQ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12915050764/job/36016055758,26 January 2025 7:00pm,Success,,1.0.0
gypQecfFNK2x4pNGDtGMDv,MATH level 5,mistral-large-2402,0.245,"normalized_string_match:0.21±0.01,sympy_equiv:0.22±0.01,model_graded_equiv:0.24±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/gypQecfFNK2x4pNGDtGMDv.eval,https://github.com/epoch-research/benchmarks/actions/runs/12915566284/job/36017770453,26 January 2025 7:00pm,Success,,1.0.0
9SSHEP95FYVZvHDdoHSRy3,GPQA diamond,Llama-3.2-90B-Vision-Instruct,0.410,choice:0.41±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F9SSHEP95FYVZvHDdoHSRy3.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/9SSHEP95FYVZvHDdoHSRy3.eval,https://github.com/epoch-research/benchmarks/actions/runs/12917603900/job/36024387598,26 January 2025 7:00pm,Success,,1.0.0
iHRmwxsh8M3FGun8x5mxkt,MATH level 5,Llama-3.2-90B-Vision-Instruct,0.394,"normalized_string_match:0.35±0.01,sympy_equiv:0.36±0.01,model_graded_equiv:0.39±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/iHRmwxsh8M3FGun8x5mxkt.eval,https://github.com/epoch-research/benchmarks/actions/runs/12917603900/job/36024388001,26 January 2025 7:00pm,Success,,1.0.0
5JgivNDZp2D7Lc7cpDhS28,GPQA diamond,Llama-3.1-Tulu-3-70B-DPO,0.463,choice:0.46±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2F5JgivNDZp2D7Lc7cpDhS28.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/5JgivNDZp2D7Lc7cpDhS28.eval,https://github.com/epoch-research/benchmarks/actions/runs/12913124170/job/36009673210,26 January 2025 7:00pm,Success,,1.0.0
RJm4u2vXF6PUmThJGzAfHE,GPQA diamond,Eurus-2-7B-PRIME,0.339,choice:0.34±0.02,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FRJm4u2vXF6PUmThJGzAfHE.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/RJm4u2vXF6PUmThJGzAfHE.eval,https://github.com/epoch-research/benchmarks/actions/runs/12919360954/job/36029766826,26 January 2025 7:00pm,Success,,1.0.0
Hf2ESooSteQK99yFjhFm8t,MATH level 5,Llama-3.1-Tulu-3-70B-DPO,0.427,"normalized_string_match:0.39±0.01,sympy_equiv:0.39±0.01,model_graded_equiv:0.43±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/Hf2ESooSteQK99yFjhFm8t.eval,https://github.com/epoch-research/benchmarks/actions/runs/12929237891/job/36058281186,26 January 2025 7:00pm,Success,,1.0.0
TxXS78Wg2pSCDbpmeJuzQn,GPQA diamond,Llama-3.3-70B-Instruct,0.474,choice:0.47±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FTxXS78Wg2pSCDbpmeJuzQn.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/TxXS78Wg2pSCDbpmeJuzQn.eval,https://github.com/epoch-research/benchmarks/actions/runs/12916909534/job/36022209746,26 January 2025 7:00pm,Success,,1.0.0
DeaDxFfiQtCTdnZPH7Awaf,MATH level 5,Llama-3.3-70B-Instruct,0.416,"normalized_string_match:0.41±0.01,sympy_equiv:0.4±0.01,model_graded_equiv:0.42±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/DeaDxFfiQtCTdnZPH7Awaf.eval,https://github.com/epoch-research/benchmarks/actions/runs/12917780478/job/36024967192,26 January 2025 7:00pm,Success,,1.0.0
ni4DJboYYpdRssVeAuro2V,MATH level 5,qwen2.5-72b-instruct,0.632,"normalized_string_match:0.59±0.01,sympy_equiv:0.59±0.01,model_graded_equiv:0.63±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/ni4DJboYYpdRssVeAuro2V.eval,https://github.com/epoch-research/benchmarks/actions/runs/12918001992/job/36025709327,26 January 2025 7:00pm,Success,,1.0.0
8Uu6XsSJCKwcwhXDt8vgzq,MATH level 5,o1-preview-2024-09-12,0.816,"normalized_string_match:0.68±0.01,sympy_equiv:0.68±0.01,model_graded_equiv:0.82±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/8Uu6XsSJCKwcwhXDt8vgzq.eval,https://github.com/epoch-research/benchmarks/actions/runs/12932410772/job/36068662769,26 January 2025 7:00pm,Success,,1.0.0
Uczhz7MKLstjLkSbbGuwdN,GPQA diamond,o1-2024-12-17_medium,0.758,choice:0.76±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FUczhz7MKLstjLkSbbGuwdN.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Uczhz7MKLstjLkSbbGuwdN.eval,https://github.com/epoch-research/benchmarks/actions/runs/12932723666/job/36069724303,26 January 2025 7:00pm,Success,,1.0.0
AzanNoyhhVrtQi63Azo6hu,MATH level 5,o1-2024-12-17_medium,0.944,"normalized_string_match:0.78±0.01,sympy_equiv:0.8±0.01,model_graded_equiv:0.94±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/AzanNoyhhVrtQi63Azo6hu.eval,https://github.com/epoch-research/benchmarks/actions/runs/12932723666/job/36069724929,26 January 2025 7:00pm,Success,,1.0.0
Vwo3nMA8g2gGBCyTdKFMQJ,GPQA diamond,DeepSeek-V3,0.565,choice:0.57±0.03,https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FVwo3nMA8g2gGBCyTdKFMQJ.eval,https://epoch-benchmarks-production-public.s3.us-east-2.amazonaws.com/inspect_ai_logs/Vwo3nMA8g2gGBCyTdKFMQJ.eval,https://github.com/epoch-research/benchmarks/actions/runs/12953658131/job/36133911159,26 January 2025 7:00pm,Success,,1.0.0
exvQCLCiga6gE7zvQNp2Z6,MATH level 5,DeepSeek-V3,0.649,"normalized_string_match:0.58±0.01,sympy_equiv:0.58±0.01,model_graded_equiv:0.65±0.01",,https://epoch-benchmarks-production-private.s3.us-east-2.amazonaws.com/inspect_ai_logs/exvQCLCiga6gE7zvQNp2Z6.eval,https://github.com/epoch-research/benchmarks/actions/runs/12953800670/job/36134354884,26 January 2025 7:00pm,Success,,1.0.0
A85Zfq2qguE4X9xXBweBHP,GPQA diamond,grok-4-0709,0.870,choice:0.87±0.02,,,,,Success,,1.0.0
cvTPRDCM38zSTn9Y3MUb9d,OTIS Mock AIME 2024-2025,grok-4-0709,0.840,model_graded:0.84±0.05,,,,,Success,,1.0.0
manual_gemini_2.5_deep_think_run_frontiermath_tiers_1_to_3,FrontierMath-2025-02-28-Private,gemini-2.5-deep-think-2025-08-01-webapp,0.290,manual_inspection:0.29±0.03,,,,,Success,checked,1.0.0
manual_gemini_2.5_deep_think_run_frontiermath_tier_4,FrontierMath-Tier-4-2025-07-01-Private,gemini-2.5-deep-think-2025-08-01-webapp,0.104,manual_inspection:0.1±0.04,,,,,Success,checked,1.0.0