Model,Count,End-to-End % Solved,Flag Success Count,Subtask Challenge Count,Subtask-Guided # Solved,Subtask-Guided % Solved,Sum of Subtask Run Scores,Subtasks % Solved,FST Standard,FST Subtask
Claude Mythos Preview ⁶,35,100,,,,,,,,
Claude Opus 4.7 ⁸,35,96,,,,,,,,
Claude Opus 4.6 ⁵,37,93,,,,,,,,
Claude Opus 4.5 ³,39,82,,,,,,,,
Muse Spark ⁷,40,65.4,,,,,,,,
Claude Sonnet 4.5 ³,39,60,,,,,,,,
Grok 4 ⁴,40,43,,,,,,,,
Claude Opus 4.1 ³,39,42,,,,,,,,
Grok 4.1 Thinking ⁴,40,39,,,,,,,,
Claude Opus 4 ²,37,38,,,,,,,,
Claude Sonnet 4 ²,37,35,,,,,,,,
Grok 4 Fast ⁴,40,30,,,,,,,,
OpenAI o3-mini ¹†,40,22.5,10,,,,,,0:42,
OpenAI o1-mini ¹†,40,10,5,,,,,,0:11,
GPT-4.5-preview ¹,40,17.5,7,,,,,,0:11,
Claude 3.7 Sonnet ¹,40,20,8,,,,,,0:11,
GPT-4o,40,12.5,5,40,7,17.5,11.8,28.7,0:11,0:52
OpenAI o1-preview,40,10,4,40,4,10,18.2,46.8,0:11,0:11
Claude 3 Opus,40,10,4,40,5,12.5,14.4,36.8,0:11,0:11
Claude 3.5 Sonnet,40,17.5,7,40,6,15,17.8,43.9,0:11,0:11
Mixtral 8x22b Instruct,40,7.5,3,40,2,5,6.1,15.2,0:09,0:07
Gemini 1.5 Pro,40,7.5,3,40,2,5,4.5,11.7,0:09,0:06
Llama 3 70b Chat,40,5,2,40,3,7.5,3.2,8.2,0:09,0:11
Llama 3.1 405B Instruct,40,7.5,3,40,6,15,8.2,20.5,0:09,0:11