Matched Opus 4 on coding at one-fifth the price, 1M context window
SI v3 uses weighted percentages from one Artificial Analysis configuration. Ranking requires all five benchmarks. Partial scores average only available weights. Scores are not comparable to SI v2 or a probability of reaching singularity.
Compared with 25 models released within six months of it, across 10 benchmarks where scores spread out.
| Benchmark | Score | Rank |
|---|---|---|
ARC-C Grade-school science questions requiring reasoning | 97.5% | #14 / 40 |
HumanEval Coding ability - generating correct Python functions | 94.1% | #16 / 50 |
HellaSwag Common sense reasoning about everyday situations | 92.4% | #21 / 36 |
SWE-bench Real-world GitHub issue resolution | 72.7% | #21 / 39 |
MMLU Tests knowledge across 57 subjects from STEM to humanities | 89.2% | #25 / 54 |
MATH Competition-level mathematics problems | 86.2% | #27 / 50 |
Arena Elo Human preference ranking via blind comparisons | 1368 | #32 / 51 |
ARC-AGIARC Prize Novel reasoning tasks requiring fluid intelligence | 5.9% | #37 / 40 |
TerminalArtificial Analysis Agentic terminal coding tasks requiring multi-step execution | 31.1% | #42 / 58 |
MMMUvals.ai College-level multimodal reasoning across 30+ disciplines | 74.9% | #43 / 51 |
MMLU-Provals.ai Harder 10-option successor to MMLU; more reasoning-focused | 83.9% | #50 / 61 |
LiveCodeBenchvals.ai Contamination-free competitive programming (filtered by cutoff date) | 62.4% | #57 / 62 |
GPQA PhD-level science questions even experts struggle with | 74.8% | #76 / 96 |
HLEArtificial Analysis Challenging multidisciplinary questions evaluated by Artificial Analysis | 10.7% | #90 / 100 |
| Benchmark / source | Score |
|---|---|
HLE Artificial AnalysisClaude 4 Sonnet (Reasoning) | 10.7% |
MMMU-Pro Artificial AnalysisClaude 4 Sonnet (Reasoning) | 61.8% |
Terminal-Bench 2.1 Artificial AnalysisClaude 4 Sonnet (Reasoning) | 36.3% |
AA-LCR Artificial AnalysisClaude 4 Sonnet (Reasoning) | 70.3% |
CritPt Artificial AnalysisClaude 4 Sonnet (Reasoning) | 0.3% |
IFBench Artificial AnalysisClaude 4 Sonnet (Reasoning) | 54.7% |