Near-frontier performance at flash pricing, native tool use and code execution
SI v3 uses weighted percentages from one Artificial Analysis configuration. Ranking requires all five benchmarks. Partial scores average only available weights. Scores are not comparable to SI v2 or a probability of reaching singularity.
Compared with models released within six months of it.
| Benchmark | Score | Rank |
|---|---|---|
HellaSwag Common sense reasoning about everyday situations | 94.1% | #16 / 36 |
ARC-C Grade-school science questions requiring reasoning | 96.2% | #28 / 40 |
HumanEval Coding ability - generating correct Python functions | 89.5% | #32 / 50 |
MMLU Tests knowledge across 57 subjects from STEM to humanities | 88.1% | #34 / 54 |
MATH Competition-level mathematics problems | 77.2% | #34 / 50 |
Arena Elo Human preference ranking via blind comparisons | 1290 | #41 / 51 |
GPQA PhD-level science questions even experts struggle with | 62.1% | #83 / 96 |
HLEArtificial Analysis Challenging multidisciplinary questions evaluated by Artificial Analysis | 6.4% | #92 / 100 |
| Benchmark / source | Score |
|---|---|
HLE Artificial AnalysisGemini 2.0 Flash Thinking Experimental (Jan '25) | 6.4% |