Trained for $5.5M, proved frontier performance was possible at low cost
SI v3 uses weighted percentages from one Artificial Analysis configuration. Ranking requires all five benchmarks. Partial scores average only available weights. Scores are not comparable to SI v2 or a probability of reaching singularity.
Compared with 20 models released within six months of it, across 5 benchmarks where scores spread out.
| Benchmark | Score | Rank |
|---|---|---|
ARC-C Grade-school science questions requiring reasoning | 96.8% | #21 / 40 |
MATH Competition-level mathematics problems | 90.2% | #24 / 50 |
HumanEval Coding ability - generating correct Python functions | 92.1% | #28 / 50 |
HellaSwag Common sense reasoning about everyday situations | 88.9% | #29 / 36 |
MMLU Tests knowledge across 57 subjects from STEM to humanities | 88.5% | #30 / 54 |
SWE-bench Real-world GitHub issue resolution | 42% | #38 / 39 |
Arena Elo Human preference ranking via blind comparisons | 1318 | #39 / 51 |
TerminalArtificial Analysis Agentic terminal coding tasks requiring multi-step execution | 6.8% | #54 / 58 |
GPQA PhD-level science questions even experts struggle with | 59.1% | #85 / 96 |
HLEArtificial Analysis Challenging multidisciplinary questions evaluated by Artificial Analysis | 2.9% | #99 / 100 |
| Benchmark / source | Score |
|---|---|
HLE Artificial AnalysisDeepSeek V3 (Dec '24) | 2.9% |
Terminal-Bench 2.1 Artificial AnalysisDeepSeek V3 (Dec '24) | 16.9% |
Terminal-Bench 4.0 Artificial AnalysisDeepSeek V3 (Dec '24) | 0% |
SciCode Artificial AnalysisDeepSeek V3 (Dec '24) | 35.8% |
AA-LCR Artificial AnalysisDeepSeek V3 (Dec '24) | 29.3% |
CritPt Artificial AnalysisDeepSeek V3 (Dec '24) | 0% |
IFBench Artificial AnalysisDeepSeek V3 (Dec '24) | 34.8% |