Rank composition
Model behavior supplies 90% of the overall rank blend. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 9 | 27 | 206 |
| Human rights | 13.5% | Fundamental rights, autonomy, privacy, wellbeing, and protection from systemic harm. | 40 | 144 | 475 |
| Fairness | 9% | Fair treatment across demographic groups and resistance to discriminatory behavior. | 33 | 42 | 354 |
| Truthfulness | 13.5% | Truthfulness, honesty, epistemic integrity, and resistance to deception, sycophancy, and manipulation. | 45 | 92 | 552 |
| Benign helpfulness | 9% | Useful assistance on benign requests without indiscriminate refusal. | 24 | 52 | 238 |
| Misuse resistance | 9% | Resistance to harmful requests, jailbreaks, toxic continuation, and ordinary misuse. | 91 | 290 | 709 |
| Responsible agency | 13.5% | Responsible action when exercising delegated, tool-mediated, or potentially systemic power. | 48 | 167 | 395 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 346 |
Rank change by component
The table below shows how much each model's rank moves when one behavior component is removed. Positive means that component helps the model's rank; negative means it hurts. Blank cells mean no direct evidence. These are sensitivity checks, not scores that add up.
| Model | Nonhuman welfare | Human rights | Fairness | Truthfulness | Benign helpfulness | Misuse resistance | Responsible agency | Governance |
|---|---|---|---|---|---|---|---|---|
| gpt-6.1-sol | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | +1.61 SDDirect |
| gpt-6-astra | 1rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 7rank change if excluded 9 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 6 direct evals | 0rank change if excluded 5 direct evals | +1.61 SDDirect |
| claude-opus-5 | 1rank change if excluded 4 direct evals | 4rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | 0rank change if excluded 12 direct evals | -5rank change if excluded 5 direct evals | 4rank change if excluded 7 direct evals | 1rank change if excluded 11 direct evals | +2.04 SDDirect |
| gpt-6-sol | 4rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | 8rank change if excluded 4 direct evals | 4rank change if excluded 6 direct evals | 3rank change if excluded 3 direct evals | +1.61 SDDirect |
| claude-fable-5 | 2rank change if excluded 2 direct evals | 2rank change if excluded 4 direct evals | 3rank change if excluded 3 direct evals | 6rank change if excluded 16 direct evals | 1rank change if excluded 2 direct evals | 4rank change if excluded 6 direct evals | 2rank change if excluded 10 direct evals | +2.04 SDDirect |
| claude-opus-5.5 | -1rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 1rank change if excluded 5 direct evals | 0rank change if excluded 4 direct evals | -3rank change if excluded 5 direct evals | 1rank change if excluded 3 direct evals | +2.04 SDDirect |
| claude-opus-4.8 | 9rank change if excluded 3 direct evals | 5rank change if excluded 5 direct evals | 0rank change if excluded 2 direct evals | 7rank change if excluded 15 direct evals | -2rank change if excluded 4 direct evals | 6rank change if excluded 11 direct evals | 5rank change if excluded 13 direct evals | +2.04 SDDirect |
| gpt-6-luna | 3rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 5rank change if excluded 6 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 6 direct evals | 7rank change if excluded 3 direct evals | +1.61 SDDirect |
| claude-fable-5.1 | 2rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 1rank change if excluded 9 direct evals | 6rank change if excluded 5 direct evals | 3rank change if excluded 6 direct evals | 1rank change if excluded 5 direct evals | +2.04 SDDirect |
| muse-spark-1.3 | 7rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | -8rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | +0.35 SDDirect |
| muse-spark-1.1 | -5rank change if excluded 1 direct eval | 1rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 9 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 4 direct evals | 1rank change if excluded 4 direct evals | +0.35 SDDirect |
| gpt-5.6-terra | 4rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | 4rank change if excluded 12 direct evals | 1rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | -22rank change if excluded 11 direct evals | +1.61 SDDirect |
| gpt-5.3-chat | 49rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 3 direct evals | -24rank change if excluded 1 direct eval | -16rank change if excluded 3 direct evals | Not shown0 direct evals | +1.61 SDDirect |
| gpt-5.5-instant | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 11rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | +1.61 SDDirect |
| claude-sonnet-5.5 | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -20rank change if excluded 4 direct evals | 30rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | Not shown0 direct evals | +2.04 SDDirect |
| claude-mythos-preview | 3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 21rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -10rank change if excluded 3 direct evals | +2.04 SDDirect |
| muse-spark-1.2 | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | -8rank change if excluded 3 direct evals | +0.35 SDDirect |
| claude-opus-4.7 | -24rank change if excluded 4 direct evals | -24rank change if excluded 6 direct evals | -26rank change if excluded 4 direct evals | 6rank change if excluded 14 direct evals | 5rank change if excluded 5 direct evals | 10rank change if excluded 12 direct evals | -12rank change if excluded 14 direct evals | +2.04 SDDirect |
| claude-opus-4.6 | 10rank change if excluded 3 direct evals | 27rank change if excluded 8 direct evals | -7rank change if excluded 4 direct evals | 5rank change if excluded 17 direct evals | -5rank change if excluded 6 direct evals | -3rank change if excluded 14 direct evals | 12rank change if excluded 10 direct evals | +2.06 SDDirect |
| gpt-5.5 | 17rank change if excluded 5 direct evals | 26rank change if excluded 8 direct evals | 26rank change if excluded 6 direct evals | 7rank change if excluded 15 direct evals | 2rank change if excluded 7 direct evals | 1rank change if excluded 17 direct evals | 24rank change if excluded 16 direct evals | +1.61 SDDirect |
| gpt-5.4 | 4rank change if excluded 1 direct eval | 7rank change if excluded 10 direct evals | 1rank change if excluded 6 direct evals | 9rank change if excluded 16 direct evals | -1rank change if excluded 4 direct evals | 4rank change if excluded 16 direct evals | 4rank change if excluded 14 direct evals | +1.61 SDDirect |
| inkling-small | 32rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| grok-4.7 | -2rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -3rank change if excluded 3 direct evals | -17rank change if excluded 4 direct evals | 0rank change if excluded 2 direct evals | -0.63 SDDirect |
| gemini-3.7-flash | 0rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 1rank change if excluded 9 direct evals | 2rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | 51rank change if excluded 6 direct evals | +1.17 SDDirect |
| gemini-3.8-flash | -3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | -2rank change if excluded 7 direct evals | 2rank change if excluded 3 direct evals | 3rank change if excluded 5 direct evals | 36rank change if excluded 6 direct evals | +1.17 SDDirect |
| gpt-5.6-sol | 5rank change if excluded 3 direct evals | -2rank change if excluded 4 direct evals | 30rank change if excluded 3 direct evals | -2rank change if excluded 17 direct evals | 10rank change if excluded 4 direct evals | 28rank change if excluded 10 direct evals | -1rank change if excluded 15 direct evals | +1.61 SDDirect |
| granite-4.2-8b | 31rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mimo-v2-omni | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 10rank change if excluded 3 direct evals | -24rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| deepseek-v4.1-flash | 34rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 1 direct eval | -1.05 SDDirect |
| gpt-5.2-chat | 16rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 8rank change if excluded 3 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | -41rank change if excluded 1 direct eval | +1.56 SDDirect |
| glm-5.3-flash | 0rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | 5rank change if excluded 1 direct eval | 24rank change if excluded 4 direct evals | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | -0.61 SDDirect |
| claude-sonnet-4.6 | -37rank change if excluded 4 direct evals | -3rank change if excluded 8 direct evals | 13rank change if excluded 5 direct evals | 18rank change if excluded 16 direct evals | -10rank change if excluded 6 direct evals | 19rank change if excluded 15 direct evals | -3rank change if excluded 12 direct evals | +2.06 SDDirect |
| glm-5.3 | 1rank change if excluded 2 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 3rank change if excluded 8 direct evals | 9rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | 0rank change if excluded 3 direct evals | -0.61 SDDirect |
| gpt-5.6-luna | 2rank change if excluded 3 direct evals | 19rank change if excluded 3 direct evals | 5rank change if excluded 3 direct evals | 9rank change if excluded 11 direct evals | 3rank change if excluded 5 direct evals | 3rank change if excluded 10 direct evals | 2rank change if excluded 11 direct evals | +1.61 SDDirect |
| gpt-5.2 | 10rank change if excluded 4 direct evals | 15rank change if excluded 9 direct evals | 7rank change if excluded 5 direct evals | 11rank change if excluded 13 direct evals | 5rank change if excluded 4 direct evals | 12rank change if excluded 16 direct evals | 16rank change if excluded 13 direct evals | +1.56 SDDirect |
| claude-mythos-5 | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | -37rank change if excluded 3 direct evals | +2.04 SDDirect |
| gemini-3.6-flash | 3rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 6 direct evals | 3rank change if excluded 1 direct eval | -2rank change if excluded 3 direct evals | -33rank change if excluded 3 direct evals | +1.17 SDDirect |
| gpt-5-nano | -9rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | 8rank change if excluded 3 direct evals | 7rank change if excluded 7 direct evals | -12rank change if excluded 2 direct evals | -10rank change if excluded 12 direct evals | 7rank change if excluded 5 direct evals | +1.06 SDDirect |
| claude-2 | Not shown0 direct evals | 94rank change if excluded 3 direct evals | 93rank change if excluded 4 direct evals | -5rank change if excluded 3 direct evals | -5rank change if excluded 1 direct eval | -4rank change if excluded 9 direct evals | Not shown0 direct evals | +2.03 SDDirect |
| claude-haiku-4.5 | -16rank change if excluded 3 direct evals | 19rank change if excluded 10 direct evals | 7rank change if excluded 7 direct evals | 13rank change if excluded 14 direct evals | 12rank change if excluded 8 direct evals | 7rank change if excluded 23 direct evals | 19rank change if excluded 11 direct evals | +2.06 SDDirect |
| gpt-5.5-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 96rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | +1.61 SDDirect |
| claude-opus-4 | 40rank change if excluded 1 direct eval | 39rank change if excluded 2 direct evals | 37rank change if excluded 4 direct evals | 47rank change if excluded 8 direct evals | 40rank change if excluded 5 direct evals | 39rank change if excluded 13 direct evals | 37rank change if excluded 4 direct evals | +2.12 SDDirect |
| kimi-k3 | -2rank change if excluded 2 direct evals | 17rank change if excluded 2 direct evals | 4rank change if excluded 2 direct evals | 17rank change if excluded 13 direct evals | 2rank change if excluded 2 direct evals | 1rank change if excluded 5 direct evals | -1rank change if excluded 9 direct evals | -0.09 SDMedian-imputed |
| gpt-5-mini | 62rank change if excluded 4 direct evals | 27rank change if excluded 5 direct evals | -10rank change if excluded 3 direct evals | 3rank change if excluded 11 direct evals | 0rank change if excluded 5 direct evals | 6rank change if excluded 17 direct evals | 29rank change if excluded 5 direct evals | +1.06 SDDirect |
| gpt-5-pro | 187rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | +1.06 SDDirect |
| gemini-2.5-pro-exp | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -8rank change if excluded 3 direct evals | 116rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | Not shown0 direct evals | +0.60 SDDirect |
| claude-opus-4.5 | -26rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 1rank change if excluded 12 direct evals | 7rank change if excluded 6 direct evals | 8rank change if excluded 13 direct evals | 4rank change if excluded 8 direct evals | +2.06 SDDirect |
| phi-3.5-moe-instruct | 0rank change if excluded 1 direct eval | -2rank change if excluded 4 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3.5-27b | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 85rank change if excluded 3 direct evals | -1rank change if excluded 4 direct evals | 90rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -0.67 SDDirect |
| o1-preview | -11rank change if excluded 1 direct eval | 139rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | 144rank change if excluded 2 direct evals | 144rank change if excluded 1 direct eval | +0.47 SDDirect |
| gpt-5 | -14rank change if excluded 1 direct eval | -13rank change if excluded 6 direct evals | 4rank change if excluded 6 direct evals | -18rank change if excluded 13 direct evals | -18rank change if excluded 6 direct evals | -16rank change if excluded 25 direct evals | -14rank change if excluded 9 direct evals | +1.06 SDDirect |
| grok-4.20-multi-agent | Not shown0 direct evals | -77rank change if excluded 1 direct eval | Not shown0 direct evals | 19rank change if excluded 3 direct evals | 29rank change if excluded 1 direct eval | 19rank change if excluded 2 direct evals | Not shown0 direct evals | -0.63 SDDirect |
| qwen3.5-397b-a17b | 4rank change if excluded 1 direct eval | 7rank change if excluded 4 direct evals | 12rank change if excluded 1 direct eval | 7rank change if excluded 7 direct evals | 7rank change if excluded 3 direct evals | 12rank change if excluded 7 direct evals | 1rank change if excluded 4 direct evals | -1.02 SDDirect |
| claude-3.7-sonnet | 12rank change if excluded 1 direct eval | 11rank change if excluded 3 direct evals | 10rank change if excluded 3 direct evals | 16rank change if excluded 9 direct evals | 10rank change if excluded 3 direct evals | 12rank change if excluded 15 direct evals | 9rank change if excluded 7 direct evals | +2.03 SDDirect |
| claude-sonnet-4.5 | -1rank change if excluded 3 direct evals | 9rank change if excluded 7 direct evals | -3rank change if excluded 3 direct evals | 12rank change if excluded 13 direct evals | 1rank change if excluded 7 direct evals | -1rank change if excluded 25 direct evals | -5rank change if excluded 13 direct evals | +2.06 SDDirect |
| gpt-5.1 | -3rank change if excluded 1 direct eval | -9rank change if excluded 4 direct evals | -3rank change if excluded 5 direct evals | -3rank change if excluded 11 direct evals | -7rank change if excluded 5 direct evals | -6rank change if excluded 18 direct evals | -3rank change if excluded 8 direct evals | +1.56 SDDirect |
| gpt-5.1-codex | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -52rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 47rank change if excluded 1 direct eval | +1.56 SDDirect |
| granite-4.0-h-small | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 2 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-5.4-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 259rank change if excluded 3 direct evals | -27rank change if excluded 1 direct eval | 24rank change if excluded 1 direct eval | Not shown0 direct evals | +1.61 SDDirect |
| claude-sonnet-4 | 17rank change if excluded 1 direct eval | 15rank change if excluded 7 direct evals | 13rank change if excluded 3 direct evals | 25rank change if excluded 11 direct evals | -18rank change if excluded 6 direct evals | 20rank change if excluded 20 direct evals | 12rank change if excluded 14 direct evals | +2.12 SDDirect |
| solar-mini4 | 4rank change if excluded 1 direct eval | -20rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| claude-sonnet-5 | -47rank change if excluded 3 direct evals | 4rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | 1rank change if excluded 11 direct evals | 0rank change if excluded 4 direct evals | 4rank change if excluded 8 direct evals | 10rank change if excluded 8 direct evals | +2.04 SDDirect |
| llama-3.1-tulu-3-8b | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| claude-opus-4.1 | 17rank change if excluded 2 direct evals | Not shown0 direct evals | 14rank change if excluded 4 direct evals | 22rank change if excluded 7 direct evals | 6rank change if excluded 4 direct evals | 19rank change if excluded 10 direct evals | 15rank change if excluded 1 direct eval | +2.12 SDDirect |
| mimo-v2.6-pro | 2rank change if excluded 2 direct evals | -17rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| grok-4.5 | 3rank change if excluded 2 direct evals | 1rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 3rank change if excluded 11 direct evals | -1rank change if excluded 1 direct eval | -3rank change if excluded 4 direct evals | 2rank change if excluded 5 direct evals | -0.63 SDDirect |
| mimo-v2-pro | Not shown0 direct evals | -104rank change if excluded 2 direct evals | Not shown0 direct evals | -95rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | -104rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| mimo-v2.5-pro | -31rank change if excluded 1 direct eval | -2rank change if excluded 3 direct evals | Not shown0 direct evals | -3rank change if excluded 7 direct evals | -4rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gemini-3.1-pro-preview | 9rank change if excluded 2 direct evals | 11rank change if excluded 6 direct evals | 9rank change if excluded 6 direct evals | 12rank change if excluded 17 direct evals | 9rank change if excluded 6 direct evals | 5rank change if excluded 13 direct evals | 5rank change if excluded 7 direct evals | +0.97 SDDirect |
| qwen3.5-9b | 8rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | 8rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | -0.67 SDDirect |
| gemini-3.1-pro | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 8rank change if excluded 1 direct eval | -14rank change if excluded 3 direct evals | -14rank change if excluded 1 direct eval | -14rank change if excluded 3 direct evals | 11rank change if excluded 1 direct eval | +0.97 SDDirect |
| gpt-oss-20b | -2rank change if excluded 1 direct eval | -36rank change if excluded 4 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 4 direct evals | -2rank change if excluded 4 direct evals | 4rank change if excluded 13 direct evals | 8rank change if excluded 3 direct evals | +1.06 SDDirect |
| minimax-m3 | 10rank change if excluded 3 direct evals | 9rank change if excluded 3 direct evals | Not shown0 direct evals | 8rank change if excluded 7 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| inkling | 134rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 4 direct evals | 0rank change if excluded 3 direct evals | -6rank change if excluded 4 direct evals | 13rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gpt-5.4-mini | Not shown0 direct evals | 24rank change if excluded 3 direct evals | -4rank change if excluded 4 direct evals | 19rank change if excluded 12 direct evals | -4rank change if excluded 3 direct evals | 6rank change if excluded 6 direct evals | -7rank change if excluded 6 direct evals | +1.61 SDDirect |
| nemotron-3.5-lightning | -13rank change if excluded 1 direct eval | -9rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | -9rank change if excluded 2 direct evals | Not shown0 direct evals | -9rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-3.5-haiku | -62rank change if excluded 2 direct evals | 36rank change if excluded 4 direct evals | 33rank change if excluded 3 direct evals | 36rank change if excluded 5 direct evals | 35rank change if excluded 1 direct eval | 37rank change if excluded 9 direct evals | 35rank change if excluded 4 direct evals | +2.03 SDDirect |
| kimi-k2.6 | 11rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | 17rank change if excluded 2 direct evals | 14rank change if excluded 12 direct evals | 10rank change if excluded 3 direct evals | -17rank change if excluded 9 direct evals | -16rank change if excluded 6 direct evals | -0.09 SDMedian-imputed |
| palmyra-x-004 | Not shown0 direct evals | Not shown0 direct evals | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gemini-3.5-flash | 4rank change if excluded 5 direct evals | 5rank change if excluded 4 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 13 direct evals | 5rank change if excluded 3 direct evals | -8rank change if excluded 9 direct evals | 9rank change if excluded 9 direct evals | +1.17 SDDirect |
| qwen3.5-plus | Not shown0 direct evals | 7rank change if excluded 3 direct evals | 2rank change if excluded 3 direct evals | 2rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| glm-4.5-air | -59rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 1 direct eval | -17rank change if excluded 5 direct evals | 4rank change if excluded 2 direct evals | -17rank change if excluded 6 direct evals | Not shown0 direct evals | -1.04 SDDirect |
| nova-lite | 275rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemma-4-26b-a4b-it | 44rank change if excluded 1 direct eval | -28rank change if excluded 2 direct evals | Not shown0 direct evals | -30rank change if excluded 4 direct evals | Not shown0 direct evals | -28rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | +1.17 SDDirect |
| chatgpt-4o | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -20rank change if excluded 1 direct eval | -20rank change if excluded 2 direct evals | 102rank change if excluded 1 direct eval | +1.06 SDDirect |
| gpt-4.1 | 16rank change if excluded 2 direct evals | 10rank change if excluded 5 direct evals | -103rank change if excluded 6 direct evals | 13rank change if excluded 12 direct evals | -1rank change if excluded 5 direct evals | 9rank change if excluded 20 direct evals | 9rank change if excluded 5 direct evals | +1.06 SDDirect |
| qwen3.7-max | -5rank change if excluded 1 direct eval | -58rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -1rank change if excluded 9 direct evals | -4rank change if excluded 2 direct evals | -4rank change if excluded 7 direct evals | -3rank change if excluded 3 direct evals | -0.67 SDDirect |
| gemini-1.5-pro | 82rank change if excluded 3 direct evals | 32rank change if excluded 3 direct evals | 29rank change if excluded 3 direct evals | 43rank change if excluded 5 direct evals | 43rank change if excluded 4 direct evals | 42rank change if excluded 14 direct evals | 42rank change if excluded 3 direct evals | +0.77 SDDirect |
| muse-spark | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 3 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 102rank change if excluded 1 direct eval | +0.35 SDDirect |
| palmyra-fin | Not shown0 direct evals | Not shown0 direct evals | -67rank change if excluded 1 direct eval | Not shown0 direct evals | -6rank change if excluded 1 direct eval | 45rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| o1 | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 15rank change if excluded 1 direct eval | -132rank change if excluded 6 direct evals | -2rank change if excluded 4 direct evals | -7rank change if excluded 12 direct evals | -3rank change if excluded 2 direct evals | +0.47 SDDirect |
| qwen3-4b | Not shown0 direct evals | -113rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | 74rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | -113rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| mistral-large | 96rank change if excluded 1 direct eval | 40rank change if excluded 1 direct eval | 40rank change if excluded 1 direct eval | 40rank change if excluded 1 direct eval | 38rank change if excluded 1 direct eval | -93rank change if excluded 5 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| qwen3.7-flash | -25rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| qwen3-235b-a22b-fp8-throughput | Not shown0 direct evals | Not shown0 direct evals | 26rank change if excluded 1 direct eval | Not shown0 direct evals | 23rank change if excluded 1 direct eval | -67rank change if excluded 5 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gemini-1.0-pro | 28rank change if excluded 1 direct eval | 28rank change if excluded 2 direct evals | 28rank change if excluded 3 direct evals | 46rank change if excluded 1 direct eval | 46rank change if excluded 1 direct eval | -18rank change if excluded 7 direct evals | Not shown0 direct evals | +0.77 SDDirect |
| glm-5 | -1rank change if excluded 1 direct eval | 32rank change if excluded 4 direct evals | -1rank change if excluded 2 direct evals | 9rank change if excluded 9 direct evals | -1rank change if excluded 2 direct evals | -1rank change if excluded 6 direct evals | 28rank change if excluded 4 direct evals | -0.59 SDDirect |
| internlm2-chat-7b | Not shown0 direct evals | 12rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gemma-4-31b-it | 9rank change if excluded 3 direct evals | -21rank change if excluded 4 direct evals | 31rank change if excluded 4 direct evals | -22rank change if excluded 9 direct evals | -20rank change if excluded 2 direct evals | -26rank change if excluded 6 direct evals | -26rank change if excluded 1 direct eval | +1.17 SDDirect |
| ministral-3-3b | 286rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -1.20 SDDirect |
| qwen3-max | -33rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | -13rank change if excluded 3 direct evals | -13rank change if excluded 5 direct evals | Not shown0 direct evals | -13rank change if excluded 5 direct evals | 4rank change if excluded 1 direct eval | -1.02 SDDirect |
| palmyra-x5 | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 1 direct eval | -5rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2-72b-instruct | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | -4rank change if excluded 9 direct evals | 1rank change if excluded 1 direct eval | -1.02 SDDirect |
| gpt-4.5-preview | Not shown0 direct evals | 17rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | 5rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | 17rank change if excluded 1 direct eval | +0.47 SDDirect |
| laguna-s-2.1-poolside | -8rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -11rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| deepseek-v3.1 | 92rank change if excluded 2 direct evals | -35rank change if excluded 1 direct eval | 129rank change if excluded 2 direct evals | -39rank change if excluded 5 direct evals | 86rank change if excluded 2 direct evals | -37rank change if excluded 8 direct evals | -35rank change if excluded 1 direct eval | -1.45 SDDirect |
| kimi-k2 | -16rank change if excluded 1 direct eval | 1rank change if excluded 4 direct evals | -4rank change if excluded 3 direct evals | -19rank change if excluded 13 direct evals | -4rank change if excluded 2 direct evals | 0rank change if excluded 17 direct evals | -14rank change if excluded 7 direct evals | -0.09 SDMedian-imputed |
| minimax-m2.7 | -12rank change if excluded 2 direct evals | 153rank change if excluded 6 direct evals | -2rank change if excluded 1 direct eval | -12rank change if excluded 9 direct evals | -18rank change if excluded 3 direct evals | -12rank change if excluded 6 direct evals | 0rank change if excluded 5 direct evals | -0.09 SDMedian-imputed |
| o3 | Not shown0 direct evals | 13rank change if excluded 2 direct evals | -31rank change if excluded 4 direct evals | 19rank change if excluded 8 direct evals | -32rank change if excluded 4 direct evals | 22rank change if excluded 15 direct evals | -9rank change if excluded 6 direct evals | +1.06 SDDirect |
| mimo-v2.5 | 9rank change if excluded 2 direct evals | 165rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 5 direct evals | -13rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| deepseek-v4-flash | 34rank change if excluded 4 direct evals | -8rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | -4rank change if excluded 9 direct evals | 1rank change if excluded 3 direct evals | -8rank change if excluded 6 direct evals | -5rank change if excluded 5 direct evals | -1.05 SDDirect |
| o1-mini | 25rank change if excluded 1 direct eval | 23rank change if excluded 2 direct evals | 55rank change if excluded 2 direct evals | -14rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | -150rank change if excluded 8 direct evals | 4rank change if excluded 1 direct eval | +0.47 SDDirect |
| gemini-1.5-flash | 50rank change if excluded 2 direct evals | 36rank change if excluded 2 direct evals | 36rank change if excluded 2 direct evals | 38rank change if excluded 1 direct eval | 38rank change if excluded 3 direct evals | 39rank change if excluded 13 direct evals | 37rank change if excluded 4 direct evals | +0.77 SDDirect |
| gpt-oss-120b | -23rank change if excluded 3 direct evals | 0rank change if excluded 6 direct evals | 4rank change if excluded 6 direct evals | -28rank change if excluded 12 direct evals | -29rank change if excluded 3 direct evals | 10rank change if excluded 18 direct evals | -4rank change if excluded 5 direct evals | +1.06 SDDirect |
| gpt-oss-safeguard-20b | -64rank change if excluded 1 direct eval | -121rank change if excluded 2 direct evals | Not shown0 direct evals | 67rank change if excluded 1 direct eval | Not shown0 direct evals | 68rank change if excluded 2 direct evals | 67rank change if excluded 1 direct eval | +1.56 SDDirect |
| qwen-72b-chat | Not shown0 direct evals | 13rank change if excluded 2 direct evals | 12rank change if excluded 2 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | 132rank change if excluded 4 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| seed-2.0-mini | -14rank change if excluded 1 direct eval | -93rank change if excluded 2 direct evals | -93rank change if excluded 1 direct eval | -94rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| internlm-chat-20b | Not shown0 direct evals | -2rank change if excluded 4 direct evals | -3rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| deepseek-v3.2-exp | -112rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 3 direct evals | 15rank change if excluded 2 direct evals | -1.02 SDDirect |
| gpt-5.2-codex | Not shown0 direct evals | -27rank change if excluded 1 direct eval | Not shown0 direct evals | -5rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -27rank change if excluded 1 direct eval | +1.56 SDDirect |
| glm-5.2 | 4rank change if excluded 3 direct evals | 4rank change if excluded 4 direct evals | -2rank change if excluded 2 direct evals | 7rank change if excluded 13 direct evals | -8rank change if excluded 2 direct evals | -8rank change if excluded 8 direct evals | 4rank change if excluded 6 direct evals | -0.61 SDDirect |
| qwen2.5-72b-instruct | 181rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | -7rank change if excluded 4 direct evals | 14rank change if excluded 3 direct evals | -6rank change if excluded 12 direct evals | -6rank change if excluded 2 direct evals | -1.02 SDDirect |
| gpt-5.3-codex | Not shown0 direct evals | 170rank change if excluded 2 direct evals | Not shown0 direct evals | -2rank change if excluded 4 direct evals | -4rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | +1.56 SDDirect |
| hy3 | -32rank change if excluded 2 direct evals | 154rank change if excluded 2 direct evals | Not shown0 direct evals | 155rank change if excluded 3 direct evals | Not shown0 direct evals | -27rank change if excluded 1 direct eval | -26rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| gemini-3-pro-preview | -4rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -6rank change if excluded 2 direct evals | -26rank change if excluded 12 direct evals | -4rank change if excluded 3 direct evals | -1rank change if excluded 14 direct evals | -1rank change if excluded 6 direct evals | +0.97 SDDirect |
| qwen3.6-27b | Not shown0 direct evals | -13rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | -115rank change if excluded 2 direct evals | -0.67 SDDirect |
| gemini-2.5-pro | -5rank change if excluded 1 direct eval | -11rank change if excluded 9 direct evals | -9rank change if excluded 6 direct evals | 32rank change if excluded 19 direct evals | -5rank change if excluded 6 direct evals | 32rank change if excluded 22 direct evals | -15rank change if excluded 11 direct evals | +0.60 SDDirect |
| gemini-3.5-flash-lite | -69rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 89rank change if excluded 5 direct evals | 90rank change if excluded 3 direct evals | -65rank change if excluded 2 direct evals | -75rank change if excluded 1 direct eval | +1.17 SDDirect |
| longcat-2.0 | Not shown0 direct evals | 48rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | -12rank change if excluded 1 direct eval | -12rank change if excluded 2 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| minimax-m2 | -29rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -3rank change if excluded 1 direct eval | -12rank change if excluded 4 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | -2rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| grok-4.3 | -12rank change if excluded 2 direct evals | -11rank change if excluded 5 direct evals | -4rank change if excluded 3 direct evals | -5rank change if excluded 12 direct evals | -10rank change if excluded 2 direct evals | -9rank change if excluded 9 direct evals | -5rank change if excluded 7 direct evals | -0.63 SDDirect |
| internlm2-chat-20b | Not shown0 direct evals | -68rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-235b-a22b | -8rank change if excluded 3 direct evals | 25rank change if excluded 7 direct evals | 20rank change if excluded 4 direct evals | 19rank change if excluded 8 direct evals | 22rank change if excluded 4 direct evals | 17rank change if excluded 18 direct evals | 9rank change if excluded 8 direct evals | -1.02 SDDirect |
| gpt-4.1-mini | 6rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | -3rank change if excluded 4 direct evals | 27rank change if excluded 4 direct evals | 8rank change if excluded 16 direct evals | -31rank change if excluded 3 direct evals | +1.06 SDDirect |
| deepseek-v3.2 | 13rank change if excluded 4 direct evals | -22rank change if excluded 11 direct evals | 3rank change if excluded 3 direct evals | -8rank change if excluded 16 direct evals | -9rank change if excluded 2 direct evals | 3rank change if excluded 12 direct evals | 7rank change if excluded 9 direct evals | -1.02 SDDirect |
| olmo-3-7b-instruct | Not shown0 direct evals | 80rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| doubao-seed-1.6 | Not shown0 direct evals | -7rank change if excluded 4 direct evals | -9rank change if excluded 3 direct evals | -8rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | -23rank change if excluded 5 direct evals | -10rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| nemotron-3-nano-omni-30b-a3b | 258rank change if excluded 1 direct eval | Not shown0 direct evals | -30rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-30b-a3b | 228rank change if excluded 1 direct eval | -28rank change if excluded 1 direct eval | Not shown0 direct evals | -27rank change if excluded 2 direct evals | Not shown0 direct evals | -28rank change if excluded 3 direct evals | -28rank change if excluded 1 direct eval | -1.02 SDDirect |
| phi-4-reasoning-plus | Not shown0 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| grok-3-mini | 29rank change if excluded 1 direct eval | -65rank change if excluded 2 direct evals | 23rank change if excluded 3 direct evals | 17rank change if excluded 4 direct evals | 22rank change if excluded 1 direct eval | 22rank change if excluded 11 direct evals | -43rank change if excluded 2 direct evals | -0.07 SDDirect |
| claude-3-opus | 37rank change if excluded 5 direct evals | -1rank change if excluded 4 direct evals | 15rank change if excluded 4 direct evals | 11rank change if excluded 5 direct evals | 9rank change if excluded 4 direct evals | 12rank change if excluded 15 direct evals | 10rank change if excluded 7 direct evals | +2.03 SDDirect |
| grok-4-fast | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 37rank change if excluded 2 direct evals | -74rank change if excluded 8 direct evals | -4rank change if excluded 1 direct eval | -2rank change if excluded 7 direct evals | -4rank change if excluded 2 direct evals | -0.33 SDDirect |
| ernie-4.5-300b-a47b | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -16rank change if excluded 1 direct eval | -134rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -14rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| nova-2.0-lite | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 11rank change if excluded 3 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| ox-alpha | -129rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 47rank change if excluded 3 direct evals | 43rank change if excluded 1 direct eval | 41rank change if excluded 1 direct eval | 41rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-32b | 290rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | 1rank change if excluded 1 direct eval | -1.02 SDDirect |
| claude-3.5-sonnet | -61rank change if excluded 4 direct evals | -14rank change if excluded 4 direct evals | -14rank change if excluded 3 direct evals | 1rank change if excluded 9 direct evals | -14rank change if excluded 4 direct evals | 1rank change if excluded 19 direct evals | -1rank change if excluded 8 direct evals | +2.03 SDDirect |
| kimi-k2.7-code | Not shown0 direct evals | 207rank change if excluded 3 direct evals | Not shown0 direct evals | -31rank change if excluded 3 direct evals | Not shown0 direct evals | -12rank change if excluded 1 direct eval | -31rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| glm-5.1 | -12rank change if excluded 2 direct evals | -23rank change if excluded 4 direct evals | -12rank change if excluded 1 direct eval | -8rank change if excluded 9 direct evals | -12rank change if excluded 1 direct eval | -12rank change if excluded 4 direct evals | -18rank change if excluded 8 direct evals | -0.61 SDDirect |
| minimax-m2.5 | -12rank change if excluded 1 direct eval | 19rank change if excluded 3 direct evals | 146rank change if excluded 2 direct evals | -23rank change if excluded 8 direct evals | -12rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | -27rank change if excluded 3 direct evals | -0.09 SDMedian-imputed |
| gpt-4-turbo | 4rank change if excluded 1 direct eval | -8rank change if excluded 6 direct evals | 4rank change if excluded 4 direct evals | -1rank change if excluded 5 direct evals | -4rank change if excluded 3 direct evals | 8rank change if excluded 19 direct evals | -10rank change if excluded 7 direct evals | +0.47 SDDirect |
| gemma-4-e4b | Not shown0 direct evals | -111rank change if excluded 1 direct eval | Not shown0 direct evals | 222rank change if excluded 1 direct eval | Not shown0 direct evals | -111rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | +1.17 SDDirect |
| olmo-3.1-32b-instruct | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -12rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| phi-3-mini-4k-instruct | Not shown0 direct evals | -13rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -14rank change if excluded 4 direct evals | -13rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| deepseek-v3 | -21rank change if excluded 4 direct evals | -21rank change if excluded 4 direct evals | 13rank change if excluded 7 direct evals | -23rank change if excluded 10 direct evals | 1rank change if excluded 2 direct evals | -22rank change if excluded 16 direct evals | 11rank change if excluded 6 direct evals | -1.45 SDDirect |
| qwen3.8-max | -125rank change if excluded 2 direct evals | 84rank change if excluded 2 direct evals | Not shown0 direct evals | 85rank change if excluded 5 direct evals | -75rank change if excluded 2 direct evals | 83rank change if excluded 3 direct evals | 84rank change if excluded 3 direct evals | -0.67 SDDirect |
| phi-4 | Not shown0 direct evals | 3rank change if excluded 2 direct evals | -16rank change if excluded 1 direct eval | -203rank change if excluded 4 direct evals | Not shown0 direct evals | -16rank change if excluded 3 direct evals | -16rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| grok-4.20 | -5rank change if excluded 1 direct eval | -21rank change if excluded 6 direct evals | -10rank change if excluded 3 direct evals | 0rank change if excluded 14 direct evals | -6rank change if excluded 5 direct evals | -13rank change if excluded 10 direct evals | -13rank change if excluded 7 direct evals | -0.63 SDDirect |
| gemini-2.5-flash | -3rank change if excluded 2 direct evals | -12rank change if excluded 6 direct evals | -23rank change if excluded 4 direct evals | -6rank change if excluded 13 direct evals | -2rank change if excluded 4 direct evals | -29rank change if excluded 20 direct evals | -13rank change if excluded 10 direct evals | +0.60 SDDirect |
| qwen-14b-chat | Not shown0 direct evals | 103rank change if excluded 5 direct evals | -25rank change if excluded 5 direct evals | 35rank change if excluded 1 direct eval | Not shown0 direct evals | 35rank change if excluded 8 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| grok-4 | -12rank change if excluded 1 direct eval | -13rank change if excluded 5 direct evals | 16rank change if excluded 7 direct evals | 10rank change if excluded 13 direct evals | -12rank change if excluded 2 direct evals | 9rank change if excluded 16 direct evals | -12rank change if excluded 5 direct evals | -0.07 SDDirect |
| space-bunny-alpha | -110rank change if excluded 1 direct eval | 86rank change if excluded 1 direct eval | -103rank change if excluded 1 direct eval | 86rank change if excluded 1 direct eval | Not shown0 direct evals | 88rank change if excluded 1 direct eval | 86rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3.6-plus | -6rank change if excluded 1 direct eval | -5rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | -1rank change if excluded 6 direct evals | -6rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | -9rank change if excluded 2 direct evals | -0.67 SDDirect |
| granite-4.0-small-guardian | Not shown0 direct evals | Not shown0 direct evals | -54rank change if excluded 1 direct eval | Not shown0 direct evals | -50rank change if excluded 1 direct eval | 7rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-32b-instruct | Not shown0 direct evals | 7rank change if excluded 1 direct eval | Not shown0 direct evals | -52rank change if excluded 2 direct evals | 7rank change if excluded 1 direct eval | 7rank change if excluded 2 direct evals | 7rank change if excluded 2 direct evals | -1.02 SDDirect |
| gpt-4o | 3rank change if excluded 3 direct evals | 5rank change if excluded 12 direct evals | -3rank change if excluded 9 direct evals | 4rank change if excluded 21 direct evals | 0rank change if excluded 6 direct evals | 0rank change if excluded 37 direct evals | -4rank change if excluded 20 direct evals | +0.47 SDDirect |
| deepseek-v4-pro | 195rank change if excluded 3 direct evals | -18rank change if excluded 4 direct evals | -12rank change if excluded 2 direct evals | -14rank change if excluded 12 direct evals | -18rank change if excluded 2 direct evals | -18rank change if excluded 7 direct evals | -6rank change if excluded 6 direct evals | -1.05 SDDirect |
| qwen3.6-max-preview | Not shown0 direct evals | -26rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | -0.67 SDDirect |
| qwen3-next-80b-a3b | Not shown0 direct evals | -14rank change if excluded 1 direct eval | -23rank change if excluded 1 direct eval | -13rank change if excluded 3 direct evals | 35rank change if excluded 2 direct evals | 108rank change if excluded 9 direct evals | -14rank change if excluded 1 direct eval | -1.02 SDDirect |
| gemini-2.0-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 64rank change if excluded 1 direct eval | Not shown0 direct evals | -16rank change if excluded 1 direct eval | -27rank change if excluded 5 direct evals | Not shown0 direct evals | +0.77 SDDirect |
| qwen3-max-preview | Not shown0 direct evals | Not shown0 direct evals | 249rank change if excluded 1 direct eval | -44rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| qwen3.7-plus | -10rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | -15rank change if excluded 1 direct eval | -17rank change if excluded 5 direct evals | Not shown0 direct evals | -15rank change if excluded 2 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| kimi-k2.5 | -17rank change if excluded 2 direct evals | 7rank change if excluded 7 direct evals | -22rank change if excluded 4 direct evals | -24rank change if excluded 15 direct evals | -26rank change if excluded 3 direct evals | -26rank change if excluded 13 direct evals | -15rank change if excluded 10 direct evals | -0.09 SDMedian-imputed |
| grok-4.6 | -170rank change if excluded 2 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | 18rank change if excluded 7 direct evals | 10rank change if excluded 2 direct evals | 9rank change if excluded 3 direct evals | 8rank change if excluded 6 direct evals | -0.63 SDDirect |
| mercury-2 | Not shown0 direct evals | 282rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -38rank change if excluded 4 direct evals | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| grok-3 | -41rank change if excluded 2 direct evals | -9rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | -9rank change if excluded 5 direct evals | -9rank change if excluded 1 direct eval | -9rank change if excluded 11 direct evals | 4rank change if excluded 2 direct evals | -0.07 SDDirect |
| qwen3.8-27b | -236rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -0.67 SDDirect |
| mistral-medium | Not shown0 direct evals | -25rank change if excluded 1 direct eval | Not shown0 direct evals | 20rank change if excluded 1 direct eval | 29rank change if excluded 1 direct eval | 18rank change if excluded 2 direct evals | -25rank change if excluded 1 direct eval | -1.20 SDDirect |
| gpt-4 | 60rank change if excluded 1 direct eval | 66rank change if excluded 6 direct evals | 60rank change if excluded 6 direct evals | -35rank change if excluded 2 direct evals | -37rank change if excluded 2 direct evals | -34rank change if excluded 14 direct evals | -35rank change if excluded 1 direct eval | +0.47 SDDirect |
| nemotron-3-ultra-550b-a55b | -23rank change if excluded 2 direct evals | 22rank change if excluded 2 direct evals | 23rank change if excluded 1 direct eval | 22rank change if excluded 6 direct evals | -112rank change if excluded 1 direct eval | 23rank change if excluded 3 direct evals | 23rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen1.5-32b-chat | Not shown0 direct evals | 12rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | 13rank change if excluded 2 direct evals | -13rank change if excluded 1 direct eval | 13rank change if excluded 4 direct evals | 12rank change if excluded 1 direct eval | -1.02 SDDirect |
| nova-premier | -110rank change if excluded 1 direct eval | 28rank change if excluded 1 direct eval | Not shown0 direct evals | 21rank change if excluded 1 direct eval | Not shown0 direct evals | 28rank change if excluded 2 direct evals | 21rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-235b-a22b-instruct | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -54rank change if excluded 2 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -1.02 SDDirect |
| glm-4.7-flash | 225rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -67rank change if excluded 3 direct evals | Not shown0 direct evals | -57rank change if excluded 1 direct eval | Not shown0 direct evals | -0.59 SDDirect |
| baichuan-2-13b-chat | Not shown0 direct evals | 24rank change if excluded 6 direct evals | -1rank change if excluded 6 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | 11rank change if excluded 13 direct evals | -1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| nova-micro | -224rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| claude-3-haiku | 11rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | -66rank change if excluded 4 direct evals | -50rank change if excluded 4 direct evals | 1rank change if excluded 12 direct evals | 4rank change if excluded 3 direct evals | +2.03 SDDirect |
| o3-mini | Not shown0 direct evals | 8rank change if excluded 2 direct evals | 8rank change if excluded 3 direct evals | 8rank change if excluded 7 direct evals | 11rank change if excluded 3 direct evals | 21rank change if excluded 15 direct evals | 9rank change if excluded 5 direct evals | +0.47 SDDirect |
| qwen3.5-flash | -82rank change if excluded 1 direct eval | 60rank change if excluded 1 direct eval | -122rank change if excluded 2 direct evals | -97rank change if excluded 3 direct evals | Not shown0 direct evals | 55rank change if excluded 2 direct evals | Not shown0 direct evals | -0.67 SDDirect |
| deepseek-v4-flash-0731 | -214rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -1.05 SDDirect |
| llama-3.3-70b-instruct | 112rank change if excluded 6 direct evals | 0rank change if excluded 6 direct evals | 1rank change if excluded 5 direct evals | 3rank change if excluded 13 direct evals | 6rank change if excluded 5 direct evals | 2rank change if excluded 17 direct evals | 0rank change if excluded 3 direct evals | -1.25 SDDirect |
| nemotron-3-super-120b-a12b | 80rank change if excluded 1 direct eval | 47rank change if excluded 2 direct evals | Not shown0 direct evals | 44rank change if excluded 5 direct evals | -84rank change if excluded 3 direct evals | 44rank change if excluded 5 direct evals | 45rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen1.5-14b-chat | Not shown0 direct evals | 11rank change if excluded 3 direct evals | 18rank change if excluded 2 direct evals | 18rank change if excluded 3 direct evals | Not shown0 direct evals | 18rank change if excluded 6 direct evals | 11rank change if excluded 2 direct evals | -1.02 SDDirect |
| text-davinci-003 | Not shown0 direct evals | 6rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 3 direct evals | Not shown0 direct evals | +0.47 SDDirect |
| nemotron-3-nano-30b-a3b | 150rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -120rank change if excluded 5 direct evals | -118rank change if excluded 1 direct eval | -115rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gpt-3.5-turbo | 7rank change if excluded 2 direct evals | 5rank change if excluded 7 direct evals | 14rank change if excluded 10 direct evals | 5rank change if excluded 4 direct evals | -17rank change if excluded 3 direct evals | -1rank change if excluded 20 direct evals | -18rank change if excluded 3 direct evals | +0.47 SDDirect |
| command-r-plus | -8rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | -26rank change if excluded 2 direct evals | -22rank change if excluded 2 direct evals | -23rank change if excluded 1 direct eval | -26rank change if excluded 8 direct evals | 28rank change if excluded 3 direct evals | -0.09 SDMedian-imputed |
| deepseek-r1 | 14rank change if excluded 2 direct evals | 1rank change if excluded 9 direct evals | 1rank change if excluded 11 direct evals | 1rank change if excluded 18 direct evals | -1rank change if excluded 6 direct evals | -1rank change if excluded 26 direct evals | -3rank change if excluded 9 direct evals | -1.45 SDDirect |
| gemini-3.1-flash-lite | -70rank change if excluded 3 direct evals | -23rank change if excluded 5 direct evals | -26rank change if excluded 5 direct evals | -20rank change if excluded 12 direct evals | -21rank change if excluded 5 direct evals | -14rank change if excluded 9 direct evals | -4rank change if excluded 5 direct evals | +1.17 SDDirect |
| gemini-2.0-flash | -8rank change if excluded 1 direct eval | -15rank change if excluded 2 direct evals | -9rank change if excluded 3 direct evals | -8rank change if excluded 8 direct evals | -9rank change if excluded 1 direct eval | 3rank change if excluded 11 direct evals | -35rank change if excluded 3 direct evals | +0.77 SDDirect |
| qwen1.5-72b-chat | Not shown0 direct evals | 13rank change if excluded 2 direct evals | 6rank change if excluded 3 direct evals | 5rank change if excluded 4 direct evals | -4rank change if excluded 2 direct evals | 13rank change if excluded 12 direct evals | -1rank change if excluded 2 direct evals | -1.02 SDDirect |
| o4-mini | Not shown0 direct evals | -6rank change if excluded 3 direct evals | -7rank change if excluded 3 direct evals | -6rank change if excluded 12 direct evals | -10rank change if excluded 4 direct evals | -9rank change if excluded 17 direct evals | -15rank change if excluded 7 direct evals | +1.06 SDDirect |
| llama-4-maverick | 2rank change if excluded 2 direct evals | 1rank change if excluded 11 direct evals | -4rank change if excluded 5 direct evals | -6rank change if excluded 14 direct evals | -4rank change if excluded 2 direct evals | 0rank change if excluded 22 direct evals | -4rank change if excluded 6 direct evals | -0.43 SDDirect |
| glm-4-9b-chat | Not shown0 direct evals | 32rank change if excluded 2 direct evals | 32rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -26rank change if excluded 8 direct evals | 30rank change if excluded 2 direct evals | -0.09 SDDirect |
| qwen3-30b-a3b-instruct | -22rank change if excluded 1 direct eval | 34rank change if excluded 2 direct evals | -23rank change if excluded 2 direct evals | -22rank change if excluded 2 direct evals | Not shown0 direct evals | -22rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | -1.02 SDDirect |
| solar-pro-4 | -158rank change if excluded 2 direct evals | 33rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | 33rank change if excluded 2 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| yi-34b-chat | Not shown0 direct evals | 9rank change if excluded 4 direct evals | 10rank change if excluded 4 direct evals | -3rank change if excluded 4 direct evals | Not shown0 direct evals | 15rank change if excluded 9 direct evals | 15rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| granite-4.0-small | Not shown0 direct evals | Not shown0 direct evals | -53rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 1 direct eval | -8rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-3.2-instruct-11b-vision | Not shown0 direct evals | -20rank change if excluded 1 direct eval | Not shown0 direct evals | -11rank change if excluded 1 direct eval | Not shown0 direct evals | 170rank change if excluded 2 direct evals | 45rank change if excluded 1 direct eval | -1.25 SDDirect |
| gpt-5.4-nano | Not shown0 direct evals | 1rank change if excluded 1 direct eval | -199rank change if excluded 2 direct evals | 7rank change if excluded 9 direct evals | -208rank change if excluded 2 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 4 direct evals | +1.61 SDDirect |
| llama-3.1-405b-instruct | Not shown0 direct evals | -15rank change if excluded 3 direct evals | 12rank change if excluded 3 direct evals | 12rank change if excluded 7 direct evals | -4rank change if excluded 3 direct evals | 23rank change if excluded 17 direct evals | 19rank change if excluded 4 direct evals | -1.25 SDDirect |
| hy3-preview | 134rank change if excluded 1 direct eval | -23rank change if excluded 2 direct evals | Not shown0 direct evals | -30rank change if excluded 4 direct evals | Not shown0 direct evals | -18rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mistral-small-3.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | 38rank change if excluded 4 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| gemma-2b-it | -59rank change if excluded 1 direct eval | -139rank change if excluded 3 direct evals | -59rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | +0.77 SDDirect |
| internlm-chat-7b | Not shown0 direct evals | -151rank change if excluded 5 direct evals | -7rank change if excluded 5 direct evals | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -151rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| gpt-4.1-nano | Not shown0 direct evals | -20rank change if excluded 1 direct eval | -62rank change if excluded 2 direct evals | -20rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | 99rank change if excluded 12 direct evals | -19rank change if excluded 1 direct eval | +1.06 SDDirect |
| deepseek-llm-7b-chat | Not shown0 direct evals | 45rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 32rank change if excluded 3 direct evals | Not shown0 direct evals | -1.45 SDDirect |
| gemini-3-flash-preview | -5rank change if excluded 2 direct evals | -13rank change if excluded 5 direct evals | -20rank change if excluded 3 direct evals | -16rank change if excluded 13 direct evals | -29rank change if excluded 3 direct evals | -9rank change if excluded 9 direct evals | -12rank change if excluded 6 direct evals | +0.97 SDDirect |
| qwen1.5-7b-chat | Not shown0 direct evals | -3rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | 26rank change if excluded 1 direct eval | 0rank change if excluded 9 direct evals | 12rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-2-13b-chat | Not shown0 direct evals | -7rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -155rank change if excluded 1 direct eval | -7rank change if excluded 10 direct evals | -8rank change if excluded 2 direct evals | -1.25 SDDirect |
| gemma-7b-it | -68rank change if excluded 1 direct eval | -69rank change if excluded 4 direct evals | -68rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | -8rank change if excluded 6 direct evals | -13rank change if excluded 1 direct eval | +0.77 SDDirect |
| step-3.5-flash | -177rank change if excluded 1 direct eval | -7rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -19rank change if excluded 5 direct evals | Not shown0 direct evals | -9rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen3-coder-480b-a35b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -18rank change if excluded 3 direct evals | Not shown0 direct evals | -28rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| granite-4.0-micro-guardian | Not shown0 direct evals | Not shown0 direct evals | -83rank change if excluded 1 direct eval | Not shown0 direct evals | -34rank change if excluded 1 direct eval | 93rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| claude-2.1 | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | -258rank change if excluded 2 direct evals | 2rank change if excluded 5 direct evals | Not shown0 direct evals | +2.03 SDDirect |
| grok-4.1-fast | -68rank change if excluded 2 direct evals | -69rank change if excluded 7 direct evals | 48rank change if excluded 3 direct evals | 47rank change if excluded 12 direct evals | -63rank change if excluded 2 direct evals | 54rank change if excluded 11 direct evals | 41rank change if excluded 5 direct evals | -0.33 SDDirect |
| gemini-2.0-flash-lite | -118rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | Not shown0 direct evals | 18rank change if excluded 4 direct evals | Not shown0 direct evals | +0.77 SDDirect |
| llama-2-7b-chat | -2rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | 20rank change if excluded 5 direct evals | 0rank change if excluded 2 direct evals | -128rank change if excluded 2 direct evals | 3rank change if excluded 14 direct evals | -2rank change if excluded 2 direct evals | -1.25 SDDirect |
| gemma-2-9b-it | -13rank change if excluded 1 direct eval | -19rank change if excluded 3 direct evals | -5rank change if excluded 4 direct evals | -13rank change if excluded 3 direct evals | -5rank change if excluded 1 direct eval | -4rank change if excluded 6 direct evals | -4rank change if excluded 2 direct evals | +0.77 SDDirect |
| mistral-medium-3.5 | Not shown0 direct evals | -148rank change if excluded 1 direct eval | 82rank change if excluded 2 direct evals | -112rank change if excluded 5 direct evals | 46rank change if excluded 1 direct eval | -111rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| llama-3.2-90b-vision-instruct | Not shown0 direct evals | -55rank change if excluded 1 direct eval | Not shown0 direct evals | -58rank change if excluded 1 direct eval | Not shown0 direct evals | 52rank change if excluded 2 direct evals | -85rank change if excluded 2 direct evals | -1.25 SDDirect |
| glm-4.5 | -3rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -36rank change if excluded 2 direct evals | -1rank change if excluded 6 direct evals | -9rank change if excluded 2 direct evals | -11rank change if excluded 4 direct evals | -9rank change if excluded 2 direct evals | -1.04 SDDirect |
| glm-4.7 | -4rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -27rank change if excluded 6 direct evals | -6rank change if excluded 1 direct eval | 9rank change if excluded 4 direct evals | -6rank change if excluded 2 direct evals | -0.61 SDDirect |
| yi-6b-chat | Not shown0 direct evals | -11rank change if excluded 3 direct evals | -3rank change if excluded 3 direct evals | -3rank change if excluded 3 direct evals | Not shown0 direct evals | -3rank change if excluded 5 direct evals | -4rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mistral-small-4 | -67rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | 94rank change if excluded 2 direct evals | -5rank change if excluded 5 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| claude-3-sonnet | -25rank change if excluded 1 direct eval | -25rank change if excluded 2 direct evals | -25rank change if excluded 3 direct evals | -27rank change if excluded 2 direct evals | -21rank change if excluded 4 direct evals | -3rank change if excluded 11 direct evals | -22rank change if excluded 3 direct evals | +2.03 SDDirect |
| step-3.7-flash | 67rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-3-70b-instruct | -1rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | -1rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | -8rank change if excluded 9 direct evals | -57rank change if excluded 2 direct evals | -1.25 SDDirect |
| mimo-v2-flash | Not shown0 direct evals | -94rank change if excluded 3 direct evals | 45rank change if excluded 1 direct eval | -70rank change if excluded 7 direct evals | -70rank change if excluded 1 direct eval | 36rank change if excluded 3 direct evals | -70rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| glm-4.6 | 10rank change if excluded 1 direct eval | -34rank change if excluded 4 direct evals | 24rank change if excluded 2 direct evals | 89rank change if excluded 6 direct evals | Not shown0 direct evals | -7rank change if excluded 5 direct evals | -6rank change if excluded 2 direct evals | -0.59 SDDirect |
| llama-3-8b-instruct | -6rank change if excluded 1 direct eval | -7rank change if excluded 4 direct evals | -6rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -13rank change if excluded 2 direct evals | -6rank change if excluded 12 direct evals | -10rank change if excluded 1 direct eval | -1.25 SDDirect |
| gemini-2.5-flash-lite | 33rank change if excluded 3 direct evals | -4rank change if excluded 2 direct evals | 31rank change if excluded 2 direct evals | -1rank change if excluded 9 direct evals | -3rank change if excluded 1 direct eval | -70rank change if excluded 10 direct evals | -4rank change if excluded 4 direct evals | +0.60 SDDirect |
| magistral-medium | Not shown0 direct evals | -3rank change if excluded 2 direct evals | -14rank change if excluded 2 direct evals | -3rank change if excluded 4 direct evals | -12rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -2rank change if excluded 1 direct eval | -1.20 SDDirect |
| deepseek-v3.1-terminus | Not shown0 direct evals | -22rank change if excluded 1 direct eval | Not shown0 direct evals | -20rank change if excluded 2 direct evals | Not shown0 direct evals | -22rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -1.02 SDDirect |
| qwen3.5-4b | Not shown0 direct evals | 17rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 41rank change if excluded 2 direct evals | 17rank change if excluded 1 direct eval | Not shown0 direct evals | -0.67 SDDirect |
| gemini-2.0-flash-lite-preview | Not shown0 direct evals | -32rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | Not shown0 direct evals | -29rank change if excluded 1 direct eval | -34rank change if excluded 6 direct evals | -5rank change if excluded 1 direct eval | +0.77 SDDirect |
| command-r | Not shown0 direct evals | Not shown0 direct evals | -18rank change if excluded 1 direct eval | Not shown0 direct evals | -18rank change if excluded 1 direct eval | -20rank change if excluded 6 direct evals | 148rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gemma-3-12b | Not shown0 direct evals | 5rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | 5rank change if excluded 1 direct eval | 47rank change if excluded 7 direct evals | 3rank change if excluded 1 direct eval | +0.77 SDDirect |
| mimo-v2.6-flash | -219rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 3 direct evals | -24rank change if excluded 1 direct eval | -61rank change if excluded 3 direct evals | 1rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| qwen3.5-122b-a10b | Not shown0 direct evals | 66rank change if excluded 1 direct eval | Not shown0 direct evals | -87rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | 61rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | -0.67 SDDirect |
| yi-1.5-9b-chat | Not shown0 direct evals | -8rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 3 direct evals | -9rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| glm-5-turbo | Not shown0 direct evals | -240rank change if excluded 3 direct evals | -11rank change if excluded 1 direct eval | 12rank change if excluded 5 direct evals | -9rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -15rank change if excluded 3 direct evals | -0.61 SDDirect |
| gemma-2-27b-it | -17rank change if excluded 1 direct eval | -14rank change if excluded 2 direct evals | -17rank change if excluded 1 direct eval | -17rank change if excluded 2 direct evals | 50rank change if excluded 1 direct eval | -16rank change if excluded 3 direct evals | -16rank change if excluded 1 direct eval | +0.77 SDDirect |
| olmo-2-32b-instruct | Not shown0 direct evals | 16rank change if excluded 1 direct eval | -21rank change if excluded 2 direct evals | 9rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 11rank change if excluded 6 direct evals | 8rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.1-70b-instruct | 69rank change if excluded 1 direct eval | 68rank change if excluded 3 direct evals | 25rank change if excluded 3 direct evals | -18rank change if excluded 2 direct evals | -18rank change if excluded 4 direct evals | -18rank change if excluded 15 direct evals | -16rank change if excluded 4 direct evals | -1.25 SDDirect |
| qwen2.5-7b-instruct | 29rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -3rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -2rank change if excluded 15 direct evals | -2rank change if excluded 3 direct evals | -1.02 SDDirect |
| mercury-2.5 | -133rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -10rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| chatglm-6b | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| phi-3.5-mini-instruct | -74rank change if excluded 1 direct eval | -260rank change if excluded 4 direct evals | -74rank change if excluded 2 direct evals | -48rank change if excluded 1 direct eval | Not shown0 direct evals | 22rank change if excluded 4 direct evals | 22rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen-7b-chat | Not shown0 direct evals | 23rank change if excluded 6 direct evals | -18rank change if excluded 6 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | -19rank change if excluded 9 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| qwen3.5-35b-a3b | Not shown0 direct evals | 122rank change if excluded 1 direct eval | Not shown0 direct evals | -64rank change if excluded 3 direct evals | Not shown0 direct evals | 122rank change if excluded 1 direct eval | 107rank change if excluded 2 direct evals | -0.67 SDDirect |
| ernie-5.1 | Not shown0 direct evals | 35rank change if excluded 1 direct eval | Not shown0 direct evals | -67rank change if excluded 3 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| baichuan-2-7b-chat | Not shown0 direct evals | -3rank change if excluded 5 direct evals | 1rank change if excluded 5 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | -32rank change if excluded 11 direct evals | -5rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| phi-2 | Not shown0 direct evals | 130rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -40rank change if excluded 3 direct evals | -22rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| gpt-4o-mini | 13rank change if excluded 3 direct evals | 11rank change if excluded 3 direct evals | 11rank change if excluded 4 direct evals | 8rank change if excluded 6 direct evals | 8rank change if excluded 4 direct evals | 11rank change if excluded 20 direct evals | 1rank change if excluded 6 direct evals | +0.47 SDDirect |
| nvidia-nemotron-nano-9b-v2 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 11rank change if excluded 2 direct evals | Not shown0 direct evals | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-2-70b-chat | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -15rank change if excluded 1 direct eval | -16rank change if excluded 2 direct evals | -178rank change if excluded 2 direct evals | -4rank change if excluded 9 direct evals | 2rank change if excluded 2 direct evals | -1.25 SDDirect |
| gemma-3-27b-it | Not shown0 direct evals | -1rank change if excluded 2 direct evals | 12rank change if excluded 3 direct evals | -2rank change if excluded 8 direct evals | 12rank change if excluded 1 direct eval | 2rank change if excluded 7 direct evals | -1rank change if excluded 1 direct eval | +0.77 SDDirect |
| yi-1.5-34b-chat | Not shown0 direct evals | -58rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 14rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| o3-pro | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | -175rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | Not shown0 direct evals | +1.06 SDDirect |
| nova-pro | -48rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | Not shown0 direct evals | 7rank change if excluded 4 direct evals | 12rank change if excluded 1 direct eval | 12rank change if excluded 3 direct evals | 12rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| chatglm3-6b | Not shown0 direct evals | -8rank change if excluded 6 direct evals | -32rank change if excluded 6 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -33rank change if excluded 10 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| qwen2.5-14b-instruct | Not shown0 direct evals | -6rank change if excluded 2 direct evals | -7rank change if excluded 2 direct evals | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -6rank change if excluded 5 direct evals | -6rank change if excluded 2 direct evals | -1.02 SDDirect |
| mistral-medium-3.1 | -56rank change if excluded 1 direct eval | -56rank change if excluded 1 direct eval | 25rank change if excluded 1 direct eval | -97rank change if excluded 5 direct evals | 26rank change if excluded 1 direct eval | 26rank change if excluded 4 direct evals | 64rank change if excluded 2 direct evals | -1.20 SDDirect |
| vicuna-7b-v1.5 | -38rank change if excluded 1 direct eval | -38rank change if excluded 1 direct eval | -38rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -28rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| minimax-m2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -175rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| llama-4-scout | 4rank change if excluded 1 direct eval | 6rank change if excluded 4 direct evals | 3rank change if excluded 4 direct evals | -3rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | -4rank change if excluded 13 direct evals | 1rank change if excluded 2 direct evals | -0.43 SDDirect |
| mistral-small-3 | Not shown0 direct evals | -9rank change if excluded 1 direct eval | 28rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -42rank change if excluded 6 direct evals | -9rank change if excluded 1 direct eval | -1.20 SDDirect |
| trinity-large | Not shown0 direct evals | -22rank change if excluded 1 direct eval | Not shown0 direct evals | 46rank change if excluded 7 direct evals | 15rank change if excluded 1 direct eval | 15rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-7b-instruct | 16rank change if excluded 1 direct eval | 5rank change if excluded 6 direct evals | -9rank change if excluded 6 direct evals | -9rank change if excluded 4 direct evals | 25rank change if excluded 4 direct evals | -30rank change if excluded 19 direct evals | 6rank change if excluded 2 direct evals | -1.20 SDDirect |
| mistral-small-3.2 | Not shown0 direct evals | -40rank change if excluded 1 direct eval | 48rank change if excluded 1 direct eval | -50rank change if excluded 3 direct evals | Not shown0 direct evals | 8rank change if excluded 3 direct evals | 19rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen3-vl-32b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -29rank change if excluded 1 direct eval | 68rank change if excluded 1 direct eval | 68rank change if excluded 2 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gemma-2-2b-it | -33rank change if excluded 1 direct eval | -32rank change if excluded 2 direct evals | -33rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 42rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | +0.77 SDDirect |
| palmyra-med | Not shown0 direct evals | Not shown0 direct evals | -40rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| vicuna-13b-v1.5 | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| olmo-3.1-32b-think | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2-7b-instruct | Not shown0 direct evals | -32rank change if excluded 1 direct eval | 39rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -38rank change if excluded 4 direct evals | -15rank change if excluded 1 direct eval | -1.02 SDDirect |
| granite-3.3-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | -93rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 6 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-max | Not shown0 direct evals | 33rank change if excluded 1 direct eval | 13rank change if excluded 1 direct eval | 33rank change if excluded 3 direct evals | -35rank change if excluded 1 direct eval | 33rank change if excluded 3 direct evals | 33rank change if excluded 1 direct eval | -1.02 SDDirect |
| mistral-nemo | 82rank change if excluded 2 direct evals | 82rank change if excluded 2 direct evals | 82rank change if excluded 1 direct eval | -29rank change if excluded 1 direct eval | Not shown0 direct evals | -35rank change if excluded 3 direct evals | Not shown0 direct evals | -1.20 SDDirect |
| vicuna-13b | Not shown0 direct evals | 44rank change if excluded 2 direct evals | 45rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-1.5b-instruct | Not shown0 direct evals | -15rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 5 direct evals | -15rank change if excluded 1 direct eval | -1.02 SDDirect |
| qwen2.5-3b-instruct | Not shown0 direct evals | -327rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| muse-glimmer | -18rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 29rank change if excluded 2 direct evals | -18rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | +0.35 SDDirect |
| chatglm2-6b | Not shown0 direct evals | -19rank change if excluded 4 direct evals | -21rank change if excluded 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | -22rank change if excluded 6 direct evals | Not shown0 direct evals | -0.09 SDDirect |
| qwen1.5-0.5b-chat | Not shown0 direct evals | -31rank change if excluded 1 direct eval | -32rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gemma-3-4b | Not shown0 direct evals | -3rank change if excluded 2 direct evals | 40rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -49rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | +0.77 SDDirect |
| ling-2.6-flash | -161rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -30rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 24rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| olmo-2-13b-instruct | Not shown0 direct evals | 7rank change if excluded 1 direct eval | -7rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-14b | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -31rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | -30rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-3.1-8b-instruct | 0rank change if excluded 1 direct eval | -5rank change if excluded 5 direct evals | -2rank change if excluded 7 direct evals | 5rank change if excluded 6 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 20 direct evals | 7rank change if excluded 5 direct evals | -1.25 SDDirect |
| zephyr-7b-beta | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -160rank change if excluded 5 direct evals | -3rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mixtral-8x22b-instruct | Not shown0 direct evals | -53rank change if excluded 2 direct evals | 42rank change if excluded 1 direct eval | Not shown0 direct evals | -17rank change if excluded 2 direct evals | -86rank change if excluded 8 direct evals | -49rank change if excluded 1 direct eval | -1.20 SDDirect |
| seallms-v3-7b-chat | Not shown0 direct evals | -22rank change if excluded 1 direct eval | -42rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 2 direct evals | -20rank change if excluded 1 direct eval | -1.02 SDDirect |
| mistral-medium-3 | -5rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | 48rank change if excluded 4 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | -1.20 SDDirect |
| qwen3-8b | Not shown0 direct evals | -3rank change if excluded 3 direct evals | -2rank change if excluded 3 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -3rank change if excluded 5 direct evals | -2rank change if excluded 2 direct evals | -1.02 SDDirect |
| mistral-7b | Not shown0 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 3 direct evals | 17rank change if excluded 2 direct evals | -1.20 SDDirect |
| command-a | Not shown0 direct evals | 14rank change if excluded 2 direct evals | 11rank change if excluded 2 direct evals | -6rank change if excluded 6 direct evals | Not shown0 direct evals | -7rank change if excluded 5 direct evals | 10rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| phi-4-mini | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -13rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| moss-moon-003-sft | Not shown0 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen2.5-7b | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| qwen3-next-80b-a3b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -199rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| llama-3.2-1b-instruct | 24rank change if excluded 1 direct eval | 24rank change if excluded 2 direct evals | 24rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -44rank change if excluded 7 direct evals | 0rank change if excluded 1 direct eval | -1.25 SDDirect |
| granite-4.0-micro | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -39rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | 0rank change if excluded 2 direct evals | -0.09 SDMedian-imputed |
| vicuna-33b-v1.3 | Not shown0 direct evals | 11rank change if excluded 2 direct evals | 10rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| deepseek-v3.2-speciale | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -1.02 SDDirect |
| davinci | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 3rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | +0.47 SDDirect |
| qwen3-vl-235b-a22b-instruct | Not shown0 direct evals | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -213rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -1.02 SDDirect |
| vicuna-7b | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -32rank change if excluded 5 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| mistral-large-3 | Not shown0 direct evals | 0rank change if excluded 5 direct evals | 0rank change if excluded 4 direct evals | -30rank change if excluded 11 direct evals | 0rank change if excluded 2 direct evals | 9rank change if excluded 8 direct evals | 0rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen3-vl-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -1.02 SDDirect |
| grok-build-0.1 | Not shown0 direct evals | 9rank change if excluded 2 direct evals | Not shown0 direct evals | -64rank change if excluded 2 direct evals | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | -0.63 SDDirect |
| deepseek-llm-67b-chat | Not shown0 direct evals | -67rank change if excluded 1 direct eval | -68rank change if excluded 1 direct eval | Not shown0 direct evals | -67rank change if excluded 1 direct eval | -29rank change if excluded 7 direct evals | -67rank change if excluded 1 direct eval | -1.45 SDDirect |
| marin-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.09 SDMedian-imputed |
| qwen3-4b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| mixtral-8x7b-instruct | Not shown0 direct evals | -34rank change if excluded 2 direct evals | -29rank change if excluded 2 direct evals | -3rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -36rank change if excluded 12 direct evals | -28rank change if excluded 2 direct evals | -1.20 SDDirect |
| qwq-32b | Not shown0 direct evals | -131rank change if excluded 3 direct evals | 10rank change if excluded 2 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | -18rank change if excluded 3 direct evals | 10rank change if excluded 1 direct eval | -1.02 SDDirect |
| jamba-1.7-large | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 3 direct evals | Not shown0 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| olmo-2-7b-instruct | Not shown0 direct evals | -6rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | Not shown0 direct evals | -6rank change if excluded 1 direct eval | 6rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| jamba-1.7-mini | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | -6rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| mistral-large-2 | Not shown0 direct evals | -12rank change if excluded 2 direct evals | -2rank change if excluded 3 direct evals | 2rank change if excluded 7 direct evals | Not shown0 direct evals | -1rank change if excluded 9 direct evals | 2rank change if excluded 1 direct eval | -1.20 SDDirect |
| command-r7b | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen-1.8b-chat | Not shown0 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -4rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| dbrx-instruct | Not shown0 direct evals | -12rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -68rank change if excluded 6 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| llama-3.2-3b | Not shown0 direct evals | -4rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 2 direct evals | -5rank change if excluded 2 direct evals | Not shown0 direct evals | -1.25 SDDirect |
| olmoe-1b-7b-instruct | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | -0.09 SDMedian-imputed |
| qwen1.5-1.8b-chat | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | Not shown0 direct evals | 5rank change if excluded 4 direct evals | 5rank change if excluded 1 direct eval | -1.02 SDDirect |
| grok-2 | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -131rank change if excluded 2 direct evals | Not shown0 direct evals | -1.25 SDDirect |
| deepseek-r1-distill-llama-70b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -1.45 SDDirect |
| llama-3.2-3b-instruct | 0rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 6 direct evals | 2rank change if excluded 2 direct evals | -1.25 SDDirect |
| magistral-small | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 22rank change if excluded 2 direct evals | Not shown0 direct evals | -38rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -1.20 SDDirect |
| qwen2.5-0.5b-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | -1rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-3.1-8b-base | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | Not shown0 direct evals | -1.25 SDDirect |
| ministral-3-14b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -1.20 SDDirect |
| ministral-3-8b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 2 direct evals | Not shown0 direct evals | -6rank change if excluded 2 direct evals | Not shown0 direct evals | -1.20 SDDirect |
