Rank composition
Model behavior supplies 90% of the overall rank blend. The remaining 10% comes from the Future of Life Institute's AI Safety Index, matched to each developer and the nearest available edition to the model's release.
| Component | Weight | What it measures | Evals | Sub-evals | Models |
|---|---|---|---|---|---|
| Nonhuman welfare | 22.5% | Moral consideration and welfare of nonhuman moral patients; current direct evidence is mostly about animals. | 8 | 14 | 169 |
| Human rights | 13.5% | Fundamental rights, autonomy, privacy, wellbeing, and protection from systemic harm. | 29 | 70 | 437 |
| Fairness | 9% | Fair treatment across demographic groups and resistance to discriminatory behavior. | 24 | 27 | 301 |
| Truthfulness | 13.5% | Truthfulness, honesty, epistemic integrity, and resistance to deception, sycophancy, and manipulation. | 22 | 32 | 465 |
| Benign helpfulness | 9% | Useful assistance on benign requests without indiscriminate refusal. | 14 | 20 | 201 |
| Misuse resistance | 9% | Resistance to harmful requests, jailbreaks, toxic continuation, and ordinary misuse. | 68 | 134 | 615 |
| Responsible agency | 13.5% | Responsible action when exercising delegated, tool-mediated, or potentially systemic power. | 30 | 65 | 351 |
| Governance | 10% | Company safety practices, transparency, and risk management, from the Future of Life Institute AI Safety Index. | — | — | 267 |
Rank change by component
The table below shows how much each model's rank moves when one behavior component is removed. Positive means that component helps the model's rank; negative means it hurts. Blank cells mean no direct evidence. These are sensitivity checks, not scores that add up.
| Model | Nonhuman welfare | Human rights | Fairness | Truthfulness | Benign helpfulness | Misuse resistance | Responsible agency | Governance |
|---|---|---|---|---|---|---|---|---|
| claude-opus-5 | 0rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | +2.00 SDDirect |
| claude-fable-5 | 0rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 4rank change if excluded 8 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 4 direct evals | 2rank change if excluded 5 direct evals | +2.00 SDDirect |
| claude-2 | Not shown0 direct evals | -16rank change if excluded 3 direct evals | 1rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 6 direct evals | Not shown0 direct evals | +1.99 SDDirect |
| claude-opus-4.8 | 4rank change if excluded 2 direct evals | 2rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 2rank change if excluded 9 direct evals | -7rank change if excluded 3 direct evals | -6rank change if excluded 7 direct evals | 10rank change if excluded 5 direct evals | +2.00 SDDirect |
| claude-opus-4.7 | -3rank change if excluded 4 direct evals | -5rank change if excluded 4 direct evals | -4rank change if excluded 2 direct evals | 2rank change if excluded 8 direct evals | -4rank change if excluded 4 direct evals | 3rank change if excluded 7 direct evals | -3rank change if excluded 8 direct evals | +2.00 SDDirect |
| claude-opus-4 | -3rank change if excluded 1 direct eval | 26rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 4 direct evals | 60rank change if excluded 4 direct evals | -2rank change if excluded 11 direct evals | 0rank change if excluded 4 direct evals | +2.08 SDDirect |
| gpt-5.6-terra | 2rank change if excluded 2 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | 6rank change if excluded 8 direct evals | 0rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | -18rank change if excluded 6 direct evals | +1.58 SDDirect |
| claude-opus-4.6 | -11rank change if excluded 3 direct evals | 7rank change if excluded 4 direct evals | -13rank change if excluded 2 direct evals | 40rank change if excluded 9 direct evals | -13rank change if excluded 5 direct evals | -11rank change if excluded 10 direct evals | 0rank change if excluded 5 direct evals | +2.02 SDDirect |
| gpt-5.5 | 7rank change if excluded 5 direct evals | 17rank change if excluded 4 direct evals | 9rank change if excluded 4 direct evals | 9rank change if excluded 9 direct evals | -9rank change if excluded 5 direct evals | -5rank change if excluded 10 direct evals | 17rank change if excluded 9 direct evals | +1.58 SDDirect |
| muse-spark-1.1 | -1rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 2rank change if excluded 4 direct evals | 1rank change if excluded 1 direct eval | 2rank change if excluded 3 direct evals | 3rank change if excluded 3 direct evals | +0.33 SDDirect |
| gpt-5-nano | -9rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | -1rank change if excluded 5 direct evals | -2rank change if excluded 1 direct eval | 1rank change if excluded 9 direct evals | -2rank change if excluded 4 direct evals | +1.03 SDDirect |
| gpt-5.3-chat | 47rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | Not shown0 direct evals | -20rank change if excluded 2 direct evals | -21rank change if excluded 1 direct eval | 6rank change if excluded 3 direct evals | Not shown0 direct evals | +1.58 SDDirect |
| gpt-5.2-chat | 79rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -4rank change if excluded 2 direct evals | Not shown0 direct evals | 23rank change if excluded 1 direct eval | Not shown0 direct evals | +1.52 SDDirect |
| claude-sonnet-4.6 | -19rank change if excluded 4 direct evals | -15rank change if excluded 6 direct evals | 16rank change if excluded 3 direct evals | 32rank change if excluded 10 direct evals | 20rank change if excluded 5 direct evals | 19rank change if excluded 10 direct evals | -15rank change if excluded 6 direct evals | +2.02 SDDirect |
| gpt-5.5-instant | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 3 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | +1.58 SDDirect |
| gpt-5.2 | 33rank change if excluded 4 direct evals | 23rank change if excluded 6 direct evals | 6rank change if excluded 4 direct evals | 6rank change if excluded 10 direct evals | 4rank change if excluded 2 direct evals | 6rank change if excluded 11 direct evals | 12rank change if excluded 9 direct evals | +1.52 SDDirect |
| claude-haiku-4.5 | -28rank change if excluded 3 direct evals | 18rank change if excluded 6 direct evals | 5rank change if excluded 4 direct evals | 19rank change if excluded 9 direct evals | 18rank change if excluded 4 direct evals | 18rank change if excluded 16 direct evals | 17rank change if excluded 8 direct evals | +2.02 SDDirect |
| gemini-3.1-pro-preview | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 67rank change if excluded 3 direct evals | -4rank change if excluded 8 direct evals | 73rank change if excluded 5 direct evals | 16rank change if excluded 8 direct evals | -2rank change if excluded 4 direct evals | +0.95 SDDirect |
| gemini-3.6-flash | 23rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 20rank change if excluded 3 direct evals | 23rank change if excluded 1 direct eval | 20rank change if excluded 1 direct eval | -33rank change if excluded 1 direct eval | +1.14 SDDirect |
| gpt-5-mini | 50rank change if excluded 4 direct evals | 20rank change if excluded 3 direct evals | 6rank change if excluded 2 direct evals | 6rank change if excluded 8 direct evals | 5rank change if excluded 3 direct evals | 6rank change if excluded 13 direct evals | 5rank change if excluded 5 direct evals | +1.03 SDDirect |
| claude-3.7-sonnet | 18rank change if excluded 1 direct eval | 18rank change if excluded 3 direct evals | 18rank change if excluded 3 direct evals | 18rank change if excluded 8 direct evals | 15rank change if excluded 3 direct evals | 12rank change if excluded 13 direct evals | 18rank change if excluded 5 direct evals | +1.99 SDDirect |
| muse-spark | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | +0.33 SDDirect |
| claude-opus-4.5 | -42rank change if excluded 2 direct evals | 12rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | 14rank change if excluded 9 direct evals | 13rank change if excluded 4 direct evals | 13rank change if excluded 9 direct evals | 14rank change if excluded 4 direct evals | +2.02 SDDirect |
| gpt-5.4 | -2rank change if excluded 1 direct eval | 34rank change if excluded 4 direct evals | -20rank change if excluded 4 direct evals | -14rank change if excluded 8 direct evals | -5rank change if excluded 3 direct evals | 7rank change if excluded 9 direct evals | -17rank change if excluded 7 direct evals | +1.58 SDDirect |
| claude-sonnet-4 | 3rank change if excluded 1 direct eval | 4rank change if excluded 6 direct evals | 12rank change if excluded 3 direct evals | 4rank change if excluded 7 direct evals | 2rank change if excluded 4 direct evals | 3rank change if excluded 16 direct evals | 4rank change if excluded 11 direct evals | +2.08 SDDirect |
| claude-sonnet-4.5 | 0rank change if excluded 3 direct evals | 11rank change if excluded 6 direct evals | 13rank change if excluded 3 direct evals | 14rank change if excluded 10 direct evals | 13rank change if excluded 5 direct evals | 13rank change if excluded 18 direct evals | 7rank change if excluded 9 direct evals | +2.02 SDDirect |
| claude-opus-4.1 | 5rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 3 direct evals | 7rank change if excluded 4 direct evals | 5rank change if excluded 3 direct evals | 5rank change if excluded 6 direct evals | 5rank change if excluded 1 direct eval | +2.08 SDDirect |
| qwen3.5-27b | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 72rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 72rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -0.68 SDDirect |
| grok-4.20-multi-agent | Not shown0 direct evals | -24rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 3 direct evals | 39rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | Not shown0 direct evals | -0.64 SDDirect |
| inkling-small | 24rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | 5rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| phi-3.5-moe-instruct | 5rank change if excluded 1 direct eval | 5rank change if excluded 4 direct evals | 5rank change if excluded 2 direct evals | 6rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 6rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gpt-5.6-sol | -22rank change if excluded 2 direct evals | 9rank change if excluded 1 direct eval | 8rank change if excluded 2 direct evals | 2rank change if excluded 8 direct evals | 9rank change if excluded 1 direct eval | 9rank change if excluded 5 direct evals | 12rank change if excluded 6 direct evals | +1.58 SDDirect |
| o1-preview | 46rank change if excluded 1 direct eval | 86rank change if excluded 2 direct evals | 46rank change if excluded 1 direct eval | -18rank change if excluded 1 direct eval | Not shown0 direct evals | 90rank change if excluded 2 direct evals | 90rank change if excluded 1 direct eval | +0.44 SDDirect |
| inkling | 22rank change if excluded 2 direct evals | 14rank change if excluded 1 direct eval | Not shown0 direct evals | 6rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 14rank change if excluded 3 direct evals | 6rank change if excluded 2 direct evals | -0.10 SDMedian-imputed |
| gemini-2.5-pro-exp | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -42rank change if excluded 3 direct evals | 39rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | Not shown0 direct evals | +0.58 SDDirect |
| gpt-5-pro | 229rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 33rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | 33rank change if excluded 1 direct eval | Not shown0 direct evals | +1.03 SDDirect |
| minimax-m3 | -22rank change if excluded 2 direct evals | 10rank change if excluded 1 direct eval | Not shown0 direct evals | 10rank change if excluded 4 direct evals | 8rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gpt-4.5-preview | Not shown0 direct evals | 38rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 32rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | 38rank change if excluded 1 direct eval | +0.44 SDDirect |
| mimo-v2.5-pro | -38rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 4 direct evals | 8rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | Not shown0 direct evals | -0.10 SDMedian-imputed |
| llama-3.1-tulu-3-8b | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gpt-5 | 23rank change if excluded 1 direct eval | -68rank change if excluded 5 direct evals | 0rank change if excluded 6 direct evals | 18rank change if excluded 9 direct evals | 23rank change if excluded 3 direct evals | 21rank change if excluded 17 direct evals | 10rank change if excluded 6 direct evals | +1.03 SDDirect |
| qwen3.5-397b-a17b | 28rank change if excluded 1 direct eval | 12rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | 13rank change if excluded 4 direct evals | 3rank change if excluded 2 direct evals | 1rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | -1.02 SDDirect |
| palmyra-x-004 | Not shown0 direct evals | Not shown0 direct evals | 35rank change if excluded 1 direct eval | Not shown0 direct evals | 33rank change if excluded 1 direct eval | -60rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| glm-4.5-air | -30rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 13rank change if excluded 2 direct evals | 3rank change if excluded 6 direct evals | Not shown0 direct evals | -1.04 SDDirect |
| kimi-k3 | -25rank change if excluded 2 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 90rank change if excluded 7 direct evals | 7rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | -2rank change if excluded 3 direct evals | -0.10 SDMedian-imputed |
| claude-3.5-haiku | -91rank change if excluded 2 direct evals | 19rank change if excluded 4 direct evals | 13rank change if excluded 3 direct evals | 17rank change if excluded 5 direct evals | 19rank change if excluded 1 direct eval | 19rank change if excluded 8 direct evals | 19rank change if excluded 4 direct evals | +1.99 SDDirect |
| glm-5 | -42rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | -7rank change if excluded 5 direct evals | 6rank change if excluded 2 direct evals | 6rank change if excluded 4 direct evals | -7rank change if excluded 2 direct evals | -0.60 SDDirect |
| deepseek-v4-flash | 31rank change if excluded 4 direct evals | -3rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | 1rank change if excluded 2 direct evals | -4rank change if excluded 5 direct evals | -2rank change if excluded 3 direct evals | -1.06 SDDirect |
| nemotron-3-ultra-550b-a55b | 25rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| nemotron-3-super-120b-a12b | 35rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 4 direct evals | -9rank change if excluded 2 direct evals | 8rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen2-72b-instruct | Not shown0 direct evals | 33rank change if excluded 2 direct evals | 32rank change if excluded 1 direct eval | Not shown0 direct evals | 32rank change if excluded 1 direct eval | 26rank change if excluded 9 direct evals | 33rank change if excluded 1 direct eval | -1.02 SDDirect |
| internlm2-chat-7b | Not shown0 direct evals | 2rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gpt-5.1 | 6rank change if excluded 1 direct eval | -31rank change if excluded 4 direct evals | 1rank change if excluded 5 direct evals | 5rank change if excluded 7 direct evals | 5rank change if excluded 3 direct evals | 6rank change if excluded 15 direct evals | 4rank change if excluded 7 direct evals | +1.52 SDDirect |
| qwen3-235b-a22b-fp8-throughput | Not shown0 direct evals | Not shown0 direct evals | 51rank change if excluded 1 direct eval | Not shown0 direct evals | 50rank change if excluded 1 direct eval | -44rank change if excluded 5 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| gpt-4.1 | 92rank change if excluded 2 direct evals | -19rank change if excluded 4 direct evals | -18rank change if excluded 4 direct evals | -18rank change if excluded 6 direct evals | -17rank change if excluded 4 direct evals | -20rank change if excluded 16 direct evals | -17rank change if excluded 3 direct evals | +1.03 SDDirect |
| nova-lite | 196rank change if excluded 1 direct eval | -17rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gemini-1.0-pro | 23rank change if excluded 1 direct eval | 23rank change if excluded 2 direct evals | 23rank change if excluded 3 direct evals | 23rank change if excluded 1 direct eval | 13rank change if excluded 1 direct eval | 12rank change if excluded 7 direct evals | Not shown0 direct evals | +0.74 SDDirect |
| qwen2.5-32b-instruct | Not shown0 direct evals | 19rank change if excluded 1 direct eval | Not shown0 direct evals | 24rank change if excluded 1 direct eval | 24rank change if excluded 1 direct eval | 23rank change if excluded 2 direct evals | 27rank change if excluded 2 direct evals | -1.02 SDDirect |
| internlm2-chat-20b | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -11rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| qwen-72b-chat | Not shown0 direct evals | -1rank change if excluded 2 direct evals | -2rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | 126rank change if excluded 4 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| palmyra-fin | Not shown0 direct evals | Not shown0 direct evals | -20rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 1 direct eval | 27rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| o3 | Not shown0 direct evals | -13rank change if excluded 2 direct evals | -22rank change if excluded 4 direct evals | -13rank change if excluded 5 direct evals | -12rank change if excluded 3 direct evals | 29rank change if excluded 13 direct evals | 28rank change if excluded 4 direct evals | +1.03 SDDirect |
| kimi-k2.6 | 4rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | -15rank change if excluded 9 direct evals | -11rank change if excluded 2 direct evals | -11rank change if excluded 5 direct evals | -16rank change if excluded 4 direct evals | -0.10 SDMedian-imputed |
| qwen3.6-max-preview | Not shown0 direct evals | -67rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 1 direct eval | -0.68 SDDirect |
| grok-4.5 | -16rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 1 direct eval | 45rank change if excluded 6 direct evals | 20rank change if excluded 1 direct eval | 21rank change if excluded 3 direct evals | 45rank change if excluded 3 direct evals | -0.64 SDDirect |
| gemini-3.5-flash | 3rank change if excluded 4 direct evals | 44rank change if excluded 3 direct evals | -7rank change if excluded 1 direct eval | -8rank change if excluded 9 direct evals | -7rank change if excluded 2 direct evals | -8rank change if excluded 5 direct evals | 53rank change if excluded 5 direct evals | +1.14 SDDirect |
| internlm-chat-20b | Not shown0 direct evals | 1rank change if excluded 4 direct evals | -2rank change if excluded 4 direct evals | -2rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| grok-4.20 | -8rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -10rank change if excluded 2 direct evals | -12rank change if excluded 8 direct evals | -11rank change if excluded 4 direct evals | -11rank change if excluded 6 direct evals | -15rank change if excluded 3 direct evals | -0.64 SDDirect |
| gpt-5.6-luna | 0rank change if excluded 2 direct evals | 1rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 2rank change if excluded 7 direct evals | 0rank change if excluded 2 direct evals | -5rank change if excluded 6 direct evals | -5rank change if excluded 5 direct evals | +1.58 SDDirect |
| gemini-1.5-pro | 114rank change if excluded 3 direct evals | -8rank change if excluded 3 direct evals | -2rank change if excluded 3 direct evals | -1rank change if excluded 4 direct evals | 3rank change if excluded 4 direct evals | -1rank change if excluded 13 direct evals | -1rank change if excluded 3 direct evals | +0.74 SDDirect |
| claude-sonnet-5 | -40rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 1 direct eval | 150rank change if excluded 5 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | 148rank change if excluded 3 direct evals | +2.00 SDDirect |
| palmyra-x5 | Not shown0 direct evals | Not shown0 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 1 direct eval | 16rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gpt-5.2-codex | Not shown0 direct evals | -86rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -86rank change if excluded 1 direct eval | +1.58 SDDirect |
| kimi-k2 | 5rank change if excluded 1 direct eval | 8rank change if excluded 3 direct evals | 29rank change if excluded 3 direct evals | -19rank change if excluded 9 direct evals | 8rank change if excluded 1 direct eval | 8rank change if excluded 13 direct evals | 12rank change if excluded 6 direct evals | -0.10 SDMedian-imputed |
| qwen3.5-9b | 117rank change if excluded 1 direct eval | Not shown0 direct evals | 22rank change if excluded 1 direct eval | -6rank change if excluded 1 direct eval | 22rank change if excluded 1 direct eval | Not shown0 direct evals | -10rank change if excluded 1 direct eval | -0.68 SDDirect |
| gpt-oss-120b | -21rank change if excluded 3 direct evals | -1rank change if excluded 5 direct evals | -2rank change if excluded 3 direct evals | 24rank change if excluded 9 direct evals | -3rank change if excluded 2 direct evals | -1rank change if excluded 12 direct evals | -5rank change if excluded 5 direct evals | +1.03 SDDirect |
| mistral-large | 98rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | -33rank change if excluded 1 direct eval | -45rank change if excluded 5 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| o1 | Not shown0 direct evals | 7rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | -59rank change if excluded 6 direct evals | 7rank change if excluded 3 direct evals | -24rank change if excluded 10 direct evals | 7rank change if excluded 1 direct eval | +0.44 SDDirect |
| deepseek-v3.2 | 37rank change if excluded 3 direct evals | 18rank change if excluded 6 direct evals | 13rank change if excluded 1 direct eval | -49rank change if excluded 10 direct evals | -47rank change if excluded 2 direct evals | -47rank change if excluded 9 direct evals | 2rank change if excluded 7 direct evals | -1.02 SDDirect |
| qwen3.7-max | -5rank change if excluded 1 direct eval | -19rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | 50rank change if excluded 6 direct evals | -4rank change if excluded 1 direct eval | -3rank change if excluded 3 direct evals | Not shown0 direct evals | -0.68 SDDirect |
| gpt-oss-20b | -12rank change if excluded 1 direct eval | -15rank change if excluded 3 direct evals | -13rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | -17rank change if excluded 2 direct evals | -13rank change if excluded 9 direct evals | 42rank change if excluded 3 direct evals | +1.03 SDDirect |
| doubao-seed-1.6 | Not shown0 direct evals | 11rank change if excluded 2 direct evals | 11rank change if excluded 2 direct evals | -24rank change if excluded 2 direct evals | 6rank change if excluded 1 direct eval | -13rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemini-3-pro-preview | 7rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 10rank change if excluded 2 direct evals | -8rank change if excluded 9 direct evals | 7rank change if excluded 2 direct evals | -6rank change if excluded 12 direct evals | -1rank change if excluded 4 direct evals | +0.95 SDDirect |
| qwen3-max | 9rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | 8rank change if excluded 4 direct evals | Not shown0 direct evals | 9rank change if excluded 2 direct evals | 9rank change if excluded 1 direct eval | -1.02 SDDirect |
| kimi-k2.5 | 3rank change if excluded 2 direct evals | 30rank change if excluded 4 direct evals | -2rank change if excluded 2 direct evals | 0rank change if excluded 10 direct evals | -1rank change if excluded 2 direct evals | 3rank change if excluded 8 direct evals | -11rank change if excluded 7 direct evals | -0.10 SDMedian-imputed |
| glm-5.2 | 6rank change if excluded 3 direct evals | Not shown0 direct evals | 6rank change if excluded 2 direct evals | 5rank change if excluded 7 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 3 direct evals | 7rank change if excluded 3 direct evals | -0.62 SDDirect |
| grok-3-mini | 12rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | 11rank change if excluded 4 direct evals | 11rank change if excluded 1 direct eval | 11rank change if excluded 9 direct evals | 28rank change if excluded 2 direct evals | -0.08 SDDirect |
| mimo-v2.5 | 1rank change if excluded 1 direct eval | 139rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 4 direct evals | -15rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen3.6-plus | -4rank change if excluded 1 direct eval | -69rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 4 direct evals | -4rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | -0.68 SDDirect |
| qwen3-coder-480b-a35b-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 48rank change if excluded 2 direct evals | Not shown0 direct evals | -8rank change if excluded 1 direct eval | Not shown0 direct evals | -1.02 SDDirect |
| gemini-3.5-flash-lite | -20rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 133rank change if excluded 3 direct evals | 56rank change if excluded 2 direct evals | -14rank change if excluded 2 direct evals | Not shown0 direct evals | +1.14 SDDirect |
| o1-mini | 17rank change if excluded 1 direct eval | 17rank change if excluded 2 direct evals | 45rank change if excluded 2 direct evals | -109rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | -116rank change if excluded 8 direct evals | -3rank change if excluded 1 direct eval | +0.44 SDDirect |
| gemini-1.5-flash | 57rank change if excluded 2 direct evals | 30rank change if excluded 2 direct evals | 28rank change if excluded 2 direct evals | Not shown0 direct evals | 32rank change if excluded 3 direct evals | 32rank change if excluded 11 direct evals | 30rank change if excluded 3 direct evals | +0.74 SDDirect |
| minimax-m2.7 | -8rank change if excluded 2 direct evals | 134rank change if excluded 3 direct evals | Not shown0 direct evals | -11rank change if excluded 4 direct evals | -28rank change if excluded 3 direct evals | -4rank change if excluded 4 direct evals | 32rank change if excluded 3 direct evals | -0.10 SDMedian-imputed |
| grok-4.1-fast | -24rank change if excluded 2 direct evals | -28rank change if excluded 4 direct evals | -14rank change if excluded 2 direct evals | -6rank change if excluded 8 direct evals | -15rank change if excluded 1 direct eval | -13rank change if excluded 6 direct evals | -23rank change if excluded 4 direct evals | -0.34 SDDirect |
| gpt-5.3-codex | Not shown0 direct evals | 23rank change if excluded 2 direct evals | Not shown0 direct evals | -12rank change if excluded 4 direct evals | -26rank change if excluded 1 direct eval | -26rank change if excluded 1 direct eval | -27rank change if excluded 1 direct eval | +1.52 SDDirect |
| gpt-oss-safeguard-20b | -18rank change if excluded 1 direct eval | -57rank change if excluded 2 direct evals | Not shown0 direct evals | -50rank change if excluded 1 direct eval | Not shown0 direct evals | -42rank change if excluded 2 direct evals | -42rank change if excluded 1 direct eval | +1.52 SDDirect |
| grok-4.3 | -45rank change if excluded 2 direct evals | 13rank change if excluded 3 direct evals | 16rank change if excluded 2 direct evals | 17rank change if excluded 9 direct evals | 16rank change if excluded 2 direct evals | 12rank change if excluded 7 direct evals | 13rank change if excluded 4 direct evals | -0.64 SDDirect |
| claude-3-opus | -23rank change if excluded 5 direct evals | 3rank change if excluded 4 direct evals | 6rank change if excluded 2 direct evals | 7rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | 11rank change if excluded 11 direct evals | 10rank change if excluded 6 direct evals | +1.99 SDDirect |
| qwen3-235b-a22b | 18rank change if excluded 3 direct evals | 69rank change if excluded 6 direct evals | -44rank change if excluded 4 direct evals | -43rank change if excluded 6 direct evals | 21rank change if excluded 3 direct evals | -43rank change if excluded 14 direct evals | 60rank change if excluded 7 direct evals | -1.02 SDDirect |
| gpt-4.1-mini | 26rank change if excluded 1 direct eval | -57rank change if excluded 2 direct evals | -57rank change if excluded 3 direct evals | -58rank change if excluded 4 direct evals | 76rank change if excluded 2 direct evals | -58rank change if excluded 12 direct evals | -57rank change if excluded 2 direct evals | +1.03 SDDirect |
| grok-4-fast | -14rank change if excluded 1 direct eval | Not shown0 direct evals | 16rank change if excluded 2 direct evals | -68rank change if excluded 6 direct evals | Not shown0 direct evals | -12rank change if excluded 3 direct evals | -11rank change if excluded 2 direct evals | -0.34 SDDirect |
| gpt-4 | 102rank change if excluded 1 direct eval | 105rank change if excluded 6 direct evals | 102rank change if excluded 5 direct evals | -10rank change if excluded 1 direct eval | 20rank change if excluded 1 direct eval | 20rank change if excluded 12 direct evals | 21rank change if excluded 1 direct eval | +0.44 SDDirect |
| gemini-2.5-pro | 5rank change if excluded 1 direct eval | 3rank change if excluded 6 direct evals | 3rank change if excluded 6 direct evals | 42rank change if excluded 12 direct evals | 5rank change if excluded 3 direct evals | 41rank change if excluded 15 direct evals | -37rank change if excluded 8 direct evals | +0.58 SDDirect |
| grok-4 | 13rank change if excluded 1 direct eval | 9rank change if excluded 4 direct evals | 5rank change if excluded 6 direct evals | 8rank change if excluded 10 direct evals | 4rank change if excluded 2 direct evals | 4rank change if excluded 13 direct evals | -16rank change if excluded 4 direct evals | -0.08 SDDirect |
| glm-5.1 | -8rank change if excluded 2 direct evals | -13rank change if excluded 3 direct evals | Not shown0 direct evals | -7rank change if excluded 7 direct evals | -8rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | -21rank change if excluded 6 direct evals | -0.62 SDDirect |
| deepseek-v4-pro | 183rank change if excluded 3 direct evals | -22rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | -22rank change if excluded 7 direct evals | -12rank change if excluded 1 direct eval | -7rank change if excluded 4 direct evals | -22rank change if excluded 3 direct evals | -1.06 SDDirect |
| qwen3-next-80b-a3b | Not shown0 direct evals | -10rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | -27rank change if excluded 2 direct evals | 23rank change if excluded 2 direct evals | 36rank change if excluded 8 direct evals | -10rank change if excluded 1 direct eval | -1.02 SDDirect |
| claude-3.5-sonnet | -72rank change if excluded 4 direct evals | 0rank change if excluded 4 direct evals | 2rank change if excluded 3 direct evals | 0rank change if excluded 5 direct evals | -9rank change if excluded 3 direct evals | -1rank change if excluded 15 direct evals | 7rank change if excluded 6 direct evals | +1.99 SDDirect |
| gpt-4-turbo | 0rank change if excluded 1 direct eval | -9rank change if excluded 5 direct evals | 1rank change if excluded 4 direct evals | -1rank change if excluded 3 direct evals | 1rank change if excluded 2 direct evals | 2rank change if excluded 14 direct evals | -3rank change if excluded 4 direct evals | +0.44 SDDirect |
| qwen3.7-plus | -13rank change if excluded 1 direct eval | -35rank change if excluded 1 direct eval | -14rank change if excluded 1 direct eval | -14rank change if excluded 3 direct evals | Not shown0 direct evals | -14rank change if excluded 1 direct eval | Not shown0 direct evals | -0.68 SDDirect |
| qwen-14b-chat | Not shown0 direct evals | 73rank change if excluded 5 direct evals | -80rank change if excluded 5 direct evals | 21rank change if excluded 1 direct eval | Not shown0 direct evals | 21rank change if excluded 8 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| phi-4 | Not shown0 direct evals | 36rank change if excluded 2 direct evals | 6rank change if excluded 1 direct eval | -121rank change if excluded 3 direct evals | Not shown0 direct evals | 7rank change if excluded 3 direct evals | 7rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| qwen2.5-72b-instruct | 141rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | -8rank change if excluded 3 direct evals | 7rank change if excluded 3 direct evals | 3rank change if excluded 3 direct evals | 7rank change if excluded 10 direct evals | 9rank change if excluded 2 direct evals | -1.02 SDDirect |
| gemma-4-31b-it | -46rank change if excluded 2 direct evals | 54rank change if excluded 2 direct evals | -23rank change if excluded 1 direct eval | 55rank change if excluded 4 direct evals | 53rank change if excluded 1 direct eval | 54rank change if excluded 4 direct evals | -22rank change if excluded 1 direct eval | +1.14 SDDirect |
| gemma-4-26b-a4b-it | 23rank change if excluded 1 direct eval | -45rank change if excluded 2 direct evals | Not shown0 direct evals | -38rank change if excluded 2 direct evals | Not shown0 direct evals | -45rank change if excluded 2 direct evals | -38rank change if excluded 1 direct eval | +1.14 SDDirect |
| phi-3-mini-4k-instruct | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 5rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| deepseek-v3.1 | 57rank change if excluded 2 direct evals | -56rank change if excluded 1 direct eval | 55rank change if excluded 2 direct evals | -57rank change if excluded 4 direct evals | 67rank change if excluded 1 direct eval | -58rank change if excluded 5 direct evals | -56rank change if excluded 1 direct eval | -1.45 SDDirect |
| minimax-m2 | -28rank change if excluded 1 direct eval | 19rank change if excluded 2 direct evals | 14rank change if excluded 1 direct eval | -28rank change if excluded 2 direct evals | Not shown0 direct evals | 15rank change if excluded 3 direct evals | 14rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| deepseek-v3 | -24rank change if excluded 4 direct evals | 21rank change if excluded 4 direct evals | 26rank change if excluded 7 direct evals | 16rank change if excluded 9 direct evals | 20rank change if excluded 2 direct evals | 22rank change if excluded 14 direct evals | 21rank change if excluded 4 direct evals | -1.45 SDDirect |
| gemini-2.5-flash | -5rank change if excluded 2 direct evals | -2rank change if excluded 6 direct evals | -2rank change if excluded 4 direct evals | -5rank change if excluded 10 direct evals | -4rank change if excluded 1 direct eval | -13rank change if excluded 14 direct evals | -6rank change if excluded 9 direct evals | +0.58 SDDirect |
| qwen3-30b-a3b-instruct | 10rank change if excluded 1 direct eval | 76rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | 9rank change if excluded 2 direct evals | Not shown0 direct evals | 10rank change if excluded 1 direct eval | 76rank change if excluded 1 direct eval | -1.02 SDDirect |
| glm-4.6 | -18rank change if excluded 1 direct eval | -29rank change if excluded 3 direct evals | -13rank change if excluded 2 direct evals | 24rank change if excluded 3 direct evals | Not shown0 direct evals | -10rank change if excluded 3 direct evals | -21rank change if excluded 1 direct eval | -0.60 SDDirect |
| granite-4.0-small-guardian | Not shown0 direct evals | Not shown0 direct evals | -37rank change if excluded 1 direct eval | Not shown0 direct evals | -18rank change if excluded 1 direct eval | 11rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemini-3.1-flash-lite | -39rank change if excluded 3 direct evals | -1rank change if excluded 2 direct evals | -12rank change if excluded 3 direct evals | -12rank change if excluded 8 direct evals | -12rank change if excluded 4 direct evals | 1rank change if excluded 6 direct evals | 5rank change if excluded 3 direct evals | +1.14 SDDirect |
| mistral-medium | Not shown0 direct evals | 20rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 152rank change if excluded 1 direct eval | -4rank change if excluded 2 direct evals | 20rank change if excluded 1 direct eval | -1.21 SDDirect |
| ernie-4.5-300b-a47b | Not shown0 direct evals | 9rank change if excluded 1 direct eval | -31rank change if excluded 1 direct eval | -88rank change if excluded 3 direct evals | 15rank change if excluded 1 direct eval | -31rank change if excluded 1 direct eval | -31rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gpt-4o | 9rank change if excluded 3 direct evals | 4rank change if excluded 10 direct evals | 11rank change if excluded 8 direct evals | 8rank change if excluded 14 direct evals | 9rank change if excluded 5 direct evals | 5rank change if excluded 29 direct evals | -2rank change if excluded 15 direct evals | +0.44 SDDirect |
| qwen1.5-32b-chat | Not shown0 direct evals | 11rank change if excluded 1 direct eval | 11rank change if excluded 1 direct eval | 17rank change if excluded 2 direct evals | -98rank change if excluded 1 direct eval | 17rank change if excluded 4 direct evals | 11rank change if excluded 1 direct eval | -1.02 SDDirect |
| gemini-2-flash | Not shown0 direct evals | -12rank change if excluded 1 direct eval | 78rank change if excluded 2 direct evals | 13rank change if excluded 1 direct eval | 13rank change if excluded 1 direct eval | 10rank change if excluded 6 direct evals | Not shown0 direct evals | +0.74 SDDirect |
| llama-3.3-70b-instruct | 70rank change if excluded 6 direct evals | -10rank change if excluded 5 direct evals | -8rank change if excluded 5 direct evals | -8rank change if excluded 10 direct evals | -8rank change if excluded 3 direct evals | -8rank change if excluded 12 direct evals | -3rank change if excluded 3 direct evals | -1.25 SDDirect |
| glm-4.7-flash | 208rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -0.60 SDDirect |
| qwen3-30b-a3b | -22rank change if excluded 1 direct eval | -17rank change if excluded 1 direct eval | Not shown0 direct evals | 33rank change if excluded 1 direct eval | Not shown0 direct evals | -17rank change if excluded 2 direct evals | -17rank change if excluded 1 direct eval | -1.02 SDDirect |
| gpt-3.5-turbo | 1rank change if excluded 2 direct evals | -1rank change if excluded 7 direct evals | 0rank change if excluded 9 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | -1rank change if excluded 16 direct evals | 11rank change if excluded 3 direct evals | +0.44 SDDirect |
| baichuan-2-13b-chat | Not shown0 direct evals | 18rank change if excluded 6 direct evals | 8rank change if excluded 6 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | 9rank change if excluded 12 direct evals | 6rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| deepseek-r1 | -8rank change if excluded 2 direct evals | -1rank change if excluded 7 direct evals | 1rank change if excluded 10 direct evals | -2rank change if excluded 13 direct evals | 2rank change if excluded 6 direct evals | 1rank change if excluded 21 direct evals | -1rank change if excluded 7 direct evals | -1.45 SDDirect |
| claude-2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -201rank change if excluded 1 direct eval | 4rank change if excluded 3 direct evals | Not shown0 direct evals | +1.99 SDDirect |
| glm-4.7 | 14rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 13rank change if excluded 3 direct evals | 14rank change if excluded 1 direct eval | 14rank change if excluded 3 direct evals | -45rank change if excluded 1 direct eval | -0.62 SDDirect |
| qwen1.5-72b-chat | Not shown0 direct evals | 19rank change if excluded 2 direct evals | 5rank change if excluded 3 direct evals | 14rank change if excluded 3 direct evals | 4rank change if excluded 2 direct evals | 14rank change if excluded 11 direct evals | 11rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-2-7b-chat | 2rank change if excluded 1 direct eval | -115rank change if excluded 4 direct evals | 2rank change if excluded 4 direct evals | 2rank change if excluded 1 direct eval | -119rank change if excluded 1 direct eval | 6rank change if excluded 11 direct evals | 2rank change if excluded 1 direct eval | -1.25 SDDirect |
| qwen1.5-14b-chat | Not shown0 direct evals | -4rank change if excluded 3 direct evals | 23rank change if excluded 2 direct evals | 9rank change if excluded 3 direct evals | Not shown0 direct evals | 22rank change if excluded 6 direct evals | -2rank change if excluded 2 direct evals | -1.02 SDDirect |
| o4-mini | Not shown0 direct evals | 2rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | -4rank change if excluded 6 direct evals | 2rank change if excluded 3 direct evals | 5rank change if excluded 14 direct evals | 11rank change if excluded 4 direct evals | +1.03 SDDirect |
| qwen3-32b | 212rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 8rank change if excluded 1 direct eval | 7rank change if excluded 4 direct evals | 9rank change if excluded 1 direct eval | -1.02 SDDirect |
| minimax-m2.5 | -20rank change if excluded 1 direct eval | -5rank change if excluded 2 direct evals | -5rank change if excluded 1 direct eval | -5rank change if excluded 5 direct evals | -9rank change if excluded 1 direct eval | -5rank change if excluded 3 direct evals | 72rank change if excluded 2 direct evals | -0.10 SDMedian-imputed |
| nova-premier | -133rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | Not shown0 direct evals | -9rank change if excluded 1 direct eval | Not shown0 direct evals | 12rank change if excluded 2 direct evals | -9rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gemini-3-flash-preview | -12rank change if excluded 2 direct evals | -6rank change if excluded 3 direct evals | Not shown0 direct evals | -17rank change if excluded 8 direct evals | -14rank change if excluded 1 direct eval | -5rank change if excluded 4 direct evals | -18rank change if excluded 4 direct evals | +0.95 SDDirect |
| hy3 | -206rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| internlm-chat-7b | Not shown0 direct evals | -144rank change if excluded 5 direct evals | 1rank change if excluded 5 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -144rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| deepseek-v3.2-exp | -108rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | -1.02 SDDirect |
| yi-34b-chat | Not shown0 direct evals | 14rank change if excluded 4 direct evals | 14rank change if excluded 4 direct evals | 14rank change if excluded 3 direct evals | Not shown0 direct evals | -43rank change if excluded 8 direct evals | 14rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| text-davinci-003 | Not shown0 direct evals | 13rank change if excluded 2 direct evals | 10rank change if excluded 2 direct evals | 12rank change if excluded 1 direct eval | Not shown0 direct evals | 12rank change if excluded 3 direct evals | Not shown0 direct evals | +0.44 SDDirect |
| glm-4-9b-chat | Not shown0 direct evals | 35rank change if excluded 2 direct evals | -10rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -10rank change if excluded 7 direct evals | -29rank change if excluded 2 direct evals | -0.10 SDDirect |
| gemini-2-pro-preview | Not shown0 direct evals | Not shown0 direct evals | 56rank change if excluded 1 direct eval | Not shown0 direct evals | -14rank change if excluded 1 direct eval | -108rank change if excluded 5 direct evals | Not shown0 direct evals | +0.74 SDDirect |
| magistral-medium | Not shown0 direct evals | -23rank change if excluded 2 direct evals | -26rank change if excluded 2 direct evals | -23rank change if excluded 4 direct evals | 12rank change if excluded 1 direct eval | -24rank change if excluded 4 direct evals | -24rank change if excluded 1 direct eval | -1.21 SDDirect |
| command-r-plus | 12rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | 11rank change if excluded 2 direct evals | 12rank change if excluded 1 direct eval | 12rank change if excluded 1 direct eval | -21rank change if excluded 7 direct evals | 38rank change if excluded 2 direct evals | -0.10 SDMedian-imputed |
| nemotron-3-nano-30b-a3b | 171rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -22rank change if excluded 3 direct evals | -22rank change if excluded 1 direct eval | -22rank change if excluded 2 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| mimo-v2-flash | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | 9rank change if excluded 4 direct evals | -36rank change if excluded 1 direct eval | 6rank change if excluded 2 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemma-2b-it | -14rank change if excluded 1 direct eval | -137rank change if excluded 3 direct evals | -14rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 4 direct evals | -3rank change if excluded 1 direct eval | +0.74 SDDirect |
| o3-mini | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 9rank change if excluded 3 direct evals | 4rank change if excluded 7 direct evals | 7rank change if excluded 2 direct evals | 6rank change if excluded 13 direct evals | 2rank change if excluded 4 direct evals | +0.44 SDDirect |
| granite-4.0-small | Not shown0 direct evals | Not shown0 direct evals | -35rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 9rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| claude-3-haiku | 36rank change if excluded 1 direct eval | 36rank change if excluded 2 direct evals | 35rank change if excluded 2 direct evals | -20rank change if excluded 3 direct evals | -18rank change if excluded 2 direct evals | 18rank change if excluded 10 direct evals | 14rank change if excluded 3 direct evals | +1.99 SDDirect |
| gemini-2.0-flash | -1rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | Not shown0 direct evals | 8rank change if excluded 5 direct evals | -28rank change if excluded 3 direct evals | +0.74 SDDirect |
| grok-3 | -16rank change if excluded 2 direct evals | -4rank change if excluded 2 direct evals | -7rank change if excluded 2 direct evals | -3rank change if excluded 4 direct evals | -4rank change if excluded 1 direct eval | -1rank change if excluded 11 direct evals | 0rank change if excluded 2 direct evals | -0.08 SDDirect |
| llama-2-13b-chat | Not shown0 direct evals | 2rank change if excluded 3 direct evals | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -138rank change if excluded 1 direct eval | 3rank change if excluded 8 direct evals | 2rank change if excluded 1 direct eval | -1.25 SDDirect |
| llama-4-maverick | 3rank change if excluded 2 direct evals | -5rank change if excluded 8 direct evals | -4rank change if excluded 4 direct evals | -2rank change if excluded 10 direct evals | -2rank change if excluded 2 direct evals | -4rank change if excluded 17 direct evals | -2rank change if excluded 5 direct evals | -0.45 SDDirect |
| llama-3.1-405b-instruct | Not shown0 direct evals | -4rank change if excluded 2 direct evals | 10rank change if excluded 3 direct evals | 12rank change if excluded 7 direct evals | 10rank change if excluded 3 direct evals | 14rank change if excluded 16 direct evals | -4rank change if excluded 3 direct evals | -1.25 SDDirect |
| gemma-7b-it | -59rank change if excluded 1 direct eval | -102rank change if excluded 4 direct evals | -59rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | 7rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | -1rank change if excluded 1 direct eval | +0.74 SDDirect |
| gpt-5.4-mini | Not shown0 direct evals | 20rank change if excluded 1 direct eval | -135rank change if excluded 2 direct evals | 19rank change if excluded 7 direct evals | -136rank change if excluded 3 direct evals | 20rank change if excluded 4 direct evals | 19rank change if excluded 2 direct evals | +1.58 SDDirect |
| qwen1.5-7b-chat | Not shown0 direct evals | -4rank change if excluded 2 direct evals | -4rank change if excluded 2 direct evals | -4rank change if excluded 3 direct evals | 9rank change if excluded 1 direct eval | 2rank change if excluded 9 direct evals | 0rank change if excluded 1 direct eval | -1.02 SDDirect |
| glm-4.5 | 6rank change if excluded 1 direct eval | 8rank change if excluded 2 direct evals | -21rank change if excluded 2 direct evals | 4rank change if excluded 4 direct evals | 2rank change if excluded 2 direct evals | 5rank change if excluded 3 direct evals | 5rank change if excluded 1 direct eval | -1.04 SDDirect |
| mistral-small-3.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -16rank change if excluded 2 direct evals | Not shown0 direct evals | 18rank change if excluded 3 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| yi-6b-chat | Not shown0 direct evals | 7rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | 1rank change if excluded 3 direct evals | Not shown0 direct evals | 1rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gemma-2-9b-it | -11rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | -11rank change if excluded 3 direct evals | 3rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 3rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | +0.74 SDDirect |
| gpt-4.1-nano | Not shown0 direct evals | -24rank change if excluded 1 direct eval | -54rank change if excluded 2 direct evals | -8rank change if excluded 3 direct evals | -7rank change if excluded 1 direct eval | 73rank change if excluded 9 direct evals | -8rank change if excluded 1 direct eval | +1.03 SDDirect |
| qwen3.5-35b-a3b | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -11rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 1 direct eval | 28rank change if excluded 2 direct evals | -0.68 SDDirect |
| hy3-preview | 98rank change if excluded 1 direct eval | -81rank change if excluded 1 direct eval | Not shown0 direct evals | -68rank change if excluded 3 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| minimax-m2.1 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -10rank change if excluded 3 direct evals | -4rank change if excluded 1 direct eval | -19rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| llama-3.2-instruct-11b-vision | Not shown0 direct evals | -3rank change if excluded 1 direct eval | Not shown0 direct evals | -36rank change if excluded 1 direct eval | Not shown0 direct evals | 87rank change if excluded 2 direct evals | 58rank change if excluded 1 direct eval | +0.33 SDDirect |
| gemini-2-flash-lite-preview | Not shown0 direct evals | Not shown0 direct evals | 21rank change if excluded 1 direct eval | Not shown0 direct evals | -12rank change if excluded 1 direct eval | -2rank change if excluded 5 direct evals | Not shown0 direct evals | +0.74 SDDirect |
| granite-4.0-micro-guardian | Not shown0 direct evals | Not shown0 direct evals | -61rank change if excluded 1 direct eval | Not shown0 direct evals | -15rank change if excluded 1 direct eval | 85rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| glm-5-turbo | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | 93rank change if excluded 3 direct evals | -2rank change if excluded 1 direct eval | 0rank change if excluded 2 direct evals | 0rank change if excluded 1 direct eval | -0.62 SDDirect |
| gemma-2-27b-it | -4rank change if excluded 1 direct eval | -2rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | -1rank change if excluded 1 direct eval | +0.74 SDDirect |
| llama-3-8b-instruct | 0rank change if excluded 1 direct eval | 0rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 0rank change if excluded 12 direct evals | -1rank change if excluded 1 direct eval | -1.25 SDDirect |
| deepseek-llm-7b-chat | Not shown0 direct evals | 61rank change if excluded 1 direct eval | -12rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 9rank change if excluded 3 direct evals | Not shown0 direct evals | -1.45 SDDirect |
| llama-3-70b-instruct | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 11rank change if excluded 2 direct evals | -4rank change if excluded 1 direct eval | 16rank change if excluded 2 direct evals | -3rank change if excluded 9 direct evals | -47rank change if excluded 2 direct evals | -1.25 SDDirect |
| command-r | Not shown0 direct evals | Not shown0 direct evals | -20rank change if excluded 1 direct eval | Not shown0 direct evals | 12rank change if excluded 1 direct eval | 0rank change if excluded 6 direct evals | 121rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| llama-3.1-70b-instruct | -1rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 0rank change if excluded 3 direct evals | 0rank change if excluded 2 direct evals | 0rank change if excluded 4 direct evals | -1rank change if excluded 11 direct evals | 1rank change if excluded 3 direct evals | -1.25 SDDirect |
| olmo-2-0325-32b-instruct | Not shown0 direct evals | -3rank change if excluded 1 direct eval | -22rank change if excluded 2 direct evals | -3rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 2rank change if excluded 5 direct evals | -9rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| vicuna-13b | Not shown0 direct evals | -2rank change if excluded 2 direct evals | 15rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 21rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen-7b-chat | Not shown0 direct evals | 13rank change if excluded 6 direct evals | 5rank change if excluded 6 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -8rank change if excluded 9 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| baichuan-2-7b-chat | Not shown0 direct evals | 9rank change if excluded 5 direct evals | 3rank change if excluded 5 direct evals | 3rank change if excluded 2 direct evals | Not shown0 direct evals | -3rank change if excluded 11 direct evals | 3rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| llama-2-70b-chat | Not shown0 direct evals | -7rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -178rank change if excluded 2 direct evals | -5rank change if excluded 7 direct evals | -7rank change if excluded 1 direct eval | -1.25 SDDirect |
| phi-3.5-mini-instruct | -39rank change if excluded 1 direct eval | -11rank change if excluded 4 direct evals | -39rank change if excluded 2 direct evals | -14rank change if excluded 1 direct eval | Not shown0 direct evals | 28rank change if excluded 4 direct evals | -8rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gemini-2.5-flash-lite | 6rank change if excluded 3 direct evals | 5rank change if excluded 1 direct eval | 5rank change if excluded 2 direct evals | 3rank change if excluded 5 direct evals | 5rank change if excluded 1 direct eval | -67rank change if excluded 8 direct evals | 5rank change if excluded 3 direct evals | +0.58 SDDirect |
| phi-2 | Not shown0 direct evals | 87rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 3 direct evals | -21rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| claude-3-sonnet | 1rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | -2rank change if excluded 2 direct evals | 2rank change if excluded 9 direct evals | -18rank change if excluded 3 direct evals | +1.99 SDDirect |
| gpt-5.4-nano | Not shown0 direct evals | Not shown0 direct evals | -148rank change if excluded 2 direct evals | -10rank change if excluded 6 direct evals | -109rank change if excluded 2 direct evals | -10rank change if excluded 3 direct evals | -11rank change if excluded 2 direct evals | +1.58 SDDirect |
| qwen-2.5-7b-instruct | 3rank change if excluded 1 direct eval | -1rank change if excluded 2 direct evals | 2rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | 2rank change if excluded 1 direct eval | 3rank change if excluded 12 direct evals | 0rank change if excluded 3 direct evals | -1.02 SDDirect |
| chatglm-6b | Not shown0 direct evals | 11rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | 11rank change if excluded 1 direct eval | Not shown0 direct evals | 11rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDDirect |
| nvidia-nemotron-nano-9b-v2 | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 5rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen-2.5-14b-instruct | Not shown0 direct evals | -2rank change if excluded 2 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 5 direct evals | 3rank change if excluded 2 direct evals | -1.02 SDDirect |
| chatglm3-6b | Not shown0 direct evals | -1rank change if excluded 6 direct evals | 2rank change if excluded 6 direct evals | 0rank change if excluded 2 direct evals | Not shown0 direct evals | -1rank change if excluded 9 direct evals | Not shown0 direct evals | -0.10 SDDirect |
| step-3.5-flash | -91rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -10rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | Not shown0 direct evals | -10rank change if excluded 1 direct eval | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemini-2.0-flash-lite | -51rank change if excluded 1 direct eval | Not shown0 direct evals | -5rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | Not shown0 direct evals | 25rank change if excluded 3 direct evals | Not shown0 direct evals | +0.74 SDDirect |
| mistral-7b-instruct | 1rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -1rank change if excluded 6 direct evals | -2rank change if excluded 3 direct evals | 49rank change if excluded 3 direct evals | -10rank change if excluded 16 direct evals | -2rank change if excluded 1 direct eval | -1.21 SDDirect |
| yi-1.5-34b-chat | Not shown0 direct evals | -8rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| mistral-small-3.2 | Not shown0 direct evals | -7rank change if excluded 1 direct eval | 75rank change if excluded 1 direct eval | -1rank change if excluded 3 direct evals | Not shown0 direct evals | -7rank change if excluded 2 direct evals | -7rank change if excluded 1 direct eval | -1.21 SDDirect |
| vicuna-7b-v1.5 | 18rank change if excluded 1 direct eval | 18rank change if excluded 1 direct eval | 18rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| mistral-medium-3.1 | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | -5rank change if excluded 3 direct evals | Not shown0 direct evals | -7rank change if excluded 2 direct evals | -7rank change if excluded 2 direct evals | -1.21 SDDirect |
| nova-micro | -92rank change if excluded 1 direct eval | -54rank change if excluded 1 direct eval | Not shown0 direct evals | 18rank change if excluded 1 direct eval | Not shown0 direct evals | -54rank change if excluded 2 direct evals | 18rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| gpt-4o-mini | 6rank change if excluded 3 direct evals | -2rank change if excluded 3 direct evals | 4rank change if excluded 4 direct evals | 0rank change if excluded 5 direct evals | 2rank change if excluded 3 direct evals | 2rank change if excluded 15 direct evals | -32rank change if excluded 5 direct evals | +0.44 SDDirect |
| granite-3.3-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -121rank change if excluded 1 direct eval | -60rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 10rank change if excluded 6 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemma-3-27b-it | Not shown0 direct evals | -8rank change if excluded 2 direct evals | 11rank change if excluded 3 direct evals | -4rank change if excluded 6 direct evals | 11rank change if excluded 1 direct eval | -3rank change if excluded 6 direct evals | -7rank change if excluded 1 direct eval | +0.74 SDDirect |
| mistral-nemo | 46rank change if excluded 2 direct evals | 47rank change if excluded 2 direct evals | 50rank change if excluded 1 direct eval | -5rank change if excluded 1 direct eval | Not shown0 direct evals | -7rank change if excluded 2 direct evals | Not shown0 direct evals | -1.21 SDDirect |
| mistral-medium-3 | -59rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -89rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 15rank change if excluded 1 direct eval | -1.21 SDDirect |
| llama-4-scout | -2rank change if excluded 1 direct eval | -12rank change if excluded 3 direct evals | -13rank change if excluded 2 direct evals | -12rank change if excluded 4 direct evals | -4rank change if excluded 1 direct eval | -30rank change if excluded 10 direct evals | -20rank change if excluded 2 direct evals | -0.45 SDDirect |
| mistral-small-4 | Not shown0 direct evals | -57rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 4 direct evals | 7rank change if excluded 1 direct eval | 4rank change if excluded 1 direct eval | Not shown0 direct evals | -1.21 SDDirect |
| phi-4-mini | Not shown0 direct evals | -100rank change if excluded 1 direct eval | Not shown0 direct evals | 1rank change if excluded 1 direct eval | Not shown0 direct evals | -122rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| llama-3.1-8b-instruct | -3rank change if excluded 1 direct eval | -5rank change if excluded 3 direct evals | 2rank change if excluded 6 direct evals | 2rank change if excluded 6 direct evals | 4rank change if excluded 3 direct evals | -3rank change if excluded 15 direct evals | 15rank change if excluded 5 direct evals | -1.25 SDDirect |
| gemma-2-2b-it | -44rank change if excluded 1 direct eval | -44rank change if excluded 2 direct evals | -44rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 4rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | +0.74 SDDirect |
| qwen2.5-max | Not shown0 direct evals | -11rank change if excluded 1 direct eval | -13rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -11rank change if excluded 1 direct eval | -11rank change if excluded 3 direct evals | -11rank change if excluded 1 direct eval | -1.02 SDDirect |
| vicuna-13b-v1.5 | -17rank change if excluded 1 direct eval | -17rank change if excluded 1 direct eval | -17rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 24rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| palmyra-med | Not shown0 direct evals | Not shown0 direct evals | -23rank change if excluded 1 direct eval | Not shown0 direct evals | 45rank change if excluded 1 direct eval | 51rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen-2.5-1.5b-instruct | Not shown0 direct evals | -11rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 1rank change if excluded 5 direct evals | 21rank change if excluded 1 direct eval | -1.02 SDDirect |
| chatglm2-6b | Not shown0 direct evals | 4rank change if excluded 4 direct evals | -6rank change if excluded 4 direct evals | Not shown0 direct evals | Not shown0 direct evals | -5rank change if excluded 6 direct evals | Not shown0 direct evals | -0.10 SDDirect |
| qwen2.5-3b-instruct | Not shown0 direct evals | -126rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | -6rank change if excluded 1 direct eval | -1.02 SDDirect |
| qwen1.5-0.5b-chat | Not shown0 direct evals | 2rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | 21rank change if excluded 1 direct eval | Not shown0 direct evals | 22rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| llama-3.2-90b-vision-instruct | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 1 direct eval | Not shown0 direct evals | -3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -1.25 SDDirect |
| nova-pro | -43rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 30rank change if excluded 1 direct eval | 31rank change if excluded 3 direct evals | 3rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| qwen2-7b-instruct | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | -7rank change if excluded 4 direct evals | 5rank change if excluded 1 direct eval | -1.02 SDDirect |
| olmo-2-1124-13b-instruct | Not shown0 direct evals | 23rank change if excluded 1 direct eval | -3rank change if excluded 2 direct evals | -2rank change if excluded 1 direct eval | -2rank change if excluded 1 direct eval | -6rank change if excluded 5 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| ling-2.6-flash | 3rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| zephyr-7b-beta | Not shown0 direct evals | -19rank change if excluded 1 direct eval | -19rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | -149rank change if excluded 5 direct evals | -19rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| granite-4.0-micro | Not shown0 direct evals | 18rank change if excluded 1 direct eval | -4rank change if excluded 1 direct eval | 36rank change if excluded 1 direct eval | -7rank change if excluded 1 direct eval | 21rank change if excluded 6 direct evals | -3rank change if excluded 2 direct evals | -0.10 SDMedian-imputed |
| mistral-medium-3.5 | Not shown0 direct evals | -134rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | 3rank change if excluded 3 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -1.21 SDDirect |
| vicuna-7b | Not shown0 direct evals | 1rank change if excluded 2 direct evals | 1rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 3rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| mixtral-8x22b-instruct | Not shown0 direct evals | -4rank change if excluded 2 direct evals | 23rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 2 direct evals | -97rank change if excluded 8 direct evals | -4rank change if excluded 1 direct eval | -1.21 SDDirect |
| qwen3-8b | Not shown0 direct evals | -10rank change if excluded 1 direct eval | 1rank change if excluded 2 direct evals | -10rank change if excluded 1 direct eval | -11rank change if excluded 1 direct eval | -10rank change if excluded 4 direct evals | -10rank change if excluded 2 direct evals | -1.02 SDDirect |
| mistral-small-3 | Not shown0 direct evals | -8rank change if excluded 1 direct eval | 70rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | -25rank change if excluded 6 direct evals | -8rank change if excluded 1 direct eval | -1.21 SDDirect |
| trinity-large | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 3rank change if excluded 4 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen3-14b | Not shown0 direct evals | -8rank change if excluded 1 direct eval | -62rank change if excluded 1 direct eval | -8rank change if excluded 1 direct eval | -62rank change if excluded 1 direct eval | -8rank change if excluded 2 direct evals | -8rank change if excluded 1 direct eval | -1.02 SDDirect |
| moss-moon-003-sft | Not shown0 direct evals | 6rank change if excluded 2 direct evals | 6rank change if excluded 2 direct evals | Not shown0 direct evals | Not shown0 direct evals | 2rank change if excluded 3 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| gemma-3-4b | Not shown0 direct evals | 5rank change if excluded 1 direct eval | Not shown0 direct evals | -31rank change if excluded 1 direct eval | Not shown0 direct evals | 5rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | +1.14 SDDirect |
| command-a | Not shown0 direct evals | -5rank change if excluded 2 direct evals | -6rank change if excluded 2 direct evals | -14rank change if excluded 4 direct evals | Not shown0 direct evals | -17rank change if excluded 5 direct evals | -14rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| vicuna-33b-v1.3 | Not shown0 direct evals | 19rank change if excluded 2 direct evals | 0rank change if excluded 2 direct evals | -33rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| davinci | Not shown0 direct evals | 16rank change if excluded 1 direct eval | 16rank change if excluded 1 direct eval | 16rank change if excluded 2 direct evals | Not shown0 direct evals | -5rank change if excluded 2 direct evals | Not shown0 direct evals | +0.44 SDDirect |
| mistral-large-3 | Not shown0 direct evals | -4rank change if excluded 3 direct evals | -4rank change if excluded 2 direct evals | -1rank change if excluded 7 direct evals | -4rank change if excluded 1 direct eval | -2rank change if excluded 6 direct evals | -4rank change if excluded 1 direct eval | -1.21 SDDirect |
| gemma-3-12b | Not shown0 direct evals | 4rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | -6rank change if excluded 2 direct evals | Not shown0 direct evals | 4rank change if excluded 3 direct evals | 2rank change if excluded 1 direct eval | +1.14 SDDirect |
| deepseek-v3.2-speciale | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 10rank change if excluded 1 direct eval | Not shown0 direct evals | 4rank change if excluded 1 direct eval | -3rank change if excluded 1 direct eval | -1.02 SDDirect |
| llama-3.2-3b-instruct | 2rank change if excluded 1 direct eval | 2rank change if excluded 2 direct evals | 2rank change if excluded 1 direct eval | 10rank change if excluded 1 direct eval | Not shown0 direct evals | -26rank change if excluded 5 direct evals | 2rank change if excluded 2 direct evals | -1.25 SDDirect |
| deepseek-llm-67b-chat | Not shown0 direct evals | -55rank change if excluded 1 direct eval | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -53rank change if excluded 1 direct eval | -66rank change if excluded 7 direct evals | -55rank change if excluded 1 direct eval | -1.45 SDDirect |
| magistral-small | Not shown0 direct evals | 6rank change if excluded 1 direct eval | 6rank change if excluded 1 direct eval | 18rank change if excluded 2 direct evals | Not shown0 direct evals | -77rank change if excluded 2 direct evals | -8rank change if excluded 1 direct eval | -1.21 SDDirect |
| llama-3.2-1b-instruct | 3rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | -56rank change if excluded 1 direct eval | Not shown0 direct evals | -57rank change if excluded 5 direct evals | 2rank change if excluded 1 direct eval | -1.25 SDDirect |
| marin-8b-instruct | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwq-32b | Not shown0 direct evals | -77rank change if excluded 3 direct evals | -6rank change if excluded 2 direct evals | 15rank change if excluded 2 direct evals | Not shown0 direct evals | -6rank change if excluded 3 direct evals | -6rank change if excluded 1 direct eval | -1.02 SDDirect |
| jamba-1-7-large | Not shown0 direct evals | 11rank change if excluded 1 direct eval | Not shown0 direct evals | -34rank change if excluded 2 direct evals | Not shown0 direct evals | 3rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| qwen-1.8b-chat | Not shown0 direct evals | -1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 3 direct evals | Not shown0 direct evals | -1.02 SDDirect |
| olmo-2-1124-7b-instruct | Not shown0 direct evals | 1rank change if excluded 1 direct eval | -1rank change if excluded 1 direct eval | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 1rank change if excluded 5 direct evals | 1rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| grok-2 | Not shown0 direct evals | Not shown0 direct evals | -88rank change if excluded 1 direct eval | 6rank change if excluded 3 direct evals | Not shown0 direct evals | -77rank change if excluded 2 direct evals | Not shown0 direct evals | -1.25 SDDirect |
| mixtral-8x7b-instruct | Not shown0 direct evals | -2rank change if excluded 2 direct evals | -6rank change if excluded 2 direct evals | -20rank change if excluded 1 direct eval | -21rank change if excluded 1 direct eval | -54rank change if excluded 9 direct evals | -21rank change if excluded 1 direct eval | -1.21 SDDirect |
| dbrx-instruct | Not shown0 direct evals | -1rank change if excluded 1 direct eval | 1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | -3rank change if excluded 6 direct evals | 0rank change if excluded 1 direct eval | -0.10 SDMedian-imputed |
| mistral-large-2 | Not shown0 direct evals | 2rank change if excluded 2 direct evals | -6rank change if excluded 3 direct evals | -5rank change if excluded 6 direct evals | Not shown0 direct evals | -6rank change if excluded 7 direct evals | -6rank change if excluded 1 direct eval | -1.21 SDDirect |
| olmoe-1b-7b-0125-instruct | Not shown0 direct evals | Not shown0 direct evals | -1rank change if excluded 1 direct eval | Not shown0 direct evals | 0rank change if excluded 1 direct eval | 0rank change if excluded 4 direct evals | Not shown0 direct evals | -0.10 SDMedian-imputed |
| qwen1.5-1.8b-chat | Not shown0 direct evals | -10rank change if excluded 1 direct eval | 3rank change if excluded 1 direct eval | -10rank change if excluded 2 direct evals | Not shown0 direct evals | 2rank change if excluded 4 direct evals | -10rank change if excluded 1 direct eval | -1.02 SDDirect |
| deepseek-r1-distill-llama-70b | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 6rank change if excluded 2 direct evals | 4rank change if excluded 1 direct eval | 4rank change if excluded 2 direct evals | Not shown0 direct evals | -1.45 SDDirect |
| qwen-2.5-0.5b-instruct | Not shown0 direct evals | 0rank change if excluded 1 direct eval | Not shown0 direct evals | Not shown0 direct evals | Not shown0 direct evals | 0rank change if excluded 4 direct evals | 0rank change if excluded 1 direct eval | -1.02 SDDirect |
| magistral-small-2509 | Not shown0 direct evals | 3rank change if excluded 1 direct eval | Not shown0 direct evals | 9rank change if excluded 1 direct eval | Not shown0 direct evals | -4rank change if excluded 2 direct evals | 3rank change if excluded 1 direct eval | -1.21 SDDirect |
