Evaluation profile
BullshitBench v2
1sub-evals
2.2%total index weight
1components
Within-component eval weight: Truthfulness 14.7%.
Model score (higher is better)Predicted score
About this eval
Detection and explanation of plausible-sounding nonsense without refusal.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| clear_pushback_ratebullshitbench-v2/bullshitbench-v2.csv:clear_pushback_rateMeasures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true. | truthfulness_honesty:1.000bullshitbench-v2 | Higher is better | 2.2% | Truthfulness 14.7% |
clear_pushback_rate
Measures whether the model identifies plausible-sounding nonsense and clearly explains that it is unsupported instead of answering as if it were true.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-opus-4.8 | 0.945 | official | |
| 2 | claude-sonnet-4.6 | 0.9 | official | |
| 3 | claude-opus-4.6 | 0.85 | official | |
| 4 | claude-opus-4.5 | 0.845 | official | |
| 5 | claude-sonnet-5 | 0.79 | official | |
| 6 | claude-opus-4.7 | 0.785 | official | |
| 7 | claude-sonnet-4.5 | 0.765 | official | |
| 8 | claude-haiku-4.5 | 0.74 | official | |
| 9 | qwen3.5-397b-a17b | 0.735 | official | |
| 10 | kimi-k3 | 0.72 | official | |
| 11 | claude-opus-5 | 0.715 | official | |
| 12 | grok-4.20-multi-agent | 0.655 | official | |
| 12 | qwen3.6-plus | 0.655 | official | |
| 14 | qwen3.7-max | 0.635 | official | |
| 15 | minimax-m3 | 0.625 | official | |
| 16 | gemini-3.5-flash-lite | 0.62 | official | |
| 17 | kimi-k2.6 | 0.575 | official | |
| 18 | grok-4.20 | 0.55 | official | |
| 19 | qwen3-max | 0.545 | official | |
| 20 | grok-4.5 | 0.54 | official | |
| 21 | claude-3.5-haiku | 0.5 | official | |
| 21 | mimo-v2.5-pro | 0.5 | official | |
| 23 | gpt-5.6-terra | 0.495 | official | |
| 24 | claude-fable-5 | 0.49 | official | |
| 25 | grok-4.3 | 0.48 | official | |
| 26 | gpt-5.6-sol | 0.465 | official | |
| 27 | claude-3.7-sonnet | 0.46 | official | |
| 28 | gpt-5.5 | 0.4567 | official | |
| 29 | claude-3.5-sonnet | 0.45 | official | |
| 29 | gpt-5.4 | 0.45 | official | |
| 31 | nemotron-3-ultra-550b-a55b | 0.445 | official | |
| 32 | nemotron-3-super-120b-a12b | 0.4375 | official | |
| 33 | claude-opus-4.1 | 0.425 | official | |
| 34 | gemini-3-pro-preview | 0.42 | official | |
| 35 | kimi-k2.5 | 0.415 | official | |
| 36 | gpt-5.2-codex | 0.4033 | official | |
| 37 | gpt-5.3-chat | 0.4 | official | |
| 38 | gpt-5-codex | 0.39 | official | |
| 38 | hunter-alpha | 0.39 | official | |
| 40 | gpt-5.6-luna | 0.38 | official | |
| 41 | gpt-5.1 | 0.35 | official | |
| 41 | gpt-5.5-pro | 0.35 | official | |
| 43 | claude-opus-4 | 0.34 | official | |
| 43 | gemini-3.1-pro-preview | 0.34 | official | |
| 43 | gpt-5.5-instant | 0.34 | official | |
| 46 | gemini-3.6-flash | 0.335 | official | |
| 47 | gpt-5.2 | 0.33 | official | |
| 48 | gpt-5.1-codex | 0.32 | official | |
| 49 | gpt-5.4-mini | 0.3167 | official | |
| 50 | healer-alpha | 0.315 | official | |
| 51 | claude-sonnet-4 | 0.295 | official | |
| 51 | mimo-v2.5 | 0.295 | official | |
| 53 | llama-4-maverick | 0.28 | official | |
| 54 | glm-5-turbo | 0.27 | official | |
| 54 | gpt-5.2-chat | 0.27 | official | |
| 56 | o3 | 0.26 | official | |
| 57 | glm-5 | 0.24 | official | |
| 57 | glm-5.2 | 0.24 | official | |
| 59 | gemma-4-31b-it | 0.225 | official | |
| 60 | gpt-5.3-codex | 0.21 | official | |
| 60 | laguna-s-2.1-poolside | 0.21 | official | |
| 62 | gemini-2.5-pro | 0.2 | official | |
| 62 | glm-5.1 | 0.2 | official | |
| 62 | qwen3-coder-480b-a35b-instruct | 0.2 | official | |
| 65 | gemini-3.5-flash | 0.195 | official | |
| 65 | gpt-5 | 0.195 | official | |
| 67 | gemini-2.5-flash | 0.19 | official | |
| 67 | llama-4-scout | 0.19 | official | |
| 67 | nemotron-3-nano-30b-a3b | 0.19 | official | |
| 70 | step-3.5-flash | 0.175 | official | |
| 71 | deepseek-v4-flash | 0.16 | official | |
| 71 | gemma-4-26b-a4b-it | 0.16 | official | |
| 73 | gemini-2.0-flash | 0.15 | official | |
| 73 | trinity-large | 0.15 | official | |
| 75 | grok-4.1-fast | 0.145 | official | |
| 75 | mimo-v2-flash | 0.145 | official | |
| 77 | deepseek-v4-pro | 0.14 | official | |
| 77 | gpt-4.1 | 0.14 | official | |
| 77 | llama-3.1-8b-instruct | 0.14 | official | |
| 80 | gpt-5.4-nano | 0.1233 | official | |
| 81 | gpt-4o | 0.12 | official | |
| 81 | hy3-preview | 0.12 | official | |
| 83 | deepseek-v3.2 | 0.115 | official | |
| 84 | gemini-3.1-flash-lite | 0.11 | official | |
| 85 | claude-3-haiku | 0.1 | official | |
| 85 | gemini-3-flash-preview | 0.1 | official | |
| 85 | kimi-k2 | 0.1 | official | |
| 88 | doubao-seed-1.6 | 0.09 | official | |
| 89 | minimax-m2.5 | 0.085 | official | |
| 90 | gpt-oss-120b | 0.08 | official | |
| 91 | deepseek-r1 | 0.075 | official | |
| 91 | minimax-m2.7 | 0.075 | official | |
| 93 | ernie-4.5-vl-424b-a47b | 0.07 | official | |
| 93 | glm-4.5 | 0.07 | official | |
| 95 | intellect-3 | 0.06 | official | |
| 95 | o4-mini | 0.06 | official | |
| 97 | jamba-1-7-large | 0.05 | official | |
| 97 | mistral-small-4 | 0.05 | official | |
| 99 | qwen3-235b-a22b | 0.045 | official | |
| 100 | deepseek-v3 | 0.04 | official | |
| 100 | ernie-4.5-300b-a47b | 0.04 | official | |
| 100 | nvidia-nemotron-nano-9b-v2 | 0.04 | official | |
| 103 | gemma-3-27b-it | 0.03 | official | |
| 104 | gpt-4o-mini | 0.02 | official | |
| 104 | mistral-large-3 | 0.02 | official |