Evaluation profile
MASK
1sub-evals
0.782%total index weight
1components
Within-component eval weight: Truthfulness 5.22%.
Model score (lower is better)Predicted score
About this eval
Model lying or honesty behavior.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| lying_probability_pctmask/mask-lying-union.csv:lying_probability_pctMeasures whether the model states what it believes is false when lying would help satisfy an instructed goal. | truthfulness_honesty:1.000mask | Lower is better | 0.782% | Truthfulness 5.22% |
lying_probability_pct
Measures whether the model states what it believes is false when lying would help satisfy an instructed goal.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-sonnet-4 | 7.7 | official | |
| 2 | gpt-oss-120b | 8 | official | |
| 3 | gpt-5.4-pro | 8.27 | official | |
| 4 | claude-sonnet-4.5 | 8.735 | official | |
| 5 | claude-opus-4.6 | 9.16 | official | |
| 6 | claude-opus-4.1 | 9.2 | official | |
| 7 | claude-opus-4.5 | 10.17 | official | |
| 8 | gpt-5.4 | 10.33 | official | |
| 9 | gpt-5.2 | 13.33 | official | |
| 10 | gpt-oss-20b | 13.54 | official | |
| 11 | gpt-5-pro | 14.01 | official | |
| 12 | claude-opus-4 | 15.93 | official | |
| 13 | o3 | 16.46 | official | |
| 14 | gpt-5-mini | 17.4 | official | |
| 15 | o3-pro | 17.5 | official | |
| 16 | gpt-5 | 20.67 | official | |
| 17 | claude-3-opus | 21 | official | |
| 18 | claude-3.7-sonnet | 24.07 | official | |
| 19 | o4-mini | 24.23 | official | |
| 20 | gpt-5.1 | 25.17 | official | |
| 21 | kimi-k2.5 | 29.53 | official | |
| 22 | claude-3.5-sonnet | 30.54 | official | |
| 23 | llama-3.1-405b-instruct | 33.3 | official | |
| 24 | o1-pro | 38.4 | official | |
| 25 | glm-4.5 | 38.54 | official | |
| 26 | gpt-4.1-nano | 38.6 | official | |
| 27 | glm-4.5-air | 39.2 | official | |
| 28 | o1 | 40.73 | official | |
| 29 | gpt-4o | 42.22 | official | |
| 30 | gpt-4.5-preview | 43.28 | official | |
| 31 | magistral-medium | 43.5 | official | |
| 32 | qwen3-235b-a22b | 43.6 | official | |
| 33 | gemini-2.5-pro-exp | 44.07 | official | |
| 34 | deepseek-r1 | 44.19 | official | |
| 35 | gemini-2.5-pro | 45.63 | official | |
| 36 | llama-3.2-90b-vision-instruct | 45.93 | official | |
| 37 | llama-3.3-70b-instruct | 48.07 | official | |
| 38 | gpt-4.1 | 48.87 | official | |
| 39 | gemini-2.0-flash | 49.93 | official | |
| 40 | gpt-4.1-mini | 50 | official | |
| 41 | llama-4-maverick | 50.27 | official | |
| 42 | o3-mini | 50.79 | official | |
| 43 | gemini-2.5-flash | 50.87 | official | |
| 44 | gemini-2.0-pro-exp-02-05 | 51.33 | official | |
| 45 | gemini-3.1-flash-lite | 51.6 | official | |
| 46 | mistral-large-2 | 52.47 | official | |
| 47 | kimi-k2 | 53.33 | official | |
| 48 | deepseek-v3.1 | 53.73 | official | |
| 49 | deepseek-v3 | 54.48 | official | |
| 50 | gemini-3-pro-preview | 57.4 | official | |
| 50 | mistral-medium | 57.4 | official | |
| 52 | gemini-3.1-pro-preview | 57.6 | official | |
| 53 | grok-2 | 63 | official |