Evaluation profile
FORTRESS
2sub-evals
1.97%total index weight
2components
Within-component eval weight: Misuse resistance 2.24% · Benign helpfulness 17.4%.
Model score (lower is better)Predicted score
About this eval
Balanced refusal behavior across harmful and benign prompts, including over-refusal.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| average_risk_scorefortress/leaderboard.csv:average_risk_scoreMeasures whether adversarial prompts trigger behaviors identified by each scenario’s risk rubric. | ordinary_harm_misuse_resistance:1.000fortress/leaderboard.csv | Lower is better | 0.224% | Misuse resistance 2.24% |
| over_refusal_scorefortress/leaderboard.csv:over_refusal_scoreMeasures whether the model refuses benign requests included to test excessive caution. | benign_helpfulness_non_overrefusal:1.000fortress/leaderboard.csv | Lower is better | 1.74% | Benign helpfulness 17.4% |
average_risk_score
Measures whether adversarial prompts trigger behaviors identified by each scenario’s risk rubric.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-oss-120b | 8.24 | official | |
| 2 | claude-opus-4.5 | 11.59 | official | |
| 3 | muse-spark-1.1 | 12.4 | official | |
| 4 | claude-3.5-sonnet | 12.96 | official | |
| 5 | gpt-5.4-pro | 14.84 | official | |
| 6 | claude-sonnet-4.5 | 14.88 | official | |
| 7 | gpt-5-pro | 15.2 | official | |
| 8 | claude-opus-4.1 | 15.47 | official | |
| 9 | o3 | 16.01 | official | |
| 10 | gpt-5.5 | 16.28 | official | |
| 11 | claude-opus-4.6 | 16.76 | official | |
| 12 | gpt-5-mini | 17 | official | |
| 13 | gpt-5 | 17.04 | official | |
| 14 | gpt-5.2 | 17.5 | official | |
| 15 | gpt-oss-20b | 17.62 | official | |
| 16 | claude-opus-4.8 | 18.19 | official | |
| 17 | o1 | 19.38 | official | |
| 18 | muse-spark | 20.22 | official | |
| 19 | llama-3.1-405b-instruct | 20.61 | official | |
| 20 | claude-sonnet-4 | 21.21 | official | |
| 21 | o4-mini | 21.49 | official | |
| 22 | claude-opus-4 | 26.19 | official | |
| 23 | gpt-5.1 | 28.04 | official | |
| 24 | gemini-3.1-pro-preview | 29.76 | official | |
| 25 | o3-mini | 30.05 | official | |
| 26 | claude-3.5-haiku | 30.41 | official | |
| 27 | claude-3.7-sonnet | 38.01 | official | |
| 28 | llama-4-maverick | 40.09 | official | |
| 29 | kimi-k2.5 | 41.09 | official | |
| 30 | gemini-3-pro-preview | 41.69 | official | |
| 31 | llama-3.1-70b-instruct | 44.18 | official | |
| 32 | llama-3.3-70b-instruct | 44.79 | official | |
| 33 | gpt-4o | 47.18 | official | |
| 34 | gpt-4o-mini | 48.07 | official | |
| 35 | gemini-1.5-flash | 50.61 | official | |
| 36 | gemini-3.1-flash-lite | 51.05 | official | |
| 37 | gpt-4.1 | 53.02 | official | |
| 38 | gemini-1.5-pro | 53.86 | official | |
| 39 | gemini-2.5-pro-exp | 54.89 | official | |
| 40 | kimi-k2 | 55.47 | official | |
| 41 | mixtral-8x22b-instruct | 56.06 | official | |
| 42 | qwen2.5-72b-instruct | 56.43 | official | |
| 43 | gpt-4.1-mini | 59.19 | official | |
| 44 | glm-4.5 | 59.58 | official | |
| 45 | deepseek-v3.1 | 60.55 | official | |
| 46 | qwen3-235b-a22b | 61.39 | official | |
| 47 | gemini-2.5-pro | 61.68 | official | |
| 48 | glm-4.5-air | 63.18 | official | |
| 49 | deepseek-r1 | 74.39 | official |
over_refusal_score
Measures whether the model refuses benign requests included to test excessive caution.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | deepseek-r1 | 0.49 | official | |
| 2 | qwen3-235b-a22b | 0.98 | official | |
| 3 | llama-3.1-70b-instruct | 1.09 | official | |
| 4 | glm-4.5-air | 1.17 | official | |
| 5 | qwen2.5-72b-instruct | 1.19 | official | |
| 6 | gemini-2.5-pro-exp | 1.38 | official | |
| 6 | gpt-4.1-mini | 1.38 | official | |
| 8 | gemini-2.5-pro | 1.58 | official | |
| 9 | gpt-4o | 1.68 | official | |
| 10 | deepseek-v3.1 | 1.76 | official | |
| 11 | gemini-3.1-flash-lite | 2.15 | official | |
| 12 | llama-3.3-70b-instruct | 2.18 | official | |
| 13 | claude-opus-4 | 2.205 | official | |
| 14 | gpt-4o-mini | 2.27 | official | |
| 15 | glm-4.5 | 2.54 | official | |
| 16 | mixtral-8x22b-instruct | 2.77 | official | |
| 17 | gpt-4.1 | 2.87 | official | |
| 18 | gemini-3.1-pro-preview | 3.13 | official | |
| 19 | gemini-1.5-pro | 3.17 | official | |
| 20 | kimi-k2.5 | 3.72 | official | |
| 21 | claude-opus-4.1 | 4.065 | official | |
| 22 | claude-3.7-sonnet | 4.45 | official | |
| 22 | gemini-1.5-flash | 4.45 | official | |
| 24 | llama-4-maverick | 4.95 | official | |
| 25 | o1 | 5.05 | official | |
| 26 | o4-mini | 5.14 | official | |
| 27 | claude-sonnet-4 | 5.14 | official | |
| 28 | claude-opus-4.5 | 5.145 | official | |
| 29 | claude-sonnet-4.5 | 5.25 | official | |
| 30 | o3-mini | 5.65 | official | |
| 31 | gpt-5.1 | 5.78 | official | |
| 32 | llama-3.1-405b-instruct | 6.03 | official | |
| 33 | muse-spark-1.1 | 6.92 | official | |
| 34 | claude-opus-4.8 | 7.42 | official | |
| 34 | o3 | 7.42 | official | |
| 36 | gpt-5.4-pro | 9.41 | official | |
| 37 | claude-opus-4.6 | 9.59 | official | |
| 38 | gpt-5.5 | 9.79 | official | |
| 39 | gpt-5 | 11.2 | official | |
| 40 | gpt-5.2 | 11.76 | official | |
| 41 | gpt-oss-20b | 11.94 | official | |
| 42 | gpt-5-pro | 12.43 | official | |
| 43 | claude-3.5-haiku | 13.16 | official | |
| 44 | gpt-5-mini | 14.17 | official | |
| 45 | gpt-oss-120b | 18 | official | |
| 46 | claude-3.5-sonnet | 21.07 | official |