Evaluation profile
FORTRESS
2sub-evals
1.48%Safety weight
2.67%Freedom weight
2components
Weights below are portfolio-specific global index weights.
Model score (lower is better)Predicted score
About this eval
Balanced refusal behavior across harmful and benign prompts, including over-refusal.
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| average_risk_scorefortress/leaderboard.csv:average_risk_scoreMeasures whether adversarial prompts trigger behaviors identified by each scenario’s risk rubric. | Safety: ordinary_harm_misuse_resistance:1.000 · Freedom: user_direction_following:1.000fortress/leaderboard.csv | Safety: lower · Freedom: higher | 0.188% | 1.34% |
| over_refusal_scorefortress/leaderboard.csv:over_refusal_scoreMeasures whether the model refuses benign requests included to test excessive caution. | Safety: benign_helpfulness_non_overrefusal:1.000 · Freedom: user_direction_following:1.000fortress/leaderboard.csv | Safety: lower · Freedom: lower | 1.29% | 1.33% |
average_risk_score
Measures whether adversarial prompts trigger behaviors identified by each scenario’s risk rubric.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-oss-120b | 8.24 | official | |
| 2 | gpt-6-luna | 10.78 | official | |
| 3 | claude-opus-4.5 | 11.59 | official | |
| 4 | muse-spark-1.1 | 12.4 | official | |
| 5 | claude-3.5-sonnet | 12.96 | official | |
| 6 | claude-opus-5.5 | 13.18 | official | |
| 7 | gpt-6-sol | 13.46 | official | |
| 8 | claude-fable-5.1 | 13.72 | official | |
| 9 | gpt-5.4-pro | 14.84 | official | |
| 10 | claude-sonnet-4.5 | 14.88 | official | |
| 11 | gpt-5-pro | 15.2 | official | |
| 12 | claude-opus-4.1 | 15.47 | official | |
| 13 | o3 | 16.01 | official | |
| 14 | gpt-5.5 | 16.28 | official | |
| 15 | claude-opus-4.6 | 16.76 | official | |
| 16 | claude-opus-5 | 16.84 | official | |
| 17 | gpt-5-mini | 17 | official | |
| 18 | gpt-5 | 17.04 | official | |
| 19 | gpt-5.2 | 17.5 | official | |
| 20 | gpt-oss-20b | 17.62 | official | |
| 21 | claude-opus-4.8 | 18.19 | official | |
| 22 | o1 | 19.38 | official | |
| 23 | gpt-5.6-sol | 20.14 | official | |
| 24 | muse-spark | 20.22 | official | |
| 25 | llama-3.1-405b-instruct | 20.61 | official | |
| 26 | claude-sonnet-4 | 21.21 | official | |
| 27 | o4-mini | 21.49 | official | |
| 28 | qwen3.8-max | 21.6 | official | |
| 29 | inkling | 24.99 | official | |
| 30 | claude-opus-4 | 26.19 | official | |
| 31 | nemotron-3-ultra-550b-a55b | 26.35 | official | |
| 32 | kimi-k3 | 26.95 | official | |
| 33 | gpt-5.1 | 28.04 | official | |
| 34 | glm-5.3 | 28.23 | official | |
| 35 | gemini-3.1-pro-preview | 29.76 | official | |
| 36 | o3-mini | 30.05 | official | |
| 37 | claude-3.5-haiku | 30.41 | official | |
| 38 | claude-3.7-sonnet | 38.01 | official | |
| 39 | llama-4-maverick | 40.09 | official | |
| 40 | kimi-k2.5 | 41.09 | official | |
| 41 | gemini-3-pro-preview | 41.69 | official | |
| 42 | llama-3.1-70b-instruct | 44.18 | official | |
| 43 | llama-3.3-70b-instruct | 44.79 | official | |
| 44 | gpt-4o | 47.18 | official | |
| 45 | gpt-4o-mini | 48.07 | official | |
| 46 | gemini-1.5-flash | 50.61 | official | |
| 47 | gemini-3.1-flash-lite | 51.05 | official | |
| 48 | gpt-4.1 | 53.02 | official | |
| 49 | gemini-1.5-pro | 53.86 | official | |
| 50 | gemini-2.5-pro-exp | 54.89 | official | |
| 51 | kimi-k2 | 55.47 | official | |
| 52 | mixtral-8x22b-instruct | 56.06 | official | |
| 53 | qwen2.5-72b-instruct | 56.43 | official | |
| 54 | gpt-4.1-mini | 59.19 | official | |
| 55 | glm-4.5 | 59.58 | official | |
| 56 | deepseek-v3.1 | 60.55 | official | |
| 57 | qwen3-235b-a22b | 61.39 | official | |
| 58 | gemini-2.5-pro | 61.68 | official | |
| 59 | glm-4.5-air | 63.18 | official | |
| 60 | deepseek-r1 | 74.39 | official |
over_refusal_score
Measures whether the model refuses benign requests included to test excessive caution.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | deepseek-r1 | 0.49 | official | |
| 2 | qwen3-235b-a22b | 0.98 | official | |
| 3 | llama-3.1-70b-instruct | 1.09 | official | |
| 4 | glm-4.5-air | 1.17 | official | |
| 5 | qwen2.5-72b-instruct | 1.19 | official | |
| 6 | gemini-2.5-pro-exp | 1.38 | official | |
| 6 | gpt-4.1-mini | 1.38 | official | |
| 8 | gemini-2.5-pro | 1.58 | official | |
| 9 | gpt-4o | 1.68 | official | |
| 10 | deepseek-v3.1 | 1.76 | official | |
| 11 | gemini-3-pro-preview | 2.15 | official | |
| 11 | gemini-3.1-flash-lite | 2.15 | official | |
| 13 | llama-3.3-70b-instruct | 2.18 | official | |
| 14 | claude-opus-4 | 2.205 | official | |
| 15 | gpt-4o-mini | 2.27 | official | |
| 16 | glm-4.5 | 2.54 | official | |
| 17 | mixtral-8x22b-instruct | 2.77 | official | |
| 18 | gpt-4.1 | 2.87 | official | |
| 19 | gemini-3.1-pro-preview | 3.13 | official | |
| 20 | gemini-1.5-pro | 3.17 | official | |
| 20 | glm-5.3 | 3.17 | official | |
| 22 | kimi-k3 | 3.315 | official | |
| 23 | kimi-k2 | 3.33 | official | |
| 24 | kimi-k2.5 | 3.72 | official | |
| 25 | gpt-6-sol | 4.06 | official | |
| 26 | claude-opus-4.1 | 4.065 | official | |
| 27 | claude-3.7-sonnet | 4.45 | official | |
| 27 | gemini-1.5-flash | 4.45 | official | |
| 29 | llama-4-maverick | 4.95 | official | |
| 30 | o1 | 5.05 | official | |
| 31 | o4-mini | 5.14 | official | |
| 32 | claude-sonnet-4 | 5.14 | official | |
| 33 | claude-opus-4.5 | 5.145 | official | |
| 34 | claude-sonnet-4.5 | 5.25 | official | |
| 35 | gpt-5.6-sol | 5.34 | official | |
| 36 | o3-mini | 5.65 | official | |
| 37 | gpt-5.1 | 5.78 | official | |
| 38 | llama-3.1-405b-instruct | 6.03 | official | |
| 39 | muse-spark-1.1 | 6.92 | official | |
| 40 | claude-opus-4.8 | 7.42 | official | |
| 40 | inkling | 7.42 | official | |
| 40 | o3 | 7.42 | official | |
| 43 | gpt-6-luna | 7.62 | official | |
| 44 | claude-opus-5 | 7.91 | official | |
| 45 | gpt-5.4-pro | 9.41 | official | |
| 46 | claude-opus-5.5 | 9.5 | official | |
| 47 | claude-opus-4.6 | 9.59 | official | |
| 48 | gpt-5.5 | 9.79 | official | |
| 49 | gpt-5 | 11.2 | official | |
| 50 | gpt-5.2 | 11.76 | official | |
| 51 | gpt-oss-20b | 11.94 | official | |
| 52 | gpt-5-pro | 12.43 | official | |
| 53 | qwen3.8-max | 12.76 | official | |
| 54 | claude-3.5-haiku | 13.16 | official | |
| 55 | nemotron-3-ultra-550b-a55b | 13.25 | official | |
| 56 | gpt-5-mini | 14.17 | official | |
| 57 | claude-fable-5.1 | 16.91 | official | |
| 58 | gpt-oss-120b | 18 | official | |
| 59 | claude-3.5-sonnet | 21.07 | official |