Evaluation profile
OpenAgentSafety
3sub-evals
0.521%total index weight
1components
Within-component eval weight: Responsible agency 3.47%.
Model score (see each sub-eval direction)Predicted score
About this eval
State-changing tool-agent safety and successful safe completion
Included in the behavior ranking.
Sub-evals
| Measure | Component | Direction | Total index weight | Within-component weight |
|---|---|---|---|---|
| llm_judge_safety_vulnerableopenagentsafety/openagentsafety.csv:llm_judge_safety_vulnerableMeasures unsafe behavior on safety-vulnerable trajectories, LLM judge. | responsible_agency_control:1.000openagentsafety | Lower is better | 0.195% | Responsible agency 1.3% |
| rule_based_safety_vulnerableopenagentsafety/openagentsafety.csv:rule_based_safety_vulnerableMeasures unsafe state change on safety-vulnerable trajectories, rule-based evaluator. | responsible_agency_control:1.000openagentsafety | Lower is better | 0.163% | Responsible agency 1.09% |
| successful_completionopenagentsafety/openagentsafety.csv:successful_completionMeasures tasks completed with the correct safe outcome. | responsible_agency_control:1.000openagentsafety | Higher is better | 0.163% | Responsible agency 1.09% |
llm_judge_safety_vulnerable
Measures unsafe behavior on safety-vulnerable trajectories, LLM judge.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-sonnet-4 | 49.06 | official | |
| 2 | claude-3.7-sonnet | 51.2 | official | |
| 3 | gpt-5 | 52.58 | official | |
| 4 | deepseek-v3 | 62.23 | official | |
| 5 | deepseek-r1 | 62.78 | official | |
| 6 | gpt-4o | 65.8 | official | |
| 7 | o3-mini | 72.73 | official |
rule_based_safety_vulnerable
Measures unsafe state change on safety-vulnerable trajectories, rule-based evaluator.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | o3-mini | 32.32 | official | |
| 2 | deepseek-v3 | 32.44 | official | |
| 3 | gpt-4o | 32.64 | official | |
| 4 | deepseek-r1 | 32.78 | official | |
| 5 | claude-3.7-sonnet | 32.85 | official | |
| 6 | claude-sonnet-4 | 49.06 | official | |
| 7 | gpt-5 | 52.15 | official |
successful_completion
Measures tasks completed with the correct safe outcome.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | gpt-5 | 40.4 | official | |
| 2 | claude-sonnet-4 | 37.1 | official | |
| 3 | claude-3.7-sonnet | 33.88 | official | |
| 4 | deepseek-v3 | 22.12 | official | |
| 5 | gpt-4o | 21.77 | official | |
| 6 | deepseek-r1 | 20.86 | official | |
| 7 | o3-mini | 17.29 | official |