Evaluation profile

AIRBench 2024 Safety Scenarios

1sub-evals

0.429%total index weight

1components

Within-component eval weight: Misuse resistance 4.29%.

Model score (higher is better)Predicted score

About this eval

AIRBench 2024 safety-scenarios performance.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction	Total index weight	Within-component weight
safety_scenarioshelm/airbench2024.csv:safety_scenariosMeasures whether the model responds safely across diverse real-world risk scenarios generated and audited for AIRBench.	ordinary_harm_misuse_resistance:1.000helm/airbench2024.csv	Higher is better	0.429%	Misuse resistance 4.29%

safety_scenarios

Measures whether the model responds safely across diverse real-world risk scenarios generated and audited for AIRBench.

Rank	Model	Value	Provenance
1	claude-haiku-4.5	0.932	official
2	claude-sonnet-4.5	0.898	official
3	claude-3.5-sonnet	0.8835	official
4	claude-sonnet-4	0.883	official
5	gpt-oss-120b	0.88	official
6	gpt-5-nano	0.878	official
7	gpt-5	0.877	official
8	qwen3-next-80b-a3b	0.867	official
9	gpt-5.1	0.862	official
10	gpt-oss-20b	0.86	official
11	claude-opus-4	0.857	official
11	gpt-5-mini	0.857	official
13	claude-3-sonnet	0.847	official
14	o3	0.845	official
15	claude-3-opus	0.844	official
16	claude-3-haiku	0.827	official
17	granite-3.3-8b-instruct-guardian	0.825	official
18	granite-4.0-small-guardian	0.821	official
19	claude-3.7-sonnet	0.818	official
20	granite-4.0-micro-guardian	0.804	official
21	o1	0.8	official
22	qwen3-235b-a22b	0.79	official
23	o4-mini	0.785	official
24	palmyra-x5	0.782	official
25	granite-3.3-8b-instruct	0.76	official
26	gemini-1.5-pro	0.7505	official
27	o3-mini	0.749	official
28	gpt-4.5-preview	0.741	official
28	kimi-k2	0.741	official
30	gemini-2.5-pro	0.736	official
31	gemini-1.5-flash	0.7325	official
32	gemini-3-pro-preview	0.732	official
33	gpt-4-turbo	0.719	official
34	granite-4.0-small	0.716	official
35	llama-3-8b-instruct	0.709	official
36	gemini-2.5-flash	0.687	official
37	llama-4-maverick	0.686	official
38	gemini-2-pro-preview	0.684	official
39	gemini-2-flash-lite-preview	0.675	official
40	palmyra-fin	0.663	official
41	gemini-2-flash	0.662	official
42	granite-4.0-micro	0.661	official
43	gemini-2.5-flash-lite	0.658	official
44	gpt-4.1	0.648	official
45	llama-3-70b-instruct	0.646	official
46	gpt-4	0.642	official
47	llama-3.1-8b-instruct	0.623	official
48	qwen2-72b-instruct	0.621	official
49	gpt-4.1-nano	0.615	official
50	gpt-4.1-mini	0.604	official
51	qwen2.5-72b-instruct	0.59	official
52	llama-3.1-405b-instruct	0.586	official
53	gemini-1.0-pro	0.582	official
54	palmyra-med	0.578	official
55	gpt-4o	0.5755	official
56	glm-4.5-air	0.571	official
57	gpt-4o-mini	0.563	official
58	qwen3-235b-a22b-fp8-throughput	0.56	official
59	gpt-3.5-turbo	0.5577	official
60	yi-34b-chat	0.536	official
61	grok-3-mini	0.535	official
62	deepseek-r1	0.529	official
63	llama-4-scout	0.523	official
64	grok-3	0.513	official
65	deepseek-llm-67b-chat	0.505	official
66	qwen1.5-72b-chat	0.486	official
67	qwen-2.5-7b-instruct	0.47	official
68	o1-mini	0.452	official
69	grok-4	0.444	official
70	palmyra-x-004	0.442	official
71	mixtral-8x22b-instruct	0.44	official
72	llama-3.1-70b-instruct	0.425	official
73	deepseek-v3	0.408	official
74	mixtral-8x7b-instruct	0.391	official
75	mistral-large-2	0.353	official
76	mistral-small-3	0.328	official
77	mistral-7b-instruct	0.326	official
78	command-r	0.318	official
79	command-r-plus	0.293	official
80	dbrx-instruct	0.254	official