Evaluation profile

SOSBench

6sub-evals

0.23%total index weight

1components

Within-component eval weight: Misuse resistance 2.3%.

Model score (lower is better)Predicted score

About this eval

Scientific misuse or unsafe open-science behavior.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction	Total index weight	Within-component weight
biology_pvrsosbench/sosbench.csv:biology_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in biology.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%
chemistry_pvrsosbench/sosbench.csv:chemistry_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in chemistry.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%
medicine_pvrsosbench/sosbench.csv:medicine_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in medicine.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%
pharmacology_pvrsosbench/sosbench.csv:pharmacology_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in pharmacology.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%
physics_pvrsosbench/sosbench.csv:physics_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in physics.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%
psychology_pvrsosbench/sosbench.csv:psychology_pvrMeasures whether the model avoids policy-violating assistance on high-risk prompts in psychology.	ordinary_harm_misuse_resistance:1.000sosbench/sosbench.csv	Lower is better	0.0383%	Misuse resistance 0.383%

biology_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in biology.

Rank	Model	Value	Provenance
1	claude-sonnet-4	0.104	official
2	gpt-5	0.108	official
3	claude-opus-4.1	0.134	official
4	o3	0.156	official
5	claude-3.7-sonnet	0.229	official
6	o4-mini	0.262	official
7	llama-4-maverick	0.288	official
8	gpt-4o	0.306	official
9	gemini-2.5-flash	0.336	official
10	gemini-2.5-pro	0.354	official
11	gpt-4.1	0.374	official
12	llama-3.3-70b-instruct	0.408	official
13	llama-4-scout	0.488	official
14	llama-3.1-405b-instruct	0.59	official
15	qwen2.5-72b-instruct	0.68	official
16	grok-3-mini	0.758	official
17	gemma-3-27b-it	0.792	official
18	qwen3-32b	0.802	official
19	deepseek-r1	0.814	official
20	deepseek-r1-distill-llama-70b	0.838	official
21	qwen3-235b-a22b	0.852	official
22	deepseek-v3	0.856	official
23	grok-3	0.894	official

chemistry_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in chemistry.

Rank	Model	Value	Provenance
1	gpt-5	0.122	official
2	claude-opus-4.1	0.147	official
3	o3	0.152	official
4	o4-mini	0.206	official
5	claude-3.7-sonnet	0.208	official
6	claude-sonnet-4	0.21	official
7	llama-4-maverick	0.238	official
8	gpt-4o	0.254	official
9	gpt-4.1	0.314	official
10	gemini-2.5-flash	0.338	official
11	gemini-2.5-pro	0.342	official
12	llama-4-scout	0.436	official
13	llama-3.1-405b-instruct	0.468	official
14	llama-3.3-70b-instruct	0.54	official
15	qwen2.5-72b-instruct	0.56	official
16	grok-3-mini	0.586	official
17	deepseek-v3	0.6	official
18	grok-3	0.638	official
19	gemma-3-27b-it	0.646	official
20	qwen3-235b-a22b	0.76	official
21	qwen3-32b	0.784	official
22	deepseek-r1	0.834	official
23	deepseek-r1-distill-llama-70b	0.904	official

medicine_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in medicine.

Rank	Model	Value	Provenance
1	claude-sonnet-4	0.213	official
2	claude-opus-4.1	0.232	official
3	gpt-5	0.332	official
4	claude-3.7-sonnet	0.35	official
5	o3	0.372	official
6	llama-4-maverick	0.426	official
7	gemini-2.5-flash	0.462	official
7	o4-mini	0.462	official
9	gpt-4o	0.476	official
10	gemini-2.5-pro	0.492	official
11	llama-3.3-70b-instruct	0.546	official
12	gpt-4.1	0.57	official
13	llama-4-scout	0.688	official
14	llama-3.1-405b-instruct	0.69	official
15	qwen2.5-72b-instruct	0.734	official
16	grok-3-mini	0.746	official
17	qwen3-32b	0.774	official
18	deepseek-r1	0.806	official
19	gemma-3-27b-it	0.814	official
20	deepseek-r1-distill-llama-70b	0.854	official
21	grok-3	0.86	official
22	qwen3-235b-a22b	0.868	official
23	deepseek-v3	0.872	official

pharmacology_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in pharmacology.

Rank	Model	Value	Provenance
1	claude-sonnet-4	0.234	official
2	claude-opus-4.1	0.249	official
3	o4-mini	0.408	official
4	gpt-5	0.418	official
5	o3	0.424	official
6	claude-3.7-sonnet	0.579	official
7	gemini-2.5-pro	0.634	official
8	llama-4-maverick	0.652	official
9	gpt-4o	0.676	official
10	gemini-2.5-flash	0.684	official
11	llama-3.1-405b-instruct	0.764	official
12	llama-3.3-70b-instruct	0.812	official
13	gpt-4.1	0.85	official
14	llama-4-scout	0.874	official
15	deepseek-v3	0.916	official
16	qwen2.5-72b-instruct	0.926	official
17	grok-3-mini	0.93	official
18	gemma-3-27b-it	0.934	official
18	qwen3-235b-a22b	0.934	official
20	qwen3-32b	0.946	official
21	grok-3	0.954	official
22	deepseek-r1	0.964	official
23	deepseek-r1-distill-llama-70b	0.972	official

physics_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in physics.

Rank	Model	Value	Provenance
1	claude-opus-4.1	0.098	official
2	gpt-5	0.104	official
3	o3	0.114	official
4	claude-sonnet-4	0.145	official
5	claude-3.7-sonnet	0.171	official
6	gpt-4o	0.194	official
7	o4-mini	0.22	official
8	llama-4-maverick	0.24	official
9	gpt-4.1	0.41	official
10	gemini-2.5-flash	0.424	official
11	llama-3.1-405b-instruct	0.444	official
12	gemini-2.5-pro	0.466	official
13	llama-4-scout	0.492	official
14	llama-3.3-70b-instruct	0.516	official
15	qwen2.5-72b-instruct	0.678	official
16	grok-3-mini	0.708	official
17	deepseek-v3	0.722	official
18	qwen3-32b	0.74	official
19	qwen3-235b-a22b	0.764	official
20	grok-3	0.804	official
21	gemma-3-27b-it	0.842	official
22	deepseek-r1	0.872	official
23	deepseek-r1-distill-llama-70b	0.886	official

psychology_pvr

Measures whether the model avoids policy-violating assistance on high-risk prompts in psychology.

Rank	Model	Value	Provenance
1	claude-opus-4.1	0.107	official
2	claude-sonnet-4	0.123	official
3	gpt-5	0.142	official
4	claude-3.7-sonnet	0.168	official
5	o3	0.196	official
6	llama-4-maverick	0.242	official
7	gemini-2.5-pro	0.294	official
8	o4-mini	0.314	official
9	gemini-2.5-flash	0.326	official
10	gpt-4o	0.396	official
11	llama-3.3-70b-instruct	0.446	official
12	gpt-4.1	0.498	official
13	llama-4-scout	0.51	official
14	llama-3.1-405b-instruct	0.568	official
15	grok-3-mini	0.7	official
16	qwen2.5-72b-instruct	0.734	official
17	qwen3-32b	0.746	official
18	gemma-3-27b-it	0.792	official
19	deepseek-r1	0.806	official
20	deepseek-r1-distill-llama-70b	0.816	official
21	deepseek-v3	0.82	official
22	qwen3-235b-a22b	0.852	official
23	grok-3	0.89	official