Evaluation profile

HELM Classic RealToxicityPrompts

2sub-evals

0.207%total index weight

1components

Within-component eval weight: Misuse resistance 2.07%.

Model score (lower is better)Predicted score

About this eval

Toxic continuation generation under toxic and non-toxic prompt strata.

Included in the behavior ranking.

Measure	Component	Direction	Total index weight	Within-component weight
non_toxic_prompt_toxic_fractionrealtoxicityprompts/realtoxicityprompts.csv:non_toxic_prompt_toxic_fractionMeasures whether the model generates toxic continuations when the prompt does not contain toxic language.	ordinary_harm_misuse_resistance:1.000realtoxicityprompts	Lower is better	0.104%	Misuse resistance 1.04%
toxic_prompt_toxic_fractionrealtoxicityprompts/realtoxicityprompts.csv:toxic_prompt_toxic_fractionMeasures whether the model generates toxic continuations when the prompt itself contains toxic language.	ordinary_harm_misuse_resistance:1.000realtoxicityprompts	Lower is better	0.104%	Misuse resistance 1.04%

Measures whether the model generates toxic continuations when the prompt does not contain toxic language.

Rank	Model	Value	Provenance
1	xlarge-20220609	0.001287	official
2	command-medium-beta	0.002317	official
2	medium-20220720	0.002317	official
4	large-20220720	0.003089	official
4	small-20220720	0.003089	official
6	palmyra-x	0.003089	official
7	xlarge-20221108	0.003346	official
8	command-xlarge-beta	0.003604	official
8	medium-20221108	0.003604	official
10	t5-11b	0.004118	official
11	t0pp	0.009266	official
12	text-davinci-003	0.01004	official
13	luminous-base	0.01055	official
14	luminous-extended	0.01313	official
15	ul2	0.01364	official
16	text-ada-001	0.01519	official
17	j1-large	0.01725	official
18	j1-grande	0.0175	official
19	gpt-j-6b	0.01905	official
20	bloom	0.01905	official
21	text-babbage-001	0.01931	official
22	ada	0.01982	official
23	babbage	0.02059	official
24	j2-large	0.02059	official
25	j1-jumbo	0.02111	official
26	tnlgv2-7b	0.02136	official
27	glm	0.02188	official
28	gpt-neox-20b	0.02188	official
29	text-curie-001	0.02214	official
30	curie	0.02291	official
31	j1-grande-v2-beta	0.02342	official
32	opt-66b	0.02368	official
33	opt-175b	0.0242	official
34	j2-jumbo	0.0242	official
35	stanford-online-all-v4-s3	0.02445	official
36	palmyra-instruct-30	0.02471	official
36	text-davinci-002	0.02471	official
38	j2-grande	0.02523	official
39	tnlgv2-530b	0.02523	official
40	luminous-supreme	0.02728	official
41	yalm	0.02754	official
42	davinci	0.03449	official

Measures whether the model generates toxic continuations when the prompt itself contains toxic language.

Rank	Model	Value	Provenance
1	palmyra-x	0.008072	official
2	t5-11b	0.01435	official
3	medium-20220720	0.01435	official
4	command-xlarge-beta	0.01614	official
4	medium-20221108	0.01614	official
6	small-20220720	0.01704	official
7	command-medium-beta	0.01794	official
8	large-20220720	0.01794	official
9	xlarge-20220609	0.01883	official
10	xlarge-20221108	0.02152	official
11	luminous-base	0.04126	official
12	luminous-extended	0.04305	official
13	text-ada-001	0.06457	official
14	text-davinci-003	0.06726	official
15	palmyra-instruct-30	0.07265	official
15	ul2	0.07265	official
17	gpt-neox-20b	0.07803	official
18	opt-175b	0.08072	official
19	gpt-j-6b	0.0852	official
20	luminous-supreme	0.0852	official
21	babbage	0.0861	official
22	j1-large	0.0861	official
23	ada	0.08789	official
24	curie	0.08969	official
25	tnlgv2-530b	0.09148	official
26	j1-jumbo	0.09148	official
27	opt-66b	0.09238	official
28	stanford-online-all-v4-s3	0.09327	official
29	j1-grande	0.09417	official
30	bloom	0.09507	official
31	glm	0.09507	official
32	tnlgv2-7b	0.09596	official
33	j2-large	0.09865	official
34	j2-jumbo	0.09955	official
35	text-davinci-002	0.1013	official
36	text-babbage-001	0.1045	official
37	davinci	0.1049	official
37	j1-grande-v2-beta	0.1049	official
39	text-curie-001	0.1067	official
40	j2-grande	0.1094	official
41	t0pp	0.1184	official
42	yalm	0.157	official