Evaluation profile

AA-Omniscience

1sub-evals

2.5%Safety weight

0%Freedom weight

1components

Weights below are portfolio-specific global index weights.

Model score (lower is better)Predicted score

About this eval

Hallucination propensity on difficult closed-book factual questions under explicit encouragement to abstain when uncertain.

Included in the behavior ranking.

Sub-evals

Measure	Component	Direction by ranking	Safety weight	Freedom weight
hallucination_rateaa-omniscience/aa-omniscience.csv:hallucination_rateMeasures whether the model avoids confidently giving incorrect answers across six thousand factual questions.	Safety: truthfulness_honesty:1.000aa-omniscience	Safety: lower	2.5%	—

hallucination_rate

Measures whether the model avoids confidently giving incorrect answers across six thousand factual questions.

Rank	Model	Value	Provenance
1	minicpm5-1b	0.009009	official
2	g9v3-3b	0.1166	official
3	g9v3-39a5b	0.1304	official
4	command-a-plus	0.1416	official
5	gemini-4-argon	0.1507	official
6	lfm2.5-2.6b	0.16	official
7	grok-4.3	0.1694	official
8	qwen3.8-27b	0.1809	official
9	minimax-m3	0.1843	official
10	grok-4.20	0.1992	official
11	quasar-438b	0.2144	official
12	minicpm5-2b	0.2191	official
13	ling-3.0-flash-vl	0.2205	official
14	k-exaone-2.0	0.226	official
15	grok-4.6	0.24	official
16	solar-pro-4	0.244	official
17	mimo-v2.5-pro	0.247	official
18	solar-open2-250b	0.2538	official
19	granite-4.2-30b	0.2558	official
20	qwen3.7-max	0.2563	official
21	claude-haiku-4.5	0.2571	official
22	grok-3-mini	0.2582	official
23	k2-horizon-375b-a23b	0.2585	official
24	glm-5.2	0.263	official
25	granite-4.2-3b	0.2634	official
26	glm-5.3-flash	0.2763	official
27	qwen3.7-plus	0.2766	official
28	qwen3.8-max	0.2884	official
29	claude-sonnet-4	0.2912	official
30	grok-4.7	0.2934	official
31	glm-5.3	0.2955	official
32	nemotron-3-ultra-550b-a55b	0.2969	official
33	glm-5.1	0.2995	official
34	mimo-v2-pro	0.2998	official
35	gemma-3-270m	0.3042	official
36	ling-3.0-tiny	0.3046	official
37	k2-horizon-mova-36b-a4b	0.3053	official
38	gemma-4-e4b	0.3094	official
39	muse-spark-1.3	0.3146	official
40	mimo-v2.5	0.3193	official
41	granite-4.2-8b	0.3197	official
42	gemma-4-e2b	0.3236	official
43	agnes-2.5-pro-beta	0.328	official
44	a-x-k2	0.3299	official
45	muse-spark-1.2	0.3329	official
46	k2-horizon-7b	0.3363	official
47	gemini-3.5-flash-lite	0.3436	official
48	qwen3.6-plus	0.3464	official
49	glm-5	0.3532	official
50	minimax-m2.7	0.3556	official
51	solar-mini4	0.3583	official
52	qwen3.5-omni-plus	0.37	official
53	nemotron-3.5-lightning	0.3757	official
54	gpt-4o	0.3792	official
55	motif-3	0.3877	official
56	claude-opus-4.8	0.3925	official
57	claude-sonnet-5	0.3937	official
58	ling-3.0-flash-fin	0.3951	official
59	claude-3.7-sonnet	0.3997	official
60	mimo-v2-omni	0.4005	official
61	kimi-k2.6	0.405	official
62	mimo-v2.6-pro	0.4062	official
63	claude-3.5-haiku	0.4115	official
64	claude-opus-4.7	0.4229	official
65	step-5-preview	0.4302	official
66	llama-3.1-8b-instruct	0.4304	official
67	jt-4.1-flash-236b-a21b	0.4367	official
68	ling-3.0-flash	0.4405	official
69	k2-horizon-3.7b	0.4414	official
70	gpt-5-mini	0.4432	official
71	gpt-6-astra	0.4477	official
72	qwen3-coder-480b-a35b-instruct	0.4493	official
73	qwen3.8-flash-next	0.4532	official
74	qwen3.6-max-preview	0.4619	official
75	lfm2.5-8b-a1b	0.4691	official
76	claude-sonnet-5.5	0.4701	official
77	agnes-3.0-flash	0.4838	official
78	mimo-v2-flash	0.4841	official
79	claude-sonnet-4.6	0.4845	official
80	claude-sonnet-4.5	0.4908	official
81	qwen3.6-27b	0.4927	official
82	gpt-6.1-sol	0.4936	official
83	kimi-k2.5	0.4996	official
84	muse-spark-1.1	0.4998	official
85	qwen3.6-35b-a3b	0.505	official
86	gpt-6-sol	0.5074	official
87	gemini-3.1-pro-preview	0.5087	official
88	gpt-5.4-nano	0.5113	official
89	glm-4.6v	0.5146	official
90	gpt-5.1-codex-mini	0.5158	official
91	gpt-5.1	0.5189	official
92	gemini-3.8-flash	0.519	official
93	llama-3.1-405b-instruct	0.5244	official
94	gpt-5-nano	0.5262	official
95	kimi-k3	0.532	official
96	deepseek-v4.1-flash	0.5399	official
97	grok-4.5	0.5415	official
98	mimo-v2.6-flash	0.5436	official
99	gemini-3.6-flash	0.5563	official
100	kat-coder-pro-v2	0.5598	official
101	nova-lite	0.5761	official
102	nanbeige4.1-3b	0.581	official
103	claude-opus-5.5	0.5861	official
104	claude-opus-5	0.5955	official
105	magistral-medium	0.5966	official
106	nvidia-nemotron-nano-9b-v2	0.6049	official
107	mistral-medium-3	0.609	official
108	claude-opus-4.5	0.61	official
109	qwen3.5-0.8b	0.6135	official
110	gpt-5.2	0.616	official
111	gemini-3.5-flash	0.6176	official
112	devstral	0.6219	official
113	glm-5-turbo	0.6264	official
114	claude-opus-4.6	0.6284	official
115	lfm2.2-6b	0.6289	official
116	inkling-small	0.6299	official
117	claude-fable-5	0.6364	official
118	jt-35b-flash	0.6384	official
119	grok-4	0.6449	official
120	gemini-3.7-flash	0.6453	official
121	nova-micro	0.6508	official
122	gpt-5.5-instant	0.653	official
123	claude-fable-5.1	0.6563	official
124	olmo-3.1-32b-think	0.6629	official
125	llama-3.2-1b-instruct	0.6634	official
126	mistral-small-4	0.665	official
127	ernie-4.5-300b-a47b	0.6693	official
128	llama-nemotron-super-49b-v1.5	0.6726	official
129	kat-coder-pro-v1	0.6736	official
130	glm-4.6	0.6757	official
131	inkling	0.6766	official
132	mistral-large-2	0.6768	official
133	grok-4-fast	0.6827	official
134	minimax-m2.1	0.6854	official
135	glm-5v-turbo	0.688	official
136	lfm2-24b-a2b	0.6903	official
137	jamba-reasoning-3b	0.6904	official
138	gpt-4	0.6941	official
139	o1	0.6963	official
140	glm-4.5	0.7015	official
141	k2-think-v2	0.7117	official
142	llama-3.1-nemotron-instruct-70b	0.7126	official
143	gemini-2.5-flash-lite	0.7171	official
144	nova-premier	0.7192	official
145	kimi-k2	0.7267	official
146	olmo-3.1-32b-instruct	0.7324	official
147	gpt-5.2-codex	0.7336	official
148	gemma-4-12b	0.7341	official
149	grok-4.1-fast	0.7341	official
150	llama-3.3-nemotron-super-49b	0.7369	official
151	hy3	0.7413	official
152	qwen3.5-2b	0.7424	official
153	deepseek-v3.1-terminus	0.7483	official
154	gpt-5.6-luna	0.75	official
155	qwen3.5-27b	0.7513	official
156	longcat-2.0	0.7539	official
157	gpt-5-codex	0.7542	official
158	qwen3-14b-instruct	0.7582	official
159	k2-v2	0.7604	official
160	granite-4.0-350m	0.7613	official
161	gpt-6-luna	0.7673	official
162	mistral-small-3.2	0.7681	official
163	gpt-5.1-codex	0.772	official
164	command-a	0.7732	official
165	qwen3-235b-a22b-instruct	0.774	official
166	mistral-small-3.1	0.7756	official
167	nova-pro	0.7771	official
168	phi-4-mini	0.7799	official
169	llama-3.1-70b-instruct	0.7821	official
170	apodex-1.1	0.7843	official
171	gpt-5	0.7902	official
172	grok-code-fast-1	0.7929	official
173	llama-4-scout	0.7935	official
174	exaone-4.5-33b	0.7992	official
175	nova-2.0-pro	0.7994	official
176	doubao-seed-code	0.7999	official
177	cogito-v2.1	0.8016	official
178	ministral-3-3b	0.8019	official
179	hermes-4-llama-3.1-405b	0.8023	official
180	qwen3-coder-30b-a3b-instruct	0.803	official
181	mercury-2.5	0.8033	official
182	o4-mini	0.8045	official
183	claude-3-haiku	0.8045	official
184	tri-21b-think-preview	0.8053	official
185	qwen3-30b-a3b-instruct	0.8073	official
186	o3-mini	0.8108	official
187	deepseek-r1-distill-llama-70b	0.8109	official
188	hyperclova-x-seed-think-32b	0.8119	official
189	phi-4	0.8119	official
190	llama-3.1-nemotron-ultra-253b-v1	0.8123	official
191	mistral-7b-instruct	0.8126	official
192	lfm2.1-2b	0.8138	official
193	mistral-medium-3.5	0.8159	official
194	devstral-small	0.8162	official
195	granite-4.0-h-nano-1b	0.8172	official
196	qwen3-4b-instruct	0.8174	official
197	llama-3.2-instruct-11b-vision	0.8188	official
198	muse-glimmer	0.8194	official
199	gemma-4-31b-it	0.8195	official
200	exaone-4.0-32b	0.821	official
201	qwen3-32b-instruct	0.8212	official
202	gemma-3-1b	0.8229	official
203	hypernova-60b	0.8233	official
204	kimi-k2.7-code	0.8238	official
205	deepseek-v3.1	0.8251	official
206	gpt-4.1-nano	0.8256	official
207	gpt-5.4	0.8258	official
208	gemini-2.5-flash	0.826	official
209	gemini-3.1-flash-lite	0.8266	official
210	hermes-4-llama-3.1-70b	0.8268	official
211	qwen3.5-397b-a17b	0.8269	official
212	apertus-70b-instruct	0.8282	official
213	north-mini-code	0.8321	official
214	nemotron-3-nano-30b-a3b	0.8327	official
215	step-3-vl-10b	0.8332	official
216	magistral-small	0.8345	official
217	deepseek-r1	0.8345	official
218	qwen3.5-9b	0.8361	official
219	qwen3-vl-32b	0.8368	official
220	muse-spark	0.8419	official
221	mistral-medium-3.1	0.8458	official
222	nova-2.0-omni	0.847	official
223	glm-4.5v	0.8478	official
224	step-3.7-flash	0.8498	official
225	qwen3-vl-235b-a22b	0.8515	official
226	deepseek-v3.2	0.8518	official
227	devstral-2	0.8527	official
228	apriel-v1.5-15b-thinker	0.8528	official
229	ring-2.6-1t	0.8533	official
230	nemotron-cascade-2-30b-a3b	0.8536	official
231	tri-21b-think-v0.5	0.8542	official
232	qwen3.5-35b-a3b	0.8543	official
233	qwen2.5-72b-instruct	0.8546	official
234	nemotron-3-nano-omni-30b-a3b	0.8565	official
235	olmo-3-32b-think	0.8567	official
236	k2-horizon-0.9b	0.8569	official
237	deepseek-v3	0.8586	official
238	nova-2.0-lite	0.8587	official
239	trinity-large	0.8589	official
240	mistral-large-3	0.8596	official
241	gemini-2.0-flash	0.8624	official
242	deepseek-r1-qwen3-8b	0.8635	official
243	seed-oss-36b-instruct	0.8639	official
244	gemma-4-26b-a4b-it	0.8644	official
245	qwen3-30b-a3b	0.8654	official
246	qwen3.5-4b	0.8655	official
247	gpt-oss-20b	0.8671	official
248	nemotron-3-super-120b-a12b	0.8696	official
249	ernie-5.0-thinking-preview	0.8701	official
250	qwen3-next-80b-a3b	0.8701	official
251	qwen3.5-122b-a10b	0.8709	official
252	granite-4.1-8b	0.8713	official
253	devstral-small-2	0.8729	official
254	grok-3	0.8732	official
255	hy3-preview	0.874	official
256	llama-3-70b-instruct	0.8754	official
257	intellect-3	0.8759	official
258	granite-4.0-h-small	0.8785	official
259	deepseek-v4-pro	0.8785	official
260	gpt-5.6-terra	0.8788	official
261	gpt-5.5	0.8804	official
262	granite-4.0-h-350m	0.8806	official
263	agnes-2.5-pro-alpha	0.8807	official
264	o3	0.8809	official
265	minimax-m2.5	0.8812	official
266	solar-pro-3	0.8822	official
267	step-3.5-flash	0.8849	official
268	nvidia-nemotron-3-nano-4b	0.8857	official
269	falcon-h1r-7b	0.8877	official
270	k-exaone	0.8884	official
271	llama-4-maverick	0.8888	official
272	deepseek-v4-flash	0.8897	official
273	qwen3-omni-30b-a3b	0.8899	official
274	deepseek-v3.2-speciale	0.8915	official
275	gpt-5.3-codex	0.8918	official
276	gpt-5.6-sol	0.8942	official
277	ling-flash-2.0	0.8948	official
278	solar-open-100b	0.8965	official
279	lfm2.5-1.2b-instruct	0.8973	official
280	ring-1t	0.8975	official
281	qwen3-vl-30b-a3b	0.8977	official
282	llama-3-8b-instruct	0.8979	official
283	gpt-5.4-mini	0.8985	official
284	qwen3-max	0.8992	official
285	qwen3-max-preview	0.8993	official
286	gemini-3-pro-preview	0.9	official
287	ring-flash-2.0	0.9007	official
288	llama-3.3-70b-instruct	0.9021	official
289	motif-2-12.7b	0.9059	official
290	sarvam-m	0.9064	official
291	qwen3-8b-instruct	0.9074	official
292	gpt-oss-120b	0.9082	official
293	diffusiongemma-26b-a4b	0.9089	official
294	gemini-2.5-pro	0.9089	official
295	minimax-m2	0.909	official
296	qwen3-0.6b-instruct	0.9094	official
297	qwen3-vl-8b-instruct	0.9101	official
298	molmo2-8b	0.911	official
299	olmo-3-7b-instruct	0.9122	official
300	mercury-2	0.9122	official
301	reka-flash-3	0.9127	official
302	jt-mini	0.913	official
303	qwen3-vl-8b	0.9141	official
304	deepseek-v4-flash-vision	0.9148	official
305	minimax-m1-80k	0.9153	official
306	qwen3-vl-235b-a22b-instruct	0.9155	official
307	deepseek-v4-flash-0731	0.917	official
308	exaone-4.0-1.2b	0.9174	official
309	qwen3-vl-4b	0.9182	official
310	qwen3-vl-32b-instruct	0.9182	official
311	apriel-v1.6-15b-thinker	0.9193	official
312	deepseek-v4-pro-0813	0.9194	official
313	gemma-3-27b-it	0.9209	official
314	lfm2-8b-a1b	0.923	official
315	nvidia-nemotron-nano-12b-v2-vl	0.9235	official
316	gemini-3-flash-preview	0.9241	official
317	ministral-3-14b	0.9252	official
318	gpt-4.1-mini	0.9272	official
319	qwen3-next-80b-a3b-instruct	0.9275	official
320	celeris-1	0.9275	official
321	ling-2.6-1t	0.9283	official
322	glm-4.5-air	0.9288	official
323	glm-4.7	0.9291	official
324	solar-pro-2	0.9303	official
325	grok-build-0.1	0.9312	official
326	olmo-3-7b-think	0.9321	official
327	gpt-4.1	0.9331	official
328	gemma-3n-e2b	0.9338	official
329	qwen3.5-omni-flash	0.934	official
330	sarvam-105b	0.9343	official
331	granite-4.0-nano-1b	0.9352	official
332	qwen3-vl-30b-a3b-instruct	0.9366	official
333	qwen3-coder-next	0.9368	official
334	glm-4.7-flash	0.9394	official
335	ministral-3-8b	0.9416	official
336	mi-dm-k-2.5-pro	0.9427	official
337	ling-1t	0.9473	official
338	qwen3-1.7b-instruct	0.9481	official
339	nex-n2-pro	0.9491	official
340	granite-4.1-30b	0.9498	official
341	jamba-1.7-large	0.9507	official
342	granite-4.1-3b	0.9508	official
343	jamba-1.7-mini	0.951	official
344	lfm2.5-vl-1.6b	0.9574	official
345	granite-4.0-micro	0.9577	official
346	ling-mini-2.0	0.9587	official
347	minicpm-v4.6-1.3b	0.9607	official
348	apertus-8b-instruct	0.9614	official
349	longcat-flash-lite	0.9616	official
350	tiny-aya-global	0.962	official
351	granite-3.3-8b-instruct	0.9622	official
352	sarvam-30b	0.9628	official
353	ling-2.6-flash	0.9672	official
354	gemma-3n-e4b-it	0.9707	official
355	gemma-3-12b	0.974	official
356	qwen3-vl-4b-instruct	0.9742	official
357	qwen3-omni-30b-a3b-instruct	0.9759	official
358	gemma-3-4b	0.9821	official