Evaluation profile
TAC
1sub-evals
6.09%Safety weight
0%Freedom weight
1components
Weights below are portfolio-specific global index weights.
Model score (higher is better)Predicted score
About this eval
Agentic animal-welfare behavior in travel-booking decisions under a neutral prompt.
Included in the behavior ranking.
Interpretation and limitations
- MiMo-V2.6-Pro completed 155 of 156 purchases (99.36%) and has 55 of 156 welfare-preserving outcomes (35.26%). MiMo-V2.6-Flash completed 146 of 156 purchases (93.59%), below the existing 95% gate, and is excluded from active TAC scoring. Neither model's behavioral non-completions were resampled. Xiaomi native enabled reasoning is not asserted to be medium-effort equivalent.
Sub-evals
| Measure | Component | Direction by ranking | Safety weight | Freedom weight |
|---|---|---|---|---|
| base_welfare_ratetac/tac-research-preview-round52.csv:base_welfare_rateMeasures whether an autonomous shopping agent preserves a stated animal-welfare constraint when making purchase decisions. | Safety: nonhuman_ethics:1.000tac | Safety: higher | 6.09% | — |
base_welfare_rate
Measures whether an autonomous shopping agent preserves a stated animal-welfare constraint when making purchase decisions.
| Rank | Model | Value | Relative performance | Provenance |
|---|---|---|---|---|
| 1 | claude-opus-4.8 | 64.74 | official | |
| 2 | claude-opus-5 | 59.62 | official | |
| 3 | claude-fable-5 | 55.77 | official | |
| 4 | qwen2.5-72b-instruct | 49.4 | self run | |
| 5 | qwen3-32b | 48.1 | self run | |
| 6 | qwen2.5-7b-instruct | 46.2 | self run | |
| 7 | deepseek-v4.1-flash | 43.59 | self run | |
| 7 | gpt-6-sol | 43.59 | self run | |
| 7 | muse-spark-1.3 | 43.59 | self run | |
| 10 | deepseek-v3 | 42.3 | self run | |
| 11 | granite-4.2-8b | 41.67 | self run | |
| 12 | gemini-2.5-flash | 39.74 | official | |
| 13 | claude-sonnet-4.6 | 37.82 | official | |
| 14 | gpt-5-nano | 37.82 | self run | |
| 15 | gpt-4.1 | 37.18 | official | |
| 16 | nemotron-3-nano-omni-30b-a3b | 36.54 | self run | |
| 17 | deepseek-v3.2 | 36.54 | official | |
| 18 | qwen3-235b-a22b | 35.9 | self run | |
| 19 | gemini-3-flash-preview | 35.9 | self run | |
| 20 | llama-3.3-70b-instruct | 35.3 | self run | |
| 21 | inkling-small | 35.26 | self run | |
| 21 | mimo-v2.6-pro | 35.26 | self run | |
| 21 | ministral-3-3b | 35.26 | self run | |
| 21 | nex-n2-mini | 35.26 | self run | |
| 25 | hy4-preview | 33.97 | self run | |
| 26 | glm-4.7-flash | 33.33 | self run | |
| 27 | deepseek-v3.1 | 32.1 | self run | |
| 28 | nemotron-3-nano-30b-a3b | 32.05 | self run | |
| 28 | solar-mini4 | 32.05 | self run | |
| 30 | seed-1.6-flash | 31.41 | self run | |
| 30 | seed-2.0-mini | 31.41 | self run | |
| 32 | gpt-oss-120b | 31.4 | self run | |
| 33 | gemini-3.8-flash | 30.77 | self run | |
| 34 | gemini-3.1-flash-lite | 30.13 | self run | |
| 35 | nemotron-3-super-120b-a12b | 29.5 | self run | |
| 36 | glm-5.3 | 29.49 | self run | |
| 36 | gpt-4.1-mini | 29.49 | self run | |
| 36 | ling-3.0-tiny | 29.49 | self run | |
| 39 | gpt-5.6-terra | 29.49 | official | |
| 40 | qwen3.5-9b | 28.85 | self run | |
| 40 | sakana-namazu | 28.85 | self run | |
| 42 | gemma-4-26b-a4b-it | 28.21 | self run | |
| 42 | gpt-oss-safeguard-20b | 28.21 | self run | |
| 42 | laguna-s-2.1-poolside | 28.21 | self run | |
| 42 | qwen3.5-397b-a17b | 28.21 | self run | |
| 46 | laguna-xs-2.1-poolside | 27.56 | self run | |
| 47 | gpt-5.2 | 26.28 | official | |
| 48 | big-pickle | 26.28 | self run | |
| 48 | gpt-5-mini | 26.28 | self run | |
| 48 | gpt-6-luna | 26.28 | self run | |
| 48 | solar-pro-4 | 26.28 | self run | |
| 52 | gemini-3.7-flash | 25.64 | official | |
| 53 | inkling | 25.64 | self run | |
| 53 | kimi-k2.6 | 25.64 | self run | |
| 55 | kimi-k3 | 25 | official | |
| 55 | space-bunny-alpha | 25 | self run | |
| 57 | glm-4.5-air | 24.36 | self run | |
| 57 | kimi-k2.5 | 24.36 | self run | |
| 57 | mercury-2.5-preview | 24.36 | self run | |
| 57 | minimax-m3 | 24.36 | self run | |
| 57 | qwen3.5-flash | 24.36 | self run | |
| 57 | qwen3.7-flash | 24.36 | self run | |
| 63 | gemini-3.5-flash-lite | 23.72 | self run | |
| 63 | gemma-4-31b-it | 23.72 | self run | |
| 63 | nemotron-3-ultra-550b-a55b | 23.72 | self run | |
| 66 | mimo-v2.5-pro | 23.7 | self run | |
| 67 | deepseek-v4-pro | 23.08 | official | |
| 68 | mimo-v2.5 | 21.79 | self run | |
| 69 | deepseek-v4-flash | 21.15 | official | |
| 70 | minimax-m2.7 | 21.15 | self run | |
| 70 | ox-alpha | 21.15 | self run | |
| 72 | deepseek-v4-flash-0731 | 20.51 | self run | |
| 72 | minimax-m2.5 | 20.51 | self run | |
| 74 | step-3.7-flash | 19.87 | self run | |
| 75 | glm-5.2 | 19.23 | official | |
| 75 | gpt-5.5 | 19.23 | official | |
| 77 | ling-2.6-flash | 19.23 | self run | |
| 77 | mercury-2.5 | 19.23 | self run | |
| 77 | seed-2.1-turbo | 19.23 | self run | |
| 80 | glm-5 | 18.59 | self run | |
| 80 | step-3.5-flash | 18.59 | self run | |
| 82 | glm-5.1 | 17.95 | self run | |
| 83 | gemini-3.5-flash | 17.31 | official | |
| 84 | hy3 | 16.67 | self run | |
| 84 | ornith-1.5-35b-a3b | 16.67 | self run | |
| 84 | qwen3.8-27b | 16.67 | self run | |
| 87 | gpt-5.6-sol | 16.67 | official | |
| 87 | grok-4.6 | 16.67 | official | |
| 87 | qwen3.8-max | 16.67 | official | |
| 90 | claude-sonnet-5 | 15.38 | official | |
| 91 | gpt-5.6-luna | 13.46 | self run | |
| 92 | muse-glimmer | 9.615 | self run |