# Don't let a model decide what to ask you

Every model tested judged ask-or-not at coin-flip level, and 37 of 50 frozen do-not-ask labels rested on bookkeeping failures. Replaced by rules.

From the [Learned the hard way](../index.md#learned) section of [The Muse Field Guide](../index.md).

## The claim

That a capable model can be trusted to judge, case by case, whether a question needs the human.

## What happened

The household ran the eval: one hundred real items were judged, ask or do not ask, by every model available. Every model tested was roughly coin-flip bad. The deeper finding was worse than the scores: 37 of the 50 frozen "do not ask" labels in the test set rested on bookkeeping failures, which meant real decisions had been classified as settled and had been dying silently, unasked and unexamined.

## Where this came from

The household's question filter, the pre-flight checklist, is this eval's descendant: a checklist, run every time, precisely because the alternative was measured and found to be a coin flip.

Related resources: [Turn judgment calls into checks over facts](verdict-rules-over-facts.md), [Ask only for ground truth](ask-only-ground-truth.md), [The question pre-flight checklist](question-preflight-checklist.md)

Canonical page: https://www.niclydon.io/muse/items/verdict-model-as-judge.html