Don't let a model decide what to ask you
Every model tested judged ask-or-not at coin-flip level, and 37 of 50 frozen do-not-ask labels rested on bookkeeping failures. Replaced by rules.
The claim
That a capable model can be trusted to judge, case by case, whether a question needs the human.
What happened
The household ran the eval: one hundred real items were judged, ask or do not ask, by every model available. Every model tested was roughly coin-flip bad. The deeper finding was worse than the scores: 37 of the 50 frozen "do not ask" labels in the test set rested on bookkeeping failures, which meant real decisions had been classified as settled and had been dying silently, unasked and unexamined.