The Screen Decides the Verdict: Operating Characteristics of Model‑Exclusion Rules in LLM Evaluation

In brief

Evaluations of hosted language models often screen out models that look unstable, then count votes among the models that remain. This paper audits a preregistered evaluation in which that two-step procedure decided the result. The registered stability gate excluded the model whose answers had changed least between runs and kept the one that had changed most. After the exclusion, the registered hypothesis could be neither supported nor refuted. Simulations show that, at the study’s design, the gate usually excluded models that had not drifted at all. Screens and vote rules should therefore be specified and simulated together.

Abstract

An evaluation of hosted language models can screen models for run-to-run stability and then vote among the survivors. We audit a preregistered evaluation in which this two-step pipeline determined the result. Four models answered a yes/no concern question about 20 behaviours written to be ambiguous, each attributed to a neutral actor or to one of six personas. The registered stability gate required a Pearson correlation above .80 between two runs over 14 conditions (two scenarios under seven frames) with 20 calls each. It excluded the model whose rates moved least on the rate scale (mean absolute change .018) and retained the model whose rates moved most (.129). A simulated model with the excluded model’s rates and no drift failed the gate in 83.9% of bootstrap draws. Across 252 simulated designs, the gate excluded models that had not drifted at a median rate of .763. At the study’s design, with an assumed spread between conditions of .5 logits, we recalibrated the threshold to a 5% false-exclusion rate. Power against an upward one-logit shift at a baseline rate of .50 (moving it to about .73) was .066. A registered hypothesis, that persona frames make a model’s answers less consistent within a condition, was to be supported if at least 3 of the 4 models showed this. After the exclusion this rule required 3 of 3, and the realised outcome, two passes and one failure, met neither the support rule nor the paired rule that would refute it. A non-confirmatory sensitivity analysis that restored the excluded model met the support rule. We study one protocol and make no prevalence claim. We show that fixed-count support and refutation rules that give every full-panel outcome exactly one verdict leave some outcomes with no verdict after a removal, and that restating the registered thresholds as fractions of survivors does not repair our case. Screens and vote rules should therefore be specified and simulated together.

About the venue

NeurIPS, the Conference on Neural Information Processing Systems, is one of the leading international conferences in machine learning and artificial intelligence; 2026 is its 40th annual meeting. Alongside the main conference it hosts workshops: smaller meetings on focused topics, each with its own call for papers and peer review.

The TAE workshop (Trust‑AI‑Eval: Can We Trust AI Evaluation?) treats the evaluation of AI systems as an object of study in its own right: what an evaluation measures, which assumptions link a protocol to its conclusions, and when the evidence is strong enough to guide decisions about deploying a system. Its topics include the reliability of human and model judges, measurement validity, benchmark auditing, and deployment risk and governance. It takes place on 12 December 2026 in Sydney, Australia. Accepted papers are presented in person as posters. They are non‑archival: they do not count as formal publication, so the work can still be submitted to a journal.

Workshop website · Accepted posters (TAE)

Cite

@inproceedings{boullineau_screen_2026,
author    = {Boullineau, Emile and Mu{\~n}oz Arciniegas, Jos{\'e} Daniel},
title     = {The Screen Decides the Verdict: Operating Characteristics
             of Model‑Exclusion Rules in LLM Evaluation},
booktitle = {NeurIPS 2026 Trust‑AI‑Eval Workshop (TAE)},
year      = {2026},
note      = {Poster, non-archival},
url       = {https://openreview.net/forum?id=PiBK1nXVZB}
}

Non-archival workshop poster. Camera-ready version, revised after workshop review. Also deposited on Zenodo 10.5281/zenodo.21366659 and SSRN 10.2139/ssrn.7490919.