Local LLM Evaluation: Why Verifiability Widens Rather than Closes the Human Alignment Gap
Abstract: Regulation pulls automated evaluation in two directions at once. Frameworks governing high-risk AI systems require that model outputs be evaluated, while data-protection law treats the evaluation of outputs containing personal data through a hosted third-party judge as processing that requires a formal agreement. For a large class of deployments the consequence is that evaluation is mandatory and a hosted judge is unavailable: the judge must run on hardware the organisation controls. What that constraint costs has not been measured. We evaluate five open-weight judges that fit on a single workstation against three independent expert raters on a public summarisation benchmark, using a reference-matched geometric test: one rater is held out, and the judge and that rater are both scored against the mean of the remainder, so both face an identical target. We report three findings. First, the gap between judge and human is nearly twice as large on the rubric with a checkable answer (+52.6 degrees) as on rubrics that are matters of taste (+27.5 degrees)- the inverse of what is reported for frontier judges, and a result that replicates across two independent cohorts to within 0.3 degrees. Second, these judges disagree with one another more than people do, by up to 54 degrees, with effective rank 4.59-4.85 out of 5: they are near-independent rather than sharing an axis. Third, because their errors are independent noise rather than correlated bias, ensembling beats the best single judge on three of four rubrics but not on the verifiable one, where averaging cannot recover a capability none of them has. One model of the six tested declined the task outright. The practical consequence is uncomfortable: the rubric a constrained deployment most needs a judge for- whether a claim is supported by its source- is the rubric on which locally-runnable judges perform worst.
Keywords: LLM-as-a-judge, evaluation, alignment, open-weight models, groundedness, data protection, reproducibility
How to Cite?: Narendar Kumar Ale, "Local LLM Evaluation: Why Verifiability Widens Rather than Closes the Human Alignment Gap", Volume 15 Issue 9, September 2026, International Journal of Science and Research (IJSR), Pages: 1615-1620, https://www.ijsr.net/getabstract.php?paperid=SR26924024938, DOI: https://dx.doi.org/10.21275/SR26924024938