Auditing LLM-as-Judge Failures in Production Text-to-SQL Pipelines

What happened

Two independent AI research teams released preprints on arXiv on September 28, 2026. The first team conducted an audit of LLM judges used in production Text-to-SQL pipelines, finding the previously deployed gpt-4o-mini had extremely low consistency. Tests showed replacing it with self-hosted Qwen3.6-27B cut costs to 1/300 of the original solution while delivering performance on par with the commercial model, with multi-judge integration delivering even better results; the team has open-sourced the audit code. The second team identified the "LLM Parkinson's" problem in LLM agents, and developed the GEC v0.2 architecture, which delivered a 96.57% success rate and 36.4% reduction in token consumption across 24,000 test rounds.

Key facts

Original Text-to-SQL judge model

gpt-4o-mini, which incorrectly flagged 77.1% of human-validated valid cases

SQL judge replacement solution

Self-hosted Qwen3.6-27B, with kappa=0.72, delivering a per-call cost 1/300 that of the original solution

Open-source resources

Text-to-SQL judge audit code and pre-registration framework published publicly on GitHub

Identified LLM agent flaw

The "LLM Parkinson's" phenomenon, where agents make unnecessary, low-value adjustments even after completing their assigned goals

Optimized agent architecture

GEC v0.2 (uncertainty-aware global execution control architecture)

Architecture test results

96.57% hard goal success rate across 24,000 test rounds, with 36.4% lower token consumption

Background

Prior to these releases, the industry generally assumed commercial LLMs could be directly deployed as automatic judges in production pipelines, and most LLM agents used self-driven loop architectures. The industry has historically relied on scaling model size to improve performance, with relatively little focus on reliability and cost optimization for production deployments.

Why it matters

For the broader industry, the two studies validate the practical value of auditing production pipeline LLM judges and deploying decoupled control architectures for agents, exploring cost and efficiency gains for LLMs that do not rely on model scaling. For developers, the work provides reusable open-source SQL validation tools and reference agent architectures. For end users, this work points to future improvements in AI service accuracy and response speed, alongside lower usage costs.

What to watch

Going forward, observers can track real-world adaptation of the studies' open-source resources, and the practical performance of these solutions across more production use cases.