← back to journal

A Critic in the Loop

2026-10-09 · Kit Wickham

I built a research mode for den: one question goes to several researcher agents in parallel, each working a different angle, and then a critic agent checks their notes before anything gets written up. For its first run I gave it a question I care about, since models grading models is how a lot of AI evaluation gets done now: how reliable is LLM-as-a-judge? How often do LLM judges agree with expert human raters, and how big are the known biases?

The run took about four minutes. The more interesting result was how much the critic caught.

How it works

A planner splits the question into angles. Here those were: benchmarks of judge–human agreement, how big the biases are, and an angle whose only job was to look for evidence against the comfortable answer. Three researchers work those angles at the same time. Each one is told what the others are covering, so they don't overlap, and is told to cite a primary source for every finding. The researchers can only search and read; nothing on a web page can make them do anything else.

Then the critic gets all three sets of notes. It picks the claims the answer would rest on, opens each cited source, and checks that the source says that, with that number. It also looks for places where the researchers contradict each other. Last, a writer builds the report using only claims the critic didn't throw out.

What the critic caught

The critic checked 19 groups of claims. By my count of its verdicts, 14 held up, 3 held only in part (one of them only in direction), and 2 were wrong as stated:

Two researchers working in good faith, each citing a primary source, still put wrong numbers into their notes. A single pass of research would have shipped those errors to the write-up. That's the case for the extra step: on this question it changed what I'm willing to say.

What the evidence says

I checked these numbers against the papers myself before writing them here.

Overall, an LLM judge is a decent stand-in for a crowd of non-expert raters on open-ended chat, especially when ranking whole models rather than grading single answers. It's a poor stand-in for experts, or for checking correctness on hard questions, unless it's a reasoning model. Most of the bias numbers come from 2023–24 models. Newer judges look more robust to answer order, but a 2026 preprint still found measurable position bias, and recent length-bias measurements are small, but I found no recent padding-attack test. So the practical advice is the same as before: swap the answer order and average, control for length, and when correctness matters, check the judge against a set of answers you've graded yourself.

What I'd change