Here is a cheap experiment for anyone screening decks with a chatbot. Take two submissions your model already ranked, A above B. Paste them again in the opposite order. A meaningful share of the time, the model now prefers B.

Nothing about the startups changed. The order on the page changed. That was enough.

You cannot prompt this away, and it says nothing about whether AI can read a deck. The problem lives in the number one. One model, asked once, is a single judge with a single set of habits. And single judges, silicon or human, have a bias problem that the judging industry has known about for decades.

Humans set a low bar first

Before pointing at the machine, look at the incumbent. When two qualified human experts score the same grant application, their agreement lands around an intraclass correlation of 0.26, measured across 23,000+ reviews at a national science fund (PLOS One). A meta-analysis of journal peer review puts the typical figure at 0.23: less than a quarter of the variance in a verdict is explained by shared judgment (PLOS One). A 2025 registered report on 134,991 grant reviews opens by noting the problem has not gone anywhere (PLOS One).

So the honest starting point is not "humans are reliable and AI is risky." It is that unstructured expert judgment disagrees with itself, whoever supplies the expert. The question worth asking about any judge, carbon or silicon, is the same: what are its habits, and does the process expose them or hide them?

One model has habits it cannot see

For a single LLM judge, researchers have moved past anecdotes into catalogues. The CALM framework quantifies twelve distinct bias types in LLM-as-a-judge setups, from position and verbosity to bandwagon and authority effects, and finds that even the strongest models keep several of them (arXiv).

Three of the twelve matter most for deck screening.

Position. A study spanning 15 LLM judges and 150,000+ evaluation instances found that preferring whichever candidate came first, or last, is systematic rather than random, and varies by judge and task (arXiv). Rubric-based follow-ups report accuracy shifts above 10% from reordering alone (arXiv). Your pile has an order. That order is scoring.

Self-preference. GPT-4 measurably favors text it finds familiar, in the perplexity sense: writing that looks like its own output scores higher regardless of quality (arXiv). Founders increasingly polish decks with the same models that then judge them. A single judge from one family quietly rewards its own accent.

Format. Reliability stress-tests find judge consistency breaks on changes as small as formatting, paraphrasing, and verbosity shifts. An April 2026 review of judge-bias benchmarks reports frontier models exceeding 50% error rates on advanced bias tests (Adaline), a long fall from the "80% agreement with humans" headline that launched the whole LLM-as-a-judge idea in 2023.

"Use a newer model" does not fix it

The reflex answer is to upgrade. The JudgeSense benchmark tested exactly that in April 2026: semantically identical paraphrases of an evaluation prompt flip verdicts, pairwise setups keep their position bias, and, in the authors' words, the largest and newest models are not the most consistent (arXiv).

That result should not surprise anyone. Scale buys capability. It does not buy the thing a verdict needs, which is independence from the judge's own habits. A taller judge is still one judge.

Panels work, but only real ones

The evidence points somewhere specific. A panel of smaller evaluators drawn from different model families correlates better with human judgment than a single large judge, carries less intra-model bias, and costs a fraction as much (arXiv).

There is a catch, and it is worth taking seriously rather than hiding. A May 2026 analysis titled "Nine Judges, Two Effective Votes" shows that when panel members share training data or a model family, their errors correlate, and nine judges collapse into roughly two independent opinions (arXiv). Running the same model six times with six prompts is not a panel. It is one judge wearing six hats.

So a working panel needs two properties at once: judges that are genuinely independent, and a process that measures their disagreement instead of averaging it into silence.

What this looks like when it is built in

This is the architecture EvalLens runs, and the research above is why it looks the way it does.

Every deck is read by six AI judges. Each judge has its own role, its own focus, and its own zone of the rubric; they run in parallel and never see each other's scores. The rubric is fixed across the whole batch, so every deck meets the same six readings in the same six ways, whatever order it arrived in.

Disagreement is not smoothed over. When judges split on a deck by 3.0 points or more, that spread is surfaced as a flag, because a conflict between independent readers is exactly where a human's attention earns the most. We wrote up why that number stays stable on re-runs in Same deck, same score, and the full bias-control list lives on Consistency & reliability.

And the verdict is never the machine's. The AI Total Score stays advisory; the leaderboard ranks on the Jury Score a person sets after reading the evidence. A panel can make the analysis harder to bias. Only a human can own the call.

Test your own screener this week

Whatever tool reads your submissions, four checks take an afternoon.

  • Reorder. Re-run ten decks in reverse order. Count how many verdicts move. Anything above zero is position bias doing your shortlisting.
  • Reformat. Export one deck to a different template and score it again. Same content, new layout. The score should not care.
  • Re-run. Score the same deck twice an hour apart. If the number drifts, you cannot defend either reading.
  • Ask where the disagreement went. If your tool returns one clean number per deck, ask what happened to the versions of that number that did not agree. A system with no visible disagreement is not a system without disagreement. It is a system that hides it.

A screener that fails these checks still has a use. Treat it as what it is, a single judge: a fast first impression, never a ranking anyone signs.

Common questions

Is a single LLM biased as a judge? Yes, measurably. Researchers have catalogued twelve systematic bias types in single-judge setups, including position bias confirmed across 150,000+ evaluation instances and self-preference bias where a model favors text resembling its own output. These are properties of any one model, not defects of a particular vendor.

Does a bigger or newer model remove judge bias? No. The JudgeSense benchmark found that the largest and newest models are not the most consistent: paraphrasing an evaluation prompt still flips verdicts. Scale improves capability, not independence from the judge's own habits.

Why is a panel of AI judges better than one strong model? Panels of evaluators from different model families track human judgment more closely and carry less bias than a single large judge. The condition is real independence: judges that share a family or training data have correlated errors, so their votes collapse into far fewer effective opinions.

How does EvalLens handle judge disagreement? It measures it instead of averaging it away. Six independent judges score each deck against one fixed rubric; when their scores spread by 3.0 or more, the deck is flagged for human attention. The final ranking always comes from the human Jury Score, with the AI score kept as an advisory reference.

The next batch you screen will be read by someone's habits, human or machine. The only real choice is whether those habits are measured. Book a demo and watch six judges disagree about your own decks, in the open.