LLM as a judge
LLM-as-a-judge is the use of a large language model to evaluate work against defined criteria in place of a human reviewer, producing a score, a label or a preference along with a written rationale. The term comes from AI engineering, where models grade the outputs of other models. A second application is emerging: evaluating documents written by people, such as grant applications and pitch decks.
The technique is the same. The stakes are not.
When the judged party is a person, the rules change.
Three things separate judging applications from judging model outputs. First, there is no ground truth. A dev-tooling judge is validated against labeled answers; an application has no correct score, so reliability has to come from consistency and cited evidence rather than accuracy against a key.
Second, the consequences land on a person. A skewed eval score mistunes a prompt. A skewed application score can end a funding round, which turns one judge’s habits from a metrics problem into a fairness problem.
Third, the loser may ask why. A rejected applicant is entitled to a reason, so the rationale has to cite the document itself, which is the discipline defined under evidence-grounded scoring. None of these constraints exist in the term’s original home, and tooling built for that home does not carry them.
Built as a panel, never as the decider.
EvalLens applies the technique to submissions under exactly those constraints. No single LLM judge: six independent judges read every submission in parallel, so one model’s habits cannot set the tone. Disagreement above the spread threshold is flagged, aggregation is deterministic math outside the model, every finding cites the deck, and the output is advisory. The human jury scores and decides. AI prepares the analysis, people decide.