Reliability you can inspect, not just trust
EvalLense shows which scores stay stable, where judges disagree, and when human review is needed. It also tracks how results change across repeated runs.
An average can hide disagreement
Two decks can have the same average score. One may have broad agreement. The other may split the judges. The average alone does not show the difference.
Clear cases
Judges broadly agree on the result.
Contested cases
The same average can hide very different judge opinions.
False precision
A precise score can still carry uncertainty.
Reliable scoring is built, not assumed
Fixed criteria, independent judges, deterministic aggregation, and visible disagreement make every result easier to inspect — reliability is engineered into the pipeline, not taken on faith.
One layer is deterministic. The other is measured.
EvalLense separates the math from the judgment and holds each to its own standard.
Aggregation is deterministic.
Once judge outputs exist, the score is calculated by a deterministic aggregation function, not another LLM call. The same judge outputs and weights produce the same AI Total Score every time, with only rounding-level tolerance.
Judges are measured for repeatability.
The AI judge layer runs on a language model, so repeated runs are not always identical. We benchmark that repeatability: in repeated runs of the same deck, our latest calibration prompt reduced run-to-run variance by about 60%, and the same deck reproduced the same dimension profile in roughly 86% of runs (internal single-deck repeatability test; see benchmark evidence below).
When judges split,the report says so
EvalLense tracks the spread between judges on each dimension and turns it into a clear label. A high spread does not lower the score automatically. It routes your attention to the decks worth a closer look. It is a signal, not a penalty.
- < 1.5ConsensusJudges agree. The dimension reads the same way across the jury.
- 1.5 – 2.99SplitJudges diverge. It is worth checking where the views split.
- ≥ 3.0ConflictStrong disagreement. The report flags this dimension for human review.
Common evaluation risks have built-in checks
Each common evaluation risk maps to a built-in control, so the process relies on structure rather than goodwill.
- RiskHalo effectControlDimensions are split across separate judges.
- RiskGeneric scoringControlDimension-specific prompts and criteria per judge.
- RiskOverweighting presentationControlPitch Quality is visible, but it does not dominate the score.
- RiskHidden disagreementControlSpread plus the judge contribution matrix.
- RiskAI overreachControlThe AI Total Score is advisory. The human decides.
- RiskAssumption-fillingControlMissing evidence becomes a gap or a question, not a guess.
We benchmark the scoring and publish what we find
We rerun controlled decks and measure whether scores hold. Here is the latest repeatability run.
Benchmark scope & targets
Internal repeatability benchmark: J-P5 Team Readiness, one deck, 24 runs, June 2026. A multi-deck regression across the full panel is in progress.
Targets for the controlled set: final-score standard deviation ≤ 3 · score-band consistency ≥ 90% · critical-risk recall ≥ 90% · schema-valid outputs ≥ 99% · regression pass ≥ 95%.
EvalLense comes from 1,000+ internal evaluation runs, starting with an Amazon Nova hackathon prototype and the earlier AI Jury system.
Reliability has an honest edge
EvalLense does not promise to predict startup success. It raises the quality of evaluation by making it structured, evidence-linked, and checkable. It points you to the decisions that need human attention most — and because absolute calibration across every deck type is still being proven, the human makes the final call.
See how stable the scores are on your own decks
Book a demo and see consensus, spread, and reproducibility on a real batch.