AI businesses are easy to demo and hard to judge. A working prototype can look like a company. A polished deck can look like traction. An AI-generated answer can sound more certain than the evidence beneath it. So when a $2,000,000 prize sets out to find the best AI business in the world, the interesting question isn't who ships the flashiest model. It's the question every organizer, investor, and grant committee is quietly failing at right now: how do you actually judge?

We don't have a horse to cheer for here. We care about the judging — evaluating things at scale, fairly, with a trail you can defend, is the entire reason EvalLense exists. So when Google publishes its rubric in the open, we read it the way a chef reads another kitchen's menu.

The bar, in three moves:
  Demo  →  Deployed  →  Operated
  2024: "best app"     2026: real customers · real revenue · AI in the loop

Demo is not a business

The Build with Gemini XPRIZE launched at Google I/O on May 19, 2026, with a $2M pool and a 90-day window that closes August 17. Stage one is pass/fail. Stage two scores three things in equal weight — Business Viability, AI-Native Operations, and Category Impact — and asks for revenue by month, cost evidence, customer evidence, and agent logs. The judges, notably, aren't required to test your product. They grade the evidence you submit.

Two of those three criteria have nothing to do with the model. The bar moved from "show me it works" to "show me it runs, who pays, and what the AI actually does." That is the whole industry growing out of its demo phase — and it lands everyone in the same hard problem: once you're judging evidence instead of demos, the quality of your decision depends entirely on how disciplined your judging is.

AI judges are useful — and fragile

Start with the good news, because it's real. A strong model can approximate human preference at scale: in the MT-Bench study, GPT-4 as a judge reached over 80% agreement with human raters — about the rate at which humans agree with each other. AI judging is not useless. The problem is reliability, not capability.

Now the bad news, which is worse than most people assume. The same judges are dangerously sensitive to things that have nothing to do with quality:

And here's what's striking: the FairEval authors' own fix is to generate the evidence and justification before assigning a score, balance the order, and route hard cases to a human. The leading research on AI judging quietly describes evidence-first scoring with a human in the loop. That's not a coincidence. It's where the problem forces you.

Humans don't agree either

The tempting conclusion — "so use human experts instead" — doesn't survive contact with the data. When researchers collected roughly 3,000 expert scores across 300 patent-grounded product ideas on six business dimensions, they found substantial disagreement on fine-grained scores, even as experts agreed more on coarse "is this good or not" calls.

And that disagreement isn't all noise. Some of it is structure — different investment theories, different risk appetites, different views of what matters. Which means the goal of a good evaluation system isn't to erase disagreement. It's to make it visible, so a human can read it instead of having it averaged away.

Even elite selection is low-signal

It gets humbling. When LLMs were benchmarked on predicting startup success in VCBench, the absolute precision of identifying outlier outcomes was tiny across the board: a market-index baseline around 1.9%, Y Combinator roughly 1.7× that, and tier-1 VC firms about 2.9× the index. Even the best startup pickers in the world operate in a low-signal regime.

That is not an argument against judgment. It's an argument for evidence discipline. If the signal is this faint, the worst thing you can do is let it get buried under presentation, order effects, and a tired reader's gut.

What should replace vibes

So the real picture isn't "AI judge versus human judge." Both are unstable: the companies are changing weekly, and the judges — silicon or carbon — bring their own bias. The answer is not to crown one of them. It's to change what the decision is made from.

That's the bet EvalLense is built on. It isn't an AI judge; it's an evidence engine for human judgment. Independent reviewers read every submission against one shared rubric, so two startups are never measured by two different yardsticks. Every score is tied back to the exact claim in the deck that produced it. Where reviewers disagree, the spread is surfaced, not smoothed over. The AI total stays advisory; the human's score is the one that ranks the field. The AI prepares the case. The human owns the verdict. Methodology over model, with the final call always yours.

This isn't theory for us. Across 500+ internal pitch-deck review runs, the property we've worked hardest to earn is repeatability. In one controlled benchmark — the same deck scored 24 times — our latest scoring iteration returned the identical dimension profile in 12 of 14 runs (85.7%) and cut run-to-run score variance by roughly 59% against the previous version. We're deliberate about what that does and doesn't mean: this is not a claim that EvalLense predicts startup success. It's a claim that pitch evaluation can be made more structured, evidence-linked, repeatable, and human-reviewable. Given everything above, that's the goal worth chasing.

Why now

Because the volume is about to break the old way of doing it. At Y Combinator's Spring 2025 batch, 70 of 144 startups were building agentic AI — and that's one cohort, at one accelerator. As AI makes building cheap, the number of things that need judging explodes, and the bottleneck moves from creation to evaluation.

You can watch the bar move in Google's own contests. The 2024 Gemini API Developer Competition rewarded beautiful apps — Jayu, Gaze Link, ViddyScribe — and handed out a custom DeLorean. Two years later, the XPRIZE rewards a business with customers and revenue. Same sponsor, harder question.

A $2M prize had to publish its rubric, require a paper trail, and grade operations over impressions to be taken seriously. Quiet decisions — the accelerator shortlist, the grant round, the dealflow funnel — deserve the same discipline. That's the part we'd love to show you: if you run a batch of submissions and want to see what evidence-backed, reproducible scoring looks like before the next deadline lands, book a demo. The judging stays yours. We just make it defensible.

Sources