To evaluate an AI startup, judge evidence that the business runs: who pays and how much each month, what it costs to operate, and what the AI does in production without a person behind it. Then score every company on one rubric, tie each score to evidence, show where reviewers disagree, and let a person make the call.

AI businesses are easy to demo and hard to judge. A prototype can look like a company, a polished deck can look like traction, and a generated answer can sound more certain than the evidence under it. Every organizer, investor and grant committee now runs into this.

We have no horse in that race. Our stake is the judging: evaluating submissions at scale, fairly, with a trail you can defend. So when a $2M prize published its rubric in the open, we read it the way a chef reads another kitchen's menu.

The bar, in three moves:
  Demo  →  Deployed  →  Operated
  2024: "best app"     2026: real customers · real revenue · AI in the loop

How do you evaluate an AI startup?

The Build with Gemini XPRIZE launched at Google I/O on May 19, 2026, with a $2M pool and a 90-day build window that closes on August 17. Judges score three criteria in equal weight: business viability, AI-native operations and category impact. Teams have to launch a real business, acquire real users and generate real revenue; projections alone will not win.

The submission asked for proof in the same spirit: revenue broken out by calendar month, total expenses and customer-acquisition spend, plus a three-minute video of the business at work. Two of the three criteria have nothing to do with the model.

Update, October 2026: the competition drew more than 26,000 registered builders, and Polyfork won the $500,000 grand prize, a business that sells 3D models as editable programs, with $4,428 in first-month revenue from 54 paying customers. The numbers are small, and they are real.

So the first test for any AI startup is the XPRIZE test, scaled down to your pile: does it run, who pays, and what does the AI do when nobody is watching?

What separates an AI business from an AI demo?

Evidence that survives a stranger. A demo works once, on stage, with its founder driving. A business works every week, for customers who pay, at a cost that leaves something over.

You can watch the bar move inside one sponsor's contests. The 2024 Gemini API Developer Competition rewarded apps (Jayu, Gaze Link, ViddyScribe) and handed out a custom electric DeLorean. Two years later, the same sponsor asked for customers and revenue by month. Same company, harder question.

Four questions sort a demo from a business, and each one has an answer you can check:

  • Who pays, and how much each month? A revenue line by month beats a market-size slide.
  • What does it cost to run? Model calls, people, acquisition spend. A business that loses more on each customer as it grows is still a demo.
  • What does the AI do in production? Name the decisions the system makes without a person. "AI-powered" on a slide names nothing.
  • What proof sits next to each claim? A logo needs a contract value. A retention figure needs a customer count.

How do you evaluate the founder of an AI startup?

On evidence, with modest hopes for anyone's crystal ball. VCBench built a test from 9,000 anonymized founder profiles and asked who would become an outlier success. A market-index baseline hit about 1.9% precision. Y Combinator did about 1.7× better than the index, and tier-1 VC firms about 2.9×. The best model in the benchmark, DeepSeek-V3, reached more than six times the baseline, and most models beat the human benchmarks.

Models matched or beat the best human pickers on this test, and the absolute hit rate still stayed low for everyone. Founder evaluation is a low-signal job, so the worst move is to let presentation, order effects or a tired reader's gut bury the little signal there is.

What you can do is judge the founder on the evidence in front of you. In EvalLens, team and founder fit is dimension P5, weighted 0.20 by default, tied with market for the highest weight. A Team Readiness judge reads for team composition, relevant experience, founder-market fit and ability to execute, and each score points to the slide behind it.

In one internal benchmark, we ran one deck through that judge 24 times across three prompt versions. The latest version returned the same dimension profile in 12 of 14 runs (85.7%) and cut the run-to-run spread of its overall score by about 59% against the previous one. That shows a steadier read of one team. It does not show that EvalLens predicts startup success.

Can AI judges evaluate an AI business fairly?

Sometimes, and only with guardrails. In the MT-Bench study, GPT-4 as a judge reached over 80% agreement with human preferences, about the rate at which humans agree with each other. AI judging works. The problem is reliability, and the failure list is long:

The FairEval authors' own fixes read like a process spec: produce evidence before the score, balance the reading order, and send hard cases to a person. That is evidence-first scoring with a human in the loop.

Do human experts agree when they evaluate business ideas?

Less than you would hope. When researchers collected roughly 3,000 expert scores across 300 patent-grounded product ideas on six business dimensions, they found substantial disagreement on fine-grained scores, while agreement rose on coarse "select or not" calls.

Some of that disagreement is structure: different investment theories and risk appetites. In the same study, AI judges personalized to one evaluator matched that evaluator better than an aggregate judge built on pooled scores. Average your panel into one number and you lose the reason they disagreed. Show the spread, and a person can read it.

What should replace gut feel when you evaluate an AI product?

A process that holds whoever reads. Both AI and human judges drift, so change what the decision is made from. Five steps work even with a spreadsheet and volunteers:

  • Write the rubric before you read. Fixed criteria and weights, decided before the first deck arrives.
  • Ask for evidence. Revenue by month, costs, named customers, what the AI does in production. Score what the material shows.
  • Read independently. Each reviewer scores alone, without seeing the others.
  • Surface the spread. Where reviewers split, look closer. That gap is information.
  • Let a person sign. The decision belongs to someone who can explain it.

That is the bet EvalLens is built on: an evidence engine for human judgment. Six independent AI judges read every submission against one shared rubric, so two startups never meet two different yardsticks. Every score ties back to the exact claim in the deck that produced it. Where judges disagree, the spread is surfaced. The AI Total Score stays advisory, and the human Jury Score ranks the field. Run after run, the default stays the same: AI prepares the analysis, and a person makes the final call.

One boundary: EvalLens prepares the screening read, the first gate that decides which companies deserve diligence time. Deal terms, references and the investment committee memo come later and stay with your team.

Why is evaluation the bottleneck for AI startups now?

Because building got cheap and judging did not. In Y Combinator's Spring 2025 batch, more than 70 of 144 startups were building agentic AI, and that is one cohort at one accelerator. As AI makes creation cheap, the pile of things that need judging grows, and the constraint moves from making to evaluating.

A $2M prize had to publish its rubric, demand a paper trail and grade operations over impressions to be taken seriously. The quiet decisions deserve the same discipline: the accelerator shortlist, the grant round, the dealflow funnel. If you run a batch of submissions and want to see evidence-linked, repeatable scoring before your next deadline, book a demo. The judging stays yours. We make it defensible.

Sources

Common questions

How do you evaluate an AI startup? Judge evidence that the business runs, not the demo. Ask who pays and how much each month, what it costs to operate, and what the AI decides in production without a person. Then score every company on one rubric, tie each score to evidence, surface reviewer disagreement, and let a person make the final call.

What separates an AI business from an AI demo? Evidence that survives a stranger. A demo works once, on stage, with its founder driving. A business works every week for paying customers at a cost that leaves margin. The Build with Gemini XPRIZE made this the bar: real users and real revenue in 90 days, with revenue reported by calendar month.

How do you evaluate the founder of an AI startup? On evidence, with modest expectations. In VCBench, built on 9,000 anonymized founder profiles, tier-1 VCs picked outliers at about 2.9 times a 1.9% market baseline. EvalLens scores team and founder fit as one of six dimensions, P5, weighted 0.20, and ties each score to the slide that supports it.

Are AI judges reliable for evaluating startups? Useful and fragile. In MT-Bench, GPT-4 agreed with human preferences over 80% of the time, about the human-to-human rate. But swapping the order of two answers flipped the winner on 66 of 80 queries in the FairEval study, and judge-bias benchmarks catalogue 12 bias types. Evidence-first scoring and human review on hard cases reduce the damage.

Are human experts better judges than AI? Not reliably. In a study of about 3,000 expert scores on 300 product ideas, experts disagreed substantially on fine-grained scores while agreeing more on coarse select-or-not calls. Part of that disagreement reflects different investment theories, so a good process shows the spread to a person instead of averaging it away.

How does EvalLens make startup evaluation repeatable? It fixes what the decision is made from. Independent judges read every submission against one rubric, each score ties to a claim in the deck, disagreement is surfaced, the AI Total Score stays advisory, and the human Jury Score ranks the field. That makes evaluation structured and reviewable. It is not a prediction of startup success.