Cases: where AI got it wrong, and where it got it right
One case per Round, written from a sealed Comparison. Every number below comes from the Full Check result and links back to the full record.
The Round 4 case comes from RealDatasetHub.com: two AI on the same pages, one perfect score and one built on invented data. The Round 1–3 cases come from the three Comparisons in which BigAIArena tested its own website. They show the Protocol applied to the Arena itself, with no exception, and they also exposed real mistakes on our pages, which we have since fixed (see Content Log).
Same pages, same questions: Grok 100, Gemini 15
Grok read every page it was asked about. It counted the 18 IP rows on the Datasets index correctly, described the single pricing plan and its JSON/CSV delivery, explained the Entry_ID format, and quoted the site’s ground-truth rule with the exact link, https://realdatasethub.com/how-it-works/ (10/10). Every Blind Spot was clean: 100/100.
Gemini, on the same pages and the same questions, invented the Datasets counts and most page details. Invented content is a Hallucination worth 0, so it finished with Access 10 · Trust 5 · Citation 0 = 15/100.
The gap is not about which AI sounds more confident. It is about which one actually opened the source and stayed with what it said.
Original record · Question-by-question audit on RealDatasetHub.com · What this means for a website
Reading the live page beats remembering it: Meta AI 80, Claude 20
Meta AI opened the site and caught a mistake that was really there: the Changelog listed version 1.9 as current while the Protocol page still showed Version 1.6. It cited the Changelog with the exact link (10/10) and correctly reported that the homepage Rankings block was only a template. It lost points on one question: it gave the right Rankings link but denied the figures on that page, so the citation scored 0.
Claude reported that robots rules blocked it and answered nothing it could not check. It invented nothing, but an honest refusal earns at most 5 of 15 on each Blind Spot and 0 on Access and Citation.
What changed on our side: the Protocol page now shows the current version, and the homepage carries real content instead of a template.
Confidence against caution: Meta AI 60, Claude 20
Meta AI answered with confidence. It was fully right about who runs the Arena and about the enterprise data page (15/15 each), but it claimed the Rankings table was empty when it held the Round 2 figures, a Hallucination worth 0.
Claude refused every question it could not verify. Refusing is safe, but it caps each Blind Spot at 5 of 15, so caution alone finished 40 points behind.
The Arena records both strategies with the same rules: a confident wrong answer scores 0, an honest refusal scores 5, and only a checked, correct answer scores 15.
Refusing is not denying: Claude said BigAIArena.com does not exist
Claude did not just decline to answer. It stated that bigaiarena.com, its comparison ledger and its scoring scale do not exist, while every one of those pages was live. Full Check scored each denial as Hallucination (0), not as a neutral refusal, and Claude finished with 10/100.
DeepSeek reached the homepage and quoted it correctly, and it explained the 5-of-15 rule for honest tool failure exactly (15/15). It also invented a fully loaded list of 17 Comparisons and “3 Phases” per Comparison, which cost it two questions.
Our own mistake in this match: the ledger list loaded only with JavaScript, so tools that do not run scripts saw “Loading results…”. The Comparisons page now also carries a static copy.
Cases are written in English from the original Vietnamese match records. When a case and its record differ, the record and its Full Check result are the reference.