BigAIArena.com · Changelog

Changelog — Protocol version history

The Protocol changes through real Comparisons, not theory. Each version below reflects one operational lesson. Current version: 2.8 (2026-11-02).

VersionDateSummary
2.82026-11-02Drawn from Comparisons #104–#112 and the AI Weak Points page. One change to the question set, no new scoring rule. (1) Q4 and Q8 now read the website’s AI Weak Points page, where every row is a verified error an AI made on that website in a sealed Comparison. Q4: the Respondent is asked about the latest weak point recorded for itself, with one wrong premise to correct, through the 3-gate test. Q8: it cites one weak point recorded for its opponent, with the link. If either AI has no row yet, another AI with a row is named; if the page has fewer than 2 rows, Q4 and Q8 use the AI Audit page as in 2.7. Q6 stays on the AI Audit page and may not repeat a homepage fact. (2) Worked examples added to the scoring cards, without new rules: a mistyped URL that returns 404 is not a hallucination; a citation from an AI that says it could not read the page scores 0; quoting the arena’s own results table for a question about the website’s page is outside content (cap 5); widening one record into a general range is a hallucination. (3) Reviewers and the Secretary never paste temporary file links into a ballot and never copy another ballot; public links to the website quoted from an answer are normal evidence. (4) The Full Check records, as a side metric, whether the scheduled Access and Blind package held up. Applies from Comparison #113.
2.72026-10-25Drawn from Comparisons #101–#103. One scoring change and a fixed page for every question. (1) Every question now has a fixed page: Q1 the homepage, Q2 the data page, Q3, Q5 and Q7 the pillar page (Q7 a required citation), Q4, Q6 and Q8 the AI Audit page (Q8 a required citation). Q6 asks about a fact on the website’s content recorded in an earlier sealed result. The pillars are what the website must pass on; the AI Audit page is recursive data that grows after every Comparison. An AI that cannot read these pages scores low. (2) Two Access and Blind packages are prepared in parallel. The package check marks every line of both and picks the sharper one, stating why; a package with a wrong line is never used. (3) Scoring change, Gate 3: only a real fact from another page of the same website earns 10. Content from outside the website, even if true, and answers with no content to check (a blank or generic refusal) are capped at 5. No outside source has to be verified any more. (4) Worked examples: a row count taken from another site’s index, without seeing the real table, scores 0 on Q2; the Secretary must build its own summary table, never copy one Reviewer’s ballot. (5) Each website’s AI Audit page now shows two lists for every Comparison: what the AI got right, which is how AI introduces and cites the website to users, and what it got wrong or missed, which is the debate that sends users to check the source. The final review adds one line: what the website should improve next. Applies from Comparison #104.
2.62026-10-22No new scoring rules. A full update of how each Comparison is prepared, and of every tested website. (1) One Access and Blind AI per Comparison instead of two, plus one Standby AI that steps in only if a Review or Secretary seat fails; every substitution is logged. (2) A separate account checks every line of the package against the live pages before the Comparison starts, marking it right, wrong or weak; a package with a wrong line is not used, and the check is kept in the record. (3) Three questions in every Comparison — two Blind Spots (Q3 and Q5) and the first citation (Q7) — test the pillars each website publishes on its own pillar page, so every Comparison checks what the website most wants AI to get right; Q6 still comes from another page of the website. The issues table and the benefits block on each AI Audit page, and website footers, are never used as test facts. (4) Full Check may propose a Protocol change — what, where, the exact wording and why it keeps the Protocol lean — and adds a comment of up to three sentences; a proposal never changes the result it comes from. (5) A worked example of a question that gives away its own answer, after it happened three times (#098–#100). (6) The 5-point score for an AI that honestly reports its own tool failure is labelled as such, not as Flattery. Each AI Audit page now lists earlier Comparisons one line each, shows the latest in full, and adds what each Comparison asked the website to fix. Applies from Comparison #101.
2.52026-10-18No new scoring rules. Six clarifications drawn from Comparisons #092–#098, so that every role applies the existing rules the same way. (1) The final check now settles minor issues itself and records each one in a single line: what happened, how it was handled and which rule covers it. Only situations that no rule covers and that could change the result go back to the Arena Master. Outside information that cannot be verified, for example a page that blocks automated readers, stays at the 5-point cap, decided by the final check without manual lookups. (2) A worked example for Garbage Substitution: an AI that cannot open the page asked about, quotes a real line from another page of the same website and says so earns 10 of 15, not 0, 5 or 15. Outside information earns 10 only when it is specific and can be verified. (3) Existing limits restated in every scoring card: wrong content about the source in Question 1 earns 0; partial credit in Question 2 needs proof that the real table was seen; a standard Markdown link or a tracking parameter on the same domain is a valid URL; every score sheet’s total must equal the sum of its eight lines. (4) The results summary re-adds every score sheet first; a sheet whose total does not match its lines is left out entirely, with the reason shown. (5) Questions never contain their own answers: the two citation questions ask for a detail without stating it, the four Blind Spots are four different facts that never repeat the citation fact, and each one is asked exactly once, in order, in Questions 3–6. (6) A test is finalised only when both interview transcripts are complete in the record. Applies from Comparison #099.
2.42026-10-14Three clarifications drawn from Comparisons #088–#090, plus one change to how results are published. (1) Question 2, the data page: an AI that clearly saw the real data table but got only part of the question right, for example a wrong row count, now earns 5 of 10; a total access failure, an invented entry, or calling a table empty when it holds data still earns 0. (2) Questions 3–6: when an AI cannot read the page asked about but gives real facts from another page of the same website, and says so, this counts as Garbage Substitution (10 of 15), the same as real outside data. (3) Preparing each test: only real data rows are counted on the data page, and partner promotion text in a website footer is never used as a test fact. (4) The /realdataset/ and /ai-audit/ tables are moving to tables built on the server from each website’s public sheet, so every reader and every AI tool sees the same up-to-date record without running JavaScript. Applies from Comparison #091.
2.32026-10-12Four changes, made before the first Comparison that uses fresh data. (1) The two fresh-data questions are now standard on every website tested, Core IP or Partner: every website has a public /ai-audit/ page before it is tested, so Question 4 always asks about the most recent sealed result on that website’s own page and Question 8 always asks the AI to cite that page. The 2.2 switch that turned these questions on or off for each website, and the one-way rule that let a Partner borrow a Core IP’s page, are no longer needed and have been removed. (2) Both questions now name the AI Audit page openly. This is the fair test: an AI that can reach and read the live page scores well, and an AI that cannot reach it, or answers from memory, does not. (3) The fact used in Question 4 is copied from the page exactly as published, with the time it was read, so the ground truth of each Comparison is the page that readers could see at that moment. (4) An account that answers as a Respondent is never used for any other role, and the accounts that hold any other role never answer as a Respondent. We also checked the main data page used in Question 2 directly on all 19 websites and listed each one in the Protocol.
2.22026-10-07Two changes that bring fresh data into every test, plus one clarification. (1) AI-Audit questions: each IP now publishes the Sealed results of its own Comparisons on a public /ai-audit/ page. When that page can be read without JavaScript and holds at least one Sealed Comparison, Question 4 asks about a fact from the most recent Sealed result on it, and Question 8 asks the AI to cite that page. That fact only exists after the previous Comparison, so an AI cannot answer from old stored knowledge; it has to read the live page. If an IP’s page is not ready yet, both questions stay as ordinary content questions. (2) One-way cross-links: a Partner IP may ask about a result published on a Core IP’s AI-Audit page, but a Core IP never asks about a Partner page, because some Partner sites block AI crawlers and the score would then measure the site’s settings instead of the AI. (3) Clarification: Question 2 checks each IP’s main data page. Most IPs use /realdataset/, but RealDatasetHub.com, the data hub for the whole ecosystem, uses /datasets/. The Protocol now lists this explicitly instead of assuming one address for every IP.
2.12026-10-05Two clarifications from Comparisons #080–#082, plus one routine that is now written down instead of handled case by case. (1) Who confirms access at the end: the closing access check may be done by the AI originally assigned to that role or by a different AI. Both are valid, and a different one is simply labeled as a substitute. The check happens after the final review, so the final reviewer no longer treats an empty closing block as missing. (2) Account labels: after the same account number was copied into the wrong report three times, every report now names the AI only. Account numbers appear in one place, the final results table, and a line that only records who confirmed access can never be read as the account of the AI being tested. (3) A documented routine for platform limits: when a tested AI hits a message limit mid-test, we continue on another account of the same AI. Question 1 is asked again on the new account only so the model learns which website is being checked. That second answer is never scored and is not added to the record: the original Question 1 answer stands, whether it was right or wrong. The remaining questions then follow in order, since the questions are almost entirely independent of each other.
2.02026-10-05A larger revision than recent steps, because it rewrites an existing rule’s wording rather than only adding to it. After five Comparisons in a row repeated the same citation mistake, we traced the cause to our own text: one old sentence said a link earns points regardless of whether it was used to confirm a fact or to admit nothing could be found. That contradicted the intended rule and led reviewers to credit empty citations. That sentence is gone, replaced by a clear two-part check: the link must be written correctly, and the content it points to must actually have been cited. A correct link used only to say “I couldn’t retrieve anything” now scores zero, separate from the mid-tier credit for a specific-but-wrong page guess, because a citation measures the real evidence produced, not how precisely a link was aimed. We also spelled out four scoring boundaries that reviewers kept blurring in practice: a total failure to reach a source scores zero at the access stage (the partial credit for an honest tool failure belongs only to the Trust questions); a statement that is false about the source is Hallucination, not Garbage Substitution; the three-gate test applies only to the four Blind Spot questions; and vague, unverifiable industry knowledge is not enough to reach the higher Garbage Substitution tier. Finally, every role card now names the default account explicitly (“Acc01”), and each role must check account numbers independently instead of copying them from whoever wrote before, after one typo was found copied through four consecutive reports.
1.92026-09-23Tightened 2 scoring boundaries: denying a fact on a page the AI actually reached no longer gets the same minimum-credit treatment as an honest “couldn’t access this” — it’s judged on its own merits instead; and a generic homepage link with no real attempt to point at the right page now scores zero, distinct from a specific-but-wrong page guess. Also closed a recurring labeling gap — when a reviewer mislabels who they were grading (but the round is still correct), the content is now checked against both parties’ real answers before anything gets thrown out, instead of an automatic discard. Several smaller process fixes: clearer formatting for independent-audit reports, tighter criteria for when a routine correction needs sign-off versus just being logged, and consistent crediting when one party fills in for another mid-process.
1.82026-09-21A citation now needs to be right, not just present — a link pointing to the correct spot no longer earns credit if the text attached to it is wrong or contradicts what the source actually says, since a confidently-placed wrong citation is more misleading than no citation at all. Also locked in that the facts recorded in a Comparison’s opening evidence check are fixed for that Comparison — a later re-check only confirms a link is still live, never overturns an already-recorded fact, which matters most for partner sources whose pages can change same-day. Narrowed an over-broad reading of the “wrapped link” rule so it no longer mistakenly catches ordinary same-site links. And closed a gap where one reviewer’s own internal math error could get silently averaged into a result instead of being excluded and explained.
1.72026-09-17When one AI substitutes for another mid-Comparison, the substitute must now clearly label its own content as a substitution at the top of what it writes — not just rely on a note elsewhere in the record. A small clarity fix, closing a gap where a stale name could sit next to genuinely new work.
1.62026-09-14Closed a leniency drift found across several Comparisons: an honest admission of tool failure alone was being credited too generously. A polite, truthful “I couldn’t verify this” now caps at the minimum passing score for that question — reaching a higher score still requires the AI to actually provide real substitute evidence, not just an honest admission. Also reinforced an existing but frequently-missed rule (a URL to the right site but wrong page scores a mid-tier credit, not zero) and closed a gap where an AI could insert an unauthorized “audit” block for itself.
1.52026-09-06Secretary now must produce 2 separate first-tier score tables — one per Phase, never merged or omitted — and must state a reason inline whenever a source is excluded from the average, closing a gap where a Comparison’s total could be computed from an incomplete, unexplained set of reviewers.
1.4Date not recordedFormatting update for video production and readability: every Comparison record closes each scoring block with a large, clearly separated Access/Trust/Citation/Total summary line, and the Comparison header lists each role on its own line.
1.32026-09-059 clarifications from Comparisons #003–008: Independent Audit’s role narrowed to reporting verified facts only (not gate classification); a citation must match the exact concept and location asked, not just “URL exists + content is real” independently; a URL only counts if attached to already-cited content, never a promise to cite later; and — most significantly — any role substitution now requires its own distinct content block, never a relabeled copy of the replaced party’s original numbers, protecting the accountability trail itself.
1.2Date not recordedAccess Gate’s “0 for total access failure” explicitly distinguished from Trust’s separate “self-reported tool failure = 5” rule; a URL inside a markdown link is scored by its actual target destination, not its displayed text.
1.12026-09-01Added that falsely claiming to have verified content directly against the source counts as Hallucination even when the content happens to be accurate.
1.02026-08-31Official public launch. The 8-question/3-criteria (Access/Trust/Citation) architecture, the 3-gate Blind Spot mechanic (Hallucination/Sycophancy/Garbage Substitution), Independent Audit on every Comparison, and Full Check as the sole official record — all built and refined through an extensive internal testing period before this launch.

The Full Protocol page always shows the current version. A detailed internal changelog of the testing period before launch is kept by the Arena Master; this page starts at the official launch.

Changes to website content (not to scoring rules) are recorded separately, with the Comparisons they affect, on the Content Log.

No Evidence. No Score.