Roles and Rotation
Eight AI fill the rotating roles of every Comparison, one role each; a fixed group of working accounts prepares, audits and finalises it; and every assignment is logged. Part of Protocol 3.9.
Three account groups
| Group | Who | Roles | Never |
|---|---|---|---|
| Rotating roles | Acc01 of each AI (Claude: Acc01B), with a Role Card for each role | Examiner, Standby 1–2, assigned by the Round schedule | Answering as Respondent; any fixed working role |
| Respondents | Acc02–Acc12 of each AI, rotated; each account is independent and shares no memory with the others | Respondent only. No Role Card, no package, no answer key, no sign of being tested | Any other role |
| Fixed working roles | One named account per role, outside the Round schedule | Package writer — Grok Acc14A · Package check — Grok Acc14B · Independent Audit — Grok Acc15A · Live Verification — Grok Acc15B · Full Check — Claude Acc15A · Assistant (outside scoring) — Claude Acc01A | Respondent, Examiner, any second role |
The fixed working accounts are fully independent of the accounts being tested. Sharing a brand with a Respondent is not a conflict: what is scored is the tested account’s answer, and what is checked is the evidence in the record and on the live page. Errors of a same-brand AI are listed to the same standard, and when Full Check overturns the Audit in favour of its own brand, Live Verification is required first. Each fixed operating role is held by one named, dedicated account for the whole version, so the same checks are applied the same way in every Comparison. Rotating roles use Acc01 of each AI (Claude: Acc01B). Respondents answer only from Acc02–Acc12. A fixed or rotating account never answers as Respondent, and a Respondent account never holds any other role. Account numbers are published; e-mail addresses are not.
Website maintenance is outside the Comparison. Updating site content and checking live pages after an update hold no role in any match, never score, and never touch a website from the package snapshot until the record is sealed. Every site edit is logged on the Content Log.
The roles
| Role | Seats | Job | Not allowed |
|---|---|---|---|
| Package writer | 1 (fixed) | Opens every page the questions use, with a timestamp; writes one package table — for each of the 10 questions the error tested, the target URL, three equivalent wordings, the string that passes and, where needed, the string that fails. If a page cannot be opened, the Comparison is not opened | Checking its own package; scoring |
| Package check | 1 (fixed) | Re-opens every page and every source of a popular figure and checks every row of the package; its check is recorded in the sealed record, and the package that passes is hashed before the first question | Rewriting the package; scoring |
| AI Arena | 2 | The two tested AI take turns as Examiner (picks one of three approved wordings per question, asks it word for word, then scores its Phase on the 10 questions) and Respondent (answers from a Respondent account) across 2 Phases | The Respondent receives no links, files or hints |
| Standby AI | 2 | Does nothing unless an Examiner fails, stalls or breaks format; then takes over that role, and the substitution is noted on the spot in the record | Taking a role that has not failed |
| Independent Audit | 1 (fixed) | Scores both Phases with live-page evidence for every result | Holding any other role |
| Full Check | 1 (fixed) | Checks every scorer line by line, asks Live Verification what is not yet certain, and issues the official result and the weak-point list; its first output that asks nothing more is the Final Result | Holding any other role |
| Live Verification | 1 (fixed) | Opens the real pages to answer Full Check word for word — the last check of a Comparison, on the live page — checks the published pages, and reviews challenges. Called Access & ReCheck up to version 3.6 | Scoring |
“AI Alpha” and “AI Beta” are only record-keeping labels for the two tested AI. Examiner and Respondent are reversed in Phase 2, on different accounts.
Why the roles are kept apart
| Principle | Rule |
|---|---|
| Package integrity | The package writer and the package check account never answer, ask or review in that Comparison. |
| Single-blind barrier | Only the Respondent is unaware; it never sees the package. |
| Independent scoring | The Examiner scores without seeing any other score; the Independent Audit scores both Phases with its own live-page evidence; every scorer scores only pages it opened itself. |
| Separate appeal | Challenges are reviewed by Live Verification, an account of a different brand from Full Check. |
| Role | Sees the answer key | Knows a Comparison is running |
|---|---|---|
| Package writer | Yes (writes it) | Yes |
| Examiner | Yes (may not reveal it) | Yes |
| Respondent | No | No |
| Independent Audit, Full Check, Live Verification | Yes | Yes |
The 8-AI roster
The roster is ordered by global user base and locked at the start of each quarter.
| Rank | AI |
|---|---|
| 1 | ChatGPT |
| 2 | Meta AI |
| 3 | Gemini |
| 4 | Claude |
| 5 | Copilot |
| 6 | DeepSeek |
| 7 | Grok |
| 8 | Perplexity |
Source: Similarweb worldwide web-visit share, cross-checked with company-published figures where available.
Round-robin, roles reversed
Every AI meets every other AI once per Round. Each Comparison covers both directions through its 2 Phases, with Phase 2 reversing the roles.
C(8,2) = 28 unique pairings = 28 Comparisons per Round
Pace. The planned pace is about 4 Comparisons a day, roughly one week per Round. Rounds 1–3 actually took between about 8 and 17 days each (Round 1: 2026-09-01 to 2026-09-17; Round 2: 2026-09-17 to 2026-09-24; Round 3: 2026-09-25 to early October). Dates per Comparison are on Comparisons.
Single-blind: the Phase 1 Respondent does not learn it was evaluated before Phase 2 begins on a different account.
Filling the other six seats
- Two of the six AI not in the match fill Standby 1–2, one seat each, by the Round schedule. Since Protocol 3.7 there are no Review seats: scores are settled against an answer key fixed before the Comparison.
- The package is written by a fixed working account (since 3.0) and checked by a second fixed account, Package check (Grok Acc14B), whose check is part of the sealed record.
- The schedule never overrides the separation rules above.
Account rotation
- 4 accounts per Comparison for the two tested AI. Each tested AI uses its working account as Examiner and a Respondent account as Respondent; no account is both aware and unaware.
- Respondent accounts are kept apart. They are used for nothing else, and no other account ever answers as a Respondent.
- Rotation between cycles. Respondent accounts rotate, and no account is reused within the same cycle. If a Respondent hits a platform limit mid-test, it continues on the next account of the same AI.
- Why. Repeated observation across models, roles, accounts and cycles helps separate model behaviour from account history.
Logging and sealing
When Full Check issues its result, the Comparison is Sealed and the Arena Master computes a SHA256 hash of the sealed record, so any later change to it can be detected. The sealed record names the AI, the role and the account for every operating seat.
Quarterly roster review
- Each quarter the roster is checked against the published user-base source.
- If an AI drops out of the Top 8, its replacement enters from the next cycle; a cycle is never cut short.
- Data of a replaced AI is never deleted and stays in Rankings and Cases.
- The replacement takes the outgoing AI’s place in the rotation from the next cycle.