When five AIs agree, how often are they right?
Five model families. Two sets of 120 questions. Every unanimous answer was correct—but agreement covered fewer questions than one model solved alone. See the data.
On our document questions, 103 out of 103 unanimous answers were correct. On a separate sample of public reasoning questions, the result was 106 out of 106.
The catch is in how often all five agreed. On the document test, unanimity covered 103 of 120 questions. Six produced disagreements. Eleven did not produce a complete five-model panel. And the Google model in this run answered all 120 correctly on its own.
Those details belong next to the percentage.
We make Bearly, where Council of Experts lets you inspect several models' answers alongside a synthesis. We wanted to measure what agreement itself tells you. This experiment tested independent first answers through model APIs, using an adaptation of Council's expert instruction. It did not test the complete Council workflow or its synthesis.
The questions, answer keys, raw responses, and scoring code are open. The sample and scoring rules were fixed before the main run. We kept the failures.
A 100% score needs a denominator
Suppose five models agree on one question, get it right, and disagree on the next 99. Their unanimous accuracy is 100%. They have settled one question.
We measured both accuracy when unanimous and coverage: how often unanimity happened.
For the document questions, 109 panels produced five valid answers. All five agreed on 103 of them: 94.5% of complete panels, or 85.8% of all scheduled questions. Every agreed answer matched the key.
A rule requiring at least four matching answers covered all 109 complete panels and was also correct on every one. Requiring the fifth vote withheld six additional correct answers without improving the observed accuracy on this set.
The reasoning sample produced 110 complete panels. Of those, 106 were unanimous: 96.4% of complete panels, or 88.3% of all 120 questions. Requiring at least four matching answers covered all 110 complete panels, also with every answer correct.
This is not evidence that five calls outperform a good single model. Here are the individual results from the September 11 run, shown as correct answers over valid responses. We label the tested models by provider here; the downloadable results preserve their exact model IDs. Failed calls are accounted for below.
| Tested model's provider | Documents | Reasoning |
|---|---|---|
| OpenAI | 116/120 | 116/120 |
| Anthropic | 119/120 | 115/115 |
| 120/120 | 120/120 | |
| xAI | 119/119 | 118/118 |
| DeepSeek (via Together) | 109/110 | 116/116 |
These are small, selected samples. Fable 5.1 and GPT 6 Astra were not included in this experiment; these are not scores for those models. We did not test whether five calls beat an equal budget spent on one model.
A disagreement worth opening
One document task supplied six cohorts for a conversion report. Every row was marked draft. The rules said to include only final rows and return INSUFFICIENT if the eligible denominator was zero.
What belongs in the report?
Show the five answers and the check
The Anthropic, Google, xAI, and DeepSeek models in the recorded run returned INSUFFICIENT. The OpenAI model returned 5,469 basis points (54.69%).
Including all six draft rows gives 1,353 conversions from 2,474 deliveries: 54.69%. But none of those rows qualifies for the report. After the required filter, the denominator is zero. The correct answer is INSUFFICIENT.
This is task rates-05-02 in the downloadable dataset, selected here to illustrate a disagreement. The explanations are preserved with the original responses.
That is a practical reason to keep individual answers visible. The disagreement gives you a specific rule to check. A polished number can still come from the wrong rows.
Before testing the models, we checked the test
Our initial reasoning sample included twenty date questions from BIG-Bench Hard. An independent computational review found five defective reference items among those twenty.
One described a couple married on January 2, 1958, celebrating their fifth anniversary. It asked for the date one week earlier. That is December 26, 1962. The published key pointed to December 26, 1960. The correct date was missing from the options.
A model answering correctly could have been marked wrong. Five models answering correctly could have become a dramatic, false example of unanimous failure.
We replaced the entire date category with word sorting before the main run. Then a separate program calculated answers for all 1,500 questions in the six retained categories. Every retained key matched. The audit preserves the rejected questions, calculations, and validation methods.
We are not claiming these annotation problems are newly discovered. We are showing why an answer key needs checking before it becomes evidence against a model.
What this result can tell you
We observed zero unanimous errors in the document sample. We also observed zero in the reasoning sample. Outside these samples, the error rate remains unknown.
The document set contains six families of short, fictional work tasks: invoices, conversion reports, policy revisions, corrected facts, dependencies, and inventory records. Five templates per family generated four instances each. These are explicit, mechanically checkable situations, not a representative sample of customer work.
The reasoning questions come from BIG-Bench Hard, built on BIG-Bench. They have been public since 2022 and may have appeared in training data. Its historical name does not establish difficulty for these models.
Different model names also do not guarantee unrelated errors. In a 2025 study of correlated model errors, pairs of models chose the same wrong answer about 60% of the time when both were wrong on the study's HELM questions. That is a different denominator from ours; it is not a 60% error rate for agreed answers.
Methods, failures, and uncertainty
We ran 1,320 evaluation calls on September 11, 2026: five models on each of 240 questions, plus fresh five-model repeats of 24 prespecified document tasks. Eight availability-pilot attempts were separate. The protocol, corpus, roster, and scorer were hashed and committed before evaluation; this was a local freeze, not an externally registered preregistration.
Every model received the same question and source text without seeing the other answers. No web search, calculator, code execution, or other tools were available. Native requests had a 2,048-token output cap; reasoning settings and temperature differed by provider and are preserved in the roster. DeepSeek ran through Together. Requested and returned model identifiers were checked. There was no automatic retry or failover.
Agreement means the same normalized final answer, not similar prose. We required a JSON answer and concise explanation. Any nonempty answer string was scored, including wrong or out-of-domain values. Formatting failures were retained separately. Across 1,200 primary calls, 22 failed the response contract or transport: 17 formatting failures, two incomplete outputs, and three timeouts. Repeat calls added five failures. A panel needed five valid answers to enter agreement scoring; incomplete panels still count against coverage of scheduled questions.
On the repeat subset, four of 112 paired valid answers changed. Among 16 panels complete in both runs, three changed unanimity status. Thirteen were unanimous both times; none changed its unanimous answer. Repeats were close together and do not measure long-term stability.
The descriptive 95% Wilson interval for document accuracy when unanimous is 96.4%–100%; for reasoning it is 96.5%–100%. These binomial reference intervals do not account for shared templates, selected task families, or training-data exposure. They are not reliability guarantees. Family counts and leave-one-family-out sensitivity are included in the summary.
AI assisted with the task templates, code, independent computational checks, analysis, and writing. We do not claim independent human adjudication. Bearly has a commercial interest in multi-model workflows. The data supports the stated sample results; it does not establish performance on open-web research, private files, or the full Council product.
Bring the evidence into the conversation
In Bearly, /council gathers answers from two to five selected experts on Pro and higher. You can open each model's answer as well as the synthesis. For a question about a document, include the relevant source material and ask each expert to identify the rule or passage supporting its conclusion.
If your next comparison includes Fable 5.1 or GPT 6 Astra, use the same approach: give each model the source, inspect its answer, and check the evidence. The scores above describe the recorded run, not a forecast for a new lineup.
When the answers differ, check that evidence. When they match, keep the same standard. A second opinion is most useful when you can inspect it.
Download the complete replication package · Open the answer CSV · Read the protocol · See Council of Experts