# Five-model agreement study — protocol

This file is written before evaluation calls. A freeze manifest records its hash, the corpus hashes, roster, and runner hash before the main run. The availability pilot uses a separate smoke question. The freeze is a local, timestamped record, not a third-party preregistration.

## Question and scope

Among questions receiving five valid independent first answers, how accurate is the answer when all five agree? Report unanimity coverage alongside conditional accuracy. Independence here means separate requests without seeing other answers; it does not mean statistically independent errors.

This is an API experiment inspired by Bearly Council of Experts. It reuses the expert instruction but adds a constrained answer format, supplies no tools, disables failover, and makes no synthesis call. It is not a measurement of the full shipped Council workflow, which can use tools, allows two to five experts, and sends the answers to the active chat model for synthesis.

## Fixed sample

Report two strata separately, never combine them into a headline reliability percentage:

- 120 synthetic source-packet questions: six task families, five variants per family, four seeded cases per variant. Answer keys come from deterministic rules and are independently recomputed from the rendered packets. These are small, authored situations, not a sample of customer work. Generated variants share templates.
- 120 questions from the public BIG-Bench Hard dataset: 20 deterministically selected questions in each of six prespecified categories: Boolean expressions, word sorting, logical deduction with seven objects, multistep arithmetic, temporal sequences, and tracking shuffled objects with seven objects. The source commit, item indices, seed, and license are preserved. The initially planned date-understanding category was replaced before evaluation after an independent key audit found defective questions; see PRECALL-AUDIT.md. These public questions may have appeared in training data. Original answer keys are the reference, not a claim of infallible truth, even on formally defined tasks.

An additional 24 source-packet tasks selected by the corpus generator before calls receive one fresh five-model run. Report answer stability and unanimous-answer changes separately, without expanding the primary sample size. No new difficulty variants are added after inspecting results.

## Models and requests

Five distinct model families, selected for availability in Bearly's model registry and existing provider access, not selected using scores on this corpus. The roster records exact requested provider IDs and actual response-reported IDs. Availability is checked on a separate smoke prompt before freezing. The lineup is not a claim to be the five strongest models or an equal-price comparison.

Each model sees the same task text and sources, but never the answer key or another model's response. The system message starts with Council's independent-expert instruction. The output must be one JSON object containing string fields `answer` and `reasoning`; the latter is a concise explanation. Actual request parameters and output limits are saved. Temperature is 0.7 where supported; native provider differences and reasoning defaults are disclosed.

There is no web search, code execution, calculator, tool use, few-shot example, debate, or synthesis. Output limits are fixed before the run. No paid request is automatically retried. Failed calls and invalid responses remain recorded. Running the command again resumes only unattempted requests, never overwrites failures. A conservative pre-dispatch reservation caps this study at $40, including pilots; this is an estimated upper bound based on recorded rates, not a reconciled invoice.

## Scoring

1. Parse the prescribed JSON object. Accept an optional single enclosing Markdown JSON fence. No model judges or hand-corrected responses. Any nonempty answer string is a valid answer, including an out-of-domain value; it is scored wrong unless it matches the key. Empty or non-string answers fail the contract.
2. Normalize answer strings by trimming whitespace; canonicalize integer spelling and optional parentheses around single-letter multiple-choice labels. Labels are case-insensitive. Do not extract a preferred answer from explanation prose. The executable parser is frozen with this protocol.
3. A complete panel has all five valid answers. A panel is unanimous only when all five normalized answers are identical. Unanimous accuracy = correct unanimous panels / all unanimous panels. Unanimity coverage = unanimous panels / complete panels, with coverage against all 120 scheduled tasks also stated if different.
4. Majority means at least three identical answers. A 2–2–1 or 2–1–1–1 split is no majority. For minimum agreement thresholds three, four, and five, report selected count, correct count, and coverage on the same complete-panel denominator. A correct `INSUFFICIENT` answer is a scored answer, not a missing response.
5. Report each model's correct count over valid responses, and failures over scheduled calls. Identify best observed performance only descriptively; there is no preselected winner, independent validation set, causal claim, or significance test of superiority.
6. Show every unanimous-wrong question, plus its key and all five raw explanations. If the set is empty, say so. Do not imply zero observed errors establishes reliability outside the sample.

## Uncertainty and limitations

Use Wilson 95% intervals for conditional accuracy as a descriptive binomial reference, with numerator and denominator always visible. These intervals do not account for shared templates, task-family sampling, shared training data, or benchmark contamination. Report per-family counts and leave-one-family-out sensitivity; do not advertise the interval as a universal reliability bound. No p-values or independence-based multiplication of error rates.

Costs are estimated from recorded usage and dated rate assumptions, not customer credits or invoices. Latency includes API request time under the runner's concurrency, not Bearly UI latency. No equal-compute claim. Dataset authoring, code, analysis, and copy are AI-assisted; key validation is deterministic, and the eventual PR requires human review. The authors make Bearly and have an interest in multi-model workflows. Null or unfavorable findings remain publishable.