# Pre-call dataset and protocol audit

Audit date: 2026-09-11. The source-packet corpus and the BBH companion were reviewed before the main model evaluation. A separate model-availability smoke question is not part of this dataset review. This record does not report any model performance.

## Decision

Replace BBH `date_understanding` with `word_sorting` before freezing the main sample. The reason is defective reference items, not observed model results. Retain the original date fixture and its manifest entry as evidence of the rejected category. Do not repair its labels silently, count its defective items as model errors, or describe this audit as the first discovery of BBH errors.

The retained six categories passed independent checks across all 1,500 upstream items. The selected 120 items preserve their original question text and answer targets, with a response-format instruction appended. Their seeded selection also matches the stated sampling rule.

## Defective date examples

All indices below are **zero-based** indices in the pinned [upstream date fixture](https://github.com/suzgunmirac/BIG-Bench-Hard/blob/9ee07bd481feebf959a6b59d61ea57bdcf30964d/bbh/date_understanding.json), also retained at `fixtures/bbh/date_understanding.json`. SHA-256: `0148d4ac5fca05b2f82373e5fef7208e15363b9c6079493034d77b3dab496bf5`. These five items were in the initially selected twenty date questions.

| Index | Controlling facts and requested date | Upstream target | Calculation from the question |
| --- | --- | --- | --- |
| 184 | Married January 2, 1958; today is the fifth anniversary; ask for one week earlier. | D: December 26, 1960 | Fifth anniversary is January 2, 1963; one week earlier is December 26, 1962. No option contains that date. |
| 180 | Same marriage and fifth-anniversary premise; ask for one month earlier. | D: December 2, 1960 | December 2, 1962. No option contains that date. |
| 117 | Thanksgiving 2001 is the fourth Thursday of November; ask for yesterday. | C: November 21, 2002 | The fourth Thursday is November 22, 2001, so yesterday is November 21, 2001. No option contains that date. |
| 164 | The year 2015 begins in 36 hours; ask for ten days earlier. | C: December 19, 2014 | Thirty-six hours before January 1 is December 30 at noon; ten days earlier is December 20, 2014, which is option F. |
| 211 | Same 36-hour premise; ask for one year earlier. | E: December 29, 2013 | December 30, 2013. No option contains that date. |

The date arithmetic was independently checked with Python's standard-library `datetime`. Reproduce the controlling calculations with:

```python
from datetime import datetime, timedelta

anniversary = datetime(1958 + 5, 1, 2)
print(anniversary - timedelta(days=7))  # 1962-12-26
print(datetime(anniversary.year - 1, 12, 2))  # 1962-12-02
november = datetime(2001, 11, 1)
thanksgiving = november + timedelta(days=(3 - november.weekday()) % 7 + 21)
print(thanksgiving - timedelta(days=1))  # 2001-11-21
anchor = datetime(2015, 1, 1) - timedelta(hours=36)
print(anchor - timedelta(days=10))  # 2014-12-20 12:00
print(anchor.replace(year=anchor.year - 1))  # 2013-12-30 12:00
```

## Retained BBH validation

The independently authored [validator](independent_bbh_audit.py) imports no code from the generator or model runner. It computes answers from each question without using its target, then compares the answer against the target. Unrecognized syntax, missing premises, nonunique answers, and mismatches stop execution. It performs no network calls or model calls and modifies no files.

| Category | Independent calculation | Upstream items checked | Mismatches |
| --- | --- | ---: | ---: |
| Boolean expressions | Parse an allowlisted syntax tree; evaluate Boolean operators. | 250 | 0 |
| Multistep arithmetic | Parse an allowlisted syntax tree; evaluate integer arithmetic. | 250 | 0 |
| Tracking shuffled objects, seven objects | Read initial assignments, execute all seven swaps, select the matching final object. | 250 | 0 |
| Temporal sequences | Compute waking/closing bounds and reject options overlapping occupied time intervals. | 250 | 0 |
| Logical deduction, seven objects | Enumerate all 7! orders; retain orders satisfying every premise; require exactly one option true in every retained order. | 250 | 0 |
| Word sorting | Sort the supplied words lexicographically and compare the complete sequence. | 250 | 0 |

During this audit, each of the six retained fixture files was also fetched from commit `9ee07bd481feebf959a6b59d61ea57bdcf30964d` and compared byte-for-byte to the local fixture. All matched. Offline reruns verify the recorded fixture hashes, all 1,500 targets, the six-category/twenty-item composition, the selected input/target pairs, and the hash-based sample indices:

```sh
python3 projects/landing/research/five-ai-agreement/independent_bbh_audit.py
```

Passed on the selected BBH corpus SHA-256 `c8450e049c570478c45ea4a93e9178df9e84e6850b35d47ee2b10bfe527a7083`. The upstream MIT license is retained. Formal checks reduce answer-key risk for these narrowly defined tasks; they do not establish benchmark novelty, freedom from training-data exposure, representative difficulty, or real-world reliability.

## Source-packet and protocol review

Read the complete source generator and its packet-reading oracle. No material answer-key defect was found. `python3 projects/landing/research/five-ai-agreement/corpus.py --check` passed: 120 tasks, 24 preselected repeats, corpus SHA-256 `911af339ce1ebb40953db604a1191069e282af0a6026b4da8f3c1dc5b7de5eec`. Its six families contain five templates each with four generated instances. This is a controlled fixture suite, not 120 independently designed task types.

The reviewed protocol separates source-packet and BBH results, defines unanimous accuracy and coverage, distinguishes separate requests from statistically independent errors, records failures, and labels the API setup as an adaptation of Council without tools or synthesis. Parser treatment of nonempty answers outside the expected domain was raised during review and resolved explicitly as valid-but-wrong. Task identifiers, condition names, and answer keys must remain outside model requests.

This audit is compatible with a ceiling result. If every unanimous answer is correct, retain that result with counts, uncertainty, and the limitations above. Do not add harder items after seeing model outcomes. Any later protocol amendment must preserve the original record and state whether results had already been observed.