# A source-index comparison reached a ceiling on this lookup task

On 7 September 2026 we ran a predeclared experiment against selected files from Evals.AX website commit `69ab3276b7cbf6ec6ef28e2732c372a08aa29232`. The task was to retrieve exact shipped authentication facts. Both conditions contained the same source and architecture documentation. The candidate additionally contained a short `AGENTS.md` index linking to relevant files. It did not add answer facts.

Both conditions succeeded on all 12 trials. The estimated success difference was zero. The conservative 95% paired Hoeffding interval was **−0.7841 to +0.7841**, and the descriptive discordant-pair p-value was 1. The result is **inconclusive**. The task was easy enough to reach the measurement ceiling, and the interval cannot rule out substantial differences on repeated draws. This is not evidence that indexing never matters or that the conditions are equivalent.

| Observed measure | Baseline | Candidate with index |
| --- | ---: | ---: |
| Independently verified success | 12/12 | 12/12 |
| Mean agent duration | 25.75 seconds | 32.89 seconds |
| Reported input tokens, sum | 641,053 | 835,919 |
| Reported output tokens, sum | 2,439 | 2,867 |
| Completed shell commands | 24 | 36 |
| Nonzero command exits | 0 | 19 |
| Reported dollar costs | Unknown | Unknown |

Timing and token usage were predeclared as descriptive secondary measures. We did not preregister an inferential efficiency test. The candidate's observed duration was 27.7% higher and input usage 30.4% higher, but these figures should not be presented as a general efficiency effect. Cache behavior and host/provider variation were not fully controlled. Codex did not report a dollar amount; unknown billing is not a zero-cost run.

The traces exposed an important limitation of our selected-file fixture. Every candidate attempted to read an API route file that was not included in the snapshot, then recovered and returned the correct answer. The index itself linked only files that were present. Twelve nonzero exits were missing-file reads; seven were searches that found no matches. A negative search result is not necessarily an error. The baseline instead listed the selected files and searched their contents. These observations cannot establish what a direct index would do in the complete repository.

This is also a concrete lesson for the framework: final task success hid recoverable retrieval failures. We added provider-event diagnostics that preserve command counts and nonzero exit positions in the private trace, while keeping raw command text out of uploaded result records. Unknown diagnostic coverage stays unknown for adapters without an event stream. We also added optional positive and negative verifier controls; an always-passing verifier now fails qualification before an agent runs. The frozen study result has not been rewritten to pretend these later fields were recorded at execution time. Derived diagnostics are a separate artifact.

## Protocol and provenance

The run used 12 pairs with randomized within-pair order, seed `20260907`, requested model `gpt-6-astra`, low reasoning and local Codex CLI `0.153.0-alpha.5`. Isolated configuration retained the host's authentication while avoiding its configuration file and session persistence. Model weights are provider controlled; the requested model identifier is not an immutable weight digest. Each trial used a fresh snapshot copy and a read-only agent sandbox. The verifier ran separately and checked every required fact against a fixed expected answer. It included a positive control and single-field wrong-answer controls in its own source.

The 24-trial plan was fixed before the first run. There were no retries, outcome exclusions or optional stopping. Each agent had a 45-second limit; the total limit was 15 minutes. Execution ran from `08:14:18.474Z` to `08:26:03.203Z` and completed in 704.7 seconds. The preregistered manifest SHA-256 is `cc6afc0f7888bfcbd201fa93b70f18c11289f55d34591a7add6bce2dfe646fef`. The report ID is `4574f3f9-9395-441d-bb11-01e7191f1b32`.

All 24 trace hashes were verified locally after execution and again after the framework's backward-compatible schema changes. A bounded inspection found no credential-pattern matches or unexpected network/credential commands. Raw traces remain private. The structured report and derived summaries are client-reported evidence, not independent cloud attestation. Compiled runner module hashes were retained separately before subsequent implementation changes.

An earlier four-run fictional configuration lookup qualified the real Codex adapter, host authentication and independent-verifier path. It was a separate integration qualification, not part of this study; it also produced an inconclusive success comparison. It was not pooled with this task or used to extend its sample size.

The next scientific question needs representative tasks and complete environments with a verified context inventory. We retain this study because it shows why a plausible-looking index needs measurement, and because it exposed a measurement gap that we repaired. It neither establishes nor falsifies the broader hypothesis that some index designs help on some tasks.

## Artifacts

- `auth-context-result.json` retains the frozen manifest, every trial and trace hashes.
- `auth-context-summary.json` retains preregistration time, compiled module hashes and the analysis produced by the frozen implementation.
- `auth-trace-diagnostics.json` is a separate, explicitly post hoc analysis using the new diagnostic extractor.

The actual source snapshots remain in the private repository and are excluded from the npm package. The publishable package includes the separate fictional `context-discovery` example.