# Run a local experiment

Bring your agent runtime. Preserve the task, environment, and outcome needed to interpret each trial.

## The experiment contract

An experiment manifest records a hypothesis, a task, baseline and candidate variants, the agent configuration, an independent verifier, and a fixed repetition count. The framework validates the manifest before execution. Keep the manifest in version control beside the environment it describes.

```sh
evx experiment init ./my-experiment
evx experiment validate ./my-experiment/manifest.json
evx experiment run ./my-experiment/manifest.json --output ./results
```

Built-in Codex and Claude Code adapters are read-only in this release: Codex uses its read-only sandbox and Claude is limited to reading and searching files. An explicit custom command can integrate other trusted local runtimes. A fresh copied workspace provides repeatability; it does not isolate an arbitrary custom command from the host.

Local execution requires macOS or Linux, including WSL. Native Windows execution stops before starting subprocesses or creating trial artifacts because this release cannot enforce Windows process-tree cleanup. Analysis, context inspection and account commands do not require local experiment execution.

## Verify the outcome independently

The verifier runs outside the agent's workspace and returns a structured result. It should inspect the actual task outcome: for example, execute acceptance tests or check an artifact against a specification. Do not substitute the agent's confidence or final message for verification.

```json
{ "passed": true, "reason": "The output satisfies the task specification", "metrics": {} }
```

Independent process execution prevents accidental coupling; it does not make an arbitrary verifier correct or tamper-proof. Review what the verifier can observe, and test it against intentionally wrong outputs.

The manifest supports positive and negative verifier controls. When configured, they run before the first agent trial; a verifier that accepts the wrong-answer control fails qualification.

## Read the comparison

Our [API workflow experiment](/evidence/production-api-handoff-2026-09-08) shows eight real Codex trials using captured production responses. It also shows why verifier coverage matters: the original verifier rejected valid quotations. The recorded verdicts remain unchanged, with a separate assessment of every answer and no claimed performance improvement.

The runner randomizes baseline/candidate order within each pair from a recorded seed. Every trial starts from a fresh copy. Agent errors and timeouts count as failures. Skipped or cancelled work remains visible and prevents a complete comparison claim. Verifier errors also suppress inference because the outcome was not reliably measured.

The reported effect is the candidate success fraction minus the baseline success fraction. The conservative 95% Hoeffding interval assumes independent pairs. The exact two-sided discordant-pair test is descriptive; it does not correct for trying many hypotheses or establish generalization.

Repetitions range from 2 to 100 per variant in early access. Small runs are useful for diagnosing the setup, but will often be inconclusive. Choose a sample size based on the effect that matters and the available budget, not on whether an early result looks favorable.

## Save automatically

```sh
evx auth status
evx projects list
evx experiment run ./my-experiment/manifest.json --output ./results --project PROJECT_ID
```

With evx 0.7.2 or later, a connected host saves the declaration, result and bounded private trace attachments directly to the selected project. Run `evx auth login` once if the host is not connected. Create a project with `evx projects create --name 'Interface experiments'`, or use the project ID shown on its web page. The website is for inspecting the evidence; no file upload or pasted JSON is required.

The manifest still belongs in your local project beside its surface and verifier files. Running it executes those declared commands with your agent/provider account and may incur costs. Results and captured streams can include prompts, source content and agent answers. To keep the experiment on the host, replace `--project PROJECT_ID` or `--experiment EXPERIMENT_ID` with `--local-only`.

```sh
evx experiment run ./my-experiment/manifest.json --output ./results --experiment EXPERIMENT_ID
```

Use `--experiment` for an existing declaration, with its matching local manifest and workspace. A downloaded manifest does not include the files needed to execute it. If a save fails after execution, keep the output directory and use the exact retry command returned by evx; do not repeat the trials to retry a save.

Saved results are immutable. A changed intervention needs a new declaration. The service validates the submitted trial contract and recomputes its comparison; it does not attest that external execution occurred or certify the verifier's coverage. Original trace files remain on the host; the [trace receipt](#traces) describes the private copies saved with this experiment.

## Inspect the recorded output

The experiment page links to the available trial and verifier-control streams. The CLI collects only files named by the saved result, checks their original hashes and bytes, and preserves the local originals. The immutable result stays separate from its attachments. Captured output is not a recording of every model-visible input or all host activity; external observer artifacts are not retained by this attachment version.

Each attachment can contain at most 2 MiB decoded and 1,400,000 bytes compressed as gzip, with 16 MiB decoded across one experiment. Account and service storage, request and daily transfer limits also apply. The API returns current limits with the trace list; the usage view reports retained capacity. Files beyond either per-file limit remain explicit omissions rather than silently truncated evidence.

Before saving, the CLI detects selected credential syntax, structured credential fields and nonempty environment values whose names indicate credentials. Ordinary operational token settings are not treated as secrets. Detection is limited: it does not guarantee that every secret was removed. A changed copy carries a separate hash, redactor version and replacement counts. Unchanged copies retain their exact original bytes. If safe redaction is uncertain, that file is omitted and the reason is retained in the receipt.

Inspect cloud.traces in the CLI response for saved, missing or unconfirmed files. If attachment saving fails after the result was saved, keep the local output directory and retry the returned upload command. This does not rerun trials. Use --local-only in place of the cloud destination flag to keep new experiments entirely on the host.

```sh
evx experiment upload ./results/result.json --experiment EXPERIMENT_ID
```