Research
Agent evaluation usually starts after the trajectory has been captured. We ask the earlier question: what did the recording fail to preserve?
W8's first published study measures that loss directly, comparing screenshots with structured evidence from our own Chromium-based browser on the same browser-agent trajectories.
Oct 2026 / W8 DOM-diff Bench
Track 01 / Evidence fidelity
When a required value falls outside the captured viewport, when an endpoint is hidden in a structured field, or when an important control isn't legible, a screenshot-based verifier can only reason from incomplete evidence.
Our DOM-diff Optimized representation preserved the required source evidence more often in the audited paired comparison.
Exhibit 1 — Criteria with asymmetric missing source evidence. Lower is better.
| Representation | Missing-evidence criteria | Rate |
|---|---|---|
| Screenshot recording | 26 of 639 | 4.07% |
| W8 DOM-diff Optimized | 2 of 639 | 0.31% |
(26 − 2) / 26 = 92.3%
Where evidence disappeared
Exhibit 2 — Source-evidence gaps by category.
| Source-evidence gap category | Screenshot | W8 DOM |
|---|---|---|
| Exact text and values | 11 | 0 |
| URLs | 3 | 0 |
| Control states | 1 | 0 |
| Table entries | 1 | 0 |
| Label–value relationships | 0 | 1 |
| Capture omissions | 10 | 1 |
| Total | 26 | 2 |
What the recorded examples show
Task 2 / Xbox cross-site research
The screenshot sequence stopped before the developer and release-date fields. W8's structured state preserved Developed by → Tiburon and Release date → 8/13/2026. The publisher relationship stayed ambiguous, even in DOM evidence.
Task 2 audit ↗Task 19 / GitHub release research
The screenshot showed only “released this last week”. The structured state preserved the exact timestamp, 02 Sep 17:40, which matched the agent's final answer.
Task 19 audit ↗Task 18 / GitHub issue search
Screenshots rendered issue dates as relative times. The structured state preserved the exact opened dates.
Task 18 audit ↗Task 5 / George H. W. Bush Foundation event lookup
A second event sat outside the screenshot viewport. The structured state preserved it, supporting the chronological comparison the rubric asked for.
Task 5 audit ↗Record once. Each task produces a trajectory of actions and ordered browser states.
Preserve both views. Capture screenshots and DOM-diff Optimized from the same execution.
Freeze the rubric. Both verifier pathways receive the same task, action history, final answer and frozen rubric.
Audit the sources. Review whether each criterion's required information actually exists in each representation.
Count asymmetric omissions. A criterion counts against a representation only when required evidence is missing there and present in the other.
Scope of the metric
The metric covers asymmetric evidence loss. Evidence missing from both representations, agent success, human agreement with judge scores, downstream learning gains, and performance on visual tasks are separate measurements.
The audit and scoring pipeline use model-based components and manual source review. The study measures evidence preservation. Deterministic checks versus LLM graders is the next benchmark.
Engineering measurement / Separate sample
4.84 KiB
Average compact browser-state record. In a separate historical sample of 650 recorded states, DOM-diff Optimized averaged 4.84 KiB of stored text per state, versus 44.88 KiB for detailed readable DOM text and 191.32 KiB for structured snapshots.
This compares stored size per state. Inference-token usage, verifier cost, and training time are separate measurements. The sample is separate from the 100-task evidence-loss study.
Post-training
A model judge cannot recover an exact value that was never saved. A deterministic checker also needs the relevant state to have been captured in the first place.
W8's work establishes a measurable reduction in missing browser source evidence. The next engineering step is to turn that evidence into task-specific checks for observable state and outcome changes, while retaining screenshot and human/model judgment when visual or subjective criteria demand it.
The measurements we're building next
Run the same unchanged harness in SDK-only and OpenShell-based conditions. Measure which model, process, tool, egress and policy events are visible under a published attack/failure corpus; report false positives, event loss, and integration work.
Status: Protocol in preparation.
Create a human-reviewed, task-specific held-out set with exact browser outcomes. Compare DOM-based deterministic checks, screenshot judges and DOM-assisted judges on agreement, false positives, false negatives, inference cost and latency.
Status: Next benchmark. Separate from the 92.3% evidence study.
Run matched model/routing/training experiments with and without W8-derived evidence. Score actual task success, regression rate, and cost per verified completion.
Status: Future controlled study.
Sources
Bring a workflow where existing traces or screenshots fail to explain what happened. We'll compare the evidence the harness emits with what the execution environment and browser can preserve.
Discuss a benchmark run