# Analysis plan (written 2026-09-27, before any gold-standard label or outcome statistic was seen)

The owner decided not to register on OSF. This file takes that role: it fixes the decision rules before the
annotations and outcome analyses exist. `drafts/listicles/preregistration.md` remains the full specification of
estimands and models. Any deviation is reported in the study's methodology as a deviation.

## Annotation

Two independent model annotators (A: Claude Opus; B: Claude Sonnet) code all 299 gold items, blind to the detector.
Disagreements are adjudicated by the analyst (Claude) reading the page text, with each decision logged in
`annotations/listicles/adjudication.csv`. Reported as **model-coded validation, not human validation**.

## Choosing the parsing rule (dev half only)

Candidates for numbered-heading pages: R-a (v1), R-b, R-c, and R-d-first (ItemList when present, else R-b).
Publisher names: domain label only (v1) vs label + page aliases.

1. Primary criterion: balanced accuracy (mean of sensitivity and specificity) of **self-first** classification over
   all dev items in the numbered strata, weighted by stratum weights.
2. Tie-break (within 0.02): higher accuracy of the first entry among gold ranked lists.
3. Then prefer the simpler rule, in the order R-a, R-b, R-c, R-d-first, and label-only over aliases.

The chosen rule is frozen and reported on the test half (sensitivity, specificity, PPV, with Wilson CIs, weighted).

## Coverage extension (sensitivity, not headline)

For heading-only and no-structure pages: ItemList, then `<ol>`, then first table column. Validated on the gold items
of those strata (dev and test pooled, because they are not used to choose anything). Reported beside the
numbered-only headline, never merged into it.

## Correction for detector error

Rogan–Gladen only if the test-half Se + Sp − 1 ≥ 0.5 and both CI half-widths ≤ 0.15; reported as a sensitivity estimate.

## Outcome analyses

As in `preregistration.md` (RQ1–RQ4, units, θ₁/θ₂, the Study C estimands, robustness list, Holm on the family
{θ₂ surface interaction, RQ3 primary, RQ4 date effect}). The RQ4 date effect needs the date-2 wave (on or after 3 Oct 2026).
All analyses are labelled **not pre-registered**.

## Deviations (logged as they occurred)

1. 2026-09-27, before any outcome estimate was used: the query–page–publisher graph formed one giant connected
   component (3,228 of 3,588 nodes), so a component bootstrap cannot produce intervals. Intervals therefore resample
   publishers (page-level units) and queries (citation-event and answer units), the two sensitivity clusterings named
   in the plan. Surface tests use publisher clusters.
2. 2026-09-27: the mention rule removes citation-style links before matching (whole name, citations removed) because
   the model-coded validation sample showed citation labels counted as mentions. Chosen after seeing that sample;
   reported with its validation figures.
3. 2026-09-27: a defect in the answer ids (AI Overview and AI Mode SERPs for the same keyword shared an id) was found
   and fixed before any result was reported; all tables were rebuilt.