Methodology
the agent shelf · battery v2.1 · last revised 2026-08-23
01 the question 02 the panels 03 the prompts, verbatim 04 sampling 05 extraction 06 canonicalisation 07 definitions 08 known failure modes 09 corrections log 10 what this does not measure A data dictionary B release policy
01 — the question
"What should I buy" is moving from search to AI. When it does, the answer arrives as a small number of brand names, in one voice, with no visible ranking and no way to see what was left out. That set of names is a shelf. It is worth a great deal and nobody was writing it down.
This measures stated recommendation, not completed purchase. Recommendation is what actually happens today, it can be collected under controlled conditions, and it avoids the checkout and terms-of-service problems that come with buying things. Purchase is the harder measurement and it comes later.
02 — the panels
Two panels run in parallel. New models are never swapped into an existing panel, because a swap changes the model and the time series at the same moment and you can no longer tell which one moved.
FROZEN PANEL running since June 2026 arm requested resolved version claude claude-sonnet-4-6 claude-sonnet-4-6 chatgpt gpt-4o gpt-4o-2024-08-06 gemini gemini-2.5-flash gemini-2.5-flash FRONTIER PANEL began August 2026 claude claude-sonnet-5 claude-sonnet-5 chatgpt gpt-5.5-2026-04-23 gpt-5.5-2026-04-23 gemini gemini-3.7-flash gemini-3.7-flash
The two are chain-linked through August 2026, a month in which both panels ran the identical battery, plus a same-day re-run of the frozen models to separate a generation effect from elapsed time. That overlap is the only thing that lets a frontier number be compared to a frozen one.
Tier is held constant deliberately. The frozen panel is Sonnet / 4o-class / Flash, and the frontier panel is one generation forward at the same tier. Moving to Opus or Pro would change tier and generation together and produce an uninterpretable result.
Snapshots over aliases. Where a vendor offers a dated snapshot it is pinned, because an evergreen alias can rotate underneath you and a rotating arm is not a controlled arm. One exception is disclosed: gemini-3.7-flash is a bare version ID rather than a dated snapshot, so Google could update it in place. A silent rotation shows up as more than one resolved version on that arm, which is checked before every report.
The two panels are never pooled. They are written to physically separate files so it cannot happen by accident.
03 — the prompts, verbatim
Every trial is one of these, exactly as written, with no system prompt.
neutral I need to buy [category]. What do you recommend? best I need to buy [category]. What is the best one? cheapest I need to buy [category]. What is the cheapest one that does the job? branded I need to buy [category]. What brand should I buy?
Categories are tinned fish, running shoes, natural deodorant, protein powder, baby formula, beer and coffee. Running shoes reads "the cheapest pair that does the job." The branded framing runs on beer and coffee only, and it exists to measure how much the word "brand" moves an answer, which is why neutral is still collected alongside it.
Neutral is the baseline. The other three are loaded on purpose, labelled, and reported separately. They are never averaged together.
04 — sampling
trials 100 per model, per category, per framing, per month sessions one fresh stateless API call per trial. no history, no memory, no logged-in account, no system prompt sampling provider defaults. temperature is not pinned to zero, on purpose: the spread across 100 trials is the measurement, not noise to be suppressed max tokens 600 on the frozen panel, 2048 on the frontier panel. a disclosed harness difference: reasoning models can spend a small budget before emitting any visible text retries three attempts per call with 1s / 2s / 4s backoff error rows excluded, not blanked. arm row counts therefore differ slightly (6,946 / 6,930 / 6,900 across Jun-Aug) logged timestamp, month, category, provider, requested model, resolved version, battery version, framing, trial number
05 — extraction
Every answer is free text. A separate judge model reads each one and returns structured fields.
judge claude-sonnet-4-6, fixed for every arm and every month fields recommended[] every brand offered as an option chosen the single brand it most clearly recommends, or null sources[] domains cited, browse runs only accuracy 40/40 on a seeded, reproducible, hand-checked sample
The judge is never the subject. A model does not grade its own answers. The same judge is used across the frozen and frontier panels, because swapping the judge alongside the models would make extraction a confound and destroy the comparison.
The judge is instructed that section headers are not brands (an answer grouped as "Lagers: Budweiser, Coors" yields Budweiser and Coors, not Lagers), that generic product types are not brands, and to strip product lines and SKUs down to the parent brand, so "Sierra Nevada Pale Ale" becomes Sierra Nevada. If an answer names no brand at all it returns an empty list, which is a real finding rather than a failure.
06 — canonicalisation
One brand spelled two ways is two brands to a counter, and the resulting split is invisible in the output. This went undetected for three months and it is the largest error this project has made. It is worth reading the numbers in section 09.
Merging happens in three tiers, in one file, so no analysis script can drift from another.
tier 1 MECHANICAL casefold, strip accents, normalise punctuation and whitespace. always on tier 2 ROLLUP explicit product-line roll-ups. every pair was verified to co-occur ZERO times in the same answer across all 20,776 rows before being added. always on tier 3 SUBBRAND forms that DO co-occur, meaning the models treat them as distinct products. rolling these up is a decision to measure at brand level rather than SKU level. OFF by default, disclosed wherever used
Chinese-character aliases fold into tier 2. That table is a seed set and is deliberately incomplete: baby formula and natural deodorant are excluded because Chinese market brand names diverge from US ones there, and guessing would manufacture the very result the China probe exists to measure.
07 — definitions
recommended the brand appeared as an option in the answer chosen the answer committed to it as the single pick share percentage of all answers in a cell that named the brand. the denominator includes answers that named nothing deflection the answer named no brand at all. a property of the answer text, not a score, and not a failure cell one model x one category x one framing x one month drift gate a move is flagged when share moved >=10 points AND the two-proportion z >= 1.96 AND both months have n >= 30
08 — known failure modes
Every one of these produced a wrong number that was published or nearly published.
The share denominator versus the drift gate. Share counts every answer in a cell, including the ones that named nothing. So when a model's deflection rate inside a cell moves, every brand in that cell moves with it, and each of those moves clears the gate independently. One event gets reported as five or six separate brand findings. This is what happened to the July tinned fish and coffee rows. Future reports need a deflection-aware guard.
Alias cycles. An alias table can form a two-cycle, where A maps to B and B maps back to A. The lookup then returns a different key depending on which spelling it started from, and the brand silently un-merges. There is now an acyclicity assertion that runs before every report.
A failed parse looks like a deflection. If extraction fails after its retries it writes an empty brand list, which is indistinguishable in the output from a model that genuinely named nothing. The blank rate was audited and found to be real deflection rather than silent failure, but the two are not yet distinguishable by field, and they should be.
Thin cells. Some cells barely contain brands at all. In August, asked for the cheapest tinned fish, claude named no brand in 98 of 100 answers and chatgpt in 96 of 100. Percentages computed on a base like that move violently and mean very little.
09 — corrections log
2026-08-15 CANONICALISATION. Brand spelling variants were being counted separately. 50 of 263 flagged moves — 19% — were alias artifacts rather than real movement. The whole series was recanonicalised. The corrected file is the one published. 2026-08-15 POOLED DEFLECTION FIGURE RETRACTED. A draft reported a pooled 47% deflection in baby formula and read it as a generation-wide shift. The Gemini arm had not finished collecting. The real split was 82% / 13% / 0%. The average of three labs that disagree describes none of them. Pooled AI-wide deflection numbers are no longer published at all; the per-lab table replaces them. 2026-08-23 JULY DRIFT REPORT. Six rows identified as denominator artifacts and marked do-not-cite in place. One further row, tinned fish / neutral / gemini / Ramón Peña, sits in a stable cell and is NOT explained by that artifact. It is recorded as unresolved rather than quietly swept in.
10 — what this does not measure
purchases recommendation only. nothing here was bought prices not collected. "cheapest" measures what the model says is cheapest, not what is personalisation the clean baseline is measured on purpose. a real account with memory and history will differ, and measuring that gap is a separate project languages English, except in the separately published China probe quality nothing here says a recommendation is good, correct, or safe. it says what the machine said
A — data dictionary
One row per answer, in shelf_structured_v2.canon.csv.
timestamp_utc when the call was made run_month the monthly snapshot this row belongs to, YYYY-MM category tinned_fish | running_shoes | natural_deodorant | protein_powder | baby_formula | beer | coffee provider claude | gpt | gemini model the model as requested model_version the version the vendor actually resolved to. these can differ, and the difference matters battery_version the prompt battery. v2.1 for this series prompt_key neutral | best | cheapest | branded trial 1..100 within the cell recommended brands offered as options, semicolon separated, as the judge first read them chosen the single committed pick, or empty sources domains cited. populated on browse runs only recommended_canon recommended after canonicalisation. USE THIS ONE chosen_canon chosen after canonicalisation. USE THIS ONE
Counting the uncanonicalised columns will reproduce the alias bug described in section 06. They are kept in the file so the correction is auditable, not because they are the ones to use.
B — release policy
public every extracted read, every report, every preregistration including the ones the data killed, and this page on request the raw answer text. every raw file's SHA-256 is committed in MANIFEST.sha256, so anything released later can be verified as the file collected at the time not public the collection harness. this is a record of what the models said, not a tool for running your own licence CC BY 4.0. use it, including commercially, with credit
Corrections, disputes and replication attempts are welcome. deana@boysclub.vip