Methodology
01 the question 02 the panels 03 the prompts, verbatim 04 sampling 05 extraction 06 canonicalisation 07 definitions 08 corrections log 09 what this does not measure A data dictionary B release policy
01 — the question
"What should I buy" is moving from search to AI. When it does, the answer arrives as a small number of brand names, in one voice, with no visible ranking and no way to see what was left out. That set of names is a shelf. It is worth a great deal and nobody was writing it down.
This measures stated recommendation, not completed purchase.
02 — the panels
FROZEN PANEL running since June 2026 arm requested resolved version claude claude-sonnet-4-6 claude-sonnet-4-6 chatgpt gpt-4o gpt-4o-2024-08-06 gemini gemini-2.5-flash gemini-2.5-flash FRONTIER PANEL began August 2026 claude claude-sonnet-5 claude-sonnet-5 chatgpt gpt-5.5-2026-04-23 gpt-5.5-2026-04-23 gemini gemini-3.7-flash gemini-3.7-flash CHINA PANEL quarterly, first collected 2026-08-15, battery china-probe-v1 deepseek deepseek-v4-flash-0731 qwen qwen3.7-flash kimi kimi-k3 glm glm-5.2
Tier is held constant deliberately. The frozen panel is Sonnet / 4o-class / Flash, and the frontier panel is one generation forward at the same tier. Moving to Opus or Pro would change tier and generation together and produce an uninterpretable result.
The two panels are never pooled. They are written to physically separate files so it cannot happen by accident.
The China panel runs quarterly, not monthly. Four Chinese models reached through OpenRouter, asked the same battery in English and then again in Chinese. Same judge as everything else, own files, never pooled with either panel.
The first collection ran 10 trials per cell, which is below the 30 the drift gate requires, so it stands as a snapshot rather than a trend. Later collections run at 30 so quarter-over-quarter movement can be tested rather than eyeballed.
Three limits. OpenRouter can route a pinned model ID to updated weights without notice, so a change between quarters may be the model moving rather than the shelf moving. The resolved model string is logged on every call, so a silent swap shows up as more than one version on an arm instead of becoming a finding. Natural deodorant is excluded: the Chinese translation read as general odour removal and started returning white vinegar and coffee grounds. Baby formula is provisional: the Chinese-character alias table does not cover it yet, because market brand names diverge there and guessing would manufacture the very result this panel exists to measure.
And I do not read Chinese. Some of the shelf change may come from translation and phrasing rather than from the language itself. If you read Chinese and want to check the prompts, please get in touch.
03 — the prompts, verbatim
Every trial is one of these, exactly as written, with no system prompt.
1/ neutral I need to buy [category]. What do you recommend? 2/ best I need to buy [category]. What is the best one? 3/ cheapest I need to buy [category]. What is the cheapest one that does the job?
Three framings run on every category: tinned fish, running shoes, natural deodorant, protein powder, baby formula, beer and coffee.
Neutral is the baseline. Best and cheapest are loaded on purpose, labelled, and reported separately. They are never averaged together.
One addition, disclosed for completeness: beer and coffee also carry a fourth branded framing (I need to buy [category]. What brand should I buy?), added to measure how much the word "brand" moves an answer. It exists in the data under prompt_key = branded. It is not part of the standard three-framing battery and is not shown in the shelf browser.
04 — sampling
trials 100 per model, per category, per framing, per month sessions one fresh stateless API call per trial. no history, no memory, no logged-in account, no system prompt sampling provider defaults. temperature is not pinned to zero, on purpose: the spread across 100 trials is the measurement, not noise to be suppressed max tokens 600 on the frozen panel, 2048 on the frontier panel. a disclosed harness difference: reasoning models can spend a small budget before emitting any visible text retries three attempts per call with 1s / 2s / 4s backoff error rows excluded, not blanked. arm row counts therefore differ slightly (6,946 / 6,930 / 6,900 across Jun-Aug) logged timestamp, month, category, provider, requested model, resolved version, battery version, framing, trial number
05 — extraction
Every answer is free text. A separate judge model reads each one and returns structured fields.
judge claude-sonnet-4-6, fixed for every arm and every month fields recommended[] every brand offered as an option chosen the single brand it most clearly recommends, or null sources[] domains cited, browse runs only accuracy 40/40 on a seeded, reproducible, hand-checked sample
The judge is never the subject. A model does not grade its own answers. The same judge is used across the frozen and frontier panels, because swapping the judge alongside the models would make extraction a confound and destroy the comparison.
The judge is instructed that section headers are not brands (an answer grouped as "Lagers: Budweiser, Coors" yields Budweiser and Coors, not Lagers), that generic product types are not brands, and to strip product lines and SKUs down to the parent brand, so "Sierra Nevada Pale Ale" becomes Sierra Nevada. If an answer names no brand at all it returns an empty list, which is a real finding rather than a failure.
06 — canonicalisation
One brand spelled two ways is two brands to a counter, and the resulting split is invisible in the output. This went undetected for three months and it is the largest error this project has made. It is worth reading the numbers in section 08.
Merging happens in three tiers, in one file, so no analysis script can drift from another.
tier 1 MECHANICAL casefold, strip accents, normalise punctuation and whitespace. always on tier 2 ROLLUP explicit product-line roll-ups. every pair was verified to co-occur ZERO times in the same answer across all 20,776 rows before being added. always on tier 3 SUBBRAND forms that DO co-occur, meaning the models treat them as distinct products. rolling these up is a decision to measure at brand level rather than SKU level. OFF by default, disclosed wherever used
Chinese-character aliases fold into tier 2. That table is a seed set and is deliberately incomplete: baby formula and natural deodorant are excluded because Chinese market brand names diverge from US ones there.
07 — definitions
recommended the brand appeared as an option in the answer chosen the answer committed to it as the single pick share percentage of all answers in a cell that named the brand. the denominator includes answers that named nothing deflection the answer named no brand at all. a property of the answer text, not a score, and not a failure cell one model x one category x one framing x one month drift gate a move is flagged when share moved >=10 points AND the two-proportion z >= 1.96 AND both months have n >= 30 "—" on site the shelf browser lists each model's top six brands only. a dash means the brand fell outside that model's top six, which is not the same as never named
08 — corrections log
2026-08-15 CANONICALISATION. Brand spelling variants were being counted separately. 50 of 263 flagged moves — 19% — were alias artifacts rather than real movement. The whole series was recanonicalised. The corrected file is the one published. 2026-08-15 POOLED DEFLECTION FIGURE RETRACTED. A draft reported a pooled 47% deflection in baby formula and read it as a generation-wide shift. The Gemini arm had not finished collecting. The real split was 82% / 13% / 0%. The average of three labs that disagree describes none of them. Pooled AI-wide deflection numbers are no longer published at all; the per-lab table replaces them. 2026-08-23 JULY DRIFT REPORT. Six rows identified as denominator artifacts and marked do-not-cite in place. One further row, tinned fish / neutral / gemini / Ramón Peña, sits in a stable cell and is NOT explained by that artifact. It is recorded as unresolved rather than quietly swept in.
09 — what this does not measure
purchases recommendation only. nothing here was bought prices not collected. "cheapest" measures what the model says is cheapest, not what is personalization the clean baseline is measured on purpose. a real account with memory and history will differ, and measuring that gap is a separate project languages English, except the quarterly China panel quality nothing here says a recommendation is good, correct, or safe. it says what the machine said
A — data dictionary
One row per answer, in shelf_structured_v2.canon.csv, available on request.
timestamp_utc when the call was made run_month the monthly snapshot this row belongs to, YYYY-MM category tinned_fish | running_shoes | natural_deodorant | protein_powder | baby_formula | beer | coffee provider claude | gpt | gemini model the model as requested model_version the version the vendor actually resolved to. these can differ, and the difference matters battery_version the prompt battery. v2.1 for this series prompt_key neutral | best | cheapest (| branded, beer and coffee only) trial 1..100 within the cell recommended brands offered as options, semicolon separated, as the judge first read them chosen the single committed pick, or empty sources domains cited. populated on browse runs only recommended_canon recommended after canonicalisation. USE THIS ONE chosen_canon chosen after canonicalisation. USE THIS ONE
Counting the uncanonicalised columns will reproduce the alias bug described in section 06. They are kept in the file so the correction is auditable, not because they are the ones to use.
B — release policy
public every extracted read, every report, every preregistration including the ones the data killed, and this page on request the raw answer text. every raw file's SHA-256 is committed in MANIFEST.sha256, so anything released later can be verified as the file collected at the time license CC BY 4.0. use it, including commercially, with credit
Corrections, disputes and replication attempts are welcome. deana@boysclub.vip