GEO advice across question variants
A small repeated-prompt study of GEO measurement and score-validation advice for independent SaaS builders.
OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.
- Repeated questions still produced different details. Grok’s strict named-recommendation sets varied across the three repeats. OpenAI API M1 also changed its proposed scoring rubric from 0–3 in the first two runs to −1/0–4 in the third. Generated rubrics are advice, not validated measurement standards.
- Matched comparisons describe lower cross-wording overlap in the Grok sample. Using the same answer-pair Jaccard metric, M averaged 35.5% within wording versus 26.0% across wording; V averaged 45.5% versus 27.0%. Six within and nine cross pairs reuse six answers per intent. These descriptive post-hoc comparisons do not identify a causal wording effect.
- The entity definition changes the headline overlap. OpenAI API M1/M2 pooled overlap is 0/7 under strict names and 1/6 when Google Analytics and explicitly named GA4 are grouped at the declared family level. This is a sensitivity result, not a correction to the original strict labels; coarse categories can increase overlap mechanically.
Repeat variation and wording variation
Grok Expert · consumer interface · M
Across minus within: -9.4 percentage points
Grok Expert · consumer interface · V
Across minus within: -18.5 percentage points
OpenAI API + web search · M
Across minus within: +0.0 percentage points
OpenAI API + web search · V
Across minus within: N/A
OpenAI API + web search results do not represent consumer ChatGPT Search. Environment and collection time differ.
Post-hoc descriptive analysis. The same Jaccard metric is used on both sides; pairs share answers and are not independent observations. The difference is across minus within, in percentage points. Both-empty sets are N/A; incomplete or uncoded answers are not zero. No significance test or causal effect is claimed.
What counts as the same tool matters
Grok Expert · consumer interface · M1 versus M2
Grok Expert · consumer interface · V1 versus V2
OpenAI API + web search · M1 versus M2
OpenAI API + web search · V1 versus V2
OpenAI API + web search results do not represent consumer ChatGPT Search. Environment and collection time differ.
The strict names are preserved. Family and category views use an explicit, reviewable mapping added after collection; they are sensitivity checks, not independent confirmation. Merging categories can mechanically increase overlap. Empty named-tool sets can still accompany substantive unbranded advice.
What this cannot show
- No platform winner or GEO effectiveness result. Grok Expert · consumer interface and OpenAI API + web search differ in interface, retrieval, model identity and collection time; all API answers were collected after Grok.
- No pure wording effect. The variants also change budget and action cues. Three responses per exact prompt cannot establish stable recommendation probabilities.
- Review remains bounded. Human-review status: awaiting independent human review. Coding is assistant semantic review; assistant audits are not independent human double-coding. Source-audit status: 6 selected sources reviewed; 5 exact pages accessible; 1 exact page unavailable. Only explicitly checked claims have external review evidence.
- Citation coverage differs. Three early Grok captures lack strict complete URL mapping. Source composition is supplementary and does not establish intrinsic platform preferences.
Implications for a GEO prompt bank
- Keep measurement and score-validation intents separate, and preserve each exact wording version.
- Repeat a fixed prompt before interpreting differences across variants; retain the answer, date, model/interface and search evidence.
- Report strict entities alongside any family mapping. Keep Citation, Mention and Recommendation separate, and name the denominator.
- Treat generated scores, weights and sampling rules as advice to evaluate, not validated measurement instruments.
Methods in brief
24 selected complete answers: two environments × four prompts × three repeats. The ledger preserves 27 actual prompt submissions, including 3 historical consumer ChatGPT UI failures or partial captures. Recorded API cost estimates total US$1.6678760; billing is not reconciled.
| Intent | Variants |
|---|---|
| Measurement (M) | M1 asks how to reliably measure whether ChatGPT and Grok recommend a product, with no GEO budget; M2 asks for a free, repeatable checking workflow as a bootstrapped builder. |
| Validation (V) | V1 asks how to tell whether an AI visibility score is trustworthy, with no GEO budget; V2 asks what checks a bootstrapped builder should perform before trusting reported visibility scores. Neither specifies that the score or tool itself is free. |
OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.
The API requested and returned the undated identifier gpt-5.6-sol with medium effort. This does not independently establish a backend snapshot or equivalence to ChatGPT web. Grok used Expert in its consumer interface; its backend was not independently established.
The pairwise and normalization comparisons were added after the original analysis. Original pooled-union metrics remain in the appendix; they must not be compared directly with pairwise means as if they measured a causal effect.
Technical appendix and data
The appendix retains every selected answer, all failures, coding evidence, original metrics, conditional C/M/R counts, source composition, and URL-versus-domain checks. All appendix sections open by default.
Read the full technical appendix
findings.json
Entity normalization map
Human-review record
Source-audit record
Review response
Prompt-bank guide
Download public data
Author: Helen Bird · Experiment date: 2026-11-02 · First published: 2026-11-02 · Last revised: 2026-11-06 · Version: v1.5
Technical maintenance: consistent canonical URLs, page-specific descriptions and social metadata, truthful author/series markup, and links to individual original-answer pages. Original answers, coding, data downloads and research outcomes are unchanged. This bundled update has no measured AI visibility effect. November 6 navigation update: linked the new technical learning record; original experiment data and findings remain unchanged.
Version metadata · Revision history · Cross-report review decisions · Stage 4 review decisions · Technical maintenance