GEO advice across question variants

A small repeated-prompt study of GEO measurement and score-validation advice for independent SaaS builders.

OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.
  • Repeated questions still produced different details. Grok’s strict named-recommendation sets varied across the three repeats. OpenAI API M1 also changed its proposed scoring rubric from 0–3 in the first two runs to −1/0–4 in the third. Generated rubrics are advice, not validated measurement standards.
  • Matched comparisons describe lower cross-wording overlap in the Grok sample. Using the same answer-pair Jaccard metric, M averaged 35.5% within wording versus 26.0% across wording; V averaged 45.5% versus 27.0%. Six within and nine cross pairs reuse six answers per intent. These descriptive post-hoc comparisons do not identify a causal wording effect.
  • The entity definition changes the headline overlap. OpenAI API M1/M2 pooled overlap is 0/7 under strict names and 1/6 when Google Analytics and explicitly named GA4 are grouped at the declared family level. This is a sensitivity result, not a correction to the original strict labels; coarse categories can increase overlap mechanically.

Repeat variation and wording variation

Mean pairwise Jaccard of recommended named-entity sets. Within-variant comparisons pool the three repeat pairs from each of two variants (6 planned pairs); across-variant comparisons use every 3 × 3 pair (9 planned pairs). M = measurement; V = validation. All bars share a 0–100% scale.

Grok Expert · consumer interface · M

Identical wording · within variants35.5% · 6/6 numeric pairs
Different wording · across variants26.0% · 9/9 numeric pairs

Across minus within: -9.4 percentage points

Grok Expert · consumer interface · V

Identical wording · within variants45.5% · 6/6 numeric pairs
Different wording · across variants27.0% · 9/9 numeric pairs

Across minus within: -18.5 percentage points

OpenAI API + web search · M

Identical wording · within variants0.0% · 5/6 numeric pairs
Different wording · across variants0.0% · 7/9 numeric pairs

Across minus within: +0.0 percentage points

OpenAI API + web search · V

Identical wording · within variantsN/A · 0/6 numeric pairs
No numeric overlap estimate; empty sets remain N/A.
Different wording · across variantsN/A · 0/9 numeric pairs
No numeric overlap estimate; empty sets remain N/A.

Across minus within: N/A

OpenAI API + web search results do not represent consumer ChatGPT Search. Environment and collection time differ.

Post-hoc descriptive analysis. The same Jaccard metric is used on both sides; pairs share answers and are not independent observations. The difference is across minus within, in percentage points. Both-empty sets are N/A; incomplete or uncoded answers are not zero. No significance test or causal effect is claimed.

What counts as the same tool matters

Pooled-union Jaccard across wording variants under three explicit mappings. Each union pools up to three eligible answers. This figure uses union overlap; it is not the pairwise mean in the previous figure. All bars share a 0–100% scale.

Grok Expert · consumer interface · M1 versus M2

Strict named entities25.9% · 7/27 intersection / union
Explicit tool families40.9% · 9/22 intersection / union
Broad functional categories100.0% · 3/3 intersection / union

Grok Expert · consumer interface · V1 versus V2

Strict named entities28.6% · 4/14 intersection / union
Explicit tool families36.4% · 4/11 intersection / union
Broad functional categories66.7% · 2/3 intersection / union

OpenAI API + web search · M1 versus M2

Strict named entities0.0% · 0/7 intersection / union
Explicit tool families16.7% · 1/6 intersection / union
Broad functional categories50.0% · 1/2 intersection / union

OpenAI API + web search · V1 versus V2

Strict named entitiesN/A · both-empty or unavailable
No numeric overlap estimate; empty sets remain N/A.
Explicit tool familiesN/A · both-empty or unavailable
No numeric overlap estimate; empty sets remain N/A.
Broad functional categoriesN/A · both-empty or unavailable
No numeric overlap estimate; empty sets remain N/A.

OpenAI API + web search results do not represent consumer ChatGPT Search. Environment and collection time differ.

The strict names are preserved. Family and category views use an explicit, reviewable mapping added after collection; they are sensitivity checks, not independent confirmation. Merging categories can mechanically increase overlap. Empty named-tool sets can still accompany substantive unbranded advice.

What this cannot show

  • No platform winner or GEO effectiveness result. Grok Expert · consumer interface and OpenAI API + web search differ in interface, retrieval, model identity and collection time; all API answers were collected after Grok.
  • No pure wording effect. The variants also change budget and action cues. Three responses per exact prompt cannot establish stable recommendation probabilities.
  • Review remains bounded. Human-review status: awaiting independent human review. Coding is assistant semantic review; assistant audits are not independent human double-coding. Source-audit status: 6 selected sources reviewed; 5 exact pages accessible; 1 exact page unavailable. Only explicitly checked claims have external review evidence.
  • Citation coverage differs. Three early Grok captures lack strict complete URL mapping. Source composition is supplementary and does not establish intrinsic platform preferences.

Implications for a GEO prompt bank

  • Keep measurement and score-validation intents separate, and preserve each exact wording version.
  • Repeat a fixed prompt before interpreting differences across variants; retain the answer, date, model/interface and search evidence.
  • Report strict entities alongside any family mapping. Keep Citation, Mention and Recommendation separate, and name the denominator.
  • Treat generated scores, weights and sampling rules as advice to evaluate, not validated measurement instruments.

Read the prompt-bank guide

Methods in brief

24 selected complete answers: two environments × four prompts × three repeats. The ledger preserves 27 actual prompt submissions, including 3 historical consumer ChatGPT UI failures or partial captures. Recorded API cost estimates total US$1.6678760; billing is not reconciled.

IntentVariants
Measurement (M)M1 asks how to reliably measure whether ChatGPT and Grok recommend a product, with no GEO budget; M2 asks for a free, repeatable checking workflow as a bootstrapped builder.
Validation (V)V1 asks how to tell whether an AI visibility score is trustworthy, with no GEO budget; V2 asks what checks a bootstrapped builder should perform before trusting reported visibility scores. Neither specifies that the score or tool itself is free.

OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.

The API requested and returned the undated identifier gpt-5.6-sol with medium effort. This does not independently establish a backend snapshot or equivalence to ChatGPT web. Grok used Expert in its consumer interface; its backend was not independently established.

The pairwise and normalization comparisons were added after the original analysis. Original pooled-union metrics remain in the appendix; they must not be compared directly with pairwise means as if they measured a causal effect.

Full protocol, original prompts and analysis plan

Technical appendix and data

The appendix retains every selected answer, all failures, coding evidence, original metrics, conditional C/M/R counts, source composition, and URL-versus-domain checks. All appendix sections open by default.

Author: Helen Bird · Experiment date: 2026-11-02 · First published: 2026-11-02 · Last revised: 2026-11-06 · Version: v1.5

Technical maintenance: consistent canonical URLs, page-specific descriptions and social metadata, truthful author/series markup, and links to individual original-answer pages. Original answers, coding, data downloads and research outcomes are unchanged. This bundled update has no measured AI visibility effect. November 6 navigation update: linked the new technical learning record; original experiment data and findings remain unchanged.

Version metadata · Revision history · Cross-report review decisions · Stage 4 review decisions · Technical maintenance