Technical appendix
Stage 3: question sensitivity in GEO advice. Original analyses and complete evidence records.
Return to the concise reader report
OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.
24 selected complete answers · 24 eligible assistant coding records · 27 actual prompt submissions.
Collection and coding coverage
24/24 selected bodies and 24/24 assistant semantic coding records are complete/eligible. OpenAI API + web search: 12/12; Grok Expert · consumer interface: 12/12. Actual submissions: 12 API + 12 Grok + 3 historical ChatGPT UI = 27. Historical failures: 1 technical, 2 incomplete capture. These are collection counts, not success rates.
| Platform | Prompt | Complete bodies | Coded eligible | Tool sets | Mapped domains | Observed domains | Actual search · complete bodies only | Primary answer types |
|---|---|---|---|---|---|---|---|---|
| OpenAI API + web search | M1 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Manual measurement workflow": 3 }, "denominator": 3 } |
| Grok Expert · consumer interface | M1 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Manual measurement workflow": 3 }, "denominator": 3 } |
| OpenAI API + web search | M2 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Manual measurement workflow": 3 }, "denominator": 3 } |
| Grok Expert · consumer interface | M2 | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Manual measurement workflow": 3 }, "denominator": 3 } |
| OpenAI API + web search | V1 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Score validation or audit checklist": 3 }, "denominator": 3 } |
| Grok Expert · consumer interface | V1 | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Score validation or audit checklist": 3 }, "denominator": 3 } |
| OpenAI API + web search | V2 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Score validation or audit checklist": 3 }, "denominator": 3 } |
| Grok Expert · consumer interface | V2 | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 3 | 3 / 3 | { "yes": 3, "no": 0, "unknown": 0, "complete_body_denominator": 3 } | { "counts": { "Score validation or audit checklist": 3 }, "denominator": 3 } |
Original identical-prompt repeat analysis
17/24 planned tool-set pairs have numeric Jaccard values. Each cell has R1–R2, R1–R3, R2–R3. Both-empty sets are N/A, not 0% or 100%; sparse named-tool sets are not by themselves an instability finding. Missing pairs remain unavailable. The observed-domain column is a separate coverage-qualified view.
M1 · OpenAI API + web search
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 0.0% (0 / 1)Set sizes: 1 / 0 | 50.0% (1 / 2)Set sizes: 1 / 2 | 50.0% (1 / 2)Set sizes: 1 / 2 |
| R1 ↔ R3 | 0.0% (0 / 5)Set sizes: 1 / 4 | 50.0% (1 / 2)Set sizes: 1 / 2 | 50.0% (1 / 2)Set sizes: 1 / 2 |
| R2 ↔ R3 | 0.0% (0 / 4)Set sizes: 0 / 4 | 33.3% (1 / 3)Set sizes: 2 / 2 | 33.3% (1 / 3)Set sizes: 2 / 2 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 0.0%–0.0%; mean 0.0%, from 3 / 3 planned pairs; 0 both-empty N/A | None observed within this eligible set |
| Cited domains · mapping complete | 33.3%–50.0%; mean 44.4%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
| Observed inline cited domains · coverage qualified | 33.3%–50.0%; mean 44.4%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
M1 · Grok Expert · consumer interface
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 35.7% (5 / 14)Set sizes: 7 / 12 | 0.0% (0 / 4)Set sizes: 4 / 0 | 0.0% (0 / 4)Set sizes: 4 / 0 |
| R1 ↔ R3 | 44.4% (4 / 9)Set sizes: 7 / 6 | 57.1% (4 / 7)Set sizes: 4 / 7 | 57.1% (4 / 7)Set sizes: 4 / 7 |
| R2 ↔ R3 | 50.0% (6 / 12)Set sizes: 12 / 6 | 0.0% (0 / 7)Set sizes: 0 / 7 | 0.0% (0 / 7)Set sizes: 0 / 7 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 35.7%–50.0%; mean 43.4%, from 3 / 3 planned pairs; 0 both-empty N/A | Bing Webmaster Tools, Google Analytics 4, Google Search Console, HubSpot AEO Grader |
| Cited domains · mapping complete | 0.0%–57.1%; mean 19.0%, from 3 / 3 planned pairs; 0 both-empty N/A | None observed within this eligible set |
| Observed inline cited domains · coverage qualified | 0.0%–57.1%; mean 19.0%, from 3 / 3 planned pairs; 0 both-empty N/A | None observed within this eligible set |
M2 · OpenAI API + web search
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 0.0% (0 / 2)Set sizes: 0 / 2 | 33.3% (1 / 3)Set sizes: 2 / 2 | 33.3% (1 / 3)Set sizes: 2 / 2 |
| R1 ↔ R3 | N/A: both eligible sets are empty. | 100.0% (2 / 2)Set sizes: 2 / 2 | 100.0% (2 / 2)Set sizes: 2 / 2 |
| R2 ↔ R3 | 0.0% (0 / 2)Set sizes: 2 / 0 | 33.3% (1 / 3)Set sizes: 2 / 2 | 33.3% (1 / 3)Set sizes: 2 / 2 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 0.0%–0.0%; mean 0.0%, from 2 / 3 planned pairs; 1 both-empty N/A | None observed within this eligible set |
| Cited domains · mapping complete | 33.3%–100.0%; mean 55.6%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
| Observed inline cited domains · coverage qualified | 33.3%–100.0%; mean 55.6%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
M2 · Grok Expert · consumer interface
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 41.7% (5 / 12)Set sizes: 10 / 7 | Unavailable: one or both sets are ineligible or unrecorded. | 0.0% (0 / 3)Set sizes: 3 / 0 |
| R1 ↔ R3 | 22.2% (4 / 18)Set sizes: 10 / 12 | Unavailable: one or both sets are ineligible or unrecorded. | 0.0% (0 / 9)Set sizes: 3 / 6 |
| R2 ↔ R3 | 18.8% (3 / 16)Set sizes: 7 / 12 | 0.0% (0 / 6)Set sizes: 0 / 6 | 0.0% (0 / 6)Set sizes: 0 / 6 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 18.8%–41.7%; mean 27.5%, from 3 / 3 planned pairs; 0 both-empty N/A | Google Sheets, Reddit, X |
| Cited domains · mapping complete | 0.0%–0.0%; mean 0.0%, from 1 / 3 planned pairs; 0 both-empty N/A | Unavailable · requires all 3 eligible answers |
| Observed inline cited domains · coverage qualified | 0.0%–0.0%; mean 0.0%, from 3 / 3 planned pairs; 0 both-empty N/A | None observed within this eligible set |
V1 · OpenAI API + web search
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | N/A: both eligible sets are empty. | 50.0% (1 / 2)Set sizes: 2 / 1 | 50.0% (1 / 2)Set sizes: 2 / 1 |
| R1 ↔ R3 | N/A: both eligible sets are empty. | 50.0% (1 / 2)Set sizes: 2 / 1 | 50.0% (1 / 2)Set sizes: 2 / 1 |
| R2 ↔ R3 | N/A: both eligible sets are empty. | 100.0% (1 / 1)Set sizes: 1 / 1 | 100.0% (1 / 1)Set sizes: 1 / 1 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | No numeric comparison · 3 both-empty N/A; 0 unavailable | None observed within this eligible set |
| Cited domains · mapping complete | 50.0%–100.0%; mean 66.7%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
| Observed inline cited domains · coverage qualified | 50.0%–100.0%; mean 66.7%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com |
V1 · Grok Expert · consumer interface
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 18.2% (2 / 11)Set sizes: 8 / 5 | Unavailable: one or both sets are ineligible or unrecorded. | 11.1% (1 / 9)Set sizes: 7 / 3 |
| R1 ↔ R3 | 25.0% (3 / 12)Set sizes: 8 / 7 | Unavailable: one or both sets are ineligible or unrecorded. | 12.5% (1 / 8)Set sizes: 7 / 2 |
| R2 ↔ R3 | 50.0% (4 / 8)Set sizes: 5 / 7 | 25.0% (1 / 4)Set sizes: 3 / 2 | 25.0% (1 / 4)Set sizes: 3 / 2 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 18.2%–50.0%; mean 31.1%, from 3 / 3 planned pairs; 0 both-empty N/A | Google Analytics 4, Google Search Console |
| Cited domains · mapping complete | 25.0%–25.0%; mean 25.0%, from 1 / 3 planned pairs; 0 both-empty N/A | Unavailable · requires all 3 eligible answers |
| Observed inline cited domains · coverage qualified | 11.1%–25.0%; mean 16.2%, from 3 / 3 planned pairs; 0 both-empty N/A | businessinsider.com |
V2 · OpenAI API + web search
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | N/A: both eligible sets are empty. | 66.7% (2 / 3)Set sizes: 3 / 2 | 66.7% (2 / 3)Set sizes: 3 / 2 |
| R1 ↔ R3 | N/A: both eligible sets are empty. | 66.7% (2 / 3)Set sizes: 3 / 2 | 66.7% (2 / 3)Set sizes: 3 / 2 |
| R2 ↔ R3 | N/A: both eligible sets are empty. | 100.0% (2 / 2)Set sizes: 2 / 2 | 100.0% (2 / 2)Set sizes: 2 / 2 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | No numeric comparison · 3 both-empty N/A; 0 unavailable | None observed within this eligible set |
| Cited domains · mapping complete | 66.7%–100.0%; mean 77.8%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | 66.7%–100.0%; mean 77.8%, from 3 / 3 planned pairs; 0 both-empty N/A | openai.com, x.ai |
V2 · Grok Expert · consumer interface
| Repeated answers | Recommended tools | Domains · complete mapping | Observed inline domains · qualified |
|---|---|---|---|
| R1 ↔ R2 | 40.0% (2 / 5)Set sizes: 5 / 2 | Unavailable: one or both sets are ineligible or unrecorded. | 12.5% (1 / 8)Set sizes: 4 / 5 |
| R1 ↔ R3 | 40.0% (2 / 5)Set sizes: 5 / 2 | Unavailable: one or both sets are ineligible or unrecorded. | 14.3% (1 / 7)Set sizes: 4 / 4 |
| R2 ↔ R3 | 100.0% (2 / 2)Set sizes: 2 / 2 | 12.5% (1 / 8)Set sizes: 5 / 4 | 12.5% (1 / 8)Set sizes: 5 / 4 |
| Set | Numeric-pair range / mean | Shared across all 3 |
|---|---|---|
| Recommended tools / services / surfaces | 40.0%–100.0%; mean 60.0%, from 3 / 3 planned pairs; 0 both-empty N/A | Google Analytics 4, X |
| Cited domains · mapping complete | 12.5%–12.5%; mean 12.5%, from 1 / 3 planned pairs; 0 both-empty N/A | Unavailable · requires all 3 eligible answers |
| Observed inline cited domains · coverage qualified | 12.5%–14.3%; mean 13.1%, from 3 / 3 planned pairs; 0 both-empty N/A | businessinsider.com |
Original pooled-union wording analysis
M1 versus M2 and V1 versus V2 compare variant unions and per-answer prevalence. Inspect repeat variation alongside these contrasts. The variants change context as well as phrasing; small unequal eligible samples cannot isolate wording effects.
OpenAI API + web search · M1 versus M2
Recommended tools / services / surfaces
Pooled union overlap: 0.0% (0 / 7)Set sizes: 5 / 2
M1 repeat context: 0.0%–0.0%; mean 0.0%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 0.0%–0.0%; mean 0.0%, from 2 / 3 planned pairs; 1 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | Google Analytics 4, Google Search Console, Google Sheets, Plausible, PostHog |
| M2 | 3 / 3 | Google Analytics, X |
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible |
|---|---|---|
| Google Analytics | 0 / 3 | 1 / 3 |
| Google Analytics 4 | 1 / 3 | 0 / 3 |
| Google Search Console | 1 / 3 | 0 / 3 |
| Google Sheets | 1 / 3 | 0 / 3 |
| Plausible | 1 / 3 | 0 / 3 |
| PostHog | 1 / 3 | 0 / 3 |
| X | 0 / 3 | 1 / 3 |
Cited domains · mapping complete
Pooled union overlap: 100.0% (3 / 3)Set sizes: 3 / 3
M1 repeat context: 33.3%–50.0%; mean 44.4%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 33.3%–100.0%; mean 55.6%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | openai.com, x.ai, x.com |
| M2 | 3 / 3 | openai.com, x.ai, x.com |
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible |
|---|---|---|
| openai.com | 3 / 3 | 3 / 3 |
| x.ai | 1 / 3 | 1 / 3 |
| x.com | 1 / 3 | 2 / 3 |
Observed inline cited domains · coverage qualified
Pooled union overlap: 100.0% (3 / 3)Set sizes: 3 / 3
M1 repeat context: 33.3%–50.0%; mean 44.4%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 33.3%–100.0%; mean 55.6%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | openai.com, x.ai, x.com |
| M2 | 3 / 3 | openai.com, x.ai, x.com |
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible |
|---|---|---|
| openai.com | 3 / 3 | 3 / 3 |
| x.ai | 1 / 3 | 1 / 3 |
| x.com | 1 / 3 | 2 / 3 |
OpenAI API + web search · V1 versus V2
Recommended tools / services / surfaces
Pooled union overlap: N/A: both eligible sets are empty.
V1 repeat context: No numeric comparison · 3 both-empty N/A; 0 unavailable
V2 repeat context: No numeric comparison · 3 both-empty N/A; 0 unavailable
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 3 / 3 | None observed within this eligible set |
| V2 | 3 / 3 | None observed within this eligible set |
No observed items in eligible inventories.
Cited domains · mapping complete
Pooled union overlap: 66.7% (2 / 3)Set sizes: 2 / 3
V1 repeat context: 50.0%–100.0%; mean 66.7%, from 3 / 3 planned pairs; 0 both-empty N/A
V2 repeat context: 66.7%–100.0%; mean 77.8%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 3 / 3 | openai.com, x.ai |
| V2 | 3 / 3 | arxiv.org, openai.com, x.ai |
| Coded item | V1 · 3/3 eligible | V2 · 3/3 eligible |
|---|---|---|
| arxiv.org | 0 / 3 | 1 / 3 |
| openai.com | 3 / 3 | 3 / 3 |
| x.ai | 1 / 3 | 3 / 3 |
Observed inline cited domains · coverage qualified
Pooled union overlap: 66.7% (2 / 3)Set sizes: 2 / 3
V1 repeat context: 50.0%–100.0%; mean 66.7%, from 3 / 3 planned pairs; 0 both-empty N/A
V2 repeat context: 66.7%–100.0%; mean 77.8%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 3 / 3 | openai.com, x.ai |
| V2 | 3 / 3 | arxiv.org, openai.com, x.ai |
| Coded item | V1 · 3/3 eligible | V2 · 3/3 eligible |
|---|---|---|
| arxiv.org | 0 / 3 | 1 / 3 |
| openai.com | 3 / 3 | 3 / 3 |
| x.ai | 1 / 3 | 3 / 3 |
Grok Expert · consumer interface · M1 versus M2
Recommended tools / services / surfaces
Pooled union overlap: 25.9% (7 / 27)Set sizes: 14 / 20
M1 repeat context: 35.7%–50.0%; mean 43.4%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 18.8%–41.7%; mean 27.5%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | Ahrefs AI visibility tools (unspecified), AthenaHQ Essential, Bing Webmaster Tools, GetMentioned free AI visibility report, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Grader, Mangools AI Search Watcher, Semrush AI visibility tools (unspecified), X, YouTube, geo-tool.com GEO check |
| M2 | 3 / 3 | Ahrefs AI Visibility Checker, Ahrefs Brand Radar, Bing Webmaster Tools, Capterra, G2, Google Analytics 4, Google People also ask, Google Search, Google Search Console, Google Sheets, HubSpot AEO Grader, Indexly Grok Visibility Checker, Mangools AI Search Watcher, Notion, Product Hunt, Ranked AI Visibility Checker, Reddit, Semrush AI Visibility Checker, Snoika brand checker, X |
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible |
|---|---|---|
| Ahrefs AI Visibility Checker | 0 / 3 | 1 / 3 |
| Ahrefs AI visibility tools (unspecified) | 1 / 3 | 0 / 3 |
| Ahrefs Brand Radar | 0 / 3 | 2 / 3 |
| AthenaHQ Essential | 1 / 3 | 0 / 3 |
| Bing Webmaster Tools | 3 / 3 | 1 / 3 |
| Capterra | 0 / 3 | 1 / 3 |
| G2 | 0 / 3 | 1 / 3 |
| GetMentioned free AI visibility report | 1 / 3 | 0 / 3 |
| Google Analytics 4 | 3 / 3 | 1 / 3 |
| Google People also ask | 0 / 3 | 1 / 3 |
| Google Search | 0 / 3 | 1 / 3 |
| Google Search Console | 3 / 3 | 2 / 3 |
| Google Sheets | 2 / 3 | 3 / 3 |
| HubSpot AEO Grader | 3 / 3 | 1 / 3 |
| Indexly Grok Visibility Checker | 0 / 3 | 1 / 3 |
| Mangools AI Search Grader | 1 / 3 | 0 / 3 |
| Mangools AI Search Watcher | 2 / 3 | 2 / 3 |
| Notion | 0 / 3 | 1 / 3 |
| Product Hunt | 0 / 3 | 1 / 3 |
| Ranked AI Visibility Checker | 0 / 3 | 1 / 3 |
| 0 / 3 | 3 / 3 | |
| Semrush AI Visibility Checker | 0 / 3 | 1 / 3 |
| Semrush AI visibility tools (unspecified) | 1 / 3 | 0 / 3 |
| Snoika brand checker | 0 / 3 | 1 / 3 |
| X | 2 / 3 | 3 / 3 |
| YouTube | 1 / 3 | 0 / 3 |
| geo-tool.com GEO check | 1 / 3 | 0 / 3 |
Cited domains · mapping complete
Pooled union overlap: 0.0% (0 / 13)Set sizes: 7 / 6
M1 repeat context: 0.0%–57.1%; mean 19.0%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 0.0%–0.0%; mean 0.0%, from 1 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | explaingeo.com, geoaura.world, gomega.ai, goodatmarketing.com, hubspot.com, shmeer.com, snezzi.com |
| M2 | 2 / 3 | 01net.com, amicited.com, indexly.ai, mangools.com, ranked.ai, snoika.com |
| Coded item | M1 · 3/3 eligible | M2 · 2/3 eligible |
|---|---|---|
| 01net.com | 0 / 3 | 1 / 2 |
| amicited.com | 0 / 3 | 1 / 2 |
| explaingeo.com | 1 / 3 | 0 / 2 |
| geoaura.world | 2 / 3 | 0 / 2 |
| gomega.ai | 2 / 3 | 0 / 2 |
| goodatmarketing.com | 2 / 3 | 0 / 2 |
| hubspot.com | 1 / 3 | 0 / 2 |
| indexly.ai | 0 / 3 | 1 / 2 |
| mangools.com | 0 / 3 | 1 / 2 |
| ranked.ai | 0 / 3 | 1 / 2 |
| shmeer.com | 2 / 3 | 0 / 2 |
| snezzi.com | 1 / 3 | 0 / 2 |
| snoika.com | 0 / 3 | 1 / 2 |
Observed inline cited domains · coverage qualified
Pooled union overlap: 0.0% (0 / 16)Set sizes: 7 / 9
M1 repeat context: 0.0%–57.1%; mean 19.0%, from 3 / 3 planned pairs; 0 both-empty N/A
M2 repeat context: 0.0%–0.0%; mean 0.0%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| M1 | 3 / 3 | explaingeo.com, geoaura.world, gomega.ai, goodatmarketing.com, hubspot.com, shmeer.com, snezzi.com |
| M2 | 3 / 3 | 01net.com, amicited.com, indexly.ai, mangools.com, ranked.ai, seozest.io, snoika.com, solcrys.com, stork.ai |
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible |
|---|---|---|
| 01net.com | 0 / 3 | 1 / 3 |
| amicited.com | 0 / 3 | 1 / 3 |
| explaingeo.com | 1 / 3 | 0 / 3 |
| geoaura.world | 2 / 3 | 0 / 3 |
| gomega.ai | 2 / 3 | 0 / 3 |
| goodatmarketing.com | 2 / 3 | 0 / 3 |
| hubspot.com | 1 / 3 | 0 / 3 |
| indexly.ai | 0 / 3 | 1 / 3 |
| mangools.com | 0 / 3 | 1 / 3 |
| ranked.ai | 0 / 3 | 1 / 3 |
| seozest.io | 0 / 3 | 1 / 3 |
| shmeer.com | 2 / 3 | 0 / 3 |
| snezzi.com | 1 / 3 | 0 / 3 |
| snoika.com | 0 / 3 | 1 / 3 |
| solcrys.com | 0 / 3 | 1 / 3 |
| stork.ai | 0 / 3 | 1 / 3 |
Grok Expert · consumer interface · V1 versus V2
Recommended tools / services / surfaces
Pooled union overlap: 28.6% (4 / 14)Set sizes: 13 / 5
V1 repeat context: 18.2%–50.0%; mean 31.1%, from 3 / 3 planned pairs; 0 both-empty N/A
V2 repeat context: 40.0%–100.0%; mean 60.0%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 3 / 3 | Ahrefs AI Visibility Checker, Ahrefs AI visibility tools (unspecified), Google Analytics 4, Google People also ask, Google Search Console, Google Sheets, HubSpot AEO Grader, HubSpot AI Search Grader, HubSpot grader (unspecified), Mangools AI Search Grader, Reddit, Semrush AI visibility tools (unspecified), X |
| V2 | 3 / 3 | GitHub, Google Analytics 4, Google Search Console, Reddit, X |
| Coded item | V1 · 3/3 eligible | V2 · 3/3 eligible |
|---|---|---|
| Ahrefs AI Visibility Checker | 1 / 3 | 0 / 3 |
| Ahrefs AI visibility tools (unspecified) | 2 / 3 | 0 / 3 |
| GitHub | 0 / 3 | 1 / 3 |
| Google Analytics 4 | 3 / 3 | 3 / 3 |
| Google People also ask | 1 / 3 | 0 / 3 |
| Google Search Console | 3 / 3 | 1 / 3 |
| Google Sheets | 1 / 3 | 0 / 3 |
| HubSpot AEO Grader | 1 / 3 | 0 / 3 |
| HubSpot AI Search Grader | 1 / 3 | 0 / 3 |
| HubSpot grader (unspecified) | 1 / 3 | 0 / 3 |
| Mangools AI Search Grader | 1 / 3 | 0 / 3 |
| 1 / 3 | 1 / 3 | |
| Semrush AI visibility tools (unspecified) | 2 / 3 | 0 / 3 |
| X | 2 / 3 | 3 / 3 |
Cited domains · mapping complete
Pooled union overlap: 9.1% (1 / 11)Set sizes: 4 / 8
V1 repeat context: 25.0%–25.0%; mean 25.0%, from 1 / 3 planned pairs; 0 both-empty N/A
V2 repeat context: 12.5%–12.5%; mean 12.5%, from 1 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 2 / 3 | amicited.com, businessinsider.com, searchenginejournal.com, visilay.com |
| V2 | 2 / 3 | aeotrack.io, brantial.ai, businessinsider.com, getmint.ai, loudface.co, paprik.ai, presenc.ai, sparktoro.com |
| Coded item | V1 · 2/3 eligible | V2 · 2/3 eligible |
|---|---|---|
| aeotrack.io | 0 / 2 | 1 / 2 |
| amicited.com | 1 / 2 | 0 / 2 |
| brantial.ai | 0 / 2 | 1 / 2 |
| businessinsider.com | 2 / 2 | 2 / 2 |
| getmint.ai | 0 / 2 | 1 / 2 |
| loudface.co | 0 / 2 | 1 / 2 |
| paprik.ai | 0 / 2 | 1 / 2 |
| presenc.ai | 0 / 2 | 1 / 2 |
| searchenginejournal.com | 1 / 2 | 0 / 2 |
| sparktoro.com | 0 / 2 | 1 / 2 |
| visilay.com | 1 / 2 | 0 / 2 |
Observed inline cited domains · coverage qualified
Pooled union overlap: 23.5% (4 / 17)Set sizes: 10 / 11
V1 repeat context: 11.1%–25.0%; mean 16.2%, from 3 / 3 planned pairs; 0 both-empty N/A
V2 repeat context: 12.5%–14.3%; mean 13.1%, from 3 / 3 planned pairs; 0 both-empty N/A
| Variant | Eligible answers | Union |
|---|---|---|
| V1 | 3 / 3 | ahrefs.com, amicited.com, businessinsider.com, cognizo.ai, higoodie.com, hubspot.com, maxaeo.ai, searchenginejournal.com, searchscope.com.au, visilay.com |
| V2 | 3 / 3 | aeotrack.io, amicited.com, brantial.ai, businessinsider.com, getmint.ai, loudface.co, paprik.ai, presenc.ai, searchscope.com.au, sparktoro.com, visilay.com |
| Coded item | V1 · 3/3 eligible | V2 · 3/3 eligible |
|---|---|---|
| aeotrack.io | 0 / 3 | 1 / 3 |
| ahrefs.com | 1 / 3 | 0 / 3 |
| amicited.com | 1 / 3 | 1 / 3 |
| brantial.ai | 0 / 3 | 1 / 3 |
| businessinsider.com | 3 / 3 | 3 / 3 |
| cognizo.ai | 1 / 3 | 0 / 3 |
| getmint.ai | 0 / 3 | 1 / 3 |
| higoodie.com | 1 / 3 | 0 / 3 |
| hubspot.com | 1 / 3 | 0 / 3 |
| loudface.co | 0 / 3 | 1 / 3 |
| maxaeo.ai | 1 / 3 | 0 / 3 |
| paprik.ai | 0 / 3 | 1 / 3 |
| presenc.ai | 0 / 3 | 1 / 3 |
| searchenginejournal.com | 1 / 3 | 0 / 3 |
| searchscope.com.au | 1 / 3 | 1 / 3 |
| sparktoro.com | 0 / 3 | 1 / 3 |
| visilay.com | 1 / 3 | 1 / 3 |
Task alignment and intent comparison · supplementary
Variant results appear separately before the pooled M and V columns. Each pooled intent has up to six eligible answers. The answer-type split is expected task alignment, not evidence of a new retrieval mechanism. Changing the requested task changes what an answer should contain; this supplementary comparison describes categories and recommendation sets.
OpenAI API + web search
Observed cited-source categories · provisional
M pooled versus V pooled: 50.0% (1 / 2)Set sizes: 1 / 2
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible | M · 6/6 eligible | V1 · 3/3 eligible | V2 · 3/3 eligible | V · 6/6 eligible |
|---|---|---|---|---|---|---|
| Academic or independent methods research | 0 / 3 | 0 / 3 | 0 / 6 | 0 / 3 | 1 / 3 | 1 / 6 |
| Platform documentation | 3 / 3 | 3 / 3 | 6 / 6 | 3 / 3 | 3 / 3 | 6 / 6 |
Recommended tools / services / surfaces
M pooled versus V pooled: 0.0% (0 / 7)Set sizes: 7 / 0
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible | M · 6/6 eligible | V1 · 3/3 eligible | V2 · 3/3 eligible | V · 6/6 eligible |
|---|---|---|---|---|---|---|
| Google Analytics | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Google Analytics 4 | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Google Search Console | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Google Sheets | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Plausible | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| PostHog | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| X | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
Grok Expert · consumer interface
Observed cited-source categories · provisional
M pooled versus V pooled: 75.0% (3 / 4)Set sizes: 3 / 4
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible | M · 6/6 eligible | V1 · 3/3 eligible | V2 · 3/3 eligible | V · 6/6 eligible |
|---|---|---|---|---|---|---|
| Editorial or industry reporting | 0 / 3 | 1 / 3 | 1 / 6 | 3 / 3 | 3 / 3 | 6 / 6 |
| Other / unresolved | 2 / 3 | 2 / 3 | 4 / 6 | 3 / 3 | 3 / 3 | 6 / 6 |
| Tool/product documentation or sales page | 1 / 3 | 1 / 3 | 2 / 6 | 1 / 3 | 1 / 3 | 2 / 6 |
| Vendor research or case study | 0 / 3 | 0 / 3 | 0 / 6 | 0 / 3 | 1 / 3 | 1 / 6 |
Recommended tools / services / surfaces
M pooled versus V pooled: 36.7% (11 / 30)Set sizes: 27 / 14
| Coded item | M1 · 3/3 eligible | M2 · 3/3 eligible | M · 6/6 eligible | V1 · 3/3 eligible | V2 · 3/3 eligible | V · 6/6 eligible |
|---|---|---|---|---|---|---|
| Ahrefs AI Visibility Checker | 0 / 3 | 1 / 3 | 1 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| Ahrefs AI visibility tools (unspecified) | 1 / 3 | 0 / 3 | 1 / 6 | 2 / 3 | 0 / 3 | 2 / 6 |
| Ahrefs Brand Radar | 0 / 3 | 2 / 3 | 2 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| AthenaHQ Essential | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Bing Webmaster Tools | 3 / 3 | 1 / 3 | 4 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Capterra | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| G2 | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| GetMentioned free AI visibility report | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| GitHub | 0 / 3 | 0 / 3 | 0 / 6 | 0 / 3 | 1 / 3 | 1 / 6 |
| Google Analytics 4 | 3 / 3 | 1 / 3 | 4 / 6 | 3 / 3 | 3 / 3 | 6 / 6 |
| Google People also ask | 0 / 3 | 1 / 3 | 1 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| Google Search | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Google Search Console | 3 / 3 | 2 / 3 | 5 / 6 | 3 / 3 | 1 / 3 | 4 / 6 |
| Google Sheets | 2 / 3 | 3 / 3 | 5 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| HubSpot AEO Grader | 3 / 3 | 1 / 3 | 4 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| HubSpot AI Search Grader | 0 / 3 | 0 / 3 | 0 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| HubSpot grader (unspecified) | 0 / 3 | 0 / 3 | 0 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| Indexly Grok Visibility Checker | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Mangools AI Search Grader | 1 / 3 | 0 / 3 | 1 / 6 | 1 / 3 | 0 / 3 | 1 / 6 |
| Mangools AI Search Watcher | 2 / 3 | 2 / 3 | 4 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Notion | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Product Hunt | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Ranked AI Visibility Checker | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| 0 / 3 | 3 / 3 | 3 / 6 | 1 / 3 | 1 / 3 | 2 / 6 | |
| Semrush AI Visibility Checker | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| Semrush AI visibility tools (unspecified) | 1 / 3 | 0 / 3 | 1 / 6 | 2 / 3 | 0 / 3 | 2 / 6 |
| Snoika brand checker | 0 / 3 | 1 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| X | 2 / 3 | 3 / 3 | 5 / 6 | 2 / 3 | 3 / 3 | 5 / 6 |
| YouTube | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
| geo-tool.com GEO check | 1 / 3 | 0 / 3 | 1 / 6 | 0 / 3 | 0 / 3 | 0 / 6 |
C/M/R among coded named entities
C/M/R is conditional on coded named tool/service/feature/channel entity × answer rows. Every inventoried row is a mention by selection; M/rows is not a general brand-mention rate. This is not a count of every source publisher. Supplied ChatGPT/Grok targets, generic methods and citation-only publishers are excluded. Citation means the entity’s own content under the codebook. C, M and R overlap and must not be added as disjoint totals.
| Interface | Entity × answer rows | C / known | M / known | R / known | Answers with named R set | Eligible empty R sets |
|---|---|---|---|---|---|---|
| OpenAI API + web search | 30 | 10 / 30 (0 unresolved) | 30 / 30 (0 unresolved) | 7 / 30 (0 unresolved) | 3 / 12 | 9 |
| Grok Expert · consumer interface | 126 | 7 / 126 (0 unresolved) | 126 / 126 (0 unresolved) | 83 / 126 (0 unresolved) | 12 / 12 | 0 |
Exact C/M/R combinations
| Interface | C only | M only | R only | C+M | C+R | M+R | C+M+R | None | Unresolved |
|---|---|---|---|---|---|---|---|---|---|
| OpenAI API + web search | 0 | 13 | 0 | 10 | 0 | 7 | 0 | 0 | 0 |
| Grok Expert · consumer interface | 0 | 43 | 0 | 0 | 0 | 76 | 7 | 0 | 0 |
Supplementary source composition
The arXiv citation supports repeated-run uncertainty, but not the accompanying wording-change claim.
The inaccessible Markets citation has a readable commercial press-release origin. Keep exact-URL unavailability and editorial-provenance qualification visible.
Original source-category counts are retained. The later selective audit does not silently recode provisional classifications or establish independent editorial provenance for a commercial release.
Read the selective source audit
Supplementary source composition: each category counts once per eligible observed-source answer. The first three Grok captures lack strict complete URL mapping; this observed sample does not establish an intrinsic platform preference. Categories overlap across sources. Provisional Other / unresolved is not evidence that a page is vendor research or that its contents were verified.
| Interface | Primary source-page category | Answers / observed-source denominator |
|---|---|---|
| OpenAI API + web search | Academic or independent methods research | 1 / 12 |
| OpenAI API + web search | Platform documentation | 12 / 12 |
| Grok Expert · consumer interface | Editorial or industry reporting | 7 / 12 |
| Grok Expert · consumer interface | Other / unresolved | 10 / 12 |
| Grok Expert · consumer interface | Tool/product documentation or sales page | 4 / 12 |
| Grok Expert · consumer interface | Vendor research or case study | 1 / 12 |
Answer types and returned-advice flags
Primary answer-type counts are in the coverage table. Nonexclusive flags below describe returned advice, not verified best practices. Each cell displays true judgments / known judgments; missing flag coding is excluded.
| Returned-advice flag | OpenAI API + web search / M1 | Grok Expert · consumer interface / M1 | OpenAI API + web search / M2 | Grok Expert · consumer interface / M2 | OpenAI API + web search / V1 | Grok Expert · consumer interface / V1 | OpenAI API + web search / V2 | Grok Expert · consumer interface / V2 |
|---|---|---|---|---|---|---|---|---|
| API consumer UI distinction | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 3 / 3 | 1 / 1 | 3 / 3 | 1 / 1 |
| awareness vs recommendation distinction | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | 0 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| branded and unbranded buyer prompts | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
| branded diagnostic separated | 3 / 3 | Unavailable · no eligible denominator | 3 / 3 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| buyer intent prompt relevance | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| citation and mention observations separate | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 2 | Unavailable · no eligible denominator | 2 / 2 | Unavailable · no eligible denominator | 2 / 2 |
| citation and mention rates separate | 2 / 3 | 1 / 3 | 3 / 3 | 1 / 3 | 3 / 3 | 1 / 3 | 3 / 3 | 2 / 3 |
| citations mentions recommendations distinguished | 3 / 3 | 2 / 3 | 2 / 3 | 0 / 3 | 3 / 3 | 0 / 3 | 3 / 3 | 1 / 3 |
| composite score caution | 0 / 3 | 0 / 3 | 1 / 3 | 0 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| composite score weighting disclosed or checked | 2 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 2 / 3 | 0 / 3 | 3 / 3 | 3 / 3 |
| downstream conversions | 3 / 3 | 2 / 3 | 0 / 3 | 0 / 3 | 1 / 3 | 2 / 3 | 3 / 3 | 1 / 3 |
| downstream traffic or impressions | 3 / 3 | 3 / 3 | 1 / 3 | 1 / 3 | Unavailable · no eligible denominator | 2 / 2 | Unavailable · no eligible denominator | 2 / 2 |
| fresh sessions | 3 / 3 | 1 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 2 / 3 | 2 / 3 |
| human checking | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| metric label conflation | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| model search version logging | 3 / 3 | 0 / 3 | 1 / 3 | 0 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| numerator denominator definition | 3 / 3 | 3 / 3 | 3 / 3 | 1 / 3 | 3 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
| paid tool dependence | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 | 0 / 3 |
| positive negative stance recording | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| posthoc source explanation caution | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| raw answer archiving explicit | 2 / 3 | 0 / 3 | 3 / 3 | 0 / 2 | 1 / 3 | 0 / 2 | 3 / 3 | 0 / 2 |
| raw answer inspection | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| repeated runs | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| search mode enablement advice | Unavailable · no eligible denominator | 2 / 2 | Unavailable · no eligible denominator | 2 / 2 | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | 2 / 2 |
| search mode logging explicit | 3 / 3 | 1 / 3 | 3 / 3 | 1 / 3 | 3 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
| stable benchmark cadence | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| uncertainty variance | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| universal rank caution | 2 / 2 | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| visibility revenue non equivalence | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
| visibility vs real exposure distinguished | 1 / 1 | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator | Unavailable · no eligible denominator |
API condition, request evidence, and estimated cost
Recorded API estimates sum to US$1.6678760 across 12/12 submissions with estimates.
| Attempt | Status | Requested model | Returned model | Effort | Search configured | Search observed | Token usage | Tool usage | Recorded estimated cost |
|---|---|---|---|---|---|---|---|---|---|
| S3-OPENAIAPI-V1-R1 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 46785,
"input_tokens_details": {
"cache_write_tokens": 4349,
"cached_tokens": 0
},
"output_tokens": 2974,
"output_tokens_details": {
"reasoning_tokens": 1871
},
"total_tokens": 49759
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.270969,
"token_cost_usd": 0.250969,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-V2-R1 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 19168,
"input_tokens_details": {
"cache_write_tokens": 32,
"cached_tokens": 4316
},
"output_tokens": 2115,
"output_tokens_details": {
"reasoning_tokens": 808
},
"total_tokens": 21283
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1234664,
"token_cost_usd": 0.1034664,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M2-R1 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21303,
"input_tokens_details": {
"cache_write_tokens": 34,
"cached_tokens": 4316
},
"output_tokens": 2319,
"output_tokens_details": {
"reasoning_tokens": 963
},
"total_tokens": 23622
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1360884,
"token_cost_usd": 0.1160884,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M1-R1 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21273,
"input_tokens_details": {
"cache_write_tokens": 33,
"cached_tokens": 4316
},
"output_tokens": 3561,
"output_tokens_details": {
"reasoning_tokens": 1879
},
"total_tokens": 24834
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1608074,
"token_cost_usd": 0.1408074,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M1-R2 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 12997,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2588,
"output_tokens_details": {
"reasoning_tokens": 1098
},
"total_tokens": 15585
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
} | {
"estimated_usd": 0.0980916,
"token_cost_usd": 0.0880916,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M2-R2 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 20861,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2433,
"output_tokens_details": {
"reasoning_tokens": 861
},
"total_tokens": 23294
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.136444,
"token_cost_usd": 0.116444,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-V1-R2 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 12913,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1470,
"output_tokens_details": {
"reasoning_tokens": 503
},
"total_tokens": 14383
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
} | {
"estimated_usd": 0.0753956,
"token_cost_usd": 0.0653956,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-V2-R2 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21174,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 1960,
"output_tokens_details": {
"reasoning_tokens": 558
},
"total_tokens": 23134
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1282432,
"token_cost_usd": 0.1082432,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M2-R3 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21065,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2390,
"output_tokens_details": {
"reasoning_tokens": 715
},
"total_tokens": 23455
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1364,
"token_cost_usd": 0.1164,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-V2-R3 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21162,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 2251,
"output_tokens_details": {
"reasoning_tokens": 694
},
"total_tokens": 23413
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1340152,
"token_cost_usd": 0.1140152,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-M1-R3 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 21109,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2499,
"output_tokens_details": {
"reasoning_tokens": 933
},
"total_tokens": 23608
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
} | {
"estimated_usd": 0.1387596,
"token_cost_usd": 0.1187596,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
| S3-OPENAIAPI-V1-R3 | completed | gpt-5.6-sol | gpt-5.6-sol | medium | Yes | Yes | {
"input_tokens": 19633,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1816,
"output_tokens_details": {
"reasoning_tokens": 872
},
"total_tokens": 21449
} | {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 3
}
} | {
"estimated_usd": 0.1291956,
"token_cost_usd": 0.09919560000000001,
"charged_searches_estimate": 3,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
} |
Only recorded estimates are summed; no new price calculation is performed. These are not reconciled provider charges. Missing usage or cost remains unknown; configured caps and reserves are not actual spend.
{
"api_attempts": 12,
"estimates_available": 12,
"missing_estimates": 0,
"known_estimate_sum_usd": "1.6678760",
"all_attempt_estimates_available": true,
"billing_reconciled": false,
"reported_billable_searches": 23,
"billable_search_count_coverage": 12,
"answer_url_citation_occurrences": 50,
"citation_occurrence_rule": "Count each recorded url_citation annotation once; corresponding Markdown representations are not added again."
}Authorized API condition and budget
{
"authorized": true,
"authorized_on": "2026-11-02",
"planned_formal_answers": 12,
"actual_submissions": 12,
"complete_answers": 12,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"budget_policy": "US$10 authorized total; sequential admission with US$6.60 reservation, usage-priced estimates, stop on unknown outcomes. No automatic budget increase. Estimates require provider-billing reconciliation.",
"status": "complete",
"planned_slot_mapping": [
{
"slot_id": "S3-CHATGPT-V1-R1",
"planned_attempt_id": "S3-OPENAIAPI-V1-R1",
"prompt_id": "V1",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-V2-R1",
"planned_attempt_id": "S3-OPENAIAPI-V2-R1",
"prompt_id": "V2",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M2-R1",
"planned_attempt_id": "S3-OPENAIAPI-M2-R1",
"prompt_id": "M2",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M1-R1",
"planned_attempt_id": "S3-OPENAIAPI-M1-R1",
"prompt_id": "M1",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M1-R2",
"planned_attempt_id": "S3-OPENAIAPI-M1-R2",
"prompt_id": "M1",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-M2-R2",
"planned_attempt_id": "S3-OPENAIAPI-M2-R2",
"prompt_id": "M2",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-V1-R2",
"planned_attempt_id": "S3-OPENAIAPI-V1-R2",
"prompt_id": "V1",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-V2-R2",
"planned_attempt_id": "S3-OPENAIAPI-V2-R2",
"prompt_id": "V2",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-M2-R3",
"planned_attempt_id": "S3-OPENAIAPI-M2-R3",
"prompt_id": "M2",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-V2-R3",
"planned_attempt_id": "S3-OPENAIAPI-V2-R3",
"prompt_id": "V2",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-M1-R3",
"planned_attempt_id": "S3-OPENAIAPI-M1-R3",
"prompt_id": "M1",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-V1-R3",
"planned_attempt_id": "S3-OPENAIAPI-V1-R3",
"prompt_id": "V1",
"repeat": 3
}
],
"historical_ui_submissions": 3,
"historical_ui_policy": "All three historical consumer-UI attempts retained and excluded from the 12 formal API answers.",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"estimated_spend_usd": 1.667876,
"access_preflight": {
"generation_submitted": false,
"model_access_established": true,
"http_status": 200,
"returned_model": "gpt-5.6-sol",
"evidence": "setup-history.json",
"earlier_failed_setup_checks_retained": true
},
"cost_estimate_complete": true,
"billing_reconciled": false,
"collection_completed_at": "2026-11-02T09:41:54.845343+00:00",
"no_further_submissions_planned": true
}Recorded collector configuration and budget guardrails
Pricing and limits below are recorded collector configuration. They were not independently rechecked by this offline report.
{
"provider": "OpenAI",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"budget_usd": 10,
"pre_request_budget_reserve_usd": 6.6,
"maximum_submissions": 12,
"minimum_submission_interval_seconds": 90,
"max_output_tokens": 32768,
"max_tool_calls": 8,
"timeout_seconds": 900,
"store": false,
"tool_choice": "required",
"web_search": {
"type": "web_search",
"external_web_access": true,
"return_token_budget": "default"
},
"include": [
"web_search_call.action.sources"
],
"pricing": {
"checked_on": "2026-11-02",
"source": "https://developers.openai.com/api/docs/pricing",
"input_per_million_usd": 4,
"cached_input_per_million_usd": 0.4,
"cache_write_per_million_usd": 5,
"output_per_million_usd": 20,
"web_search_per_call_usd": 0.01
},
"budget_notes": "Sequential requests only. Reserve US$6.60 before each request; reconcile returned usage at published rates. Missing usage or unknown transport outcomes block all further calls, with the reservation retained. Estimated costs are not audited billing records. No automatic retries, model changes, budget increases or fallback providers.",
"condition_notes": "API is a new experimental condition, not a consumer ChatGPT Search session. Exact original single user prompt, no system/developer additions, conversation ID, previous response or history. Eight built-in calls and 32768 generated tokens are fixed cost controls; reaching either limit is flagged. No paid connectivity probe; the first formal response is the capture preflight."
}Historical setup/access checks · not answer submissions
The ledger below contains 3 access-check records and 1 historical status snapshot(s). A snapshot is not another request. TLS trust-chain failure, an expired-key HTTP 401, and the later successful GET model-access check are setup history. They do not submit the study prompt, are not API answer failures, and do not increase the attempts ledger. Earlier blocked status is historical; see completed generation records for subsequent collection.
| Time · UTC+8 | Recorded check / historical status | HTTP status | Generation submitted | Evidence summary | Public record |
|---|---|---|---|---|---|
| 2026-11-02 17:08:12 +0800 | non-generation model-access preflight | Unknown / not recorded | No | {
"requested_model": "gpt-5.6-sol",
"accessible": false,
"error_type": "URLError",
"error_summary": "Python trust configuration could not validate the certificate chain; later resolved using the system trusted CA bundle with verification retained.",
"record_kind": "non_generation_model_access_check"
} | setup-history.json |
| 2026-11-02 17:09:02 +0800 | non-generation model-access preflight | 401 | No | {
"requested_model": "gpt-5.6-sol",
"accessible": false,
"error_code": "expired_secret_key",
"record_kind": "non_generation_model_access_check"
} | setup-history.json |
| 2026-11-02 17:18:02 +0800 | non-generation model-access preflight | 200 | No | {
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"accessible": true,
"record_kind": "non_generation_model_access_check"
} | setup-history.json |
| 2026-11-02 17:10:18 +0800 | blocked_expired_api_key | N/A · status snapshot | No request · status snapshot | {
"record_kind": "historical_setup_status_snapshot",
"status": "blocked_expired_api_key",
"formal_generation_submissions": 0,
"note": "Status snapshot after the earlier failed credential check, not an additional request. A later model-access check and all twelve API answers succeeded."
} | setup-history.json |
Recorded collection chronology
All API requests were collected after Grok in the recorded timestamps. Time/order is therefore confounded with interface and platform; these observations do not isolate a causal platform effect.
{
"platform_periods": {
"OpenAI API + web search": {
"timestamped_submissions": 12,
"first_submission": "2026-11-02T09:18:30.426126+00:00",
"last_submission": "2026-11-02T09:40:11.574647+00:00",
"minimum_same_platform_interval_seconds": 103.041462
},
"Grok": {
"timestamped_submissions": 12,
"first_submission": "2026-11-02T06:14:01.029000+00:00",
"last_submission": "2026-11-02T06:51:16.980000+00:00",
"minimum_same_platform_interval_seconds": 113.949
}
},
"all_recorded_API_after_Grok": true,
"interpretation": "Collection order and elapsed time are confounded with interface/platform; not a causal platform effect."
}Post-hoc sensitivity analyses and review status
These analyses were added after collection in response to review. They preserve the original metrics and use explicit mappings, with no new model answers. Pairwise means and pooled-union overlap answer different questions and are not interchangeable.
Same-metric within-versus-cross pairs
| Environment | Intent | Pair kind | Left | Right | Jaccard |
|---|---|---|---|---|---|
| OpenAI API + web search | M | within | S3-CHATGPT-M1-R1 | S3-CHATGPT-M1-R2 | 0/1 · 0.0% |
| OpenAI API + web search | M | within | S3-CHATGPT-M1-R1 | S3-CHATGPT-M1-R3 | 0/5 · 0.0% |
| OpenAI API + web search | M | within | S3-CHATGPT-M1-R2 | S3-CHATGPT-M1-R3 | 0/4 · 0.0% |
| OpenAI API + web search | M | within | S3-CHATGPT-M2-R1 | S3-CHATGPT-M2-R2 | 0/2 · 0.0% |
| OpenAI API + web search | M | within | S3-CHATGPT-M2-R1 | S3-CHATGPT-M2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | M | within | S3-CHATGPT-M2-R2 | S3-CHATGPT-M2-R3 | 0/2 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R1 | S3-CHATGPT-M2-R1 | 0/1 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R1 | S3-CHATGPT-M2-R2 | 0/3 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R1 | S3-CHATGPT-M2-R3 | 0/1 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R2 | S3-CHATGPT-M2-R1 | N/A · both empty or unavailable |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R2 | S3-CHATGPT-M2-R2 | 0/2 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R2 | S3-CHATGPT-M2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R3 | S3-CHATGPT-M2-R1 | 0/4 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R3 | S3-CHATGPT-M2-R2 | 0/6 · 0.0% |
| OpenAI API + web search | M | cross | S3-CHATGPT-M1-R3 | S3-CHATGPT-M2-R3 | 0/4 · 0.0% |
| OpenAI API + web search | V | within | S3-CHATGPT-V1-R1 | S3-CHATGPT-V1-R2 | N/A · both empty or unavailable |
| OpenAI API + web search | V | within | S3-CHATGPT-V1-R1 | S3-CHATGPT-V1-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | within | S3-CHATGPT-V1-R2 | S3-CHATGPT-V1-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | within | S3-CHATGPT-V2-R1 | S3-CHATGPT-V2-R2 | N/A · both empty or unavailable |
| OpenAI API + web search | V | within | S3-CHATGPT-V2-R1 | S3-CHATGPT-V2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | within | S3-CHATGPT-V2-R2 | S3-CHATGPT-V2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R1 | S3-CHATGPT-V2-R1 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R1 | S3-CHATGPT-V2-R2 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R1 | S3-CHATGPT-V2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R2 | S3-CHATGPT-V2-R1 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R2 | S3-CHATGPT-V2-R2 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R2 | S3-CHATGPT-V2-R3 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R3 | S3-CHATGPT-V2-R1 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R3 | S3-CHATGPT-V2-R2 | N/A · both empty or unavailable |
| OpenAI API + web search | V | cross | S3-CHATGPT-V1-R3 | S3-CHATGPT-V2-R3 | N/A · both empty or unavailable |
| Grok Expert · consumer interface | M | within | S3-GROK-M1-R1 | S3-GROK-M1-R2 | 5/14 · 35.7% |
| Grok Expert · consumer interface | M | within | S3-GROK-M1-R1 | S3-GROK-M1-R3 | 4/9 · 44.4% |
| Grok Expert · consumer interface | M | within | S3-GROK-M1-R2 | S3-GROK-M1-R3 | 6/12 · 50.0% |
| Grok Expert · consumer interface | M | within | S3-GROK-M2-R1 | S3-GROK-M2-R2 | 5/12 · 41.7% |
| Grok Expert · consumer interface | M | within | S3-GROK-M2-R1 | S3-GROK-M2-R3 | 4/18 · 22.2% |
| Grok Expert · consumer interface | M | within | S3-GROK-M2-R2 | S3-GROK-M2-R3 | 3/16 · 18.8% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R1 | S3-GROK-M2-R1 | 5/12 · 41.7% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R1 | S3-GROK-M2-R2 | 2/12 · 16.7% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R1 | S3-GROK-M2-R3 | 1/18 · 5.6% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R2 | S3-GROK-M2-R1 | 7/15 · 46.7% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R2 | S3-GROK-M2-R2 | 3/16 · 18.8% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R2 | S3-GROK-M2-R3 | 3/21 · 14.3% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R3 | S3-GROK-M2-R1 | 6/10 · 60.0% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R3 | S3-GROK-M2-R2 | 2/11 · 18.2% |
| Grok Expert · consumer interface | M | cross | S3-GROK-M1-R3 | S3-GROK-M2-R3 | 2/16 · 12.5% |
| Grok Expert · consumer interface | V | within | S3-GROK-V1-R1 | S3-GROK-V1-R2 | 2/11 · 18.2% |
| Grok Expert · consumer interface | V | within | S3-GROK-V1-R1 | S3-GROK-V1-R3 | 3/12 · 25.0% |
| Grok Expert · consumer interface | V | within | S3-GROK-V1-R2 | S3-GROK-V1-R3 | 4/8 · 50.0% |
| Grok Expert · consumer interface | V | within | S3-GROK-V2-R1 | S3-GROK-V2-R2 | 2/5 · 40.0% |
| Grok Expert · consumer interface | V | within | S3-GROK-V2-R1 | S3-GROK-V2-R3 | 2/5 · 40.0% |
| Grok Expert · consumer interface | V | within | S3-GROK-V2-R2 | S3-GROK-V2-R3 | 2/2 · 100.0% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R1 | S3-GROK-V2-R1 | 4/9 · 44.4% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R1 | S3-GROK-V2-R2 | 2/8 · 25.0% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R1 | S3-GROK-V2-R3 | 2/8 · 25.0% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R2 | S3-GROK-V2-R1 | 2/8 · 25.0% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R2 | S3-GROK-V2-R2 | 1/6 · 16.7% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R2 | S3-GROK-V2-R3 | 1/6 · 16.7% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R3 | S3-GROK-V2-R1 | 3/9 · 33.3% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R3 | S3-GROK-V2-R2 | 2/7 · 28.6% |
| Grok Expert · consumer interface | V | cross | S3-GROK-V1-R3 | S3-GROK-V2-R3 | 2/7 · 28.6% |
URL versus domain overlap · supplementary
Domain equality does not establish page equality. URL tracking cleanup is documented in the analysis policy. Strict URL results retain the three early Grok capture exclusions; observed URL results remain coverage-qualified.
| Environment | Intent | Source lens | Pooled-union Jaccard | Left eligible answers | Right eligible answers |
|---|---|---|---|---|---|
| OpenAI API + web search | M | strict_urls | 4/9 · 44.4% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | M | observed_urls | 4/9 · 44.4% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | M | cited_domains | 3/3 · 100.0% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | M | observed_cited_domains | 3/3 · 100.0% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | V | strict_urls | 1/11 · 9.1% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | V | observed_urls | 1/11 · 9.1% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | V | cited_domains | 2/3 · 66.7% | 3 / 3 | 3 / 3 |
| OpenAI API + web search | V | observed_cited_domains | 2/3 · 66.7% | 3 / 3 | 3 / 3 |
| Grok Expert · consumer interface | M | strict_urls | 0/13 · 0.0% | 3 / 3 | 2 / 3 |
| Grok Expert · consumer interface | M | observed_urls | 0/16 · 0.0% | 3 / 3 | 3 / 3 |
| Grok Expert · consumer interface | M | cited_domains | 0/13 · 0.0% | 3 / 3 | 2 / 3 |
| Grok Expert · consumer interface | M | observed_cited_domains | 0/16 · 0.0% | 3 / 3 | 3 / 3 |
| Grok Expert · consumer interface | V | strict_urls | 1/11 · 9.1% | 2 / 3 | 2 / 3 |
| Grok Expert · consumer interface | V | observed_urls | 2/19 · 10.5% | 3 / 3 | 3 / 3 |
| Grok Expert · consumer interface | V | cited_domains | 1/11 · 9.1% | 2 / 3 | 2 / 3 |
| Grok Expert · consumer interface | V | observed_cited_domains | 4/17 · 23.5% | 3 / 3 | 3 / 3 |
Complete post-hoc analysis output and validation
{
"schema_version": 1,
"analysis_status": "posthoc_descriptive_sensitivity",
"environments": [
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search"
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface"
}
],
"within_vs_cross": [
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "M",
"metric": "recommended_tools",
"within": {
"pairs": [
{
"left": "S3-CHATGPT-M1-R1",
"right": "S3-CHATGPT-M1-R2",
"left_attempt_id": "S3-OPENAIAPI-M1-R1",
"right_attempt_id": "S3-OPENAIAPI-M1-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 0,
"intersection": 0,
"union": 1,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R1",
"right": "S3-CHATGPT-M1-R3",
"left_attempt_id": "S3-OPENAIAPI-M1-R1",
"right_attempt_id": "S3-OPENAIAPI-M1-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 4,
"intersection": 0,
"union": 5,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R2",
"right": "S3-CHATGPT-M1-R3",
"left_attempt_id": "S3-OPENAIAPI-M1-R2",
"right_attempt_id": "S3-OPENAIAPI-M1-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 0,
"right_size": 4,
"intersection": 0,
"union": 4,
"shared": []
},
{
"left": "S3-CHATGPT-M2-R1",
"right": "S3-CHATGPT-M2-R2",
"left_attempt_id": "S3-OPENAIAPI-M2-R1",
"right_attempt_id": "S3-OPENAIAPI-M2-R2",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 0,
"right_size": 2,
"intersection": 0,
"union": 2,
"shared": []
},
{
"left": "S3-CHATGPT-M2-R1",
"right": "S3-CHATGPT-M2-R3",
"left_attempt_id": "S3-OPENAIAPI-M2-R1",
"right_attempt_id": "S3-OPENAIAPI-M2-R3",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-M2-R2",
"right": "S3-CHATGPT-M2-R3",
"left_attempt_id": "S3-OPENAIAPI-M2-R2",
"right_attempt_id": "S3-OPENAIAPI-M2-R3",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 0,
"intersection": 0,
"union": 2,
"shared": []
}
],
"summary": {
"planned_pairs": 6,
"observed_pairs": 6,
"eligible_pairs": 6,
"numeric_pairs": 5,
"both_empty_pairs": 1,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.0,
"minimum": 0.0,
"maximum": 0.0
}
},
"cross": {
"pairs": [
{
"left": "S3-CHATGPT-M1-R1",
"right": "S3-CHATGPT-M2-R1",
"left_attempt_id": "S3-OPENAIAPI-M1-R1",
"right_attempt_id": "S3-OPENAIAPI-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 0,
"intersection": 0,
"union": 1,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R1",
"right": "S3-CHATGPT-M2-R2",
"left_attempt_id": "S3-OPENAIAPI-M1-R1",
"right_attempt_id": "S3-OPENAIAPI-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 2,
"intersection": 0,
"union": 3,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R1",
"right": "S3-CHATGPT-M2-R3",
"left_attempt_id": "S3-OPENAIAPI-M1-R1",
"right_attempt_id": "S3-OPENAIAPI-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 0,
"intersection": 0,
"union": 1,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R2",
"right": "S3-CHATGPT-M2-R1",
"left_attempt_id": "S3-OPENAIAPI-M1-R2",
"right_attempt_id": "S3-OPENAIAPI-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R2",
"right": "S3-CHATGPT-M2-R2",
"left_attempt_id": "S3-OPENAIAPI-M1-R2",
"right_attempt_id": "S3-OPENAIAPI-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 0,
"right_size": 2,
"intersection": 0,
"union": 2,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R2",
"right": "S3-CHATGPT-M2-R3",
"left_attempt_id": "S3-OPENAIAPI-M1-R2",
"right_attempt_id": "S3-OPENAIAPI-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R3",
"right": "S3-CHATGPT-M2-R1",
"left_attempt_id": "S3-OPENAIAPI-M1-R3",
"right_attempt_id": "S3-OPENAIAPI-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 0,
"intersection": 0,
"union": 4,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R3",
"right": "S3-CHATGPT-M2-R2",
"left_attempt_id": "S3-OPENAIAPI-M1-R3",
"right_attempt_id": "S3-OPENAIAPI-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 2,
"intersection": 0,
"union": 6,
"shared": []
},
{
"left": "S3-CHATGPT-M1-R3",
"right": "S3-CHATGPT-M2-R3",
"left_attempt_id": "S3-OPENAIAPI-M1-R3",
"right_attempt_id": "S3-OPENAIAPI-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 0,
"intersection": 0,
"union": 4,
"shared": []
}
],
"summary": {
"planned_pairs": 9,
"observed_pairs": 9,
"eligible_pairs": 9,
"numeric_pairs": 7,
"both_empty_pairs": 2,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.0,
"minimum": 0.0,
"maximum": 0.0
}
},
"delta_pp": 0.0,
"answer_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"unique_answer_denominator": 6,
"unit": "Answer-pair Jaccard, reusing the same six answers; pairs are not independent trials."
},
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "V",
"metric": "recommended_tools",
"within": {
"pairs": [
{
"left": "S3-CHATGPT-V1-R1",
"right": "S3-CHATGPT-V1-R2",
"left_attempt_id": "S3-OPENAIAPI-V1-R1",
"right_attempt_id": "S3-OPENAIAPI-V1-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R1",
"right": "S3-CHATGPT-V1-R3",
"left_attempt_id": "S3-OPENAIAPI-V1-R1",
"right_attempt_id": "S3-OPENAIAPI-V1-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R2",
"right": "S3-CHATGPT-V1-R3",
"left_attempt_id": "S3-OPENAIAPI-V1-R2",
"right_attempt_id": "S3-OPENAIAPI-V1-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V2-R1",
"right": "S3-CHATGPT-V2-R2",
"left_attempt_id": "S3-OPENAIAPI-V2-R1",
"right_attempt_id": "S3-OPENAIAPI-V2-R2",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V2-R1",
"right": "S3-CHATGPT-V2-R3",
"left_attempt_id": "S3-OPENAIAPI-V2-R1",
"right_attempt_id": "S3-OPENAIAPI-V2-R3",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V2-R2",
"right": "S3-CHATGPT-V2-R3",
"left_attempt_id": "S3-OPENAIAPI-V2-R2",
"right_attempt_id": "S3-OPENAIAPI-V2-R3",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
],
"summary": {
"planned_pairs": 6,
"observed_pairs": 6,
"eligible_pairs": 6,
"numeric_pairs": 0,
"both_empty_pairs": 6,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": null,
"minimum": null,
"maximum": null
}
},
"cross": {
"pairs": [
{
"left": "S3-CHATGPT-V1-R1",
"right": "S3-CHATGPT-V2-R1",
"left_attempt_id": "S3-OPENAIAPI-V1-R1",
"right_attempt_id": "S3-OPENAIAPI-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R1",
"right": "S3-CHATGPT-V2-R2",
"left_attempt_id": "S3-OPENAIAPI-V1-R1",
"right_attempt_id": "S3-OPENAIAPI-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R1",
"right": "S3-CHATGPT-V2-R3",
"left_attempt_id": "S3-OPENAIAPI-V1-R1",
"right_attempt_id": "S3-OPENAIAPI-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R2",
"right": "S3-CHATGPT-V2-R1",
"left_attempt_id": "S3-OPENAIAPI-V1-R2",
"right_attempt_id": "S3-OPENAIAPI-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R2",
"right": "S3-CHATGPT-V2-R2",
"left_attempt_id": "S3-OPENAIAPI-V1-R2",
"right_attempt_id": "S3-OPENAIAPI-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R2",
"right": "S3-CHATGPT-V2-R3",
"left_attempt_id": "S3-OPENAIAPI-V1-R2",
"right_attempt_id": "S3-OPENAIAPI-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R3",
"right": "S3-CHATGPT-V2-R1",
"left_attempt_id": "S3-OPENAIAPI-V1-R3",
"right_attempt_id": "S3-OPENAIAPI-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R3",
"right": "S3-CHATGPT-V2-R2",
"left_attempt_id": "S3-OPENAIAPI-V1-R3",
"right_attempt_id": "S3-OPENAIAPI-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
},
{
"left": "S3-CHATGPT-V1-R3",
"right": "S3-CHATGPT-V2-R3",
"left_attempt_id": "S3-OPENAIAPI-V1-R3",
"right_attempt_id": "S3-OPENAIAPI-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
],
"summary": {
"planned_pairs": 9,
"observed_pairs": 9,
"eligible_pairs": 9,
"numeric_pairs": 0,
"both_empty_pairs": 9,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": null,
"minimum": null,
"maximum": null
}
},
"delta_pp": null,
"answer_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"unique_answer_denominator": 6,
"unit": "Answer-pair Jaccard, reusing the same six answers; pairs are not independent trials."
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "M",
"metric": "recommended_tools",
"within": {
"pairs": [
{
"left": "S3-GROK-M1-R1",
"right": "S3-GROK-M1-R2",
"left_attempt_id": "S3-GROK-M1-R1",
"right_attempt_id": "S3-GROK-M1-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.35714285714285715,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 12,
"intersection": 5,
"union": 14,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"HubSpot AEO Grader",
"X"
]
},
{
"left": "S3-GROK-M1-R1",
"right": "S3-GROK-M1-R3",
"left_attempt_id": "S3-GROK-M1-R1",
"right_attempt_id": "S3-GROK-M1-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.4444444444444444,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 6,
"intersection": 4,
"union": 9,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"HubSpot AEO Grader"
]
},
{
"left": "S3-GROK-M1-R2",
"right": "S3-GROK-M1-R3",
"left_attempt_id": "S3-GROK-M1-R2",
"right_attempt_id": "S3-GROK-M1-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M1",
"value": 0.5,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 12,
"right_size": 6,
"intersection": 6,
"union": 12,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher"
]
},
{
"left": "S3-GROK-M2-R1",
"right": "S3-GROK-M2-R2",
"left_attempt_id": "S3-GROK-M2-R1",
"right_attempt_id": "S3-GROK-M2-R2",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": 0.4166666666666667,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 10,
"right_size": 7,
"intersection": 5,
"union": 12,
"shared": [
"Ahrefs Brand Radar",
"Google Search Console",
"Google Sheets",
"Reddit",
"X"
]
},
{
"left": "S3-GROK-M2-R1",
"right": "S3-GROK-M2-R3",
"left_attempt_id": "S3-GROK-M2-R1",
"right_attempt_id": "S3-GROK-M2-R3",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": 0.2222222222222222,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 10,
"right_size": 12,
"intersection": 4,
"union": 18,
"shared": [
"Google Sheets",
"Mangools AI Search Watcher",
"Reddit",
"X"
]
},
{
"left": "S3-GROK-M2-R2",
"right": "S3-GROK-M2-R3",
"left_attempt_id": "S3-GROK-M2-R2",
"right_attempt_id": "S3-GROK-M2-R3",
"left_prompt_id": "M2",
"right_prompt_id": "M2",
"value": 0.1875,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 12,
"intersection": 3,
"union": 16,
"shared": [
"Google Sheets",
"Reddit",
"X"
]
}
],
"summary": {
"planned_pairs": 6,
"observed_pairs": 6,
"eligible_pairs": 6,
"numeric_pairs": 6,
"both_empty_pairs": 0,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.35466269841269843,
"minimum": 0.1875,
"maximum": 0.5
}
},
"cross": {
"pairs": [
{
"left": "S3-GROK-M1-R1",
"right": "S3-GROK-M2-R1",
"left_attempt_id": "S3-GROK-M1-R1",
"right_attempt_id": "S3-GROK-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.4166666666666667,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 10,
"intersection": 5,
"union": 12,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"HubSpot AEO Grader",
"X"
]
},
{
"left": "S3-GROK-M1-R1",
"right": "S3-GROK-M2-R2",
"left_attempt_id": "S3-GROK-M1-R1",
"right_attempt_id": "S3-GROK-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.16666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 7,
"intersection": 2,
"union": 12,
"shared": [
"Google Search Console",
"X"
]
},
{
"left": "S3-GROK-M1-R1",
"right": "S3-GROK-M2-R3",
"left_attempt_id": "S3-GROK-M1-R1",
"right_attempt_id": "S3-GROK-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.05555555555555555,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 12,
"intersection": 1,
"union": 18,
"shared": [
"X"
]
},
{
"left": "S3-GROK-M1-R2",
"right": "S3-GROK-M2-R1",
"left_attempt_id": "S3-GROK-M1-R2",
"right_attempt_id": "S3-GROK-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.4666666666666667,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 12,
"right_size": 10,
"intersection": 7,
"union": 15,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"X"
]
},
{
"left": "S3-GROK-M1-R2",
"right": "S3-GROK-M2-R2",
"left_attempt_id": "S3-GROK-M1-R2",
"right_attempt_id": "S3-GROK-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.1875,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 12,
"right_size": 7,
"intersection": 3,
"union": 16,
"shared": [
"Google Search Console",
"Google Sheets",
"X"
]
},
{
"left": "S3-GROK-M1-R2",
"right": "S3-GROK-M2-R3",
"left_attempt_id": "S3-GROK-M1-R2",
"right_attempt_id": "S3-GROK-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.14285714285714285,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 12,
"right_size": 12,
"intersection": 3,
"union": 21,
"shared": [
"Google Sheets",
"Mangools AI Search Watcher",
"X"
]
},
{
"left": "S3-GROK-M1-R3",
"right": "S3-GROK-M2-R1",
"left_attempt_id": "S3-GROK-M1-R3",
"right_attempt_id": "S3-GROK-M2-R1",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.6,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 6,
"right_size": 10,
"intersection": 6,
"union": 10,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher"
]
},
{
"left": "S3-GROK-M1-R3",
"right": "S3-GROK-M2-R2",
"left_attempt_id": "S3-GROK-M1-R3",
"right_attempt_id": "S3-GROK-M2-R2",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.18181818181818182,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 6,
"right_size": 7,
"intersection": 2,
"union": 11,
"shared": [
"Google Search Console",
"Google Sheets"
]
},
{
"left": "S3-GROK-M1-R3",
"right": "S3-GROK-M2-R3",
"left_attempt_id": "S3-GROK-M1-R3",
"right_attempt_id": "S3-GROK-M2-R3",
"left_prompt_id": "M1",
"right_prompt_id": "M2",
"value": 0.125,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 6,
"right_size": 12,
"intersection": 2,
"union": 16,
"shared": [
"Google Sheets",
"Mangools AI Search Watcher"
]
}
],
"summary": {
"planned_pairs": 9,
"observed_pairs": 9,
"eligible_pairs": 9,
"numeric_pairs": 9,
"both_empty_pairs": 0,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.26030343113676446,
"minimum": 0.05555555555555555,
"maximum": 0.6
}
},
"delta_pp": -9.435926727593397,
"answer_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"unique_answer_denominator": 6,
"unit": "Answer-pair Jaccard, reusing the same six answers; pairs are not independent trials."
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "V",
"metric": "recommended_tools",
"within": {
"pairs": [
{
"left": "S3-GROK-V1-R1",
"right": "S3-GROK-V1-R2",
"left_attempt_id": "S3-GROK-V1-R1",
"right_attempt_id": "S3-GROK-V1-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": 0.18181818181818182,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 5,
"intersection": 2,
"union": 11,
"shared": [
"Google Analytics 4",
"Google Search Console"
]
},
{
"left": "S3-GROK-V1-R1",
"right": "S3-GROK-V1-R3",
"left_attempt_id": "S3-GROK-V1-R1",
"right_attempt_id": "S3-GROK-V1-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": 0.25,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 7,
"intersection": 3,
"union": 12,
"shared": [
"Google Analytics 4",
"Google Search Console",
"X"
]
},
{
"left": "S3-GROK-V1-R2",
"right": "S3-GROK-V1-R3",
"left_attempt_id": "S3-GROK-V1-R2",
"right_attempt_id": "S3-GROK-V1-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V1",
"value": 0.5,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 7,
"intersection": 4,
"union": 8,
"shared": [
"Ahrefs AI visibility tools (unspecified)",
"Google Analytics 4",
"Google Search Console",
"Semrush AI visibility tools (unspecified)"
]
},
{
"left": "S3-GROK-V2-R1",
"right": "S3-GROK-V2-R2",
"left_attempt_id": "S3-GROK-V2-R1",
"right_attempt_id": "S3-GROK-V2-R2",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": 0.4,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 2,
"union": 5,
"shared": [
"Google Analytics 4",
"X"
]
},
{
"left": "S3-GROK-V2-R1",
"right": "S3-GROK-V2-R3",
"left_attempt_id": "S3-GROK-V2-R1",
"right_attempt_id": "S3-GROK-V2-R3",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": 0.4,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 2,
"union": 5,
"shared": [
"Google Analytics 4",
"X"
]
},
{
"left": "S3-GROK-V2-R2",
"right": "S3-GROK-V2-R3",
"left_attempt_id": "S3-GROK-V2-R2",
"right_attempt_id": "S3-GROK-V2-R3",
"left_prompt_id": "V2",
"right_prompt_id": "V2",
"value": 1.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 2,
"intersection": 2,
"union": 2,
"shared": [
"Google Analytics 4",
"X"
]
}
],
"summary": {
"planned_pairs": 6,
"observed_pairs": 6,
"eligible_pairs": 6,
"numeric_pairs": 6,
"both_empty_pairs": 0,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.4553030303030303,
"minimum": 0.18181818181818182,
"maximum": 1.0
}
},
"cross": {
"pairs": [
{
"left": "S3-GROK-V1-R1",
"right": "S3-GROK-V2-R1",
"left_attempt_id": "S3-GROK-V1-R1",
"right_attempt_id": "S3-GROK-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.4444444444444444,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 5,
"intersection": 4,
"union": 9,
"shared": [
"Google Analytics 4",
"Google Search Console",
"Reddit",
"X"
]
},
{
"left": "S3-GROK-V1-R1",
"right": "S3-GROK-V2-R2",
"left_attempt_id": "S3-GROK-V1-R1",
"right_attempt_id": "S3-GROK-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.25,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 2,
"intersection": 2,
"union": 8,
"shared": [
"Google Analytics 4",
"X"
]
},
{
"left": "S3-GROK-V1-R1",
"right": "S3-GROK-V2-R3",
"left_attempt_id": "S3-GROK-V1-R1",
"right_attempt_id": "S3-GROK-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.25,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 2,
"intersection": 2,
"union": 8,
"shared": [
"Google Analytics 4",
"X"
]
},
{
"left": "S3-GROK-V1-R2",
"right": "S3-GROK-V2-R1",
"left_attempt_id": "S3-GROK-V1-R2",
"right_attempt_id": "S3-GROK-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.25,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 5,
"intersection": 2,
"union": 8,
"shared": [
"Google Analytics 4",
"Google Search Console"
]
},
{
"left": "S3-GROK-V1-R2",
"right": "S3-GROK-V2-R2",
"left_attempt_id": "S3-GROK-V1-R2",
"right_attempt_id": "S3-GROK-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.16666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 1,
"union": 6,
"shared": [
"Google Analytics 4"
]
},
{
"left": "S3-GROK-V1-R2",
"right": "S3-GROK-V2-R3",
"left_attempt_id": "S3-GROK-V1-R2",
"right_attempt_id": "S3-GROK-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.16666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 1,
"union": 6,
"shared": [
"Google Analytics 4"
]
},
{
"left": "S3-GROK-V1-R3",
"right": "S3-GROK-V2-R1",
"left_attempt_id": "S3-GROK-V1-R3",
"right_attempt_id": "S3-GROK-V2-R1",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.3333333333333333,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 5,
"intersection": 3,
"union": 9,
"shared": [
"Google Analytics 4",
"Google Search Console",
"X"
]
},
{
"left": "S3-GROK-V1-R3",
"right": "S3-GROK-V2-R2",
"left_attempt_id": "S3-GROK-V1-R3",
"right_attempt_id": "S3-GROK-V2-R2",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.2857142857142857,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 2,
"intersection": 2,
"union": 7,
"shared": [
"Google Analytics 4",
"X"
]
},
{
"left": "S3-GROK-V1-R3",
"right": "S3-GROK-V2-R3",
"left_attempt_id": "S3-GROK-V1-R3",
"right_attempt_id": "S3-GROK-V2-R3",
"left_prompt_id": "V1",
"right_prompt_id": "V2",
"value": 0.2857142857142857,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 2,
"intersection": 2,
"union": 7,
"shared": [
"Google Analytics 4",
"X"
]
}
],
"summary": {
"planned_pairs": 9,
"observed_pairs": 9,
"eligible_pairs": 9,
"numeric_pairs": 9,
"both_empty_pairs": 0,
"unavailable_pairs": 0,
"missing_pair_records": 0,
"mean": 0.2702821869488536,
"minimum": 0.16666666666666666,
"maximum": 0.4444444444444444
}
},
"delta_pp": -18.502084335417674,
"answer_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"unique_answer_denominator": 6,
"unit": "Answer-pair Jaccard, reusing the same six answers; pairs are not independent trials."
}
],
"entity_lenses": {
"mapping": {
"file": "entity-normalization.json",
"input_schema_version": 1,
"rules": [
"Strict names and original coding remain the primary frozen inventory.",
"Family is an explicitly listed branded product-line sensitivity grouping, not a universal corporate-parent identity. Different services of a large provider remain separate (Google Analytics, Google Sheets, Google Search Console).",
"Family grouping does not establish equivalent products, versions, prices or capabilities; unspecified aliases remain unspecified in the strict lens.",
"Category grouping is deliberately coarse and can mechanically increase overlap. It is not evidence of greater recommendation stability.",
"Category-filtered analyses retain strict names and original per-answer recommendation decisions. Channel discussion is not automatically recommendation or publication advice.",
"Question-supplied targets, generic methods and source-publisher-only observations remain excluded by the original codebook."
],
"categories": {
"geo_visibility_product": "Specialized GEO visibility or readiness offering",
"general_measurement_tool": "General measurement, analytics or recordkeeping tool",
"research_or_publication_channel": "Research, evidence, review or publication channel",
"other_ai_surface_or_feature": "Other AI surface, model or retrieval feature"
},
"mapped_entity_count": 52,
"coded_entity_count": 52,
"missing_entities": [],
"interpretation": "Post-hoc declared family/category sensitivity, not a correction of frozen strict coding."
},
"per_answer": {
"S3-CHATGPT-M1-R1": {
"strict": [
"Google Search Console"
],
"family": [
"Google Search Console"
],
"category": [
"general_measurement_tool"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Search Console"
],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-M1-R2": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-M1-R3": {
"strict": [
"Google Analytics 4",
"Google Sheets",
"Plausible",
"PostHog"
],
"family": [
"Google Analytics",
"Google Sheets",
"Plausible",
"PostHog"
],
"category": [
"general_measurement_tool"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Analytics 4",
"Google Sheets",
"Plausible",
"PostHog"
],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M1-R1": {
"strict": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"X",
"geo-tool.com GEO check"
],
"family": [
"Bing Webmaster Tools",
"Google Analytics",
"Google Search Console",
"HubSpot grader family",
"Mangools AI search family",
"X",
"geo-tool.com GEO readiness family"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"geo-tool.com GEO check"
],
"general_measurement_tool": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console"
],
"research_or_publication_channel": [
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M1-R2": {
"strict": [
"Ahrefs AI visibility tools (unspecified)",
"AthenaHQ Essential",
"Bing Webmaster Tools",
"GetMentioned free AI visibility report",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"Semrush AI visibility tools (unspecified)",
"X",
"YouTube"
],
"family": [
"Ahrefs AI visibility family",
"AthenaHQ visibility family",
"Bing Webmaster Tools",
"GetMentioned visibility family",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Semrush AI visibility family",
"X",
"YouTube"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs AI visibility tools (unspecified)",
"AthenaHQ Essential",
"GetMentioned free AI visibility report",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"Semrush AI visibility tools (unspecified)"
],
"general_measurement_tool": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"research_or_publication_channel": [
"X",
"YouTube"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M1-R3": {
"strict": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher"
],
"family": [
"Bing Webmaster Tools",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family"
],
"category": [
"general_measurement_tool",
"geo_visibility_product"
],
"type_filtered": {
"geo_visibility_product": [
"HubSpot AEO Grader",
"Mangools AI Search Watcher"
],
"general_measurement_tool": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-M2-R1": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-M2-R2": {
"strict": [
"Google Analytics",
"X"
],
"family": [
"Google Analytics",
"X"
],
"category": [
"general_measurement_tool",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Analytics"
],
"research_or_publication_channel": [
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-M2-R3": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M2-R1": {
"strict": [
"Ahrefs Brand Radar",
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"Reddit",
"Semrush AI Visibility Checker",
"X"
],
"family": [
"Ahrefs AI visibility family",
"Bing Webmaster Tools",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Reddit",
"Semrush AI visibility family",
"X"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs Brand Radar",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"Semrush AI Visibility Checker"
],
"general_measurement_tool": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"research_or_publication_channel": [
"Reddit",
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M2-R2": {
"strict": [
"Ahrefs Brand Radar",
"Google People also ask",
"Google Search Console",
"Google Sheets",
"Notion",
"Reddit",
"X"
],
"family": [
"Ahrefs AI visibility family",
"Google Search",
"Google Search Console",
"Google Sheets",
"Notion",
"Reddit",
"X"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs Brand Radar"
],
"general_measurement_tool": [
"Google Search Console",
"Google Sheets",
"Notion"
],
"research_or_publication_channel": [
"Google People also ask",
"Reddit",
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-M2-R3": {
"strict": [
"Ahrefs AI Visibility Checker",
"Capterra",
"G2",
"Google Search",
"Google Sheets",
"Indexly Grok Visibility Checker",
"Mangools AI Search Watcher",
"Product Hunt",
"Ranked AI Visibility Checker",
"Reddit",
"Snoika brand checker",
"X"
],
"family": [
"Ahrefs AI visibility family",
"Capterra",
"G2",
"Google Search",
"Google Sheets",
"Indexly visibility family",
"Mangools AI search family",
"Product Hunt",
"Ranked visibility family",
"Reddit",
"Snoika visibility family",
"X"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs AI Visibility Checker",
"Indexly Grok Visibility Checker",
"Mangools AI Search Watcher",
"Ranked AI Visibility Checker",
"Snoika brand checker"
],
"general_measurement_tool": [
"Google Sheets"
],
"research_or_publication_channel": [
"Capterra",
"G2",
"Google Search",
"Product Hunt",
"Reddit",
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V1-R1": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V1-R2": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V1-R3": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V1-R1": {
"strict": [
"Ahrefs AI Visibility Checker",
"Google Analytics 4",
"Google People also ask",
"Google Search Console",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"Reddit",
"X"
],
"family": [
"Ahrefs AI visibility family",
"Google Analytics",
"Google Search",
"Google Search Console",
"HubSpot grader family",
"Mangools AI search family",
"Reddit",
"X"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs AI Visibility Checker",
"HubSpot AEO Grader",
"Mangools AI Search Grader"
],
"general_measurement_tool": [
"Google Analytics 4",
"Google Search Console"
],
"research_or_publication_channel": [
"Google People also ask",
"Reddit",
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V1-R2": {
"strict": [
"Ahrefs AI visibility tools (unspecified)",
"Google Analytics 4",
"Google Search Console",
"HubSpot grader (unspecified)",
"Semrush AI visibility tools (unspecified)"
],
"family": [
"Ahrefs AI visibility family",
"Google Analytics",
"Google Search Console",
"HubSpot grader family",
"Semrush AI visibility family"
],
"category": [
"general_measurement_tool",
"geo_visibility_product"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs AI visibility tools (unspecified)",
"HubSpot grader (unspecified)",
"Semrush AI visibility tools (unspecified)"
],
"general_measurement_tool": [
"Google Analytics 4",
"Google Search Console"
],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V1-R3": {
"strict": [
"Ahrefs AI visibility tools (unspecified)",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AI Search Grader",
"Semrush AI visibility tools (unspecified)",
"X"
],
"family": [
"Ahrefs AI visibility family",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Semrush AI visibility family",
"X"
],
"category": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [
"Ahrefs AI visibility tools (unspecified)",
"HubSpot AI Search Grader",
"Semrush AI visibility tools (unspecified)"
],
"general_measurement_tool": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"research_or_publication_channel": [
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V2-R1": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V2-R2": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-CHATGPT-V2-R3": {
"strict": [],
"family": [],
"category": [],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [],
"research_or_publication_channel": [],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V2-R1": {
"strict": [
"GitHub",
"Google Analytics 4",
"Google Search Console",
"Reddit",
"X"
],
"family": [
"GitHub",
"Google Analytics",
"Google Search Console",
"Reddit",
"X"
],
"category": [
"general_measurement_tool",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Analytics 4",
"Google Search Console"
],
"research_or_publication_channel": [
"GitHub",
"Reddit",
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V2-R2": {
"strict": [
"Google Analytics 4",
"X"
],
"family": [
"Google Analytics",
"X"
],
"category": [
"general_measurement_tool",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Analytics 4"
],
"research_or_publication_channel": [
"X"
],
"other_ai_surface_or_feature": []
}
},
"S3-GROK-V2-R3": {
"strict": [
"Google Analytics 4",
"X"
],
"family": [
"Google Analytics",
"X"
],
"category": [
"general_measurement_tool",
"research_or_publication_channel"
],
"type_filtered": {
"geo_visibility_product": [],
"general_measurement_tool": [
"Google Analytics 4"
],
"research_or_publication_channel": [
"X"
],
"other_ai_surface_or_feature": []
}
}
},
"wording_contrasts": [
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "M",
"views": {
"strict": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics 4": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
},
"Plausible": {
"numerator": 1,
"denominator": 3
},
"PostHog": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics",
"X"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 0,
"union": 7,
"shared": []
}
},
"family": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"Google Analytics",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
},
"Plausible": {
"numerator": 1,
"denominator": 3
},
"PostHog": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics",
"X"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.16666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 2,
"intersection": 1,
"union": 6,
"shared": [
"Google Analytics"
]
}
},
"category": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"general_measurement_tool"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"general_measurement_tool",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 1,
"denominator": 3
},
"research_or_publication_channel": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.5,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 2,
"intersection": 1,
"union": 2,
"shared": [
"general_measurement_tool"
]
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics 4": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
},
"Plausible": {
"numerator": 1,
"denominator": 3
},
"PostHog": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 1,
"intersection": 0,
"union": 6,
"shared": []
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"X"
],
"partial_coverage": false,
"prevalence": {
"X": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 0,
"right_size": 1,
"intersection": 0,
"union": 1,
"shared": []
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
}
},
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "V",
"views": {
"strict": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"family": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"category": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
}
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "M",
"views": {
"strict": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"Ahrefs AI visibility tools (unspecified)",
"AthenaHQ Essential",
"Bing Webmaster Tools",
"GetMentioned free AI visibility report",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"Mangools AI Search Watcher",
"Semrush AI visibility tools (unspecified)",
"X",
"YouTube",
"geo-tool.com GEO check"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 3
},
"AthenaHQ Essential": {
"numerator": 1,
"denominator": 3
},
"Bing Webmaster Tools": {
"numerator": 3,
"denominator": 3
},
"GetMentioned free AI visibility report": {
"numerator": 1,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 2,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 3,
"denominator": 3
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Watcher": {
"numerator": 2,
"denominator": 3
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 2,
"denominator": 3
},
"YouTube": {
"numerator": 1,
"denominator": 3
},
"geo-tool.com GEO check": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs Brand Radar",
"Bing Webmaster Tools",
"Capterra",
"G2",
"Google Analytics 4",
"Google People also ask",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Indexly Grok Visibility Checker",
"Mangools AI Search Watcher",
"Notion",
"Product Hunt",
"Ranked AI Visibility Checker",
"Reddit",
"Semrush AI Visibility Checker",
"Snoika brand checker",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Ahrefs Brand Radar": {
"numerator": 2,
"denominator": 3
},
"Bing Webmaster Tools": {
"numerator": 1,
"denominator": 3
},
"Capterra": {
"numerator": 1,
"denominator": 3
},
"G2": {
"numerator": 1,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 1,
"denominator": 3
},
"Google People also ask": {
"numerator": 1,
"denominator": 3
},
"Google Search": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 2,
"denominator": 3
},
"Google Sheets": {
"numerator": 3,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 3
},
"Indexly Grok Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Watcher": {
"numerator": 2,
"denominator": 3
},
"Notion": {
"numerator": 1,
"denominator": 3
},
"Product Hunt": {
"numerator": 1,
"denominator": 3
},
"Ranked AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 3,
"denominator": 3
},
"Semrush AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Snoika brand checker": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.25925925925925924,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 14,
"right_size": 20,
"intersection": 7,
"union": 27,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Watcher",
"X"
]
}
},
"family": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"Ahrefs AI visibility family",
"AthenaHQ visibility family",
"Bing Webmaster Tools",
"GetMentioned visibility family",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Semrush AI visibility family",
"X",
"YouTube",
"geo-tool.com GEO readiness family"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility family": {
"numerator": 1,
"denominator": 3
},
"AthenaHQ visibility family": {
"numerator": 1,
"denominator": 3
},
"Bing Webmaster Tools": {
"numerator": 3,
"denominator": 3
},
"GetMentioned visibility family": {
"numerator": 1,
"denominator": 3
},
"Google Analytics": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 2,
"denominator": 3
},
"HubSpot grader family": {
"numerator": 3,
"denominator": 3
},
"Mangools AI search family": {
"numerator": 3,
"denominator": 3
},
"Semrush AI visibility family": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 2,
"denominator": 3
},
"YouTube": {
"numerator": 1,
"denominator": 3
},
"geo-tool.com GEO readiness family": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI visibility family",
"Bing Webmaster Tools",
"Capterra",
"G2",
"Google Analytics",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Indexly visibility family",
"Mangools AI search family",
"Notion",
"Product Hunt",
"Ranked visibility family",
"Reddit",
"Semrush AI visibility family",
"Snoika visibility family",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility family": {
"numerator": 3,
"denominator": 3
},
"Bing Webmaster Tools": {
"numerator": 1,
"denominator": 3
},
"Capterra": {
"numerator": 1,
"denominator": 3
},
"G2": {
"numerator": 1,
"denominator": 3
},
"Google Analytics": {
"numerator": 1,
"denominator": 3
},
"Google Search": {
"numerator": 2,
"denominator": 3
},
"Google Search Console": {
"numerator": 2,
"denominator": 3
},
"Google Sheets": {
"numerator": 3,
"denominator": 3
},
"HubSpot grader family": {
"numerator": 1,
"denominator": 3
},
"Indexly visibility family": {
"numerator": 1,
"denominator": 3
},
"Mangools AI search family": {
"numerator": 2,
"denominator": 3
},
"Notion": {
"numerator": 1,
"denominator": 3
},
"Product Hunt": {
"numerator": 1,
"denominator": 3
},
"Ranked visibility family": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 3,
"denominator": 3
},
"Semrush AI visibility family": {
"numerator": 1,
"denominator": 3
},
"Snoika visibility family": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.4090909090909091,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 13,
"right_size": 18,
"intersection": 9,
"union": 22,
"shared": [
"Ahrefs AI visibility family",
"Bing Webmaster Tools",
"Google Analytics",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Semrush AI visibility family",
"X"
]
}
},
"category": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 3,
"denominator": 3
},
"geo_visibility_product": {
"numerator": 3,
"denominator": 3
},
"research_or_publication_channel": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 3,
"denominator": 3
},
"geo_visibility_product": {
"numerator": 3,
"denominator": 3
},
"research_or_publication_channel": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 1.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 3,
"intersection": 3,
"union": 3,
"shared": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
]
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"Ahrefs AI visibility tools (unspecified)",
"AthenaHQ Essential",
"GetMentioned free AI visibility report",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"Mangools AI Search Watcher",
"Semrush AI visibility tools (unspecified)",
"geo-tool.com GEO check"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 3
},
"AthenaHQ Essential": {
"numerator": 1,
"denominator": 3
},
"GetMentioned free AI visibility report": {
"numerator": 1,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 3,
"denominator": 3
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Watcher": {
"numerator": 2,
"denominator": 3
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 3
},
"geo-tool.com GEO check": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs Brand Radar",
"HubSpot AEO Grader",
"Indexly Grok Visibility Checker",
"Mangools AI Search Watcher",
"Ranked AI Visibility Checker",
"Semrush AI Visibility Checker",
"Snoika brand checker"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Ahrefs Brand Radar": {
"numerator": 2,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 3
},
"Indexly Grok Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Watcher": {
"numerator": 2,
"denominator": 3
},
"Ranked AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Semrush AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Snoika brand checker": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.14285714285714285,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 8,
"intersection": 2,
"union": 14,
"shared": [
"HubSpot AEO Grader",
"Mangools AI Search Watcher"
]
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"partial_coverage": false,
"prevalence": {
"Bing Webmaster Tools": {
"numerator": 3,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Notion"
],
"partial_coverage": false,
"prevalence": {
"Bing Webmaster Tools": {
"numerator": 1,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 2,
"denominator": 3
},
"Google Sheets": {
"numerator": 3,
"denominator": 3
},
"Notion": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.8,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 5,
"intersection": 4,
"union": 5,
"shared": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
]
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"X",
"YouTube"
],
"partial_coverage": false,
"prevalence": {
"X": {
"numerator": 2,
"denominator": 3
},
"YouTube": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Capterra",
"G2",
"Google People also ask",
"Google Search",
"Product Hunt",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"Capterra": {
"numerator": 1,
"denominator": 3
},
"G2": {
"numerator": 1,
"denominator": 3
},
"Google People also ask": {
"numerator": 1,
"denominator": 3
},
"Google Search": {
"numerator": 1,
"denominator": 3
},
"Product Hunt": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 3,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.125,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 7,
"intersection": 1,
"union": 8,
"shared": [
"X"
]
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
}
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "V",
"views": {
"strict": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"Google Analytics 4",
"Google People also ask",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"HubSpot AI Search Grader",
"HubSpot grader (unspecified)",
"Mangools AI Search Grader",
"Reddit",
"Semrush AI visibility tools (unspecified)",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google People also ask": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 3
},
"HubSpot AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"HubSpot grader (unspecified)": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 3
},
"X": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"GitHub",
"Google Analytics 4",
"Google Search Console",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"GitHub": {
"numerator": 1,
"denominator": 3
},
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.2857142857142857,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 13,
"right_size": 5,
"intersection": 4,
"union": 14,
"shared": [
"Google Analytics 4",
"Google Search Console",
"Reddit",
"X"
]
}
},
"family": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"Ahrefs AI visibility family",
"Google Analytics",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Reddit",
"Semrush AI visibility family",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility family": {
"numerator": 3,
"denominator": 3
},
"Google Analytics": {
"numerator": 3,
"denominator": 3
},
"Google Search": {
"numerator": 1,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
},
"HubSpot grader family": {
"numerator": 3,
"denominator": 3
},
"Mangools AI search family": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"Semrush AI visibility family": {
"numerator": 2,
"denominator": 3
},
"X": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"GitHub",
"Google Analytics",
"Google Search Console",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"GitHub": {
"numerator": 1,
"denominator": 3
},
"Google Analytics": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.36363636363636365,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 10,
"right_size": 5,
"intersection": 4,
"union": 11,
"shared": [
"Google Analytics",
"Google Search Console",
"Reddit",
"X"
]
}
},
"category": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 3,
"denominator": 3
},
"geo_visibility_product": {
"numerator": 3,
"denominator": 3
},
"research_or_publication_channel": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"general_measurement_tool",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 3,
"denominator": 3
},
"research_or_publication_channel": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.6666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 2,
"intersection": 2,
"union": 3,
"shared": [
"general_measurement_tool",
"research_or_publication_channel"
]
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"HubSpot AEO Grader",
"HubSpot AI Search Grader",
"HubSpot grader (unspecified)",
"Mangools AI Search Grader",
"Semrush AI visibility tools (unspecified)"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 3
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 3
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 3
},
"HubSpot AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"HubSpot grader (unspecified)": {
"numerator": 1,
"denominator": 3
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 3
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 0,
"intersection": 0,
"union": 7,
"shared": []
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 3,
"denominator": 3
},
"Google Sheets": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"Google Analytics 4",
"Google Search Console"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics 4": {
"numerator": 3,
"denominator": 3
},
"Google Search Console": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.6666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 2,
"intersection": 2,
"union": 3,
"shared": [
"Google Analytics 4",
"Google Search Console"
]
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"Google People also ask",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"Google People also ask": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 2,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"GitHub",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"GitHub": {
"numerator": 1,
"denominator": 3
},
"Reddit": {
"numerator": 1,
"denominator": 3
},
"X": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.5,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 3,
"intersection": 2,
"union": 4,
"shared": [
"Reddit",
"X"
]
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
}
}
],
"intent_contrasts": [
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"views": {
"strict": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog",
"X"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 6
},
"Google Analytics 4": {
"numerator": 1,
"denominator": 6
},
"Google Search Console": {
"numerator": 1,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
},
"Plausible": {
"numerator": 1,
"denominator": 6
},
"PostHog": {
"numerator": 1,
"denominator": 6
},
"X": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 0,
"intersection": 0,
"union": 7,
"shared": []
}
},
"family": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog",
"X"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 2,
"denominator": 6
},
"Google Search Console": {
"numerator": 1,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
},
"Plausible": {
"numerator": 1,
"denominator": 6
},
"PostHog": {
"numerator": 1,
"denominator": 6
},
"X": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 6,
"right_size": 0,
"intersection": 0,
"union": 6,
"shared": []
}
},
"category": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"general_measurement_tool",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 3,
"denominator": 6
},
"research_or_publication_channel": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 0,
"intersection": 0,
"union": 2,
"shared": []
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"Google Analytics",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Plausible",
"PostHog"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics": {
"numerator": 1,
"denominator": 6
},
"Google Analytics 4": {
"numerator": 1,
"denominator": 6
},
"Google Search Console": {
"numerator": 1,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
},
"Plausible": {
"numerator": 1,
"denominator": 6
},
"PostHog": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 6,
"right_size": 0,
"intersection": 0,
"union": 6,
"shared": []
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"X"
],
"partial_coverage": false,
"prevalence": {
"X": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 1,
"right_size": 0,
"intersection": 0,
"union": 1,
"shared": []
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3",
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3",
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
},
"unit": "Pooled M-versus-V sets; different tasks, not a wording-only effect."
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"views": {
"strict": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"Ahrefs Brand Radar",
"AthenaHQ Essential",
"Bing Webmaster Tools",
"Capterra",
"G2",
"GetMentioned free AI visibility report",
"Google Analytics 4",
"Google People also ask",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Indexly Grok Visibility Checker",
"Mangools AI Search Grader",
"Mangools AI Search Watcher",
"Notion",
"Product Hunt",
"Ranked AI Visibility Checker",
"Reddit",
"Semrush AI Visibility Checker",
"Semrush AI visibility tools (unspecified)",
"Snoika brand checker",
"X",
"YouTube",
"geo-tool.com GEO check"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Ahrefs Brand Radar": {
"numerator": 2,
"denominator": 6
},
"AthenaHQ Essential": {
"numerator": 1,
"denominator": 6
},
"Bing Webmaster Tools": {
"numerator": 4,
"denominator": 6
},
"Capterra": {
"numerator": 1,
"denominator": 6
},
"G2": {
"numerator": 1,
"denominator": 6
},
"GetMentioned free AI visibility report": {
"numerator": 1,
"denominator": 6
},
"Google Analytics 4": {
"numerator": 4,
"denominator": 6
},
"Google People also ask": {
"numerator": 1,
"denominator": 6
},
"Google Search": {
"numerator": 1,
"denominator": 6
},
"Google Search Console": {
"numerator": 5,
"denominator": 6
},
"Google Sheets": {
"numerator": 5,
"denominator": 6
},
"HubSpot AEO Grader": {
"numerator": 4,
"denominator": 6
},
"Indexly Grok Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Watcher": {
"numerator": 4,
"denominator": 6
},
"Notion": {
"numerator": 1,
"denominator": 6
},
"Product Hunt": {
"numerator": 1,
"denominator": 6
},
"Ranked AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 3,
"denominator": 6
},
"Semrush AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Snoika brand checker": {
"numerator": 1,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
},
"YouTube": {
"numerator": 1,
"denominator": 6
},
"geo-tool.com GEO check": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"GitHub",
"Google Analytics 4",
"Google People also ask",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"HubSpot AI Search Grader",
"HubSpot grader (unspecified)",
"Mangools AI Search Grader",
"Reddit",
"Semrush AI visibility tools (unspecified)",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 6
},
"GitHub": {
"numerator": 1,
"denominator": 6
},
"Google Analytics 4": {
"numerator": 6,
"denominator": 6
},
"Google People also ask": {
"numerator": 1,
"denominator": 6
},
"Google Search Console": {
"numerator": 4,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 6
},
"HubSpot AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"HubSpot grader (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 2,
"denominator": 6
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 0.36666666666666664,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 27,
"right_size": 14,
"intersection": 11,
"union": 30,
"shared": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"Google Analytics 4",
"Google People also ask",
"Google Search Console",
"Google Sheets",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"Reddit",
"Semrush AI visibility tools (unspecified)",
"X"
]
}
},
"family": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI visibility family",
"AthenaHQ visibility family",
"Bing Webmaster Tools",
"Capterra",
"G2",
"GetMentioned visibility family",
"Google Analytics",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Indexly visibility family",
"Mangools AI search family",
"Notion",
"Product Hunt",
"Ranked visibility family",
"Reddit",
"Semrush AI visibility family",
"Snoika visibility family",
"X",
"YouTube",
"geo-tool.com GEO readiness family"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility family": {
"numerator": 4,
"denominator": 6
},
"AthenaHQ visibility family": {
"numerator": 1,
"denominator": 6
},
"Bing Webmaster Tools": {
"numerator": 4,
"denominator": 6
},
"Capterra": {
"numerator": 1,
"denominator": 6
},
"G2": {
"numerator": 1,
"denominator": 6
},
"GetMentioned visibility family": {
"numerator": 1,
"denominator": 6
},
"Google Analytics": {
"numerator": 4,
"denominator": 6
},
"Google Search": {
"numerator": 2,
"denominator": 6
},
"Google Search Console": {
"numerator": 5,
"denominator": 6
},
"Google Sheets": {
"numerator": 5,
"denominator": 6
},
"HubSpot grader family": {
"numerator": 4,
"denominator": 6
},
"Indexly visibility family": {
"numerator": 1,
"denominator": 6
},
"Mangools AI search family": {
"numerator": 5,
"denominator": 6
},
"Notion": {
"numerator": 1,
"denominator": 6
},
"Product Hunt": {
"numerator": 1,
"denominator": 6
},
"Ranked visibility family": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 3,
"denominator": 6
},
"Semrush AI visibility family": {
"numerator": 2,
"denominator": 6
},
"Snoika visibility family": {
"numerator": 1,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
},
"YouTube": {
"numerator": 1,
"denominator": 6
},
"geo-tool.com GEO readiness family": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"Ahrefs AI visibility family",
"GitHub",
"Google Analytics",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Reddit",
"Semrush AI visibility family",
"X"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI visibility family": {
"numerator": 3,
"denominator": 6
},
"GitHub": {
"numerator": 1,
"denominator": 6
},
"Google Analytics": {
"numerator": 6,
"denominator": 6
},
"Google Search": {
"numerator": 1,
"denominator": 6
},
"Google Search Console": {
"numerator": 4,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
},
"HubSpot grader family": {
"numerator": 3,
"denominator": 6
},
"Mangools AI search family": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 2,
"denominator": 6
},
"Semrush AI visibility family": {
"numerator": 2,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 0.43478260869565216,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 22,
"right_size": 11,
"intersection": 10,
"union": 23,
"shared": [
"Ahrefs AI visibility family",
"Google Analytics",
"Google Search",
"Google Search Console",
"Google Sheets",
"HubSpot grader family",
"Mangools AI search family",
"Reddit",
"Semrush AI visibility family",
"X"
]
}
},
"category": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 6,
"denominator": 6
},
"geo_visibility_product": {
"numerator": 6,
"denominator": 6
},
"research_or_publication_channel": {
"numerator": 5,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
],
"partial_coverage": false,
"prevalence": {
"general_measurement_tool": {
"numerator": 6,
"denominator": 6
},
"geo_visibility_product": {
"numerator": 3,
"denominator": 6
},
"research_or_publication_channel": {
"numerator": 5,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 1.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 3,
"intersection": 3,
"union": 3,
"shared": [
"general_measurement_tool",
"geo_visibility_product",
"research_or_publication_channel"
]
}
},
"type_filtered": {
"geo_visibility_product": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"Ahrefs Brand Radar",
"AthenaHQ Essential",
"GetMentioned free AI visibility report",
"HubSpot AEO Grader",
"Indexly Grok Visibility Checker",
"Mangools AI Search Grader",
"Mangools AI Search Watcher",
"Ranked AI Visibility Checker",
"Semrush AI Visibility Checker",
"Semrush AI visibility tools (unspecified)",
"Snoika brand checker",
"geo-tool.com GEO check"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Ahrefs Brand Radar": {
"numerator": 2,
"denominator": 6
},
"AthenaHQ Essential": {
"numerator": 1,
"denominator": 6
},
"GetMentioned free AI visibility report": {
"numerator": 1,
"denominator": 6
},
"HubSpot AEO Grader": {
"numerator": 4,
"denominator": 6
},
"Indexly Grok Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Watcher": {
"numerator": 4,
"denominator": 6
},
"Ranked AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Semrush AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Snoika brand checker": {
"numerator": 1,
"denominator": 6
},
"geo-tool.com GEO check": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"HubSpot AEO Grader",
"HubSpot AI Search Grader",
"HubSpot grader (unspecified)",
"Mangools AI Search Grader",
"Semrush AI visibility tools (unspecified)"
],
"partial_coverage": false,
"prevalence": {
"Ahrefs AI Visibility Checker": {
"numerator": 1,
"denominator": 6
},
"Ahrefs AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 6
},
"HubSpot AEO Grader": {
"numerator": 1,
"denominator": 6
},
"HubSpot AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"HubSpot grader (unspecified)": {
"numerator": 1,
"denominator": 6
},
"Mangools AI Search Grader": {
"numerator": 1,
"denominator": 6
},
"Semrush AI visibility tools (unspecified)": {
"numerator": 2,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 0.3125,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 14,
"right_size": 7,
"intersection": 5,
"union": 16,
"shared": [
"Ahrefs AI Visibility Checker",
"Ahrefs AI visibility tools (unspecified)",
"HubSpot AEO Grader",
"Mangools AI Search Grader",
"Semrush AI visibility tools (unspecified)"
]
}
},
"general_measurement_tool": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Bing Webmaster Tools",
"Google Analytics 4",
"Google Search Console",
"Google Sheets",
"Notion"
],
"partial_coverage": false,
"prevalence": {
"Bing Webmaster Tools": {
"numerator": 4,
"denominator": 6
},
"Google Analytics 4": {
"numerator": 4,
"denominator": 6
},
"Google Search Console": {
"numerator": 5,
"denominator": 6
},
"Google Sheets": {
"numerator": 5,
"denominator": 6
},
"Notion": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
],
"partial_coverage": false,
"prevalence": {
"Google Analytics 4": {
"numerator": 6,
"denominator": 6
},
"Google Search Console": {
"numerator": 4,
"denominator": 6
},
"Google Sheets": {
"numerator": 1,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 0.6,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 3,
"intersection": 3,
"union": 5,
"shared": [
"Google Analytics 4",
"Google Search Console",
"Google Sheets"
]
}
},
"research_or_publication_channel": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"Capterra",
"G2",
"Google People also ask",
"Google Search",
"Product Hunt",
"Reddit",
"X",
"YouTube"
],
"partial_coverage": false,
"prevalence": {
"Capterra": {
"numerator": 1,
"denominator": 6
},
"G2": {
"numerator": 1,
"denominator": 6
},
"Google People also ask": {
"numerator": 1,
"denominator": 6
},
"Google Search": {
"numerator": 1,
"denominator": 6
},
"Product Hunt": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 3,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
},
"YouTube": {
"numerator": 1,
"denominator": 6
}
}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"GitHub",
"Google People also ask",
"Reddit",
"X"
],
"partial_coverage": false,
"prevalence": {
"GitHub": {
"numerator": 1,
"denominator": 6
},
"Google People also ask": {
"numerator": 1,
"denominator": 6
},
"Reddit": {
"numerator": 2,
"denominator": 6
},
"X": {
"numerator": 5,
"denominator": 6
}
}
},
"pooled_overlap": {
"value": 0.3333333333333333,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 8,
"right_size": 4,
"intersection": 3,
"union": 9,
"shared": [
"Google People also ask",
"Reddit",
"X"
]
}
},
"other_ai_surface_or_feature": {
"left": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3",
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"right": {
"eligible_denominator": 6,
"planned_positions": 6,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3",
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [],
"partial_coverage": false,
"prevalence": {}
},
"pooled_overlap": {
"value": null,
"reason": "N/A: both eligible sets are empty.",
"eligible": true,
"both_empty": true,
"left_size": 0,
"right_size": 0,
"intersection": 0,
"union": 0,
"shared": []
}
}
}
},
"unit": "Pooled M-versus-V sets; different tasks, not a wording-only effect."
}
]
},
"url_lens": {
"normalization_policy": {
"removed_query_keys": [
"_bhlid",
"_ga",
"_gl",
"_rsc",
"access_token",
"aff",
"aff_id",
"affiliate",
"api_key",
"apikey",
"authorization",
"awsaccesskeyid",
"campaign",
"ck_subscriber_id",
"client_id",
"cluster",
"cluster_id",
"credential",
"expires",
"fbclid",
"fpr",
"gclid",
"googleaccessid",
"igshid",
"isappinstalled",
"key",
"key-pair-id",
"mc_cid",
"mc_eid",
"mkt_tok",
"password",
"policy",
"ref",
"referrer",
"refresh_token",
"request_id",
"response_id",
"roistat_visit",
"secret",
"session_id",
"sh_kit",
"sig",
"signature",
"spm",
"srsltid",
"st_source",
"token",
"trk",
"via"
],
"removed_query_prefixes": [
"utm_",
"autm_",
"x-amz-",
"x-goog-"
],
"basis": "Same declared query cleanup as export_stage3_public.public_url; private/invalid/userinfo URLs withheld.",
"preserved": [
"scheme",
"hostname including www",
"path including encoded text",
"fragment",
"non-sensitive query parameters"
],
"not_performed": [
"redirect resolution",
"path/fragment decoding or repair",
"independent source verification"
],
"representation_rule": "Deduplicate annotation, source-table and explicit-body representations by cleaned URL; unreviewed body URLs are diagnostic only.",
"privacy": "Only cleaned comparison URLs are emitted. Exact returned URLs remain in the input records; signed copies may cease to be fetchable after cleanup."
},
"per_answer": {
"S3-CHATGPT-M1-R1": {
"observed_urls": [
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"strict_urls": [
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 2,
"coded_explicit_body_links": 2
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-M1-R2": {
"observed_urls": [
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"strict_urls": [
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.x.com/en/using-x/about-grok",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-M1-R3": {
"observed_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"strict_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://docs.x.ai/developers/model-capabilities/text/best-practices",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 3,
"coded_answer_sources": 3,
"explicit_body_URLs": 3,
"coded_explicit_body_links": 3
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M1-R1": {
"observed_urls": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://www.gomega.ai/blog/how-to-measure-geo-performance"
],
"strict_urls": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://www.gomega.ai/blog/how-to-measure-geo-performance"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://geoaura.world/blog/how-to-measure-geo-visibility",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://shmeer.com/blog/how-to-measure-geo",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.gomega.ai/blog/how-to-measure-geo-performance",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M1-R2": {
"observed_urls": [],
"strict_urls": [],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M1-R3": {
"observed_urls": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/",
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/",
"https://www.gomega.ai/blog/how-to-measure-geo-performance",
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool"
],
"strict_urls": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/",
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/",
"https://www.gomega.ai/blog/how-to-measure-geo-performance",
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://geoaura.world/blog/how-to-measure-geo-visibility",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://shmeer.com/blog/how-to-measure-geo",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.gomega.ai/blog/how-to-measure-geo-performance",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-M2-R1": {
"observed_urls": [
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"strict_urls": [
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.x.com/en/using-x/about-grok",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-M2-R2": {
"observed_urls": [
"https://docs.x.ai/developers/tools/web-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"strict_urls": [
"https://docs.x.ai/developers/tools/web-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://docs.x.ai/developers/tools/web-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 2,
"coded_explicit_body_links": 2
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-M2-R3": {
"observed_urls": [
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"strict_urls": [
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 2,
"coded_explicit_body_links": 2
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.x.com/en/using-x/about-grok",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 2,
"coded_explicit_body_links": 2
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M2-R1": {
"observed_urls": [
"https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/",
"https://solcrys.com/free-chatgpt-visibility-tools/",
"https://stork.ai/blog/does-chatgpt-recommend-your-product"
],
"strict_urls": null,
"citation_mapping_complete": false,
"coverage_qualified": true,
"reason": "Observed URL inventory only; complete citation mapping not confirmed.",
"records": [
{
"comparison_url": "https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://solcrys.com/free-chatgpt-visibility-tools/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://stork.ai/blog/does-chatgpt-recommend-your-product",
"representation_counts": {
"recorded_answer_links": 3,
"coded_answer_sources": 3,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M2-R2": {
"observed_urls": [],
"strict_urls": [],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-M2-R3": {
"observed_urls": [
"https://indexly.ai/grok-visibility-checker",
"https://mangools.com/ai-search-watcher",
"https://snoika.com/free-tools/brand-ai-visibility-checker",
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/",
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/",
"https://www.ranked.ai/free-tools/ai-visibility-checker"
],
"strict_urls": [
"https://indexly.ai/grok-visibility-checker",
"https://mangools.com/ai-search-watcher",
"https://snoika.com/free-tools/brand-ai-visibility-checker",
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/",
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/",
"https://www.ranked.ai/free-tools/ai-visibility-checker"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://indexly.ai/grok-visibility-checker",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://mangools.com/ai-search-watcher",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://snoika.com/free-tools/brand-ai-visibility-checker",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.ranked.ai/free-tools/ai-visibility-checker",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V1-R1": {
"observed_urls": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://platform.openai.com/docs/guides/evaluation-best-practices"
],
"strict_urls": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://platform.openai.com/docs/guides/evaluation-best-practices"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://docs.x.ai/docs/guides/live-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://docs.x.ai/docs/models",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://platform.openai.com/docs/guides/evaluation-best-practices",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V1-R2": {
"observed_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"strict_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V1-R3": {
"observed_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"strict_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"representation_counts": {
"recorded_answer_links": 3,
"coded_answer_sources": 3,
"explicit_body_URLs": 3,
"coded_explicit_body_links": 3
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V1-R1": {
"observed_urls": [
"https://ahrefs.com/free-ai-visibility",
"https://cognizo.ai/blog/ai-visibility-score",
"https://higoodie.com/blog/how-to-rank-on-grok/",
"https://hubspot.com/aeo-grader",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://maxaeo.ai/blog/ai-visibility-sample-size/",
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/"
],
"strict_urls": null,
"citation_mapping_complete": false,
"coverage_qualified": true,
"reason": "Observed URL inventory only; complete citation mapping not confirmed.",
"records": [
{
"comparison_url": "https://ahrefs.com/free-ai-visibility",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://cognizo.ai/blog/ai-visibility-score",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://higoodie.com/blog/how-to-rank-on-grok/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://hubspot.com/aeo-grader",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://maxaeo.ai/blog/ai-visibility-sample-size/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V1-R2": {
"observed_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/",
"https://www.visilay.com/blog/ai-visibility-score/"
],
"strict_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/",
"https://www.visilay.com/blog/ai-visibility-score/"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.visilay.com/blog/ai-visibility-score/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V1-R3": {
"observed_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/"
],
"strict_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V2-R1": {
"observed_urls": [
"https://arxiv.org/abs/2410.03492",
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"strict_urls": [
"https://arxiv.org/abs/2410.03492",
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://arxiv.org/abs/2410.03492",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://docs.x.ai/developers/model-capabilities/text/best-practices",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 4,
"coded_answer_sources": 4,
"explicit_body_URLs": 4,
"coded_explicit_body_links": 4
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V2-R2": {
"observed_urls": [
"https://docs.x.ai/cookbook/examples/x-post-fact-check",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"strict_urls": [
"https://docs.x.ai/cookbook/examples/x-post-fact-check",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://docs.x.ai/cookbook/examples/x-post-fact-check",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 3,
"coded_answer_sources": 3,
"explicit_body_URLs": 3,
"coded_explicit_body_links": 3
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-CHATGPT-V2-R3": {
"observed_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"strict_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://docs.x.ai/developers/model-capabilities/text/best-practices",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/8313428",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 1,
"coded_explicit_body_links": 1
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V2-R1": {
"observed_urls": [
"https://amicited.com/blog/grok-is-citing-brands-too/",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/",
"https://visilay.com/blog/ai-visibility-score/"
],
"strict_urls": null,
"citation_mapping_complete": false,
"coverage_qualified": true,
"reason": "Observed URL inventory only; complete citation mapping not confirmed.",
"records": [
{
"comparison_url": "https://amicited.com/blog/grok-is-citing-brands-too/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://visilay.com/blog/ai-visibility-score/",
"representation_counts": {
"recorded_answer_links": 2,
"coded_answer_sources": 2,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V2-R2": {
"observed_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui",
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"https://www.aeotrack.io/methodology",
"https://www.loudface.co/blog/what-is-ai-visibility"
],
"strict_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui",
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"https://www.aeotrack.io/methodology",
"https://www.loudface.co/blog/what-is-ai-visibility"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.aeotrack.io/methodology",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.loudface.co/blog/what-is-ai-visibility",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
},
"S3-GROK-V2-R3": {
"observed_urls": [
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands",
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt"
],
"strict_urls": [
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands",
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt"
],
"citation_mapping_complete": true,
"coverage_qualified": false,
"reason": null,
"records": [
{
"comparison_url": "https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://presenc.ai/compare/grok-vs-chatgpt-for-brands",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
},
{
"comparison_url": "https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt",
"representation_counts": {
"recorded_answer_links": 1,
"coded_answer_sources": 1,
"explicit_body_URLs": 0,
"coded_explicit_body_links": 0
},
"count_in_URL_set": 1,
"exact_returned_URLs_retained_in_input_only": true
}
],
"unreviewed_explicit_comparison_urls": []
}
},
"wording_contrasts": [
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "M",
"metrics": {
"strict_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/developers/model-capabilities/text/best-practices": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://help.x.com/en/using-x/about-grok": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"https://docs.x.ai/developers/tools/web-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/developers/tools/web-search": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt": {
"numerator": 3,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://help.x.com/en/using-x/about-grok": {
"numerator": 2,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.4444444444444444,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 6,
"intersection": 4,
"union": 9,
"shared": [
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
]
}
},
"observed_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/developers/model-capabilities/text/best-practices": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://help.x.com/en/using-x/about-grok": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"https://docs.x.ai/developers/tools/web-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/developers/tools/web-search": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt": {
"numerator": 3,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://help.x.com/en/using-x/about-grok": {
"numerator": 2,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.4444444444444444,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 6,
"intersection": 4,
"union": 9,
"shared": [
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.x.com/en/using-x/about-grok"
]
}
},
"cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"openai.com",
"x.ai",
"x.com"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
},
"x.com": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"openai.com",
"x.ai",
"x.com"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
},
"x.com": {
"numerator": 2,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 1.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 3,
"intersection": 3,
"union": 3,
"shared": [
"openai.com",
"x.ai",
"x.com"
]
}
},
"observed_cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M1-R1",
"S3-CHATGPT-M1-R2",
"S3-CHATGPT-M1-R3"
],
"union": [
"openai.com",
"x.ai",
"x.com"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
},
"x.com": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-M2-R1",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-M2-R3"
],
"union": [
"openai.com",
"x.ai",
"x.com"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
},
"x.com": {
"numerator": 2,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 1.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 3,
"right_size": 3,
"intersection": 3,
"union": 3,
"shared": [
"openai.com",
"x.ai",
"x.com"
]
}
}
}
},
{
"platform": "OpenAI API + web search",
"environment_label": "OpenAI API + web search",
"intent": "V",
"metrics": {
"strict_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://platform.openai.com/docs/guides/evaluation-best-practices"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/docs/guides/live-search": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/docs/models": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://platform.openai.com/docs/guides/evaluation-best-practices": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [
"https://arxiv.org/abs/2410.03492",
"https://docs.x.ai/cookbook/examples/x-post-fact-check",
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"partial_coverage": false,
"prevalence": {
"https://arxiv.org/abs/2410.03492": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/cookbook/examples/x-post-fact-check": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/developers/model-capabilities/text/best-practices": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/8313428": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.09090909090909091,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 7,
"intersection": 1,
"union": 11,
"shared": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
]
}
},
"observed_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://platform.openai.com/docs/guides/evaluation-best-practices"
],
"partial_coverage": false,
"prevalence": {
"https://docs.x.ai/docs/guides/live-search": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/docs/models": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt": {
"numerator": 2,
"denominator": 3
},
"https://platform.openai.com/docs/guides/evaluation-best-practices": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [
"https://arxiv.org/abs/2410.03492",
"https://docs.x.ai/cookbook/examples/x-post-fact-check",
"https://docs.x.ai/developers/model-capabilities/text/best-practices",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"partial_coverage": false,
"prevalence": {
"https://arxiv.org/abs/2410.03492": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/cookbook/examples/x-post-fact-check": {
"numerator": 1,
"denominator": 3
},
"https://docs.x.ai/developers/model-capabilities/text/best-practices": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results": {
"numerator": 2,
"denominator": 3
},
"https://help.openai.com/en/articles/8313428": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt": {
"numerator": 1,
"denominator": 3
},
"https://help.openai.com/en/articles/9237897-chatgpt-search": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.09090909090909091,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 5,
"right_size": 7,
"intersection": 1,
"union": 11,
"shared": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
]
}
},
"cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [
"openai.com",
"x.ai"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [
"arxiv.org",
"openai.com",
"x.ai"
],
"partial_coverage": false,
"prevalence": {
"arxiv.org": {
"numerator": 1,
"denominator": 3
},
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.6666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 3,
"intersection": 2,
"union": 3,
"shared": [
"openai.com",
"x.ai"
]
}
},
"observed_cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R2",
"S3-CHATGPT-V1-R3"
],
"union": [
"openai.com",
"x.ai"
],
"partial_coverage": false,
"prevalence": {
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-CHATGPT-V2-R1",
"S3-CHATGPT-V2-R2",
"S3-CHATGPT-V2-R3"
],
"union": [
"arxiv.org",
"openai.com",
"x.ai"
],
"partial_coverage": false,
"prevalence": {
"arxiv.org": {
"numerator": 1,
"denominator": 3
},
"openai.com": {
"numerator": 3,
"denominator": 3
},
"x.ai": {
"numerator": 3,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.6666666666666666,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 2,
"right_size": 3,
"intersection": 2,
"union": 3,
"shared": [
"openai.com",
"x.ai"
]
}
}
}
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "M",
"metrics": {
"strict_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/",
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/",
"https://www.gomega.ai/blog/how-to-measure-geo-performance",
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool"
],
"partial_coverage": false,
"prevalence": {
"https://geoaura.world/blog/how-to-measure-geo-visibility": {
"numerator": 2,
"denominator": 3
},
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/": {
"numerator": 2,
"denominator": 3
},
"https://shmeer.com/blog/how-to-measure-geo": {
"numerator": 2,
"denominator": 3
},
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/": {
"numerator": 1,
"denominator": 3
},
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/": {
"numerator": 1,
"denominator": 3
},
"https://www.gomega.ai/blog/how-to-measure-geo-performance": {
"numerator": 2,
"denominator": 3
},
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"https://indexly.ai/grok-visibility-checker",
"https://mangools.com/ai-search-watcher",
"https://snoika.com/free-tools/brand-ai-visibility-checker",
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/",
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/",
"https://www.ranked.ai/free-tools/ai-visibility-checker"
],
"partial_coverage": true,
"prevalence": {
"https://indexly.ai/grok-visibility-checker": {
"numerator": 1,
"denominator": 2
},
"https://mangools.com/ai-search-watcher": {
"numerator": 1,
"denominator": 2
},
"https://snoika.com/free-tools/brand-ai-visibility-checker": {
"numerator": 1,
"denominator": 2
},
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/": {
"numerator": 1,
"denominator": 2
},
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/": {
"numerator": 1,
"denominator": 2
},
"https://www.ranked.ai/free-tools/ai-visibility-checker": {
"numerator": 1,
"denominator": 2
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 6,
"intersection": 0,
"union": 13,
"shared": []
}
},
"observed_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"https://geoaura.world/blog/how-to-measure-geo-visibility",
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/",
"https://shmeer.com/blog/how-to-measure-geo",
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/",
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/",
"https://www.gomega.ai/blog/how-to-measure-geo-performance",
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool"
],
"partial_coverage": false,
"prevalence": {
"https://geoaura.world/blog/how-to-measure-geo-visibility": {
"numerator": 2,
"denominator": 3
},
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/": {
"numerator": 2,
"denominator": 3
},
"https://shmeer.com/blog/how-to-measure-geo": {
"numerator": 2,
"denominator": 3
},
"https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/": {
"numerator": 1,
"denominator": 3
},
"https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/": {
"numerator": 1,
"denominator": 3
},
"https://www.gomega.ai/blog/how-to-measure-geo-performance": {
"numerator": 2,
"denominator": 3
},
"https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"https://indexly.ai/grok-visibility-checker",
"https://mangools.com/ai-search-watcher",
"https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/",
"https://snoika.com/free-tools/brand-ai-visibility-checker",
"https://solcrys.com/free-chatgpt-visibility-tools/",
"https://stork.ai/blog/does-chatgpt-recommend-your-product",
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/",
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/",
"https://www.ranked.ai/free-tools/ai-visibility-checker"
],
"partial_coverage": false,
"prevalence": {
"https://indexly.ai/grok-visibility-checker": {
"numerator": 1,
"denominator": 3
},
"https://mangools.com/ai-search-watcher": {
"numerator": 1,
"denominator": 3
},
"https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/": {
"numerator": 1,
"denominator": 3
},
"https://snoika.com/free-tools/brand-ai-visibility-checker": {
"numerator": 1,
"denominator": 3
},
"https://solcrys.com/free-chatgpt-visibility-tools/": {
"numerator": 1,
"denominator": 3
},
"https://stork.ai/blog/does-chatgpt-recommend-your-product": {
"numerator": 1,
"denominator": 3
},
"https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/": {
"numerator": 1,
"denominator": 3
},
"https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/": {
"numerator": 1,
"denominator": 3
},
"https://www.ranked.ai/free-tools/ai-visibility-checker": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 9,
"intersection": 0,
"union": 16,
"shared": []
}
},
"cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"explaingeo.com",
"geoaura.world",
"gomega.ai",
"goodatmarketing.com",
"hubspot.com",
"shmeer.com",
"snezzi.com"
],
"partial_coverage": false,
"prevalence": {
"explaingeo.com": {
"numerator": 1,
"denominator": 3
},
"geoaura.world": {
"numerator": 2,
"denominator": 3
},
"gomega.ai": {
"numerator": 2,
"denominator": 3
},
"goodatmarketing.com": {
"numerator": 2,
"denominator": 3
},
"hubspot.com": {
"numerator": 1,
"denominator": 3
},
"shmeer.com": {
"numerator": 2,
"denominator": 3
},
"snezzi.com": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"01net.com",
"amicited.com",
"indexly.ai",
"mangools.com",
"ranked.ai",
"snoika.com"
],
"partial_coverage": true,
"prevalence": {
"01net.com": {
"numerator": 1,
"denominator": 2
},
"amicited.com": {
"numerator": 1,
"denominator": 2
},
"indexly.ai": {
"numerator": 1,
"denominator": 2
},
"mangools.com": {
"numerator": 1,
"denominator": 2
},
"ranked.ai": {
"numerator": 1,
"denominator": 2
},
"snoika.com": {
"numerator": 1,
"denominator": 2
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 6,
"intersection": 0,
"union": 13,
"shared": []
}
},
"observed_cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M1-R3"
],
"union": [
"explaingeo.com",
"geoaura.world",
"gomega.ai",
"goodatmarketing.com",
"hubspot.com",
"shmeer.com",
"snezzi.com"
],
"partial_coverage": false,
"prevalence": {
"explaingeo.com": {
"numerator": 1,
"denominator": 3
},
"geoaura.world": {
"numerator": 2,
"denominator": 3
},
"gomega.ai": {
"numerator": 2,
"denominator": 3
},
"goodatmarketing.com": {
"numerator": 2,
"denominator": 3
},
"hubspot.com": {
"numerator": 1,
"denominator": 3
},
"shmeer.com": {
"numerator": 2,
"denominator": 3
},
"snezzi.com": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-M2-R1",
"S3-GROK-M2-R2",
"S3-GROK-M2-R3"
],
"union": [
"01net.com",
"amicited.com",
"indexly.ai",
"mangools.com",
"ranked.ai",
"seozest.io",
"snoika.com",
"solcrys.com",
"stork.ai"
],
"partial_coverage": false,
"prevalence": {
"01net.com": {
"numerator": 1,
"denominator": 3
},
"amicited.com": {
"numerator": 1,
"denominator": 3
},
"indexly.ai": {
"numerator": 1,
"denominator": 3
},
"mangools.com": {
"numerator": 1,
"denominator": 3
},
"ranked.ai": {
"numerator": 1,
"denominator": 3
},
"seozest.io": {
"numerator": 1,
"denominator": 3
},
"snoika.com": {
"numerator": 1,
"denominator": 3
},
"solcrys.com": {
"numerator": 1,
"denominator": 3
},
"stork.ai": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.0,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 7,
"right_size": 9,
"intersection": 0,
"union": 16,
"shared": []
}
}
}
},
{
"platform": "Grok",
"environment_label": "Grok Expert · consumer interface",
"intent": "V",
"metrics": {
"strict_urls": {
"left": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/",
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/",
"https://www.visilay.com/blog/ai-visibility-score/"
],
"partial_coverage": true,
"prevalence": {
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461": {
"numerator": 2,
"denominator": 2
},
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/": {
"numerator": 1,
"denominator": 2
},
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/": {
"numerator": 1,
"denominator": 2
},
"https://www.visilay.com/blog/ai-visibility-score/": {
"numerator": 1,
"denominator": 2
}
}
},
"right": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui",
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands",
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"https://www.aeotrack.io/methodology",
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt",
"https://www.loudface.co/blog/what-is-ai-visibility"
],
"partial_coverage": true,
"prevalence": {
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/": {
"numerator": 1,
"denominator": 2
},
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461": {
"numerator": 2,
"denominator": 2
},
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui": {
"numerator": 1,
"denominator": 2
},
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands": {
"numerator": 1,
"denominator": 2
},
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/": {
"numerator": 1,
"denominator": 2
},
"https://www.aeotrack.io/methodology": {
"numerator": 1,
"denominator": 2
},
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt": {
"numerator": 1,
"denominator": 2
},
"https://www.loudface.co/blog/what-is-ai-visibility": {
"numerator": 1,
"denominator": 2
}
}
},
"pooled_overlap": {
"value": 0.09090909090909091,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 8,
"intersection": 1,
"union": 11,
"shared": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461"
]
}
},
"observed_urls": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"https://ahrefs.com/free-ai-visibility",
"https://cognizo.ai/blog/ai-visibility-score",
"https://higoodie.com/blog/how-to-rank-on-grok/",
"https://hubspot.com/aeo-grader",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://maxaeo.ai/blog/ai-visibility-sample-size/",
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/",
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/",
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/",
"https://www.visilay.com/blog/ai-visibility-score/"
],
"partial_coverage": false,
"prevalence": {
"https://ahrefs.com/free-ai-visibility": {
"numerator": 1,
"denominator": 3
},
"https://cognizo.ai/blog/ai-visibility-score": {
"numerator": 1,
"denominator": 3
},
"https://higoodie.com/blog/how-to-rank-on-grok/": {
"numerator": 1,
"denominator": 3
},
"https://hubspot.com/aeo-grader": {
"numerator": 1,
"denominator": 3
},
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461": {
"numerator": 3,
"denominator": 3
},
"https://maxaeo.ai/blog/ai-visibility-sample-size/": {
"numerator": 1,
"denominator": 3
},
"https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/": {
"numerator": 1,
"denominator": 3
},
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/": {
"numerator": 1,
"denominator": 3
},
"https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/": {
"numerator": 1,
"denominator": 3
},
"https://www.visilay.com/blog/ai-visibility-score/": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"https://amicited.com/blog/grok-is-citing-brands-too/",
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/",
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui",
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands",
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/",
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"https://visilay.com/blog/ai-visibility-score/",
"https://www.aeotrack.io/methodology",
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt",
"https://www.loudface.co/blog/what-is-ai-visibility"
],
"partial_coverage": false,
"prevalence": {
"https://amicited.com/blog/grok-is-citing-brands-too/": {
"numerator": 1,
"denominator": 3
},
"https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/": {
"numerator": 1,
"denominator": 3
},
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461": {
"numerator": 3,
"denominator": 3
},
"https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui": {
"numerator": 1,
"denominator": 3
},
"https://presenc.ai/compare/grok-vs-chatgpt-for-brands": {
"numerator": 1,
"denominator": 3
},
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/": {
"numerator": 1,
"denominator": 3
},
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/": {
"numerator": 1,
"denominator": 3
},
"https://visilay.com/blog/ai-visibility-score/": {
"numerator": 1,
"denominator": 3
},
"https://www.aeotrack.io/methodology": {
"numerator": 1,
"denominator": 3
},
"https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt": {
"numerator": 1,
"denominator": 3
},
"https://www.loudface.co/blog/what-is-ai-visibility": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.10526315789473684,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 10,
"right_size": 11,
"intersection": 2,
"union": 19,
"shared": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/"
]
}
},
"cited_domains": {
"left": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"amicited.com",
"businessinsider.com",
"searchenginejournal.com",
"visilay.com"
],
"partial_coverage": true,
"prevalence": {
"amicited.com": {
"numerator": 1,
"denominator": 2
},
"businessinsider.com": {
"numerator": 2,
"denominator": 2
},
"searchenginejournal.com": {
"numerator": 1,
"denominator": 2
},
"visilay.com": {
"numerator": 1,
"denominator": 2
}
}
},
"right": {
"eligible_denominator": 2,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"aeotrack.io",
"brantial.ai",
"businessinsider.com",
"getmint.ai",
"loudface.co",
"paprik.ai",
"presenc.ai",
"sparktoro.com"
],
"partial_coverage": true,
"prevalence": {
"aeotrack.io": {
"numerator": 1,
"denominator": 2
},
"brantial.ai": {
"numerator": 1,
"denominator": 2
},
"businessinsider.com": {
"numerator": 2,
"denominator": 2
},
"getmint.ai": {
"numerator": 1,
"denominator": 2
},
"loudface.co": {
"numerator": 1,
"denominator": 2
},
"paprik.ai": {
"numerator": 1,
"denominator": 2
},
"presenc.ai": {
"numerator": 1,
"denominator": 2
},
"sparktoro.com": {
"numerator": 1,
"denominator": 2
}
}
},
"pooled_overlap": {
"value": 0.09090909090909091,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 4,
"right_size": 8,
"intersection": 1,
"union": 11,
"shared": [
"businessinsider.com"
]
}
},
"observed_cited_domains": {
"left": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V1-R2",
"S3-GROK-V1-R3"
],
"union": [
"ahrefs.com",
"amicited.com",
"businessinsider.com",
"cognizo.ai",
"higoodie.com",
"hubspot.com",
"maxaeo.ai",
"searchenginejournal.com",
"searchscope.com.au",
"visilay.com"
],
"partial_coverage": false,
"prevalence": {
"ahrefs.com": {
"numerator": 1,
"denominator": 3
},
"amicited.com": {
"numerator": 1,
"denominator": 3
},
"businessinsider.com": {
"numerator": 3,
"denominator": 3
},
"cognizo.ai": {
"numerator": 1,
"denominator": 3
},
"higoodie.com": {
"numerator": 1,
"denominator": 3
},
"hubspot.com": {
"numerator": 1,
"denominator": 3
},
"maxaeo.ai": {
"numerator": 1,
"denominator": 3
},
"searchenginejournal.com": {
"numerator": 1,
"denominator": 3
},
"searchscope.com.au": {
"numerator": 1,
"denominator": 3
},
"visilay.com": {
"numerator": 1,
"denominator": 3
}
}
},
"right": {
"eligible_denominator": 3,
"planned_positions": 3,
"eligible_ids": [
"S3-GROK-V2-R1",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3"
],
"union": [
"aeotrack.io",
"amicited.com",
"brantial.ai",
"businessinsider.com",
"getmint.ai",
"loudface.co",
"paprik.ai",
"presenc.ai",
"searchscope.com.au",
"sparktoro.com",
"visilay.com"
],
"partial_coverage": false,
"prevalence": {
"aeotrack.io": {
"numerator": 1,
"denominator": 3
},
"amicited.com": {
"numerator": 1,
"denominator": 3
},
"brantial.ai": {
"numerator": 1,
"denominator": 3
},
"businessinsider.com": {
"numerator": 3,
"denominator": 3
},
"getmint.ai": {
"numerator": 1,
"denominator": 3
},
"loudface.co": {
"numerator": 1,
"denominator": 3
},
"paprik.ai": {
"numerator": 1,
"denominator": 3
},
"presenc.ai": {
"numerator": 1,
"denominator": 3
},
"searchscope.com.au": {
"numerator": 1,
"denominator": 3
},
"sparktoro.com": {
"numerator": 1,
"denominator": 3
},
"visilay.com": {
"numerator": 1,
"denominator": 3
}
}
},
"pooled_overlap": {
"value": 0.23529411764705882,
"reason": null,
"eligible": true,
"both_empty": false,
"left_size": 10,
"right_size": 11,
"intersection": 4,
"union": 17,
"shared": [
"amicited.com",
"businessinsider.com",
"searchscope.com.au",
"visilay.com"
]
}
}
}
}
]
},
"warnings": [
"All analyses here are post-hoc descriptive extensions; the frozen strict coding and original metrics are unchanged.",
"Within-wording six pairs and cross-wording nine pairs reuse the same six answers; do not treat them as 15 independent observations.",
"Delta is cross-pair mean minus within-pair mean in percentage points; numeric denominators exclude both-empty N/A and unavailable pairs.",
"M1 includes explicit no-GEO-budget/reliability cues; M2 explicitly asks for a free repeatable workflow. V1 and V2 change budget/context and action wording. These are not pure lexical substitutions.",
"M-versus-V changes the task; no p-values, causal effects or universal wording penalty are estimated.",
"OpenAI API + web search is not ChatGPT web/app performance; the API batch followed the Grok consumer batch, confounding time and interface.",
"Family/category labels come only from the explicit reviewed file. Corporate-parent or keyword merging is not inferred.",
"Observed URLs are a separate coverage-qualified view. The first native Grok records do not establish complete citation mapping or exact original hrefs.",
"URL cleanup preserves encoded paths/fragments and does not repair suspicious or malformed-looking destinations.",
"Human/source-validation status is not upgraded by this extension; existing assistant coding and provisional source judgments remain qualified."
],
"validation_issues": []
}Human-review status · assistant audit is not human verification
Status: awaiting independent human review
{
"schema_version": 1,
"status": "awaiting_independent_human_review",
"human_completed_cases": 0,
"planned_cases": 8,
"independent_human_validation_completed": false,
"purpose": "Targeted spot-check of ambiguous judgments and headline-sensitive normalization; not a random accuracy sample.",
"selection_method": "Purposive selection of seven entity decisions and one metric-definition decision from six answers; selected after expert review. No claim of representative error rate or independence from existing public coding.",
"reviewer_instructions": [
"Read the full linked answer, not only the excerpt.",
"Use the original codebook: C=own content cited, M=substantive explicit naming, R=active suggestion for this task.",
"Do not infer a product version from a parent/family label. Classify product identity, role and recommendation separately.",
"Record your independent decision and reason before comparing with the published assistant coding where possible. The originals are publicly accessible, so this is not a guaranteed blinded study.",
"Do not put private contact details in this public form. A public reviewer name is optional and requires that person’s consent.",
"No agreement score, human validation rate or corrected gold standard is computed until genuine human judgments exist."
],
"cases": [
{
"case_id": "H01",
"answer_id": "S3-OPENAIAPI-M2-R2",
"review_type": "entity_CMR",
"entity_to_assess": "X",
"question": "A suggestion to include X-centric prompts: does this actively recommend using a channel for the task, or merely describe source coverage?",
"evidence_excerpt": "For Grok, include a few X-centric prompts because its search capabilities can use both real-time web and X information.",
"public_answer_record": "records/S3-OPENAIAPI-M2-R2.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H02",
"answer_id": "S3-OPENAIAPI-M2-R2",
"review_type": "entity_CMR",
"entity_to_assess": "Google Analytics",
"question": "The answer does not specify GA4. Review the strict name, conditional recommendation and whether any citation is the tool’s own content.",
"evidence_excerpt": "making them identifiable in tools such as Google Analytics.",
"public_answer_record": "records/S3-OPENAIAPI-M2-R2.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H03",
"answer_id": "S3-OPENAIAPI-M1-R1",
"review_type": "entity_CMR",
"entity_to_assess": "Google Search Console",
"question": "Is sourcing buyer wording from Search Console an active measurement-support recommendation? Keep supporting tools distinct from specialized GEO products.",
"evidence_excerpt": "Search Console queries",
"public_answer_record": "records/S3-OPENAIAPI-M1-R1.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H04",
"answer_id": "S3-OPENAIAPI-M1-R3",
"review_type": "entity_CMR",
"entity_to_assess": "Google Analytics 4",
"question": "Review the explicitly named analytics option and distinguish recommendation from own-content citation.",
"evidence_excerpt": "In GA4, Plausible, PostHog, or server logs, create a segment for:",
"public_answer_record": "records/S3-OPENAIAPI-M1-R3.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H05",
"answer_id": "S3-OPENAIAPI-M2-R1",
"review_type": "entity_CMR",
"entity_to_assess": "Reddit",
"question": "Review whether a recurring cited-source pattern is an active tool/channel recommendation.",
"evidence_excerpt": "Reddit or community discussions",
"public_answer_record": "records/S3-OPENAIAPI-M2-R1.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H06",
"answer_id": "S3-GROK-V2-R1",
"review_type": "entity_CMR",
"entity_to_assess": "GitHub",
"question": "Review a corroborating developer/community signal: active suggestion, contextual mention, or neither?",
"evidence_excerpt": "Community and third-party presence (Reddit, review sites, GitHub, directories, X engagement)—these heavily influence what the models say.",
"public_answer_record": "records/S3-GROK-V2-R1.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H07",
"answer_id": "S3-GROK-V1-R2",
"review_type": "entity_CMR",
"entity_to_assess": "HubSpot grader (unspecified)",
"question": "Review the unspecified product label without inferring an AEO/AI Search product identity. Family grouping is a separate sensitivity lens.",
"evidence_excerpt": "Free one-off checkers (Ahrefs, Semrush limited free checks, HubSpot grader, and others) can give a directional snapshot. Treat them as low-trust starting points and always verify the important prompts yourself. They are useful for spotting gaps, not for precise scores.",
"public_answer_record": "records/S3-GROK-V1-R2.json",
"human_decision": {
"citation": null,
"mention": null,
"recommendation": null,
"strict_entity_label": null,
"category": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
},
{
"case_id": "H08",
"answer_id": "S3-OPENAIAPI-M2-R2",
"review_type": "metric_definition",
"question": "Does the returned Recommendation visibility formula measure mention rate, recommendation rate, or an ambiguous combination? Is the answer internally consistent elsewhere?",
"evidence_excerpt": "### Recommendation visibility\n\n```text\nAnswers mentioning you ÷ total tested answers\n```",
"public_answer_record": "records/S3-OPENAIAPI-M2-R2.json",
"human_decision": {
"metric_interpretation": null,
"rationale": null
},
"reviewed_by_independent_human": false,
"reviewed_at": null
}
]
}Bounded source audit · unreviewed claims remain unverified
Status: 6 selected sources reviewed; 5 exact pages accessible; 1 exact page unavailable
{
"checked_on": "2026-11-02",
"timezone": "Asia/Shanghai",
"auditor": "assistant",
"independent_human_review": false,
"all_claims_verified": false,
"scope": {
"sample": "Six purposively selected captured source URLs, with one verbatim captured claim selected per source.",
"selection_is_representative": false,
"post_collection_review": true,
"method": "Read source pages and relevant sections through web retrieval. Follow the arXiv full text and, for one inaccessible cited URL, identify related primary-origin material. Separate URL access, claim support and source role.",
"not_tested": [
"The full source corpus",
"Commercial score accuracy or actual tool results",
"Free-plan operation, pricing or entitlement",
"The raw calculations or reproducibility of cited studies",
"GEO optimization effectiveness",
"Independent human agreement with assistant coding"
],
"original_records_changed": false
},
"items": [
{
"id": "SA-01",
"source_url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"related_answer_ids": [
"S3-OPENAIAPI-M2-R1",
"S3-OPENAIAPI-M1-R1",
"S3-OPENAIAPI-M1-R2",
"S3-OPENAIAPI-V1-R2",
"S3-OPENAIAPI-M2-R3",
"S3-OPENAIAPI-V1-R3"
],
"selected_claim_answer_id": "S3-OPENAIAPI-M1-R1",
"claim_excerpt": "For ChatGPT search eligibility, OpenAI specifically recommends allowing `OAI-SearchBot`; inclusion or placement is still not guaranteed.",
"checked_on": "2026-11-02",
"access": {
"status": "accessible",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "Readable official help article. The website-eligibility section states that the crawler and its published traffic must be allowed; the preceding sentence disclaims guaranteed placement."
},
"support": {
"verdict": "supported_selected_claim",
"rationale": "The eligibility advice and non-guarantee are present. This is documentation support for the selected claim, not a crawl test of a real product site or proof that allowing a bot increases recommendations."
},
"source_role": {
"original": "Platform documentation",
"reviewed": "Platform documentation",
"status": "verified",
"qualification": "Official OpenAI help; current text checked after collection. Other claims in the answer, including cross-platform behavior, were not checked."
},
"short_source_quote": "Placement is not guaranteed.",
"evidence_locator": "Make your website available in search results",
"live_generation_or_product_test_performed": false
},
{
"id": "SA-02",
"source_url": "https://docs.x.ai/developers/model-capabilities/text/best-practices",
"related_answer_ids": [
"S3-OPENAIAPI-V2-R1",
"S3-OPENAIAPI-V2-R3",
"S3-OPENAIAPI-M1-R3"
],
"selected_claim_answer_id": "S3-OPENAIAPI-V2-R1",
"claim_excerpt": "xAI’s own evaluation guidance warns that model judges can exhibit position bias, favor longer answers, and drift from human judgment.",
"checked_on": "2026-11-02",
"access": {
"status": "accessible",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "Readable official best-practices page. Its grader table lists the three risks; the next paragraph recommends rubrics, randomized order and comparison with human labels."
},
"support": {
"verdict": "supported_selected_claim",
"rationale": "The selected warning is directly supported. It does not supply a measured error rate for this experiment or establish that assistant coding agrees with a person."
},
"source_role": {
"original": "Platform documentation",
"reviewed": "Platform documentation",
"status": "verified",
"qualification": "Provider guidance, not independent validation of Stage 3 coding. Genuine human review remains pending."
},
"short_source_quote": "Position bias, preference for longer answers, drift from human judgment",
"evidence_locator": "Evaluate Grok models / Design the eval",
"live_generation_or_product_test_performed": false
},
{
"id": "SA-03",
"source_url": "https://arxiv.org/abs/2410.03492",
"related_answer_ids": [
"S3-OPENAIAPI-V2-R1"
],
"selected_claim_answer_id": "S3-OPENAIAPI-V2-R1",
"claim_excerpt": "LLM evaluations can vary across repeated runs and small wording changes, so aggregate rankings without uncertainty can look more definitive than they are.",
"checked_on": "2026-11-02",
"access": {
"status": "accessible",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "Abstract and full text were readable. Limitations excludes linguistic-variation analysis; experiments concern repeated spatial-reasoning tests."
},
"support": {
"verdict": "partially_supported",
"rationale": "Supports repeat uncertainty, not the wording-change clause: linguistic variation was explicitly untested. GEO metrics were not evaluated."
},
"source_role": {
"original": "Academic or independent methods research",
"reviewed": "Academic or independent methods research",
"status": "verified",
"qualification": "Original spatial-reasoning methods preprint; peer review and replication were not established."
},
"short_source_quote": "We have not looked at the impact of linguistic variation",
"evidence_locator": "Version 2, Conclusions and Limitations",
"readable_full_text_url": "https://arxiv.org/html/2410.03492v2",
"material_caution": "Partial sentence support; original coding remains unchanged."
},
{
"id": "SA-04",
"source_url": "https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/",
"related_answer_ids": [
"S3-GROK-V2-R2"
],
"selected_claim_answer_id": "S3-GROK-V2-R2",
"claim_excerpt": "SparkToro research (nearly 3,000 runs across ChatGPT, Claude, and Google AI) found less than a 1-in-100 chance of identical brand lists and roughly 1-in-1,000 for the same list in the same order.",
"checked_on": "2026-11-02",
"access": {
"status": "accessible",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "Readable original research post. It reports 2,961 volunteer runs across three services, approximate list/order repetition statements, and a collaboration with Gumshoe."
},
"support": {
"verdict": "supported_as_reported_with_scope_limits",
"rationale": "The attributed study and approximations are present. The text gives a slightly different qualification for Claude, and Grok was not tested. Statistical calculations, raw responses and population-level probabilities were not independently reproduced."
},
"source_role": {
"original": "Vendor research or case study",
"reviewed": "Vendor research or case study",
"status": "verified",
"qualification": "Original commercial research collaboration with disclosed interests; not an independent audit of all visibility vendors or of Stage 3. This is support that the authors reported the results, not validation of universal rates."
},
"evidence_locator": "Experiment setup, list/order variability and stated limitations",
"commercial_provenance": "SparkToro study with a Gumshoe collaborator; interests disclosed in the article."
},
{
"id": "SA-05",
"source_url": "https://ahrefs.com/free-ai-visibility",
"related_answer_ids": [
"S3-GROK-V1-R1"
],
"selected_claim_answer_id": "S3-GROK-V1-R1",
"claim_excerpt": "Ahrefs free AI Visibility Checker (brand name in → mentions across several engines, top topics/domains)",
"checked_on": "2026-11-02",
"access": {
"status": "accessible_after_redirect",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "The supplied path redirected to the Ahrefs AI Visibility Checker page. The readable vendor page describes brand-name input, multi-platform mentions, topics and cited domains."
},
"support": {
"verdict": "supported_vendor_description_only",
"rationale": "The named product and described report features are advertised. No query was submitted, no account or payment was used, and no score accuracy, plan entitlement or free-price claim was tested. The listed platforms do not establish Grok coverage."
},
"source_role": {
"original": "Tool/product documentation or sales page",
"reviewed": "Tool/product documentation or sales page",
"status": "verified",
"qualification": "First-party product description, not an independently validated measurement method. The captured recommendation remains an observed model recommendation."
},
"evidence_locator": "How to check your brand’s AI visibility / What’s in your report",
"resolved_url": "https://ahrefs.com/ai-visibility-checker",
"live_generation_or_product_test_performed": false
},
{
"id": "SA-06",
"source_url": "https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461",
"related_answer_ids": [
"S3-GROK-V1-R1",
"S3-GROK-V2-R1",
"S3-GROK-V1-R2",
"S3-GROK-V2-R2",
"S3-GROK-V2-R3",
"S3-GROK-V1-R3"
],
"selected_claim_answer_id": "S3-GROK-V1-R1",
"claim_excerpt": "One 2026 benchmark found average top-five shortlist overlap of about 43% when the *same* engine answered the same questions again, and only ~17% *across* engines.",
"checked_on": "2026-11-02",
"access": {
"status": "unavailable_via_retrieval_tool",
"method": "Read public source with web retrieval open/find; no sign-in, product execution or paid API request.",
"http_status": null,
"evidence": "The exact Markets URL returned a tool access error, without a verifiable HTTP status. A same-title originating Crackle PR release on GlobeNewswire and the agency’s methodology page were readable."
},
"support": {
"verdict": "corroborated_by_origin_not_exact_cited_page",
"rationale": "The originating release reports the selected figures for a 500-answer, four-API B2B PR benchmark and defines overlap as Jaccard. This does not verify the inaccessible Markets copy, reproduce the calculations or generalize to consumer interfaces."
},
"source_role": {
"original": "Editorial or industry reporting",
"reviewed": "First-party agency research press release; exact Markets copy unavailable",
"status": "qualified",
"qualification": "The readable origin is a commercially interested agency’s release, not independent editorial research. The original Markets category needs a qualification; exact syndication identity was not authenticated. No original category is silently overwritten."
},
"evidence_locator": "Originating release: experiment description and Scope and Limits; agency methodology: actual collection",
"related_primary_sources": [
{
"url": "https://www.globenewswire.com/fr/news-release/2026/10/20/3384392/0/en/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists.html",
"role": "Originating release; labelled Source: Crackle PR",
"access_status": "accessible"
},
{
"url": "https://www.cracklepr.com/ai-visibility-index/methodology",
"role": "Original agency methodology",
"access_status": "accessible"
}
],
"material_caution": "Do not treat the host name or a syndicated release as proof of independent editorial provenance. The inaccessible exact URL remains unverified."
}
],
"summary": {
"requested_source_URLs": 6,
"exact_source_content_accessible": 5,
"exact_source_URL_unavailable": 1,
"support_verdict_counts": {
"supported_selected_claim": 2,
"partially_supported": 1,
"supported_as_reported_with_scope_limits": 1,
"supported_vendor_description_only": 1,
"corroborated_by_origin_not_exact_cited_page": 1
},
"material_findings": [
"The arXiv citation supports repeated-run uncertainty, but not the accompanying wording-change claim.",
"The inaccessible Markets citation has a readable commercial press-release origin. Keep exact-URL unavailability and editorial-provenance qualification visible."
],
"interpretation": "A supported attribution or feature description is narrower than an independently verified empirical claim. These checks do not validate every source or replace actual human review."
}
}Method, controls, and limits
The user authorized replacing the consumer ChatGPT web condition with OpenAI Responses API web_search, requesting gpt-5.6-sol with medium reasoning and a total US$10 API budget. The original prompts and S3-CHATGPT-* position IDs are retained; those IDs are historical identifiers, not a claim that API answers came from consumer ChatGPT Search. New API attempt IDs use S3-OPENAIAPI-*.
The formal comparison is Grok in its consumer interface versus OpenAI through the Responses API. Interface, retrieval implementation, request context, model identity and time can all differ. Same-name model labels do not prove equivalent execution or answers. Historical ChatGPT UI failures remain in the attempt ledger and are not pooled with the API answers.
The requested and returned gpt-5.6-sol value is an undated model identifier, not an independently established backend or dated snapshot. A fixed identifier plus medium effort does not prove that the backend stayed identical, or that consumer ChatGPT Search uses the same backend. All API requests were collected after the Grok batch; this is not a platform causal-effect experiment.
API search enabled in the request, actual returned web-search calls, and answer citation annotations are separate evidence fields. Transport completion does not establish a reviewed complete answer. Awaiting credentials, awaiting body review, missing usage and unknown cost remain unresolved rather than zero.
Coded entities are assistant semantic judgments, not keyword classification or independent human double-coding. Tools/services/surfaces retain their reviewed canonical names. Supplied ChatGPT/Grok targets, generic methods and citation-only publishers are excluded from default tool sets.
A missing or unresolved judgment is not false. Entire metric sets are withheld when relevant judgments are unresolved. Prevalence denominators include only complete selected bodies with matching source digests and eligible coding for that metric. Zero prevalence refers only to those reviewed inventories.
Jaccard = intersection / union. Both-empty eligible sets are N/A; one empty and one nonempty eligible set yields zero. Sparse or empty named-tool lists may accompany a substantive unbranded workflow; zero set overlap is not by itself evidence of unstable advice. Means use only numeric eligible pairs and expose the count. Shared-across-three is withheld until all three are eligible.
Cited-domain overlap with complete mapping requires citation_mapping_complete=true. Observed-inline-domain overlap is separate and coverage-qualified. The first three native Grok captures may omit URL schemes or leading www; standardized HTTPS links are not independently verified exact hrefs. Later DOM hrefs do not repair earlier URL precision.
Domain normalization uses recorded manual hostname mappings and leading-www removal; otherwise it retains the exact hostname. It never takes the last two labels blindly. Exact source URL records remain separate from domain deduplication.
Source categories use only the manually assigned primary category and explicit secondary categories. Candidate categories are not promoted. Category prevalence is per answer, not per citation occurrence; unresolved and provisional categories remain labeled. The observed-source denominator is not a complete retrieval-inventory denominator.
A retrieval/source badge is not an answer-citation count. A complete answer with zero observable inline anchors can have many retrieval hits. Workspace-explored activity and visible personalization controls do not establish audited account isolation.
Wording contrasts change registered phrase packages, not just synonyms: M1 asks to reliably measure with no GEO budget; M2 asks for a free, repeatable workflow as a bootstrapped builder. V1 explicitly says no GEO budget; V2 says bootstrapped. Budget and action cues differ. Interpret sensitivity to these full versions, not a pure lexical effect.
M-versus-V changes the requested task. None of these contrasts supports GEO intervention effectiveness, tool-score validity, or stable platform recommendation probabilities. Generated 0–3 or −1/0–4 rubrics, weights, sample sizes and illustrative dashboard percentages are model advice, not this experiment’s validated measurement instrument. In particular, the M2-R2 recommendation-visibility numerator counts mentions in the returned advice; it is not substituted for our separate R coding.
Evidence for the authored findings
The same question did not freeze the detailed advice.
Within-intent phrasing changed tool lists more than some source-domain pools.
Intent was a consistent signal for answer type.
The two collected interfaces selected visibly different source mixes.
Citation, mention and recommendation remained separate observations.
Registered prompts
| Prompt | Original wording |
|---|---|
| M1 | I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers? |
| M2 | As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations? |
| V1 | I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy? |
| V2 | As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product? |
Historical collection protocol, controls and amendments
This archived protocol preserves authorization, setup and collection status as recorded at the time. The public release is a separately authorized redacted edition; historical restrictions do not describe the current publication status.
{
"title": "Stage 3: Sensitivity to question design in GEO measurement advice",
"registered_at": "2026-11-02T14:03:10.188434+08:00",
"timezone": "Asia/Shanghai",
"design": {
"intents": 2,
"wordings_per_intent": 2,
"platforms": 2,
"repeats_per_prompt_platform": 3,
"planned_answers": 24,
"unit": "One Grok fresh conversation or one independent stateless OpenAI Responses API request receiving exactly one registered prompt.",
"order": "Three repeat blocks, shuffled prompts and platform order with seed 20261102; planned order retained, actual order and deviations recorded.",
"planned_order": [
"S3-GROK-V1-R1",
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V2-R1",
"S3-GROK-V2-R1",
"S3-GROK-M2-R1",
"S3-CHATGPT-M2-R1",
"S3-GROK-M1-R1",
"S3-CHATGPT-M1-R1",
"S3-GROK-M1-R2",
"S3-CHATGPT-M1-R2",
"S3-GROK-M2-R2",
"S3-CHATGPT-M2-R2",
"S3-CHATGPT-V1-R2",
"S3-GROK-V1-R2",
"S3-CHATGPT-V2-R2",
"S3-GROK-V2-R2",
"S3-GROK-M2-R3",
"S3-CHATGPT-M2-R3",
"S3-GROK-V2-R3",
"S3-CHATGPT-V2-R3",
"S3-GROK-M1-R3",
"S3-CHATGPT-M1-R3",
"S3-GROK-V1-R3",
"S3-CHATGPT-V1-R3"
],
"unit_at_registration": "One fresh conversation receiving exactly one registered prompt.",
"current_comparison_surfaces": [
"Grok consumer web interface, Expert mode",
"OpenAI Responses API, gpt-5.6-sol / medium, native web_search required"
],
"interpretation_amendment": "The OpenAI arm is an API configuration, not a measurement of consumer ChatGPT Search. Cross-arm differences include interface, retrieval and execution-time differences; they cannot be interpreted as pure consumer-platform or wording effects."
},
"controls": {
"chatgpt": {
"interface_requested": "desktop",
"desktop_preflight": "Computer-use tool denied protected application [Protected desktop application]; no prompt submitted.",
"web_fallback": "Explicitly approved by user on 2026-11-02, conditional on complete answer/source capture preflight.",
"model": "gpt-5.6-sol (requested and returned undated API identifier)",
"reasoning_effort": "medium",
"search": "Responses API hosted web_search; tool_choice required; live access true; actual returned calls recorded.",
"personalization": "Stateless API requests, exactly one original user prompt, no prior-response/conversation/history, no system/developer additions.",
"historical_ui_blocker": "One Cloudflare failure followed by a searched but unreadable answer. Copy yields loading error, Save Chat disabled. Five source-panel URLs recovered; body remains unavailable. No batch submission until capture preflight succeeds or user chooses manual/delayed completion.",
"collection_status": "12 complete API answers selected; all three incomplete historical consumer-UI submissions retained separately.",
"current_formal_interface": "OpenAI Responses API web_search, explicitly approved; not consumer ChatGPT Search.",
"current_blocker": null,
"historical_ui_model": "GPT-5.6 Sol (explicit UI selection)",
"historical_ui_reasoning_effort": "Medium",
"historical_ui_search": "Select native web search; capture actual search separately.",
"historical_ui_personalization": "Use non-personalized fresh temporary chats where available; document actual controls.",
"backend_snapshot": "Undated API ID; exact underlying snapshot not independently established."
},
"grok": {
"model": "Expert; exact backend version not exposed",
"search": "Native web/X retrieval; UI enabled state and actual activity recorded separately.",
"personalization": "X/history personalization disabled for all twelve tests; restored to pre-test values afterward. Original preference values omitted.",
"settings_restoration_pending": false
},
"openai_api": {
"provider": "OpenAI",
"interface": "Responses API",
"model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"configuration_status": "awaiting_credentials_and_setup_validation",
"model_availability_verified": false,
"search_required": true,
"search_evidence_policy": "Require search in API configuration; record actual returned web-search tool calls separately from configured search and citation annotations.",
"request_context": "Stateless independent requests, no prior response ID, conversation history, personal account context or seeded prior answers. Exact registered user prompt only; no additional user/system steering.",
"minimum_seconds_between_same_platform_submissions": 90,
"fallback_policy": "No silent model, effort, provider or search-setting substitution. Report an unavailable condition instead.",
"platform": "OpenAI API + web search"
}
},
"capture_policy": [
"Prompts remain verbatim; no output-format directions, identity/product seed, prior answers or follow-up questions.",
"Record complete text and explicit returned URLs/domains; do not equate retrieval panels or whole-page anchors with citations.",
"A failed or partial capture does not establish a negative result. Preserve every actual submission and recovery separately.",
"Before changing fixed model/effort, additional submission policy or accepted scope, obtain confirmation.",
"No publication authorized by this collection request. Public edition requires separate instruction.",
"For the public edition, historical publication restrictions are superseded by the dated public-release authorization; private originals remain local."
],
"analysis": "analysis-plan.md",
"preflight_submissions": 1,
"status": "complete_redacted_public_edition",
"actual_order": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R1-A2",
"S3-GROK-V1-R1",
"S3-GROK-V2-R1",
"S3-GROK-M2-R1",
"S3-GROK-M1-R1",
"S3-GROK-M1-R2",
"S3-GROK-M2-R2",
"S3-GROK-V1-R2",
"S3-GROK-V2-R2",
"S3-GROK-M2-R3",
"S3-GROK-V2-R3",
"S3-GROK-M1-R3",
"S3-GROK-V1-R3",
"S3-CHATGPT-V1-R1-A3",
"S3-OPENAIAPI-V1-R1",
"S3-OPENAIAPI-V2-R1",
"S3-OPENAIAPI-M2-R1",
"S3-OPENAIAPI-M1-R1",
"S3-OPENAIAPI-M1-R2",
"S3-OPENAIAPI-M2-R2",
"S3-OPENAIAPI-V1-R2",
"S3-OPENAIAPI-V2-R2",
"S3-OPENAIAPI-M2-R3",
"S3-OPENAIAPI-V2-R3",
"S3-OPENAIAPI-M1-R3",
"S3-OPENAIAPI-V1-R3"
],
"submission_pacing": {
"minimum_seconds_between_same_platform_submissions": 90,
"policy": "Alternate platforms where possible. No burst retries. Record all actual submission timestamps.",
"requested_by_user": "2026-11-02"
},
"supplement_policy": {
"authorized": true,
"target": "24 complete selected answer bodies",
"failures": "Retain each failed or partial submission separately; extra submissions use a new attempt ID and unchanged prompt/model settings."
},
"amendments": [
{
"date": "2026-11-02",
"change": "User approved web fallback after native desktop app was denied by computer-use tool."
},
{
"date": "2026-11-02",
"change": "User approved extra supplements with failures retained; requested longer intervals."
},
{
"date": "2026-11-02",
"change": "Chrome browser connection became available after initial native UI collection. Same browser/profile and platform settings retained. New records use scoped visible article text and exact DOM hrefs. First three Grok records preserve native display-target URL precision limitations."
},
{
"date": "2026-11-02",
"recorded_at": "2026-11-02T09:03:25.362981+00:00",
"change": "User explicitly authorized replacing the incomplete ChatGPT consumer-web arm with 12 new formal OpenAI Responses API web_search answers using gpt-5.6-sol / Medium, total budget US$10. Retain all UI failures; stateless exact prompts, required search and minimum 90-second same-platform pacing remain.",
"scope": "Original S3-CHATGPT slot IDs retained for position mapping, with explicit API platform/interface metadata; new actual attempt IDs planned as S3-OPENAIAPI-{prompt}-R{repeat}. No API submission has occurred.",
"limitation": "API web_search is not equivalent to consumer ChatGPT Search; cross-arm differences must preserve that condition change."
},
{
"date": "2026-11-02",
"change": "Redacted public edition authorized. All 24 selected answers, 27 prompt submissions and 24 coding records retained. The formal OpenAI arm used the Responses API with web search; it cannot represent consumer ChatGPT Search."
}
],
"restoration": {
"completed": true,
"restored_at": "2026-11-02T06:54:34.518Z",
"scope": "Grok personalization restored to pre-test values. Original preference values and account UI omitted.",
"evidence": "settings-summary.json",
"other_settings_changed": false
},
"api_collection": {
"authorized": true,
"authorized_on": "2026-11-02",
"planned_formal_answers": 12,
"actual_submissions": 12,
"complete_answers": 12,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"budget_policy": "US$10 authorized total; sequential admission with US$6.60 reservation, usage-priced estimates, stop on unknown outcomes. No automatic budget increase. Estimates require provider-billing reconciliation.",
"status": "complete",
"planned_slot_mapping": [
{
"slot_id": "S3-CHATGPT-V1-R1",
"planned_attempt_id": "S3-OPENAIAPI-V1-R1",
"prompt_id": "V1",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-V2-R1",
"planned_attempt_id": "S3-OPENAIAPI-V2-R1",
"prompt_id": "V2",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M2-R1",
"planned_attempt_id": "S3-OPENAIAPI-M2-R1",
"prompt_id": "M2",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M1-R1",
"planned_attempt_id": "S3-OPENAIAPI-M1-R1",
"prompt_id": "M1",
"repeat": 1
},
{
"slot_id": "S3-CHATGPT-M1-R2",
"planned_attempt_id": "S3-OPENAIAPI-M1-R2",
"prompt_id": "M1",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-M2-R2",
"planned_attempt_id": "S3-OPENAIAPI-M2-R2",
"prompt_id": "M2",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-V1-R2",
"planned_attempt_id": "S3-OPENAIAPI-V1-R2",
"prompt_id": "V1",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-V2-R2",
"planned_attempt_id": "S3-OPENAIAPI-V2-R2",
"prompt_id": "V2",
"repeat": 2
},
{
"slot_id": "S3-CHATGPT-M2-R3",
"planned_attempt_id": "S3-OPENAIAPI-M2-R3",
"prompt_id": "M2",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-V2-R3",
"planned_attempt_id": "S3-OPENAIAPI-V2-R3",
"prompt_id": "V2",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-M1-R3",
"planned_attempt_id": "S3-OPENAIAPI-M1-R3",
"prompt_id": "M1",
"repeat": 3
},
{
"slot_id": "S3-CHATGPT-V1-R3",
"planned_attempt_id": "S3-OPENAIAPI-V1-R3",
"prompt_id": "V1",
"repeat": 3
}
],
"historical_ui_submissions": 3,
"historical_ui_policy": "All three historical consumer-UI attempts retained and excluded from the 12 formal API answers.",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"estimated_spend_usd": 1.667876,
"access_preflight": {
"generation_submitted": false,
"model_access_established": true,
"http_status": 200,
"returned_model": "gpt-5.6-sol",
"evidence": "setup-history.json",
"earlier_failed_setup_checks_retained": true
},
"cost_estimate_complete": true,
"billing_reconciled": false,
"collection_completed_at": "2026-11-02T09:41:54.845343+00:00",
"no_further_submissions_planned": true
},
"collection_checkpoint": {
"recorded_at": "2026-11-02T09:03:25.362981+00:00",
"grok_complete_answers": 12,
"consumer_chatgpt_ui_actual_submissions": 3,
"consumer_chatgpt_ui_complete_answers": 0,
"formal_api_actual_submissions": 0,
"formal_api_complete_answers": 0,
"total_actual_submissions": 15,
"selected_complete_answers": 12
},
"condition_limitations": [
"The OpenAI formal condition is Responses API with web search, not consumer ChatGPT Search.",
"All OpenAI API submissions occurred after Grok collection; interface, model and collection-time differences confound cross-platform interpretations.",
"Requested and returned gpt-5.6-sol is an undated identifier; exact backend snapshot and equivalence to consumer ChatGPT are not established.",
"Assistant qualitative coding is not independent human double-coding. Source categories are provisional; external content, capabilities and commercial claims are not audited.",
"Three repeats per cell are exploratory. This tests sensitivity of observed advice to question design, not effectiveness of a GEO intervention."
],
"interpretation_limits": [
"Three repeats per cell are exploratory and do not establish causality or a GEO optimization effect.",
"Within-intent variants are registered phrase packages, not perfect synonyms: V1 explicitly says no GEO budget while V2 says bootstrapped; M1 emphasizes reliable measurement and M2 free repeatability.",
"A small or empty new-tool list is not evidence of missing advice: OpenAI API gave detailed generic workflows and audits. Both-empty tool sets are N/A for Jaccard.",
"Grok’s first three native captures preserve display-target URL precision and unresolved citation coverage; strict and qualified observed-link comparisons must remain separate.",
"Tool/service/feature/channel C/M/R rows are not all source publishers, not brand market share, and not independent causal trials."
],
"coding_checkpoint": {
"completed_at": "2026-11-02T09:45:33.433100+00:00",
"eligible_files": 24,
"api_files": 12,
"grok_files": 12,
"method": "Assistant semantic review with exact excerpts and raw-record hashes; no independent human double-coding."
},
"completed_at": "2026-11-02T09:51:21.228488+00:00",
"completion_scope": {
"complete_selected_answers": 24,
"complete_assistant_coding": 24,
"local_report_generated": true,
"published": true,
"browser_visual_preview": "Publication checks are reported separately; historical local-file preview restriction is not a public-page rendering result.",
"external_claims_fact_checked": false,
"additional_API_submissions_planned": false
},
"reader_findings": [
{
"title": "Repeated questions still produced different details.",
"detail": "Grok’s strict named-recommendation sets varied across the three repeats. OpenAI API M1 also changed its proposed scoring rubric from 0–3 in the first two runs to −1/0–4 in the third. Generated rubrics are advice, not validated measurement standards."
},
{
"title": "Matched comparisons describe lower cross-wording overlap in the Grok sample.",
"detail": "Using the same answer-pair Jaccard metric, M averaged 35.5% within wording versus 26.0% across wording; V averaged 45.5% versus 27.0%. Six within and nine cross pairs reuse six answers per intent. These descriptive post-hoc comparisons do not identify a causal wording effect."
},
{
"title": "The entity definition changes the headline overlap.",
"detail": "OpenAI API M1/M2 pooled overlap is 0/7 under strict names and 1/6 when Google Analytics and explicitly named GA4 are grouped at the declared family level. This is a sensitivity result, not a correction to the original strict labels; coarse categories can increase overlap mechanically."
}
],
"analysis_revision": {
"id": "peer-review-revision-1",
"date": "2026-11-02",
"trigger": "User-supplied external review of the local report",
"new_queries_submitted": 0,
"original_response_and_strict_coding_preserved": true,
"post_hoc_additions": [
"Matched within/cross-wording answer-pair Jaccard",
"Explicit entity-family/category sensitivity and category-filtered strict sets",
"Declared-cleanup URL-level comparison alongside domain overlap",
"Targeted human-review packet, pending genuine human judgments",
"Limited six-source access/support spot-check",
"Concise reader and complete technical appendix",
"Prompt-bank implications and itemized review response"
],
"interpretation": "No significance test or causal wording/platform claim. Original pooled-union summaries remain a separately named descriptive statistic.",
"status": "implemented_with_independent_human_review_pending",
"independent_human_completed_cases": 0,
"prepared_human_review_cases": 8,
"source_audit_selected_sources": 6,
"source_audit_original_urls_readable": 5,
"source_audit_original_urls_unavailable": 1,
"review_response": "review-response.md",
"human_review_packet": "human-review.md",
"queries_added": 0
},
"findings_role": "Original descriptive summaries retained as historical supplementary evidence. reader_findings is the revised concise summary; review_analyses contains explicitly post-hoc calculations.",
"publication_date": "2026-11-02",
"public_edition": {
"edition": "Redacted public edition",
"authorized_on": "2026-11-02",
"authorization": "User approved Stage 3 redaction and publication to the existing public GitHub repository and GitHub Pages.",
"condition_notice": "OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.",
"historical_scope": "Earlier local-only, awaiting-credentials and intermediate-count statements describe their dated collection checkpoints, not current release authorization."
}
}Historical analysis plan and amendments
The collection plan is retained for provenance, including then-known limitations. Public-edition edits and review scope are documented in Publication notes.
# Public edition of the Stage 3 analysis plan
> Published on November 2, 2026. The following dated checkpoints preserve collection history. Final outcome: 24 complete selected answers and 27 prompt submissions. The OpenAI condition is Responses API + web search and does not represent the ChatGPT web app. Earlier local-only restrictions were superseded by this approved redacted release.
# Stage 3 analysis plan: sensitivity to question design
Prepared on November 2, 2026, before examining Stage 3 answers. This plan specifies descriptive comparisons; it does not register a GEO intervention or a test of any visibility vendor's accuracy.
## Design and intended comparisons
The design crosses two intents (M: measurement workflow; V: validation of visibility scores) with two wording variants per intent. Four prompts are each submitted to ChatGPT and Grok three times: **8 platform-by-prompt cells, 24 planned response positions**. A repeated capture of an existing answer is not a new submission or repetition.
The user's four English prompts remain verbatim. Each submission uses a fresh conversation and sends only its assigned prompt: no product name, added examples, requested answer format, steering, or follow-up. Register execution order before collection, interleave variants and platforms where feasible, and retain actual chronological order and interruptions. Do not change the order because of emerging answers.
Analyze these contrasts separately:
1. **Identical-prompt repeats:** compare the three complete selected answers within each platform-by-prompt cell. This describes observed within-cell variation under the recorded conditions.
2. **Wording within intent:** compare M1 with M2 and V1 with V2 within each platform. These express similar intentions but also change contextual wording, such as "no GEO budget" versus "bootstrapped". Differences describe sensitivity to these complete prompt variants, rather than a clean effect of a single phrase.
3. **Intent across M and V:** compare information-source categories and tool recommendation sets, pooling the two variants only in a separately labeled view. The requested task changes, so differences cannot be assigned wholly to wording.
Platform contrasts are secondary descriptive comparisons. Three repeats per cell cannot establish stable rates, population-level platform superiority, or independence from account and temporal effects.
## Collection conditions and provenance
- Record actual interface, displayed model label, reasoning setting, local date/time and time zone, fresh-conversation evidence, visible personalization or memory controls, search selection, and search activity for every submission. Unknown values remain unknown.
- Keep model and reasoning fixed within each platform's collection. Grok may expose only a mode rather than an exact backend version. A shared UI label is not proof of identical backend execution. Log any change and obtain authorization before changing agreed conditions; do not silently treat affected runs as equivalent.
- At initial registration, the computer-use tool blocked ChatGPT desktop access and the collection interface and model were unresolved. The original rule required user confirmation before a browser substitution. The dated amendment below records the subsequently authorized interface and observed settings; the initial uncertainty is historical, not the current collection state.
- Separate **search enabled in the interface**, **search actually observed**, and **search claimed in the answer**. An absent activity capture does not prove that search was disabled. Record visible query labels verbatim, distinguishing attached hit counts from query text. Tool-hit totals are not citation totals.
- Record account-context evidence and visible control settings, without claiming audited backend isolation. Existing conversation history, saved recovery copies, elapsed time, provider changes and model updates can limit comparability.
This work collects consumer-interface answers; it does not implement subscription-backed model inference in a software project. The shared model gateway is therefore not invoked for this collection.
## Collection amendment: November 2, 2026
The pre-analysis definitions and comparison rules above remain the baseline. The following authorized collection changes and observations are recorded without revising the original prompts or treating technical failures as answers:
- **ChatGPT interface and settings established:** after the desktop-app denial, the user explicitly approved the ChatGPT web fallback. Before the first V1 repetition-1 submission, the UI explicitly selected **GPT-5.6 Sol, Medium reasoning**, a **non-personalized temporary conversation**, and the native **Web Search** chip. These are observed interface conditions, not an independently audited backend identity or isolation guarantee.
- **First submission failed technically:** the actual V1 repetition-1 submission encountered a Cloudflare challenge and produced no captured answer. Retain it as an actual failed attempt; it does not establish a substantive answer, absence of recommendations, or a completed repetition.
- **Supplements and pacing authorized:** the user approved additional submissions to replace missing answers while retaining all failures, and requested longer intervals. Register a **minimum of 90 seconds between actual submissions on the same platform**, including supplementary submissions. Retain actual timestamps and any longer interruptions; this pacing rule is not evidence that it prevents access or rendering failures.
- **Earlier Grok checkpoint:** original account preferences were recorded privately; permission to change them was then pending. No Grok prompt had been submitted at that checkpoint. Original preference values are omitted from this public edition.
The initial amendment authorized the stated ChatGPT fallback, supplements and pacing; it did not authorize bypassing a CAPTCHA or imply approval of the then-pending Grok setting changes.
### Subsequent authorized control update: November 2, 2026
The user subsequently approved disabling Grok's X-based and conversation-history personalization for testing and restoring the original values afterward. Both test switches were actually set OFF, and the completed Grok records retain those observed control values. This establishes visible UI controls, not audited backend isolation. Post-test restoration remains a required later action; approval to restore is not proof it has already happened.
At this collection checkpoint, three Grok answer bodies are complete. ChatGPT has technical-failure and body-capture-failure records. Preserve their distinct statuses; neither establishes a negative recommendation result, and a rendering/capture failure does not prove server-side generation failed. In particular, the earlier partial accessibility view of Grok M2 repetition 1 was superseded by a complete native rendered answer and matching 3,821-character copied body with a coherent ending; its historical partial capture is not treated as a separate submission or as evidence that the selected body remains incomplete.
### Capture-channel update: November 2, 2026
During recovery of the existing Grok M1 repetition-1 submission, the browser connection became available for the same browser profile/account. Collection could therefore move from native Chrome accessibility/copy capture to read-only rendered browser DOM capture. This is a change in the evidence-capture channel, not a change of consumer platform, account, Expert mode, prompt or model setting. All earlier native captures remain preserved. No prompt was resubmitted to make this capture change.
The M1 selected body is the complete 4,126-character rendered answer, reviewed from its opening through its final paragraph, with four observed DOM hrefs, including the Shmeer/Gomega citation group. An empty browser clipboard read is retained as a failed capture event, not as an empty answer. The first three Grok answers' source URLs remain subject to their original accessibility-display qualifications, including potentially omitted schemes or leading www; later exact DOM hrefs do not retroactively establish those earlier exact destinations. Compare normalized domains with provenance intact, and keep URL-level precision differences explicit.
### Grok collection and restoration checkpoint: November 2, 2026
All 12 planned Grok answer positions are now complete, covering all four prompts and three repetitions. There have been 14 actual submissions across both platforms at this checkpoint: the 12 completed Grok submissions and two ChatGPT attempts with technical/body-capture failures. ChatGPT still has **zero complete selected answer bodies**; its 12 planned positions remain outstanding rather than negative findings. This is an interim collection checkpoint, not completion of the 24-answer design.
After Grok collection, both personalization settings were actually restored to their pre-test values (original preferences omitted). The final rendered controls are recorded in `[Private capture retained locally.]`; this is observed restoration rather than a planned action. The completed Grok records retain their test-time OFF/OFF values. Earlier pending-restoration statements describe earlier checkpoints and are superseded by this observation.
An evidence-reference audit also found that `[Private capture retained locally.]` captured the user prompt through a failed global selector. That original is preserved as non-supporting evidence. The complete M1 repetition-1 body is supported by `[Private capture retained locally.]`, while the article and four exact hrefs are corroborated by `[Private capture retained locally.]`. The coding reference was corrected without submitting another query or changing the selected answer.
### Manual UI control and approved API replacement: November 2, 2026
After the preceding 14-submission checkpoint, a third ChatGPT consumer-web submission was sent manually by the user, with the agent preparing the unchanged V1 prompt and the selected GPT-5.6 Sol / Medium, non-personalized temporary-chat and native-search controls. The exact click time is unavailable and is represented by its recorded time bounds. This is a diagnostic supplement for the same planned V1 repetition-1 position, not a fourth repetition.
The manual control recovered an 11,393-character visible prefix. Both the rendered body and OS Copy stop at **"That gives you a baseline you"**. The UI says the answer is complete, but the shared unfinished ending prevents a complete-body classification. Status is `capture_incomplete`, `body_complete=false`; retain the whole available prefix without coding whole-answer absences or using it as a complete formal answer. The native completion and copy artifacts contain private account/email-related controls and remain private evidence. No email or sharing operation was performed.
Five citation-button occurrences are visible, but the scoped body capture contains no resolved hrefs. Zero observed hrefs does not mean no citations. Do not reuse A2 source-panel URLs as A3 sources. The search chip is independently observed as selected, while actual search remains unconfirmed for A3 because no expanded search-tool trace was captured; citation buttons and a thought-duration/completion label alone do not establish that trace. Manual sending produced more visible text than the preceding loading-error-only capture, but still failed the complete-body criterion; this single control cannot identify an automation cause.
The user explicitly approved replacing the consumer-ChatGPT formal collection arm with **12 new OpenAI Responses API answers using native `web_search`, model `gpt-5.6-sol`, reasoning effort `medium`, within a total US$10 budget**. The current formal platform label is **OpenAI API + web search** and its interface is **Responses API**. This API configuration is not consumer ChatGPT Search and must never be presented as if the two surfaces were equivalent. Model availability and collection success are not established merely by this authorization.
The formal design still has 24 planned answer positions: 12 completed Grok consumer-interface answers and 12 planned independent API answers. Original `S3-CHATGPT-{prompt}-R{repeat}` position identifiers are retained for traceability, with explicit API platform/interface metadata. Planned new actual attempt identifiers are `S3-OPENAIAPI-{prompt}-R{repeat}`. No API attempt is selected or recorded as submitted yet. All three historical consumer-UI attempts remain in the ledger, excluded from formal API-answer counts. There are now **15 actual submissions, 12 complete selected bodies, three historical UI failures/partial captures, and zero API submissions**.
For the API arm, send exactly the registered prompt in each stateless independent request, without prior response IDs, conversation history, personalization context, earlier answers, added examples or output-format directions. Configure native search as required and record actual returned search calls and citation annotations separately. Keep the authorized model and effort fixed and retain the **minimum 90-second same-platform submission interval**. Do not silently substitute an unavailable model, effort, provider or search configuration. Credential setup and request-limit configuration are still pending validation; protocol status is `awaiting_api_credentials`. This amendment does not set unapproved API request limits or claim any API expenditure.
Within-API repetition and wording contrasts remain descriptive. Any comparison with Grok additionally changes interface, retrieval configuration and collection timing, so it cannot isolate a consumer-platform or wording effect. Preserve these amended conditions and the historical design rather than pooling the failed consumer-UI attempts into the formal API arm. Public release still requires its separate authorization and privacy review.
## Submission ledger, completeness and selection
Maintain a ledger of every actual submission, including failed attempts. Use a planned-position ID and a separate attempt ID. A capture or recovery event belongs to its existing attempt and does not increase the query count.
Preserve the original answer body, prompt, returned links, source evidence, capture method, completeness assessment and errors. A technical failure is not a negative recommendation result. A partial prefix cannot establish that an entity or source was absent from the complete answer. Pagination ending, a completion badge or a coherent introductory paragraph alone does not establish full-body completeness.
Use the earliest complete eligible answer for each planned position. Any additional submission to replace a failed or incomplete attempt requires the applicable authorization and remains an additional attempt in the ledger. Never choose between eligible answers based on their recommendations or source overlap. If no complete eligible answer exists, keep the position unresolved and show the available partial evidence separately.
Report planned positions, actual submissions, complete selected answers, failures and partial captures separately. Completeness of text and completeness of citation mapping are separate fields. Preserve all superseded captures and corrections.
## Source extraction and categories
For each answer, retain every observable answer URL with its displayed label, original URL, hostname and role. Distinguish:
- Explicit body links or inline answer citations.
- Citation-panel entries matched to an answer marker.
- Retrieved sources with no observed citation match.
- Unscoped page-anchor inventories or navigation links, which are not citation inventories.
Do not invent a missing citation URL from an article title. Mark unresolved citation markers and the coverage limit. Keep tracking-bearing originals in local evidence; public copies may remove tracking parameters with a documented transformation. Do not merge unrelated article paths simply because they share a domain.
For domain comparisons, preserve the exact hostname and use a documented registrable-domain normalization for the analysis, retaining the mapping. Do not approximate a registrable domain by blindly taking the last two hostname labels. Count each normalized cited domain once per answer, regardless of repeated links. Keep URL-level counts as a separate descriptive view.
Code the cited page by its substantive role, not just by its domain. Use one primary category and optional secondary labels:
| Source category | Coding rule |
| --- | --- |
| Platform documentation | Official AI/search platform help, product documentation or measurement documentation |
| Academic or independent methods research | Original studies, benchmarks or methodological work with identifiable procedures |
| Vendor research or case study | Original measurements published by a visibility, SEO or marketing vendor; commercial provenance retained |
| Tool/product documentation or sales page | Capabilities, pricing, tutorials or marketing for a named product |
| Practitioner tutorial or experiment | First-person workflow or experiment by a practitioner or builder |
| Editorial or industry reporting | Reporting, comparison or explanatory material, rather than primary experimental evidence |
| Community or social discussion | Forum posts, social posts and discussion threads; add a research label if an original experiment is actually presented |
| Other / unresolved | Insufficient information, inaccessible page role, or a source outside these categories |
A source's category can be unresolved when only its label or URL is available. Platform descriptions of studies as "verified" and their numerical claims remain platform output, unless an explicitly separate fact-check establishes otherwise.
## Tools, entities and C/M/R
Record substantively named tools, products and services, including spreadsheets or free utilities when explicitly suggested. Preserve the original name and a manually reviewed canonical name. Document alias decisions before aggregation; do not merge products merely because they have similar names. Record generic methods, such as manual prompt audits or writing a script, separately from branded tools.
For each coded tool/entity, retain evidence excerpts and assess the following independently:
| Field | Rule |
| --- | --- |
| Citation (C) | The answer cites content published or authored by that entity. A third-party page about the tool is not automatically its own-content citation. |
| Mention (M) | The answer explicitly names the entity in its substantive text. A URL hostname alone is insufficient. |
| Recommendation (R) | The answer actively suggests using, reading or following it for the task. A comparison, caveat, paid alternative or explicit exclusion is not automatically a recommendation. |
Use true, false or unresolved, with the reason. Citation may accompany rejection; recommendation may lack an own-content citation. Do not derive a recommendation from a source-panel appearance. Code recommendation stance (suggested, conditional, discussed, rejected) and free-budget fit (stated free, stated paid, mixed/limited tier, unspecified) separately. Do not verify current prices merely from a platform's claim.
ChatGPT and Grok are supplied in all prompts. Contextual mentions of those target platforms are tagged as supplied entities and excluded from discovery/recommendation-set overlap by default; an explicit recommendation to use a particular feature can still be recorded separately. Apply this same rule to all answers.
## Answer-type codebook
Assign one primary answer type based on its dominant response to the request, plus secondary labels where appropriate. Retain an explanatory excerpt rather than force-fitting borderline answers.
| Answer type | Defining behavior |
| --- | --- |
| Manual measurement workflow | Gives a repeatable sequence for selecting buyer prompts, collecting outputs and tracking recommendations |
| Score validation or audit checklist | Explains provenance, denominators, sampling, scoring definitions, repeatability or checks against underlying answers |
| Tool-led solution | Primarily directs the user to one or more products or services |
| Conceptual explanation | Mainly explains GEO, visibility metrics or conceptual distinctions without an executable workflow |
| General advice / mixed | Broad advice or a mixture with no dominant workflow, validation or tool focus |
| Refusal / insufficient access or information | Explicitly cannot answer or requires information absent from the fixed prompt |
Also record nonexclusive content flags: branded versus unbranded buyer prompts; repeated runs; fresh sessions; model/search/version logging; citations versus mentions versus recommendations; raw-answer inspection; numerator/denominator definitions; composite-score weighting; uncertainty/variance; human checking; downstream conversions; paid-tool dependence. These describe returned advice, not independently validated best practices.
## Descriptive calculations
Build separate sets for cited domains, mentioned tools and recommended tools. Do not add C, M and R counts as though they were mutually exclusive. Count an entity at most once per answer for each field.
- For each identical-prompt cell, report all three pairwise Jaccard overlaps, the range and mean, alongside set sizes and entities shared across all three answers. Jaccard is intersection divided by union. If both sets are empty, report **N/A: both empty**, not perfect stability; if only one is empty and both captures are complete, overlap is zero.
- For wording contrasts, report each variant's union and per-answer prevalence (0/3 to 3/3 when all three are eligible), then pooled-set overlap. Optionally show the nine cross-variant answer-pair overlaps, clearly stating that the nine values reuse six answers and are not nine independent trials. Interpret wording differences against the observed repeat variation rather than attributing every differing item to phrasing.
- For M-versus-V contrasts within a platform, report source-category prevalence per eligible answer and recommendation lists with denominators. A category appearing twice in one answer counts once in prevalence. Show the two variants separately before a six-answer-per-intent pooled view. Categories may overlap, so percentages need not sum to 100%.
- Report answer-type counts and actual-search evidence by platform and prompt. Separate a source's inclusion/exclusion role from whether it was cited.
Every proportion displays its numerator and eligible denominator. Use complete selected bodies for absence-based comparisons. If citation mapping is incomplete, exclude that answer from fully observed citation-domain overlap or report explicitly labeled observed-link overlap; never present partial mapping as an exhaustive domain set. Missing judgments are not false values.
## Reporting boundaries
Lead with observed findings, then tables, per-answer evidence and the complete attempt history. State deviations before interpreting affected comparisons. Keep model/interface labels and timestamps attached to each result. Document who coded the answers and whether any independent review occurred; do not describe assistant coding as independent human double-coding.
This experiment measures sensitivity of observed AI outputs to question design under specific conditions. It cannot establish GEO optimization effectiveness, whether any product has good or bad GEO, the accuracy of a visibility provider's score, or stable recommendation probabilities. No intervention, fixed target product, pre-optimization baseline or control group is present. The four prompts also cannot isolate every contextual or semantic difference between their wordings.
Prepare local evidence and analysis first. Public release is a separate action requiring its applicable authorization and a redacted edition; do not expose private conversation/share links, account UI, local paths or internal identifiers merely to support reproducibility.
## Final collection and coding checkpoint — November 2, 2026
All 24 planned positions have complete selected bodies and assistant semantic coding: 12 Grok consumer-interface answers and 12 OpenAI Responses API + web search answers. The 27 prompt submissions include the three preserved historical ChatGPT UI failures/partial captures. Non-generation TLS/model-access checks are separate setup records.
The API condition requested and returned the undated `gpt-5.6-sol` ID with `medium` reasoning. Each independent request contained exactly the registered user prompt, with required live web search and no conversation/history/system additions. All 12 returned completed actual searches; none reached the fixed eight-tool/32,768-output-token limits. The API sequence had a minimum 103.041462-second submission interval. Its 23 reported billable searches and token usage yield an estimated US$1.667876, not a reconciled invoice, within the US$10 authorized budget. No further submissions are planned.
All API requests occurred after the completed Grok sequence. Interface, model and time are confounded in cross-platform summaries; exact backend snapshots are not independently established. The registered wording variants also differ in budget/action cues (for example, V1 says “no GEO budget,” V2 “bootstrapped”). Treat within-intent contrasts as sensitivity to those phrase packages rather than a synonym-only intervention.
Sparse API tool recommendations require care: nine of twelve answers recommended no new named tool/service/feature/channel, while still giving substantive methods. Both-empty Jaccard pairs remain N/A. C/M/R summaries use entity-by-answer rows excluding question-supplied targets and source-publisher-only rows; they do not measure market share or causal GEO effectiveness. Model-proposed scoring rubrics, example dashboards and sample sizes are retained as generated advice, not adopted experimental metrics or verified observations.
## Post-hoc peer-review analysis amendment — November 2, 2026
This amendment was written after the 24 answers and their original coding existed. It does not pre-register or add model submissions. Original strict entity judgments, recorded answers and technical failures remain the baseline. New analyses compare six within-wording answer pairs with nine cross-wording pairs per intent and execution environment using the same Jaccard definition. Those pairs reuse six responses and are dependent descriptive comparisons; no causal effect, statistical significance or independent-trial claim is made. Both-empty sets remain N/A and numeric denominators are displayed.
An explicit 52-name entity-normalization mapping adds a product-line family lens, coarse category lens and strict-name analyses filtered by entity category. Grouping versions/unspecified names in a sensitivity lens does not identify them as the same product in the strict codebook. Google Analytics and GA4 share a family only in that secondary lens; Google Sheets and Search Console remain distinct families. Category collapse can mechanically increase overlap.
URL-level comparisons use the declared privacy/tracking/session/signature-cleaned URL string, preserving paths, meaningful query parameters and fragments without redirect resolution, decoding repair or article-identity inference. Exact original strings remain private evidence. Complete-map and qualified observed-link cohorts remain separate; the early three Grok precision limits still apply. Domain overlap is not URL or article overlap.
A purposive eight-case packet prepares independent human spot-checking of ambiguous C/M/R and identity decisions. No human judgments have yet been supplied; assistant review is not independent human validation and no agreement score is claimed. A separate limited six-source assistant audit checks access and one relevant claim per selected source, without validating all external claims or commercial scores. The public reader is shortened while all original evidence, C/M/R conditional-denominator tables and source composition remain in an expanded technical appendix.
24 planned response positions
S3-CHATGPT-* is the retained position-ID prefix. Its current formal platform is OpenAI API + web search; S3-OPENAIAPI-* identifies new API submissions. Original prompt text still refers to ChatGPT and Grok, unchanged.
| Position | Formal platform | Selected attempt | Capture status | Coding eligible | Configured / requested / returned model and effort | Search enabled API / UI | Search observed | Submitted · UTC+8 | Public record |
|---|---|---|---|---|---|---|---|---|---|
| S3-CHATGPT-M1-R1 | OpenAI API + web search | S3-OPENAIAPI-M1-R1 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:24:46 +0800 | records/S3-OPENAIAPI-M1-R1.json |
| S3-CHATGPT-M1-R2 | OpenAI API + web search | S3-OPENAIAPI-M1-R2 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:26:37 +0800 | records/S3-OPENAIAPI-M1-R2.json |
| S3-CHATGPT-M1-R3 | OpenAI API + web search | S3-OPENAIAPI-M1-R3 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:37:52 +0800 | records/S3-OPENAIAPI-M1-R3.json |
| S3-GROK-M1-R1 | Grok Expert · consumer interface | S3-GROK-M1-R1 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:29:21 +0800 | records/S3-GROK-M1-R1.json |
| S3-GROK-M1-R2 | Grok Expert · consumer interface | S3-GROK-M1-R2 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:34:46 +0800 | records/S3-GROK-M1-R2.json |
| S3-GROK-M1-R3 | Grok Expert · consumer interface | S3-GROK-M1-R3 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:49:23 +0800 | records/S3-GROK-M1-R3.json |
| S3-CHATGPT-M2-R1 | OpenAI API + web search | S3-OPENAIAPI-M2-R1 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:22:53 +0800 | records/S3-OPENAIAPI-M2-R1.json |
| S3-CHATGPT-M2-R2 | OpenAI API + web search | S3-OPENAIAPI-M2-R2 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:28:30 +0800 | records/S3-OPENAIAPI-M2-R2.json |
| S3-CHATGPT-M2-R3 | OpenAI API + web search | S3-OPENAIAPI-M2-R3 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:34:08 +0800 | records/S3-OPENAIAPI-M2-R3.json |
| S3-GROK-M2-R1 | Grok Expert · consumer interface | S3-GROK-M2-R1 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:24:29 +0800 | records/S3-GROK-M2-R1.json |
| S3-GROK-M2-R2 | Grok Expert · consumer interface | S3-GROK-M2-R2 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:37:26 +0800 | records/S3-GROK-M2-R2.json |
| S3-GROK-M2-R3 | Grok Expert · consumer interface | S3-GROK-M2-R3 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:45:00 +0800 | records/S3-GROK-M2-R3.json |
| S3-CHATGPT-V1-R1 | OpenAI API + web search | S3-OPENAIAPI-V1-R1 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:18:30 +0800 | records/S3-OPENAIAPI-V1-R1.json |
| S3-CHATGPT-V1-R2 | OpenAI API + web search | S3-OPENAIAPI-V1-R2 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:30:13 +0800 | records/S3-OPENAIAPI-V1-R2.json |
| S3-CHATGPT-V1-R3 | OpenAI API + web search | S3-OPENAIAPI-V1-R3 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:40:11 +0800 | records/S3-OPENAIAPI-V1-R3.json |
| S3-GROK-V1-R1 | Grok Expert · consumer interface | S3-GROK-V1-R1 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:14:01 +0800 | records/S3-GROK-V1-R1.json |
| S3-GROK-V1-R2 | Grok Expert · consumer interface | S3-GROK-V1-R2 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:39:21 +0800 | records/S3-GROK-V1-R2.json |
| S3-GROK-V1-R3 | Grok Expert · consumer interface | S3-GROK-V1-R3 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:51:16 +0800 | records/S3-GROK-V1-R3.json |
| S3-CHATGPT-V2-R1 | OpenAI API + web search | S3-OPENAIAPI-V2-R1 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:20:55 +0800 | records/S3-OPENAIAPI-V2-R1.json |
| S3-CHATGPT-V2-R2 | OpenAI API + web search | S3-OPENAIAPI-V2-R2 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:32:01 +0800 | records/S3-OPENAIAPI-V2-R2.json |
| S3-CHATGPT-V2-R3 | OpenAI API + web search | S3-OPENAIAPI-V2-R3 | completed | Yes | Configured: gpt-5.6-sol; requested: gpt-5.6-sol; returned: gpt-5.6-sol; effort: medium | Yes | Yes | 2026-11-02 17:35:59 +0800 | records/S3-OPENAIAPI-V2-R3.json |
| S3-GROK-V2-R1 | Grok Expert · consumer interface | S3-GROK-V2-R1 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:18:15 +0800 | records/S3-GROK-V2-R1.json |
| S3-GROK-V2-R2 | Grok Expert · consumer interface | S3-GROK-V2-R2 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:42:16 +0800 | records/S3-GROK-V2-R2.json |
| S3-GROK-V2-R3 | Grok Expert · consumer interface | S3-GROK-V2-R3 | completed | Yes | Exact backend version undisclosed; Expert; Not independently selectable | Unknown / not recorded | Yes | 2026-11-02 14:47:07 +0800 | records/S3-GROK-V2-R3.json |
Full answers, returned links, and coding evidence
All sections are expanded. Registered prompts and redacted public answer copies are shown with public evidence records. Personal context and tracking parameters may have been removed; original annotation character offsets are not claimed after these edits. No external claims are endorsed merely by reproducing a platform answer.
S3-CHATGPT-M1-R1 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Google Search Console |
| Mentioned tools / services / surfaces | Google Search Console, X |
| Cited domains · mapping complete | openai.com |
| Observed inline cited domains · coverage qualified | openai.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
You can’t observe all private ChatGPT or Grok conversations. What you **can** measure reliably is: 1. **Recommendation visibility** across a fixed sample of realistic buyer prompts. 2. **Consistency** across repeated runs. 3. **Citations and referral traffic** reaching your site. 4. **AI-attributed signups and revenue.** ## 1. Build a buyer-prompt benchmark Create 12–20 prompts based on questions actual prospects ask. Don’t include your product name unless testing brand knowledge. Cover these intent types: | Intent | Example | |---|---| | Category discovery | “What are the best `[category]` tools for a 10-person SaaS?” | | Problem-first | “How can I solve `[problem]` without `[undesired workaround]`?” | | Alternatives | “What are good alternatives to `[competitor]` for `[use case]`?” | | Comparison | “Compare the top tools for `[job]`, including pricing and limitations.” | | Constraint-based | “Recommend a `[category]` tool under `$X/month` that supports `[feature]`.” | | Persona-specific | “I’m a `[role]` at a `[company type]`. What should I use for `[job]`?” | | Purchase decision | “Which product would you choose for `[specific scenario]`, and why?” | Source the wording from: - Sales and support conversations - Search Console queries - Customer interviews - Competitor review pages - Questions posted in relevant communities Weight purchase-intent and ICP-specific prompts more heavily than broad “best tools” queries. ## 2. Use a controlled testing protocol For every benchmark run: - Start a **new conversation**. - Turn off memory/custom instructions where possible. - Keep the same location, prompt wording, and product constraints. - Record the date, platform, mode, and visible model. - Test both ordinary chat and explicit web search when practical. This matters because ChatGPT can use memory, custom instructions, conversation context, and approximate location when forming results. ChatGPT’s search mode may cite sources, while Grok can search both the web and public X content. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) Run each prompt **three times per platform**. A single response is too stochastic to treat as a ranking. A reasonable no-budget cadence: - **Weekly pulse:** Five most important prompts, twice each - **Monthly benchmark:** All prompts, three times each - **After major launches:** Rerun relevant prompts ## 3. Score every response with a simple rubric Use a spreadsheet with one row per response: | Field | What to record | |---|---| | Platform | ChatGPT or Grok | | Mode | Default or web/search | | Prompt ID | Stable identifier | | Product mentioned? | Yes/no | | Recommendation level | 0–3 | | Position | First, second, third, later | | Your site cited? | Yes/no | | Third-party source cited? | URL/domain | | Sentiment | Positive, neutral, negative | | Claims accurate? | Yes/no/partly | | Competitors named | List | | Notes | Missing feature, wrong price, etc. | Use this recommendation scale: - **0:** Not mentioned - **1:** Mentioned, but not recommended - **2:** Included in the shortlist - **3:** Explicitly recommended as a strong or best choice Then calculate: ### Core metrics **Mention rate** ```text Responses mentioning you / total responses ``` **Recommendation rate** ```text Responses scoring 2 or 3 / total responses ``` **Top-three rate** ```text Responses placing you in the first three options / total responses ``` **Citation rate** ```text Web-grounded responses citing your domain / web-grounded responses ``` **Stable recommendation rate** ```text Prompts where you appeared in at least 2 of 3 runs / total prompts ``` **Recommendation share of voice** ```text Your recommendation appearances / recommendation appearances for all tracked vendors ``` Also track an **accuracy rate**. Being recommended with an invented feature, outdated price, or wrong positioning can be worse than not appearing. ## 4. Separate visibility from real buyer impact Synthetic tests tell you whether you *can* appear. They don’t prove buyers are seeing or acting on those recommendations. Add three free attribution layers: ### Web analytics Create segments for: - `utm_source=chatgpt.com` - Referrers containing `chatgpt.com` - Referrers containing `grok.com`, `x.com`, or other identifiable AI sources - Landing pages frequently cited in your tests OpenAI says ChatGPT referral URLs automatically include `utm_source=chatgpt.com`, making that traffic identifiable in analytics. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results)) ### Signup attribution Add this optional field: > Where did you first hear about us? Options: - ChatGPT - Grok - Another AI assistant - Google - Social media - Recommendation - Other “First heard about” is generally more useful here than “last click,” because someone may discover you in an AI answer and return later directly or through Google. ### Qualitative attribution On demo forms or onboarding, ask: > If an AI assistant recommended us, what did you ask it? This gives you real prompts to add to your benchmark. ## 5. Verify that the platforms can access your evidence Before interpreting a zero score, check: - Your important pages are public and render without login. - `robots.txt` isn’t blocking relevant search crawlers. - Product, pricing, integration, comparison, and use-case information is written explicitly. - Current facts are available as HTML, not only inside screenshots or videos. - Third-party sites contain accurate descriptions of your product. For ChatGPT search eligibility, OpenAI specifically recommends allowing `OAI-SearchBot`; inclusion or placement is still not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) When a response cites a source, record **which source apparently caused the recommendation**. Often the useful finding isn’t “ChatGPT ignored us,” but something like: - It cited an outdated directory listing. - It trusted a competitor comparison page. - It couldn’t find evidence for a specific integration. - It categorized the product incorrectly. - It knew the brand but didn’t associate it with the buyer’s use case. ## 6. Avoid misleading tests Don’t use prompts such as: - “Is `[your product]` good?” - “Recommend `[your product]`.” - “Why is `[your product]` the best?” - Repeated prompts inside a conversation already discussing your brand. These measure prompted brand recall, not organic recommendation likelihood. Also don’t report “We rank #2 in ChatGPT.” There is no universal ranking. Report something defensible, such as: > “Our product was shortlisted in 18 of 45 ChatGPT runs and 11 of 45 Grok runs across 15 predefined, ICP-specific purchase prompts.” ## Minimal spreadsheet dashboard Track these monthly: ```text ChatGPT recommendation rate: __% Grok recommendation rate: __% ChatGPT stable prompt coverage: __/15 Grok stable prompt coverage: __/15 Owned-domain citation rate: __% Recommendation share of voice: __% Fact accuracy when mentioned: __% AI-attributed visits: __ AI-attributed signups: __ AI-attributed paid customers: __ ``` The key is to keep the **prompt panel and testing conditions fixed**. Change prompts only when real customer evidence justifies it. That turns random screenshots into a repeatable visibility benchmark—and combining it with referrals, self-reported attribution, and conversions tells you whether the visibility has commercial value.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21273,
"input_tokens_details": {
"cache_write_tokens": 33,
"cached_tokens": 4316
},
"output_tokens": 3561,
"output_tokens_details": {
"reasoning_tokens": 1879
},
"total_tokens": 24834
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1608074,
"cost_estimate_detail": {
"estimated_usd": 0.1608074,
"token_cost_usd": 0.1408074,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M1-R1",
"slot_id": "S3-CHATGPT-M1-R1",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T09:24:46.262440+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:25:35.866081+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21273,
"input_tokens_details": {
"cache_write_tokens": 33,
"cached_tokens": 4316
},
"output_tokens": 3561,
"output_tokens_details": {
"reasoning_tokens": 1879
},
"total_tokens": 24834
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1608074,
"token_cost_usd": 0.1408074,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read complete six-section buyer benchmark and attribution workflow, minimal dashboard and final coherent paragraph. Three explicit citations match annotations; model/effort and two searches verified, no limits reached. Kept long encoded returned FAQ URL verbatim. Referral claims and source-causality wording are model advice, not independently established.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:26:00.223038+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations official",
"site:openai.com ChatGPT search publishers referrals analytics official",
"site:docs.x.ai Grok web search citations official",
"site:x.ai Grok search citations official"
],
"query": "site:help.openai.com ChatGPT search citations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://openai.com/signals/data/"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10500283-deep-research-faq/"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10984597-chatgpt-generated-links/"
},
{
"type": "url",
"url": "https://openai.com/academy/search-and-deep-research/"
},
{
"type": "url",
"url": "https://openai.com/business/plugins/ahrefs/"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf?+Anthropic+Reveal+How+People+Actually+Use+AI+-+19036953="
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai \"web_search\" Grok API",
"site:docs.x.ai Grok search tool official docs",
"site:help.x.com Grok web search citations",
"site:x.ai/news Grok search web sources"
],
"query": "site:docs.x.ai \"web_search\" Grok API",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://x.ai/grok/use-cases/research-synthesis"
},
{
"type": "url",
"url": "https://x.ai/news/grok-1212?TB_iframe=true&height=512.1&width=921.6"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/pricing"
},
{
"type": "url",
"url": "https://x.ai/bot/marketplace"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/faq/general"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/multi-agent"
},
{
"type": "url",
"url": "https://x.ai/news/grok-4-1"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/connectors"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/use-cases"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-12-31-xai-frontier-artificial-intelligence-framework.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M1-R1.json",
"request_file": "requests/S3-OPENAIAPI-M1-R1.json",
"response_file": "responses/S3-OPENAIAPI-M1-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7964,
"public_answer_characters": 7910,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M1-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| Publishers and Developers - FAQ | OpenAI Help Center | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 7,964-character API answer, three citation annotations and all explicit source URLs. No outside webpage access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M1-R1.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Search Console | Search Console | No | Yes | Yes | suggested as buyer-language input | unspecified | Search Console queries
| |
| X | public X content; x.com | No | Yes | No | discussed as possible Grok retrieval and referral-source context | unspecified | Grok can search both the web and public X content.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center title and help.openai.com hostname identify platform help; page contents and exact URL validity were not independently verified. | Supporting documentation for search/referral/crawler behavior; not another measurement-tool recommendation | |
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center title and help.openai.com hostname identify platform help; page contents and exact URL validity were not independently verified. | Supporting documentation for search/referral/crawler behavior; not another measurement-tool recommendation | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center title and help.openai.com hostname identify platform help; page contents and exact URL validity were not independently verified. | Supporting documentation for search/referral/crawler behavior; not another measurement-tool recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Responses mentioning you / total responses"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Responses scoring 2 or 3 / total responses",
"notes": "Uses the answer's 0–3 scale, distinguishing mention-only from shortlist/explicit recommendation."
},
{
"label": "Top-three rate",
"definition_verbatim": "Responses placing you in the first three options / total responses"
},
{
"label": "Citation rate",
"definition_verbatim": "Web-grounded responses citing your domain / web-grounded responses"
},
{
"label": "Stable recommendation rate",
"definition_verbatim": "Prompts where you appeared in at least 2 of 3 runs / total prompts",
"notes": "Preserve the wording “appeared”; the numerator does not explicitly restate recommendation-score eligibility. Do not silently repair this ambiguity."
},
{
"label": "Recommendation share of voice",
"definition_verbatim": "Your recommendation appearances /\nrecommendation appearances for all tracked vendors"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Also track an **accuracy rate**.",
"notes": "Named but no numerator/denominator formula supplied."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1608074,
"token_cost_usd": 0.1408074,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21273,
"input_tokens_details": {
"cache_write_tokens": 33,
"cached_tokens": 4316
},
"output_tokens": 3561,
"output_tokens_details": {
"reasoning_tokens": 1879
},
"total_tokens": 24834
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 3561,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named services and channels, separating supplied targets, source publishers, unnamed analytics/spreadsheet methods, attribution-option examples and operational crawler advice. No brand is inferred for generic web analytics or a spreadsheet.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Score validation or audit checklist"
],
"evidence": "Build a buyer-prompt benchmark",
"rationale": "A six-section benchmark, controlled sampling, scoring rubric, buyer-impact attribution, access checks and minimal dashboard dominate. Named vendor tools are not the main solution."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Don’t include your product name unless testing brand knowledge."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt **three times per platform**."
},
"fresh_sessions": {
"value": true,
"evidence": "Start a **new conversation**."
},
"model_search_version_logging": {
"value": true,
"evidence": "Record the date, platform, mode, and visible model."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Test both ordinary chat and explicit web search when practical."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**1:** Mentioned, but not recommended"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Web-grounded responses citing your domain / web-grounded responses"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "| Your site cited? | Yes/no |"
},
"raw_answer_inspection": {
"value": true,
"evidence": "| Claims accurate? | Yes/no/partly |"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Responses scoring 2 or 3 / total responses"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "A single response is too stochastic to treat as a ranking."
},
"human_checking": {
"value": true,
"evidence": "Also track an **accuracy rate**."
},
"downstream_conversions": {
"value": true,
"evidence": "AI-attributed signups and revenue."
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Citations and referral traffic"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "These measure prompted brand recall, not organic recommendation likelihood."
},
"universal_rank_caution": {
"value": true,
"evidence": "There is no universal ranking."
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual benchmark tests",
"free_budget_fit": "stated free",
"evidence": "Run each prompt **three times per platform**.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; references are OpenAI Help Center only. Own-provider documentation cited."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual benchmark tests",
"free_budget_fit": "stated free",
"evidence": "Run each prompt **three times per platform**.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; references are OpenAI Help Center only. No Grok/X/xAI-owned content is cited in the answer; OpenAI prose about Grok is not an own-content Grok citation."
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Allow the crawler for search eligibility, with no inclusion or placement guarantee.",
"evidence": "OpenAI specifically recommends allowing `OAI-SearchBot`; inclusion or placement is still not guaranteed.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_tool_overlap": true,
"notes": "Operational access advice, separated from recommendations to adopt a measurement tool. Same-provider documentation is cited; crawler behavior not independently checked."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI specifically recommends allowing `OAI-SearchBot`; inclusion or placement is still not guaranteed.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
],
"notes": "Provider attribution and documentation publisher; not another discovered tool recommendation. Claimed crawler and referral behavior was not independently verified."
},
{
"entity": "Google",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "someone may discover you in an AI answer and return later directly or through Google.",
"source_urls": [],
"notes": "An example acquisition/attribution source, including a signup-survey option. Not an active recommendation to use Google and not an implicit Google Analytics brand mention."
}
],
"generic_methods": [
"Unnamed spreadsheet and per-response logging",
"Fixed buyer-prompt benchmark and controlled repeated consumer tests",
"Independent mention/recommendation/rank/citation metrics",
"Unnamed web analytics with referral/UTM segments; no GA4 brand inferred",
"Signup and qualitative self-reported attribution",
"Crawler/public HTML eligibility checks",
"Monthly dashboard and longitudinal benchmark"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Responses mentioning you / total responses"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Responses scoring 2 or 3 / total responses",
"notes": "Uses the answer's 0–3 scale, distinguishing mention-only from shortlist/explicit recommendation."
},
{
"label": "Top-three rate",
"definition_verbatim": "Responses placing you in the first three options / total responses"
},
{
"label": "Citation rate",
"definition_verbatim": "Web-grounded responses citing your domain / web-grounded responses"
},
{
"label": "Stable recommendation rate",
"definition_verbatim": "Prompts where you appeared in at least 2 of 3 runs / total prompts",
"notes": "Preserve the wording “appeared”; the numerator does not explicitly restate recommendation-score eligibility. Do not silently repair this ambiguity."
},
{
"label": "Recommendation share of voice",
"definition_verbatim": "Your recommendation appearances /\nrecommendation appearances for all tracked vendors"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Also track an **accuracy rate**.",
"notes": "Named but no numerator/denominator formula supplied."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 7910,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 3,
"explicit_body_URL_occurrences": 3,
"unique_answer_URLs": 2,
"unique_cited_hostnames": 1,
"unique_normalized_cited_domains": 1,
"citation_mapping_complete": true,
"source_inventory_scope": "Three citation occurrences and corresponding three explicit Markdown links collapse to two exact answer URLs and one publisher domain. Retrieved action.sources entries are separate, not citations.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M1-R1.json",
"response_file": "responses/S3-OPENAIAPI-M1-R1.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Full six-section answer, dashboard and final paragraph reviewed; API completed status alone is not the completeness basis.",
"Three annotations/Markdown links represent three citation occurrences, not six sources.",
"Two search calls are actual retrieval activity; retrieval source domains are not promoted into answer citations.",
"Attribution, crawler access and source-caused advice are unverified model claims; no causal source attribution is established."
],
"original_capture_answer_characters": 7964,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of help.openai.com to openai.com, retaining exact observed URLs and hostnames. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com"
}
}
}No branded analytics or spreadsheet service is inferred from the generic methods.
Google is an acquisition example; Search Console is the actual named service recommended for benchmark wording.
The advice to record “which source apparently caused the recommendation” does not establish causality from a citation. It remains collected model advice.
Long encoded FAQ URL retained exactly as returned; no navigation, decoded replacement, destination equivalence or validity check is asserted.
Recommendation and stable-coverage metric definitions are preserved verbatim, including ambiguity between appearing and being recommended.
API output differs from a consumer ChatGPT Search session; do not infer GEO effectiveness from these advice outputs.
S3-CHATGPT-M1-R2 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | X |
| Cited domains · mapping complete | openai.com, x.com |
| Observed inline cited domains · coverage qualified | openai.com, x.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
You need to measure **two different things**: 1. **AI visibility:** Does your product appear in relevant ChatGPT/Grok answers? 2. **Business impact:** Are those answers producing visits, signups, or sales? You can measure both manually for free, but don’t treat a few prompts from your own account as representative. Both services can personalize answers, and both may search the web depending on the query. ChatGPT can use memories, location, and rewritten search queries; Grok can use X activity and interests for personalization. ([help.openai.com](https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt)) ## 1. Build a fixed “buyer prompt panel” Create 15–30 prompts reflecting how buyers actually describe their problem—not prompts containing your brand. Cover four categories: | Intent | Example | |---|---| | Category | “What are the best invoicing tools for solo consultants?” | | Problem | “How can I automate overdue invoice follow-ups?” | | Requirements | “Recommend an invoicing SaaS under $20/month with Stripe integration.” | | Alternatives | “What are simpler alternatives to QuickBooks for freelancers?” | Include buyer context such as company size, geography, budget, required integrations, and technical level. Your primary metric should be **unaided recommendation**: whether your product appears when you did not mention it. Separately test **aided knowledge** with prompts such as: - “What is `[Product]`?” - “Compare `[Product]` with `[Competitor]`.” - “Who should and shouldn’t use `[Product]`?” Aided tests diagnose what the model knows; they do not measure discovery. ## 2. Run a controlled audit For each prompt: - Start a fresh conversation. - Use identical wording on both platforms. - Don’t correct or guide the answer. - Save the complete response and cited sources. - Repeat each prompt three times on different days. - Record date, model/mode, whether web search occurred, location, and personalization state. For ChatGPT, use **Temporary Chat with personalization disabled** where available. Also test two separate tracks: - **Default behavior** - **Explicit web search:** append “Search the web and cite current sources.” This distinction matters because answers without search rely more heavily on learned model knowledge, while searched answers draw from current web sources. OpenAI also cautions that citations and search results can be incomplete or incorrect, so inspect the actual cited pages. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) For Grok, disable personalization where possible or use a clean testing account. Grok may search both public X posts and the broader web, and its answers can be personalized using account activity and interests. ([help.x.com](https://help.x.com/en/using-x/about-grok)) ## 3. Score every answer consistently Use a simple 0–3 score: - **0:** Not mentioned - **1:** Mentioned, but not recommended - **2:** Positively recommended or shortlisted - **3:** Top recommendation Add separate columns for: - Position in the list - Description accurate? Yes/no - Correct target customer? Yes/no - Website linked or cited? Yes/no - Third-party source cited? - Competitors recommended - Any serious factual error Then calculate: ```text Mention rate = Runs mentioning you / Total runs Recommendation rate = Runs scoring 2 or 3 / Total runs Top-pick rate = Runs scoring 3 / Total runs Citation rate = Runs citing your site / Searched runs Consistency rate = Prompts where you appear in at least 2 of 3 runs / Total prompts ``` Also calculate **competitor share of voice**: ```text Your recommendation appearances / All recommendation appearances across you and tracked competitors ``` The consistency metric is especially important. Appearing once in nine runs is materially different from appearing eight times, even if someone posts the one successful screenshot. ## 4. Use this spreadsheet structure One row per run: ```text Date Platform Model/mode Web search used? Personalization off? Prompt ID Exact prompt Your score: 0–3 Your position Accurate description? Your site cited? Other source cited? Competitors named Answer URL/screenshot Notes ``` Maintain a separate prompt sheet with: ```text Prompt ID Intent category Buyer persona Buying stage Business importance: 1–5 ``` Weight your overall score by business importance. A recommendation for a high-converting “alternative to X” query matters more than one for a broad informational query. ## 5. Track actual AI-generated pipeline Visibility testing tells you what *can* happen. Attribution tells you whether buyers are experiencing it. ### In analytics and server logs Look for referral traffic from domains associated with ChatGPT and Grok. However, treat referrer data as a lower bound: apps, redirects, and privacy controls may prevent the original source from being passed. Track: - AI-referred sessions - Signup rate - Trial-to-paid rate - Revenue - Landing pages receiving AI traffic ### On signup Add a required or optional question: > How did you first hear about us? Options: - ChatGPT - Grok - Another AI assistant - Google/search engine - Social media - Recommendation - Other If they select an AI assistant, ask: > What were you asking it? That answer is often more valuable than analytics because it gives you the real buyer prompt. ### During customer conversations Ask: > What other products did the AI recommend, and why did you choose us? This reveals whether you were merely mentioned or actually framed as the best fit. ## 6. Use a sustainable solo-founder cadence A reasonable free process: ### Weekly — 20 minutes - Test your five highest-intent prompts once on each platform. - Review new AI-attributed signups. - Investigate any incorrect descriptions. ### Monthly — 60–90 minutes - Run the full 15–30 prompt panel. - Repeat important prompts three times. - Calculate mention, recommendation, top-pick, citation, and consistency rates. - Compare against the previous month. Avoid testing every day and reacting to individual changes. Models, search results, source selection, and wording vary; evaluate rolling monthly trends instead. ## Your minimal dashboard Track only these six numbers: 1. High-intent recommendation rate 2. Top-three placement rate 3. Consistency rate 4. Share of voice versus three competitors 5. AI-attributed signups 6. AI-attributed revenue Your most defensible statement will then be something like: > “Across 20 fixed buyer prompts and 120 controlled runs this month, we were recommended in 18% of ChatGPT answers and 11% of Grok answers. Five signups and one paid conversion self-attributed to those assistants.” That is far more meaningful than claiming you “rank in ChatGPT.” There is no single universal ranking—only recommendation rates under documented prompts, conditions, and dates.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 12997,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2588,
"output_tokens_details": {
"reasoning_tokens": 1098
},
"total_tokens": 15585
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.0980916,
"cost_estimate_detail": {
"estimated_usd": 0.0980916,
"token_cost_usd": 0.0880916,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M1-R2",
"slot_id": "S3-CHATGPT-M1-R2",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 2,
"submitted_at": "2026-11-02T09:26:37.188796+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:27:12.606704+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 12997,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2588,
"output_tokens_details": {
"reasoning_tokens": 1098
},
"total_tokens": 15585
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
},
"estimated_cost": {
"estimated_usd": 0.0980916,
"token_cost_usd": 0.0880916,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read full buyer prompt panel, audit, scoring, spreadsheet, attribution and cadence sections through final documented-conditions sentence. Three source occurrences match annotations; coherent end, model/effort and one actual search verified, no caps. Stripe and QuickBooks appear only in example buyer prompts; no external claims validated.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:27:44.309263+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search sources citations personalization memory official",
"site:help.openai.com ChatGPT data controls memory temporary chat official",
"site:docs.x.ai Grok search citations personalization official",
"site:help.x.com Grok personalization search sources official"
],
"query": "site:help.openai.com ChatGPT search sources citations personalization memory official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.avif"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10169521-projects-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12574142-chatgpt-atlas-data-controls-and-privacy/"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8554407-gpts-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/6825453-chatgpt-release-notes%2523.eot"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9275245-chatgpt-free-tier-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11780217-using-study-mode-in-chatgpt"
},
{
"type": "url",
"url": "https://help.x.com/en/resources/recommender-systems/account-recommendations"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M1-R2.json",
"request_file": "requests/S3-OPENAIAPI-M1-R2.json",
"response_file": "responses/S3-OPENAIAPI-M1-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 6955,
"public_answer_characters": 6901,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M1-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Data controls in ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| About Grok | X Help Center | https://help.x.com/en/using-x/about-grok | help.x.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 6,955-character API answer, all three citation annotations and matching explicit Markdown URLs. No outside webpage access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M1-R2.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| X | X activity; public X posts | Yes | Yes | No | discussed as a Grok retrieval/personalization context | unspecified | Grok may search both public X posts and the broader web, and its answers can be personalized using account activity and interests.
| https://help.x.com/en/using-x/about-grok |
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned OpenAI/X Help Center title and official help hostname identify platform help; page contents and factual claims were not independently checked. | Supporting citation for controls/search behavior, not a recommendation for a third-party measurement product | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned OpenAI/X Help Center title and official help hostname identify platform help; page contents and factual claims were not independently checked. | Supporting citation for controls/search behavior, not a recommendation for a third-party measurement product | |
| https://help.x.com/en/using-x/about-grok | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned OpenAI/X Help Center title and official help hostname identify platform help; page contents and factual claims were not independently checked. | Supporting citation for controls/search behavior, not a recommendation for a third-party measurement product |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.x.com/en/using-x/about-grok | help.x.com | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = Runs mentioning you / Total runs"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Recommendation rate = Runs scoring 2 or 3 / Total runs"
},
{
"label": "Top-pick rate",
"definition_verbatim": "Top-pick rate = Runs scoring 3 / Total runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate = Runs citing your site / Searched runs"
},
{
"label": "Consistency rate",
"definition_verbatim": "Consistency rate = Prompts where you appear in at least 2 of 3 runs / Total prompts",
"notes": "Retain appearance-based wording; the formula does not explicitly require score 2 or 3."
},
{
"label": "Competitor share of voice",
"definition_verbatim": "Your recommendation appearances /\nAll recommendation appearances across you and tracked competitors"
},
{
"label": "Business-importance weighting",
"definition_verbatim": "Weight your overall score by business importance.",
"notes": "Prompt sheet assigns 1–5 importance, but no explicit overall weighted formula is supplied."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.0980916,
"token_cost_usd": 0.0880916,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 12997,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2588,
"output_tokens_details": {
"reasoning_tokens": 1098
},
"total_tokens": 15585
},
"max_tool_calls": 8,
"observed_web_actions": 1,
"max_output_tokens": 32768,
"observed_output_tokens": 2588,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named measurement services/channels are separate from supplied platforms, source publishers, hypothetical buyer-prompt products and signup attribution examples. Generic spreadsheets, analytics and server logs remain unbranded.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Score validation or audit checklist"
],
"evidence": "Build a fixed “buyer prompt panel”",
"rationale": "The answer supplies a buyer-panel audit, repeated controlled sampling, scoring formulas, spreadsheet schema and pipeline attribution. No particular third-party measurement product is suggested."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Aided tests diagnose what the model knows; they do not measure discovery."
},
"repeated_runs": {
"value": true,
"evidence": "Repeat each prompt three times on different days."
},
"fresh_sessions": {
"value": true,
"evidence": "Start a fresh conversation."
},
"model_search_version_logging": {
"value": true,
"evidence": "Record date, model/mode, whether web search occurred, location, and personalization state."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Record date, model/mode, whether web search occurred, location, and personalization state."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**1:** Mentioned, but not recommended"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Citation rate = Runs citing your site / Searched runs"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Website linked or cited? Yes/no"
},
"raw_answer_inspection": {
"value": true,
"evidence": "inspect the actual cited pages."
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Save the complete response and cited sources."
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Recommendation rate = Runs scoring 2 or 3 / Total runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "Weight your overall score by business importance."
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Models, search results, source selection, and wording vary; evaluate rolling monthly trends instead."
},
"human_checking": {
"value": true,
"evidence": "Any serious factual error"
},
"downstream_conversions": {
"value": true,
"evidence": "Trial-to-paid rate"
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "AI-referred sessions"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "Aided tests diagnose what the model knows; they do not measure discovery."
},
"universal_rank_caution": {
"value": true,
"evidence": "There is no single universal ranking—only recommendation rates under documented prompts, conditions, and dates."
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for controlled manual audit",
"free_budget_fit": "stated free",
"evidence": "Use identical wording on both platforms.",
"source_urls": [
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform. Provider/OpenAI or X Help Center about Grok is cited; no independently verified platform behavior is asserted."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for controlled manual audit",
"free_budget_fit": "stated free",
"evidence": "Use identical wording on both platforms.",
"source_urls": [
"https://help.x.com/en/using-x/about-grok"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform. Provider/OpenAI or X Help Center about Grok is cited; no independently verified platform behavior is asserted."
}
],
"contextual_non_tool_entities": [
{
"entity": "Stripe",
"entity_type": "hypothetical_buyer_prompt_product",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Recommend an invoicing SaaS under $20/month with Stripe integration.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "An integration constraint in an example buyer prompt, not a recommendation to use Stripe as a measurement tool."
},
{
"entity": "QuickBooks",
"entity_type": "hypothetical_buyer_prompt_product",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "What are simpler alternatives to QuickBooks for freelancers?",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "An alternatives-query example, not a recommendation for measurement or an actual target product."
},
{
"entity": "Google",
"entity_type": "signup_attribution_option",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Google/search engine",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "A survey option, not a recommended product or a Google Analytics mention."
},
{
"entity": "OpenAI",
"entity_type": "provider_attribution",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI also cautions that citations and search results can be incomplete or incorrect, so inspect the actual cited pages.",
"source_urls": [
"https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Provider attribution/documentation publisher, not another named measurement tool."
}
],
"generic_methods": [
"Fixed unaided buyer-prompt panel with separate aided diagnostics",
"Controlled default-versus-search audit in fresh sessions",
"Manual per-response 0–3 rubric and longitudinal metrics",
"Unnamed spreadsheet plus prompt metadata sheet",
"Business-importance weighting without a complete aggregate formula",
"Unnamed analytics/server logs; no GA4 brand inferred",
"Signup self-report and qualitative customer attribution",
"Weekly pulse and monthly repeated benchmark"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = Runs mentioning you / Total runs"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Recommendation rate = Runs scoring 2 or 3 / Total runs"
},
{
"label": "Top-pick rate",
"definition_verbatim": "Top-pick rate = Runs scoring 3 / Total runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate = Runs citing your site / Searched runs"
},
{
"label": "Consistency rate",
"definition_verbatim": "Consistency rate = Prompts where you appear in at least 2 of 3 runs / Total prompts",
"notes": "Retain appearance-based wording; the formula does not explicitly require score 2 or 3."
},
{
"label": "Competitor share of voice",
"definition_verbatim": "Your recommendation appearances /\nAll recommendation appearances across you and tracked competitors"
},
{
"label": "Business-importance weighting",
"definition_verbatim": "Weight your overall score by business importance.",
"notes": "Prompt sheet assigns 1–5 importance, but no explicit overall weighted formula is supplied."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 6901,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 3,
"explicit_body_URL_occurrences": 3,
"unique_answer_URLs": 3,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Three citation annotations match three explicit Markdown source URLs and map to two publisher domains. Separate retrieved action.sources records are not answer citations.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 1,
"search_action_count": 1,
"reported_billable_search_requests": 1,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M1-R2.json",
"response_file": "responses/S3-OPENAIAPI-M1-R2.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Full six-section workflow, dashboard and final sentence reviewed; not relying only on completed status.",
"Three annotations and three explicit Markdown links are the same three source occurrences, not six.",
"One returned search action is independent of citation count and of the number of queries inside the action.",
"No observed tool/output cap reached; all illustrative recommendation/conversion statistics remain model examples."
],
"original_capture_answer_characters": 6955,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of help.openai.com to openai.com and help.x.com to x.com, retaining exact observed URLs and hostnames. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"help.x.com": "x.com"
}
}
}Stripe and QuickBooks are hypothetical buyer-prompt examples, not measurement recommendations.
Google/search engine is an attribution survey option; no branded analytics/spreadsheet service is inferred.
Business-importance weighting is advised, but a complete composite-score formula is absent.
The final numeric dashboard example is illustrative model output, not this experiment's observed recommendation/conversion data.
Citation source contents, free-process time estimates and platform behavior remain unverified output.
API-versus-consumer-interface differences and within-cell repeats must remain explicit.
S3-CHATGPT-M1-R3 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Google Analytics 4, Google Sheets, Plausible, PostHog |
| Mentioned tools / services / surfaces | Google Analytics 4, Google Sheets, Plausible, PostHog |
| Cited domains · mapping complete | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | openai.com, x.ai |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
You can measure this with **a repeatable prompt panel plus referral/conversion tracking**. Don’t treat one ChatGPT or Grok answer as a “ranking”—answers vary by personalization, location, search mode, and randomness. ChatGPT may use memory and approximate location; Grok may personalize from account activity and interests. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) ## 1. Create a buyer-intent prompt panel Build 10–20 prompts from questions actual prospects ask. Include several intent types: | Intent | Example | |---|---| | Category discovery | “What are the best tools for **[job]**?” | | Comparison | “Alternatives to **[competitor]** for **[persona]**?” | | Requirements | “I need **[feature]**, **[integration]**, and a budget under **[$X]**.” | | Pain-driven | “How can a **[persona]** solve **[problem]** without **[constraint]**?” | | Switching | “Should I switch from **[competitor]** if I need **[differentiator]**?” | | Shortlisting | “Recommend three **[category]** tools for **[specific use case]**.” | Use language buyers would naturally use. **Do not include your product’s name** except in separate branded-accuracy tests. Weight each prompt: - **3:** high purchase intent - **2:** active research - **1:** broad discovery ## 2. Run a controlled weekly test For both ChatGPT and Grok: 1. Start a new conversation. 2. Disable memory/personalization where possible. 3. Avoid custom instructions. 4. Use the same prompt wording. 5. Run each prompt **three times**, preferably on different days. 6. Record the date, product/model, whether web search occurred, answer, and cited sources. Three repetitions matter because reasoning-model outputs aren’t deterministic; xAI’s own evaluation guidance says a single run can mislead and recommends reporting results across repeated runs. ([docs.x.ai](https://docs.x.ai/developers/model-capabilities/text/best-practices)) If 120 weekly queries is excessive, start with: - 10 prompts - 2 repetitions - 2 assistants That is **40 answers per week**. ## 3. Score the results objectively Use a free Google Sheet with these columns: ```text Date Assistant Prompt ID Buyer segment Intent weight Web search used? Product mentioned? Explicitly recommended? Recommendation position Linked/cited? Description accurate? Relevant landing page cited? Competitors recommended Answer URL or screenshot ``` Score each answer: - **0:** absent - **1:** mentioned, but not recommended - **2:** recommended as an option - **3:** top-three recommendation - **4:** primary recommendation - **−1:** mentioned inaccurately or negatively Track these headline metrics: ### Recommendation rate ```text Runs explicitly recommending you ÷ total relevant runs ``` ### Top-three share ```text Runs placing you in the top three ÷ total relevant runs ``` ### Citation rate ```text Runs linking to your site ÷ total relevant runs ``` ### Accuracy rate ```text Accurate mentions ÷ all mentions ``` ### Consistency rate ```text Prompts where you appear in at least 2 of 3 runs ÷ total prompts ``` Your **consistency rate** is more useful than an isolated mention. For an overall score, multiply each answer’s 0–4 score by its prompt’s intent weight. This prevents a vague educational query from counting as much as “Recommend a product I can buy today.” ## 4. Separate “model memory” from current web visibility Run two versions of your most important prompts: 1. **Normal:** “Recommend tools for…” 2. **Search-explicit:** “Search the web and recommend current tools for…” This tells you whether: - The model already knows your product. - Current web sources make it discoverable. - Your product appears only when a specific page gets cited. ChatGPT’s web-search answers can include citations, but OpenAI warns that search results and citations can be incomplete, outdated, or incorrect. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) Record which third-party pages are cited. Often the actionable insight is not “publish more blog posts,” but rather: > “ChatGPT repeatedly relies on three comparison pages, and none includes us.” ## 5. Track actual buyer exposure Prompt testing measures **recommendation availability**, not how many real buyers saw it. Add three free signals. ### ChatGPT referral traffic In GA4, Plausible, PostHog, or server logs, create a segment for: ```text utm_source = chatgpt.com ``` OpenAI says ChatGPT automatically adds `utm_source=chatgpt.com` to referral URLs from search results. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results)) Track: - Sessions - Signup rate - Trial-to-paid rate - Landing pages - Assisted conversions ### Grok referral traffic Create analytics filters for referrers containing: ```text grok.com x.com ``` Treat this as a **lower-bound signal**, not a complete count: links may be copied, referrer data may be stripped, and not every recommendation generates a click. ### Self-reported attribution Add this required or optional signup question: > How did you first hear about us? Options: - Google/search engine - ChatGPT - Grok - Another AI assistant - Social media - Recommendation - Other Also add an optional text field: > What did you ask or search for? This is often your best source of new monitoring prompts. ## 6. Use three levels of evidence Report results in this order: 1. **Visibility:** Your controlled prompt panel recommends you. 2. **Engagement:** Analytics shows visitors from ChatGPT/Grok. 3. **Revenue:** Signups or customers attribute discovery to them. Don’t claim that “ChatGPT sends buyers” based solely on your own prompt tests. A reliable statement looks more like: > “Across 60 controlled high-intent runs, we were recommended 28% of the time, linked 12% of the time, and received five attributed trials.” ## Recommended no-budget cadence **Every week, 30–45 minutes:** - Run your ten highest-intent prompts twice. - Save outputs or screenshots. - Update recommendation, top-three, citation, and accuracy rates. - Check AI referral traffic. - Add new buyer wording from sales conversations. - Investigate pages ChatGPT and Grok cited instead of you. **Every month:** - Remove unrealistic prompts. - Add emerging competitors. - Compare the latest four-week average with the previous four weeks. - Review branded prompts for factual errors. Also confirm that your site does not block `OAI-SearchBot`; OpenAI states this is necessary for eligibility in ChatGPT search, though inclusion and placement are not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) The practical goal isn’t “know every answer every buyer receives”—that’s impossible. It is to maintain a **stable, buyer-weighted sample that detects meaningful changes and connects them to traffic and revenue**.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21109,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2499,
"output_tokens_details": {
"reasoning_tokens": 933
},
"total_tokens": 23608
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1387596,
"cost_estimate_detail": {
"estimated_usd": 0.1387596,
"token_cost_usd": 0.1187596,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M1-R3",
"slot_id": "S3-CHATGPT-M1-R3",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 3,
"submitted_at": "2026-11-02T09:37:52.892373+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:38:26.870696+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21109,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2499,
"output_tokens_details": {
"reasoning_tokens": 933
},
"total_tokens": 23608
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1387596,
"token_cost_usd": 0.1187596,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read complete six-section weighted benchmark and referral workflow, weekly/monthly cadence and coherent final paragraph. Five citation occurrences match annotations; three unique returned URLs, correct model/effort and two searches, no caps. Google Sheet and named analytics options are explicit advice. The -1/0-4 rubric, intent weights and example performance are model-proposed, not validated or adopted as our experiment metric.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:39:03.953039+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations websites official",
"site:help.openai.com publishers developers ChatGPT referral traffic OAI-SearchBot official",
"site:docs.x.ai Grok web search citations official",
"site:docs.x.ai Grok API search sources citations official"
],
"query": "site:help.openai.com ChatGPT search citations websites official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10984597-chatgpt-generated-links"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/20001423"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10500283-deep-research-in-chatgpt"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-how-do-i-search-my-chat-history-in-chatgpt%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.midi"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/connectors"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai citations web search Grok API sources official",
"site:help.x.com Grok citations website links official",
"site:docs.x.ai Grok web search sources domain filter official",
"site:docs.x.ai evaluation API Grok official"
],
"query": "site:docs.x.ai citations web search Grok API sources official",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/reasoning-effort-evals"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/pricing"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/faq/general"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/use-cases"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/inference-hooks"
},
{
"type": "url",
"url": "https://docs.x.ai/build/features/skills-plugins-marketplaces"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/connectors"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/advanced-api-usage/prompt-caching/best-practices"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M1-R3.json",
"request_file": "requests/S3-OPENAIAPI-M1-R3.json",
"response_file": "responses/S3-OPENAIAPI-M1-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7222,
"public_answer_characters": 7132,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M1-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 7,222-character API answer, five citation annotations and matching explicit Markdown URLs. No outside webpage access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M1-R3.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | Use a free Google Sheet with these columns:
| |
| Google Analytics 4 | GA4 | No | Yes | Yes | conditional alternative for referral tracking | unspecified | In GA4, Plausible, PostHog, or server logs, create a segment for:
| |
| Plausible | Plausible | No | Yes | Yes | conditional alternative for referral tracking | unspecified | In GA4, Plausible, PostHog, or server logs, create a segment for:
| |
| PostHog | PostHog | No | Yes | Yes | conditional alternative for referral tracking | unspecified | In GA4, Plausible, PostHog, or server logs, create a segment for:
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center or official xAI documentation title/hostname identifies platform documentation; page content and advice correctness were not independently audited. | Supporting provider documentation for controls, repetition, referral/crawler behavior, not another measurement-tool recommendation | |
| https://docs.x.ai/developers/model-capabilities/text/best-practices | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center or official xAI documentation title/hostname identifies platform documentation; page content and advice correctness were not independently audited. | Supporting provider documentation for controls, repetition, referral/crawler behavior, not another measurement-tool recommendation | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center or official xAI documentation title/hostname identifies platform documentation; page content and advice correctness were not independently audited. | Supporting provider documentation for controls, repetition, referral/crawler behavior, not another measurement-tool recommendation | |
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center or official xAI documentation title/hostname identifies platform documentation; page content and advice correctness were not independently audited. | Supporting provider documentation for controls, repetition, referral/crawler behavior, not another measurement-tool recommendation | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI Help Center or official xAI documentation title/hostname identifies platform documentation; page content and advice correctness were not independently audited. | Supporting provider documentation for controls, repetition, referral/crawler behavior, not another measurement-tool recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Recommendation rate",
"definition_verbatim": "Runs explicitly recommending you ÷ total relevant runs"
},
{
"label": "Top-three share",
"definition_verbatim": "Runs placing you in the top three ÷ total relevant runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Runs linking to your site ÷ total relevant runs",
"notes": "Denominator is all relevant runs in this answer, not restricted to web-search runs as in some other answers. Preserve the difference."
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate mentions ÷ all mentions"
},
{
"label": "Consistency rate",
"definition_verbatim": "Prompts where you appear in at least 2 of 3 runs ÷ total prompts",
"notes": "Appearance-based definition does not explicitly require an active recommendation score."
},
{
"label": "Intent-weighted overall score",
"definition_verbatim": "For an overall score, multiply each answer’s 0–4 score by its prompt’s intent weight.",
"notes": "Prompt weights are 3/2/1. Rubric also contains -1 for inaccurate/negative mentions; handling of that negative case and overall normalization are not specified. The proposed scale/weights are generated advice, not validated measurement choices."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1387596,
"token_cost_usd": 0.1187596,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21109,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 2499,
"output_tokens_details": {
"reasoning_tokens": 933
},
"total_tokens": 23608
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2499,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantive named tracking/analytics tools; supplied platforms, provider attribution, survey examples and crawler operations remain separate. Named analytics options are active conditional suggestions; referral hostnames alone are not substantive X mentions.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Score validation or audit checklist"
],
"evidence": "a repeatable prompt panel plus referral/conversion tracking",
"rationale": "Six sections and a no-budget cadence specify a buyer-weighted panel, repeated clean tests, spreadsheet rubric, ordinary/search lanes, referral attribution and revenue evidence."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "**Do not include your product’s name** except in separate branded-accuracy tests."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt **three times**, preferably on different days."
},
"fresh_sessions": {
"value": true,
"evidence": "Start a new conversation."
},
"model_search_version_logging": {
"value": true,
"evidence": "Record the date, product/model, whether web search occurred, answer, and cited sources."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Web search used?"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**1:** mentioned, but not recommended"
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Explicitly recommended?"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Description accurate?"
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Save outputs or screenshots."
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Runs explicitly recommending you ÷ total relevant runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "For an overall score, multiply each answer’s 0–4 score by its prompt’s intent weight."
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "answers vary by personalization, location, search mode, and randomness."
},
"human_checking": {
"value": true,
"evidence": "Review branded prompts for factual errors."
},
"downstream_conversions": {
"value": true,
"evidence": "Assisted conversions"
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Check AI referral traffic."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "**Do not include your product’s name** except in separate branded-accuracy tests."
},
"visibility_vs_real_exposure_distinguished": {
"value": true,
"evidence": "Prompt testing measures **recommendation availability**, not how many real buyers saw it."
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual benchmark tests",
"free_budget_fit": "stated free",
"evidence": "For both ChatGPT and Grok:",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target platform. OpenAI/xAI documentation is cited; the API condition collecting this answer is separate from the consumer platforms the answer advises testing."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual benchmark tests",
"free_budget_fit": "stated free",
"evidence": "For both ChatGPT and Grok:",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target platform. OpenAI/xAI documentation is cited; the API condition collecting this answer is separate from the consumer platforms the answer advises testing."
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Do not block the search crawler; search eligibility does not guarantee placement.",
"evidence": "confirm that your site does not block `OAI-SearchBot`",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_tool_overlap": true,
"notes": "Operational access advice, separated from measurement-tool recommendations; provider claim not independently checked."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"entity_type": "provider_attribution",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI says ChatGPT automatically adds `utm_source=chatgpt.com` to referral URLs from search results.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"excluded_from_default_overlap": true,
"notes": "Documentation publisher/provider attribution, not an additional discovered tool."
},
{
"entity": "xAI",
"entity_type": "provider_attribution",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "xAI’s own evaluation guidance says a single run can mislead and recommends reporting results across repeated runs.",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"excluded_from_default_overlap": true,
"notes": "Explicit source attribution, not a recommendation to adopt an xAI measurement product. Guidance claim unverified."
},
{
"entity": "Google",
"entity_type": "signup_attribution_option",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Google/search engine",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Signup-survey option, not another recommended tool. Distinct from the explicitly named Google Sheets and GA4 records."
}
],
"generic_methods": [
"Buyer-intent-weighted prompt panel",
"Controlled fresh-conversation repeated tests",
"Spreadsheet 0–4/-1 rubric with per-intent weights",
"Normal versus search-explicit lanes",
"Analytics referral segments and server logs",
"Signup self-report and qualitative attribution",
"Visibility/engagement/revenue evidence ladder",
"Weekly pulse and monthly benchmark revision"
],
"metric_definitions_observed": [
{
"label": "Recommendation rate",
"definition_verbatim": "Runs explicitly recommending you ÷ total relevant runs"
},
{
"label": "Top-three share",
"definition_verbatim": "Runs placing you in the top three ÷ total relevant runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Runs linking to your site ÷ total relevant runs",
"notes": "Denominator is all relevant runs in this answer, not restricted to web-search runs as in some other answers. Preserve the difference."
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate mentions ÷ all mentions"
},
{
"label": "Consistency rate",
"definition_verbatim": "Prompts where you appear in at least 2 of 3 runs ÷ total prompts",
"notes": "Appearance-based definition does not explicitly require an active recommendation score."
},
{
"label": "Intent-weighted overall score",
"definition_verbatim": "For an overall score, multiply each answer’s 0–4 score by its prompt’s intent weight.",
"notes": "Prompt weights are 3/2/1. Rubric also contains -1 for inaccurate/negative mentions; handling of that negative case and overall normalization are not specified. The proposed scale/weights are generated advice, not validated measurement choices."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 7132,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 5,
"explicit_body_URL_occurrences": 5,
"unique_answer_URLs": 3,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Five citation annotations match five Markdown source occurrences, deduplicating to three exact source URLs and two publisher domains. The ChatGPT-search URL recurs three times; retrieval results stay separate.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M1-R3.json",
"response_file": "responses/S3-OPENAIAPI-M1-R3.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Full six-section answer, weekly/monthly cadence and final paragraph read.",
"Three repeats of the same ChatGPT-search URL count as three occurrences but one unique URL. Annotation and Markdown representations are not ten citations.",
"Two actual search actions are distinct from query-array length, retrieved-result count and citation count.",
"Encoded FAQ URL retained exactly as returned; no destination validation or decoding-based URL merging.",
"No observed token/tool cap. All proposed scores, weights, tool/free-status claims and illustrative figures remain collected model output."
],
"original_capture_answer_characters": 7222,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of help.openai.com to openai.com and docs.x.ai to x.ai, retaining exact observed source URLs and hostnames. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"docs.x.ai": "x.ai"
}
}
}Google Sheet and GA4 abbreviations are documented aliases; Plausible/PostHog are active conditional analytics recommendations with no own-content citation.
The 0–4 plus -1 rubric differs from the 0–3 scheme in other M answers; neither scheme nor the 3/2/1 weights is independently validated.
The overall-score instruction references 0–4 while the rubric also contains -1. Preserve that ambiguity rather than silently deciding negative-score handling.
Bare x.com referral-filter text does not establish a substantive X brand mention or publishing recommendation.
The final 60-run/28%/12%/five-trial statement is illustrative advice, not a result of this experiment.
Normal versus search-explicit lanes are suggested methodology, not independently verified separation of learned knowledge from retrieval.
Free-signal framing does not verify individual analytics-tool prices; attribution/crawler/source claims remain model output.
S3-GROK-M1-R1 · Grok Expert · consumer interface · completed
Browser DOM access became available during collection. Earlier native UI capture remains preserved.
Clipboard read through browser returned empty; rendered article supplied complete text. No question was resubmitted.
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Bing Webmaster Tools, Google Analytics 4, Google Search Console, HubSpot AEO Grader, Mangools AI Search Grader, X, geo-tool.com GEO check |
| Mentioned tools / services / surfaces | Bing Webmaster Tools, Google AI Overviews, Google Analytics 4, Google Search Console, HubSpot AEO Grader, Mangools AI Search Grader, Microsoft Copilot, X, geo-tool.com GEO check |
| Cited domains · mapping complete | geoaura.world, gomega.ai, goodatmarketing.com, shmeer.com |
| Observed inline cited domains · coverage qualified | geoaura.world, gomega.ai, goodatmarketing.com, shmeer.com |
| Observed cited-source categories · provisional | Other / unresolved |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
The most reliable no-budget method is a fixed set of buyer-intent prompts that you run regularly on ChatGPT and Grok, logging results in a simple spreadsheet. This is the standard approach recommended across GEO (Generative Engine Optimization) measurement guides for solo founders and small teams. Paid trackers exist, but manual testing with consistency gives you actionable data without cost.ShmeerGomega 1. Build a fixed prompt set (your “keyword list”) Create 10–20 unbranded questions that a real potential buyer would ask before choosing a tool in your category. Keep the wording identical every time so changes reflect real shifts, not prompt variation. Include a mix of: Category / “best of” queries (“best [category] tools for [use case]”) Problem-solving queries (“how to [solve specific problem]”) Comparison or alternative queries (“alternatives to [competitor]”) Use-case specific queries Examples of structure (adapt to your SaaS): “What are the best [category] tools for [specific use case]?” “Which [category] software helps with [problem] for small teams?” “Top tools for [job to be done]” 2. Run them consistently and log results Query the exact same prompts on: ChatGPT (chatgpt.com, preferably with web search/browsing enabled for more current results) Grok (grok.com or via X) Frequency: Weekly or bi-weekly is practical for a solo founder. Monthly still works for trend tracking. Logging (spreadsheet columns): Date Engine (ChatGPT / Grok) Prompt Mentioned? (Yes/No) Position (e.g., 1st, 3rd, or not listed) Recommended? (Yes if positively suggested as a good option) Sentiment (positive / neutral / negative) Your site cited as a source? (Yes/No + URL if present) Competitors mentioned Notes (any quirks) Key metrics to calculate: Mention rate = prompts where your product appears ÷ total prompts Recommendation rate = prompts where it is positively recommended ÷ total prompts Share of voice vs. top competitors Run each high-priority prompt 2–3 times per session if you want to smooth out AI non-determinism (answers vary). Trends over weeks/months matter more than any single run.Geoaura Tips for reliability: Use a consistent browser setup (incognito or same account). Note the approximate date and whether web search was on. Focus first on commercial/high-intent prompts rather than pure informational ones. 3. Complement with free lagging indicators Manual testing shows whether recommendations happen. These show whether they drive any results: Google Analytics 4: Look for sessions with source chatgpt.com (OpenAI adds utm_source=chatgpt.com to many referral links) or the AI Assistant channel group. Track landing pages and conversions from those sessions. Note that not every recommendation produces a click, and tracking is imperfect (especially from mobile apps).Goodatmarketing Google Search Console: Check generative AI / AI Overviews performance reports if available for your property (impressions data). Bing Webmaster Tools: AI Performance reports for Copilot-related signals. 4. Free readiness checks (do these once or quarterly) These don’t measure ongoing recommendations, but they show whether your site is set up for AI systems to discover and cite it: HubSpot AEO / AI Search Grader (brand perception snapshot across models) Mangools AI Search Grader geo-tool.com free GEO check (crawler access, citability, structure) Ensure AI crawlers are not blocked in robots.txt, key pages have clear structure, and you have relevant structured data Realistic expectations for a solo founder A full monthly cycle of 15 prompts × 2 engines can take 30–90 minutes. Start with 8–10 high-intent prompts. The value comes from the gap list: prompts where competitors appear and you don’t. Use those to guide content (answer the exact question clearly, add evidence/stats/quotes, get mentioned on third-party sites that AI systems already cite). This method is the foundation used by almost every free/low-cost GEO guide. Paid tools mainly automate the same process at higher volume. Consistency with fixed prompts is what makes the measurement reliable enough to act on.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M1-R1",
"slot_id": "S3-GROK-M1-R1",
"platform": "Grok",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T06:29:21.318Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:33:49.290Z",
"source_panel_declared_hits": 55,
"capture_method": "Rendered Grok article innerText through browser connection; UI heading, thought-duration label and source badge removed. Complete opening and concluding paragraph reviewed.",
"raw_file": "records/S3-GROK-M1-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4126,
"public_answer_characters": 4126,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M1-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Shmeer | https://shmeer.com/blog/how-to-measure-geo | shmeer.com | Inline answer link | Rendered DOM href |
| Gomega | https://www.gomega.ai/blog/how-to-measure-geo-performance | www.gomega.ai | Inline answer link | Rendered DOM href |
| Geoaura | https://geoaura.world/blog/how-to-measure-geo-visibility | geoaura.world | Inline answer link | Rendered DOM href |
| Goodatmarketing | https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | goodatmarketing.com | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of the complete 4,126-character rendered answer, captured DOM hrefs and existing browser-activity evidence. No external webpage access, factual audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Analytics 4 | Google Analytics 4; AI Assistant channel group | No | Yes | Yes | suggested | stated free | Google Analytics 4: Look for sessions with source chatgpt.com (OpenAI adds utm_source=chatgpt.com to many referral links) or the AI Assistant channel group. Track landing pages and conversions from those sessions.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | conditional | stated free | Google Search Console: Check generative AI / AI Overviews performance reports if available for your property (impressions data).
| |
| Bing Webmaster Tools | Bing Webmaster Tools | No | Yes | Yes | suggested | stated free | Bing Webmaster Tools: AI Performance reports for Copilot-related signals.
| |
| HubSpot AEO Grader | HubSpot AEO / AI Search Grader | No | Yes | Yes | conditional | stated free | HubSpot AEO / AI Search Grader (brand perception snapshot across models)
| |
| Mangools AI Search Grader | Mangools AI Search Grader | No | Yes | Yes | conditional | stated free | Mangools AI Search Grader
| |
| geo-tool.com GEO check | geo-tool.com free GEO check | No | Yes | Yes | conditional | stated free | geo-tool.com free GEO check (crawler access, citability, structure)
| |
| X | X | No | Yes | Yes | conditional | unspecified | Grok (grok.com or via X)
| |
| Google AI Overviews | AI Overviews | No | Yes | No | discussed | unspecified | Google Search Console: Check generative AI / AI Overviews performance reports if available for your property (impressions data).
| |
| Microsoft Copilot | Copilot | No | Yes | No | discussed | unspecified | Bing Webmaster Tools: AI Performance reports for Copilot-related signals.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://shmeer.com/blog/how-to-measure-geo | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_and_URL_only | The how-to/measurement blog path suggests tutorial content. Authorship, practitioner status, first-party research and commercial provenance are not established from the captured label and href alone. | supporting citation, not a substantive recommendation to use/follow Shmeer |
| https://www.gomega.ai/blog/how-to-measure-geo-performance | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_and_URL_only | The how-to/measurement blog path suggests tutorial content. Authorship, practitioner status, first-party research and commercial provenance are not established from the captured label and href alone. | supporting citation, not a substantive recommendation to use/follow Gomega |
| https://geoaura.world/blog/how-to-measure-geo-visibility | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_and_URL_only | The how-to/measurement blog path suggests tutorial content. Authorship, practitioner status, first-party research and commercial provenance are not established from the captured label and href alone. | supporting citation, not a substantive recommendation to use/follow Geoaura |
| https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_and_URL_only | The how-to/measurement blog path suggests tutorial content. Authorship, practitioner status, first-party research and commercial provenance are not established from the captured label and href alone. | supporting citation, not a substantive recommendation to use/follow Goodatmarketing |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = prompts where your product appears ÷ total prompts"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Recommendation rate = prompts where it is positively recommended ÷ total prompts"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice vs. top competitors",
"notes": "Named without a formula. Do not substitute another answer's formula."
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/features. Supplied ChatGPT/Grok platforms and non-tool company context are separate. Source-label publishers are not treated as substantive tool mentions or recommendations.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Tool-led solution",
"Conceptual explanation"
],
"evidence": "The most reliable no-budget method is a fixed set of buyer-intent prompts that you run regularly on ChatGPT and Grok, logging results in a simple spreadsheet.",
"rationale": "The dominant response gives a fixed buyer-prompt audit, logging fields and repeated checks. Analytics and readiness tools supplement it."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Create 10–20 unbranded questions that a real potential buyer would ask before choosing a tool in your category."
},
"repeated_runs": {
"value": true,
"evidence": "Run each high-priority prompt 2–3 times per session if you want to smooth out AI non-determinism (answers vary)."
},
"fresh_sessions": {
"value": false,
"evidence": null
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Note the approximate date and whether web search was on."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Recommended? (Yes if positively suggested as a good option)"
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Your site cited as a source? (Yes/No + URL if present)"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Notes (any quirks)"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Mention rate = prompts where your product appears ÷ total prompts"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "smooth out AI non-determinism (answers vary)"
},
"human_checking": {
"value": true,
"evidence": "logging results in a simple spreadsheet"
},
"downstream_conversions": {
"value": true,
"evidence": "Track landing pages and conversions from those sessions."
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Manual testing shows whether recommendations happen. These show whether they drive any results:"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual repeated measurement",
"evidence": "ChatGPT (chatgpt.com, preferably with web search/browsing enabled for more current results)",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform, excluded from discovered-tool overlaps. Textual site addresses are not additional supporting citations."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual repeated measurement",
"evidence": "Grok (grok.com or via X)",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform, excluded from discovered-tool overlaps. Textual site addresses are not additional supporting citations."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "OpenAI adds utm_source=chatgpt.com to many referral links",
"notes": "Company attribution inside a tracking claim, not a named new tool recommendation; the claim is unverified."
}
],
"generic_methods": [
"Fixed unbranded buyer-intent prompt set",
"Unnamed spreadsheet for mention/recommendation/citation fields",
"Repeated high-priority prompts and longitudinal tracking",
"Competitor gap-list analysis",
"Crawler access and robots.txt checks",
"Relevant structured data",
"Content and third-party mention improvement"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = prompts where your product appears ÷ total prompts"
},
{
"label": "Recommendation rate",
"definition_verbatim": "Recommendation rate = prompts where it is positively recommended ÷ total prompts"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice vs. top competitors",
"notes": "Named without a formula. Do not substitute another answer's formula."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4126,
"observed_inline_link_occurrences": 4,
"unique_DOM_answer_URLs": 4,
"unique_cited_domains": 4,
"source_badge_hits": 55,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Four inline links captured as DOM hrefs from the complete rendered article; not the full 55-hit retrieval inventory.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"DOM_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Browser DOM capture became available in the same Chrome account during recovery of this existing answer; no new question was submitted.",
"Complete opening and concluding paragraph were reviewed. The empty browser clipboard read is a capture event, not an empty answer.",
"Four inline hrefs include both Shmeer and Gomega in the introductory citation group. Source-label names remain embedded in innerText; they do not establish substantive tool recommendations.",
"Native/browser activity contains grouped three-search and two-search controls. Exact query strings are not present in the supplied expanded activity snapshot.",
"Workspace-explored controls are visible, but their underlying contents and influence are unresolved.",
"Exact DOM href provenance applies to this record. It does not retroactively resolve omitted schemes/www in earlier native accessibility captures.",
"All reliability claims, sample-frequency advice, free-tool status, feature coverage, crawler and tracking claims remain platform output rather than independently verified facts.",
"Evidence-reference correction: the original -dom.json global-selector capture contains the user prompt, not the answer. Body evidence is the complete rendered-text file; the browser-activity DOM snapshot corroborates the article and all four hrefs. The failed capture is preserved and is non-supporting."
],
"rendered_body_evidence_file": "[Private capture retained locally.]",
"non_supporting_capture_files": [
{
"status": "failed_global_selector_capture",
"reason": "Captured the user prompt rather than the assistant answer. Original retained, but it does not support answer completeness or the citation href inventory."
}
],
"original_capture_answer_characters": 4126,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping; remove leading www from www.gomega.ai for normalized-domain comparison and retain the exact observed hostname/href. Other displayed hosts are base domains; no external public-suffix lookup used.",
"mapping": {
"shmeer.com": "shmeer.com",
"www.gomega.ai": "gomega.ai",
"geoaura.world": "geoaura.world",
"goodatmarketing.com": "goodatmarketing.com"
}
}
}No Google Sheets product is named: keep the generic spreadsheet method separate rather than inferring a branded tool.
X is an optional platform-access channel here, not a content-publication recommendation. Its role differs from X recommendations in other responses.
No instruction explicitly requires a fresh chat for every prompt; consistent incognito or same-account setup is not equivalent to disabling memory or personalization.
The full mention/recommendation/citation distinction is present in the logging fields; citation rate itself is not defined.
Only date and search state logging are explicitly requested, not full model-version logging.
No supporting publisher census is included in tool-set counts; all four source categories remain provisional/unresolved without reading the destinations.
S3-GROK-M1-R2 · Grok Expert · consumer interface · completed
Complete answer has no observable inline citation anchors despite an 85-source badge. Badge entries are not assigned as answer citations.
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs AI visibility tools (unspecified), AthenaHQ Essential, Bing Webmaster Tools, GetMentioned free AI visibility report, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher, Semrush AI visibility tools (unspecified), X, YouTube |
| Mentioned tools / services / surfaces | Ahrefs AI visibility tools (unspecified), AthenaHQ Essential, Bing Webmaster Tools, Claude, Gemini, GetMentioned free AI visibility report, Google AI Overviews, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher, Microsoft Copilot, Perplexity, Reddit, Semrush AI visibility tools (unspecified), X, YouTube |
| Cited domains · mapping complete | None observed within this eligible set |
| Observed inline cited domains · coverage qualified | None observed within this eligible set |
| Observed cited-source categories · provisional | None observed within this eligible set |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
There is no official “how often we recommend you” dashboard from OpenAI or xAI. Measurement is sampling-based. For a solo founder with zero GEO budget, the most reliable approach is a disciplined manual process plus free snapshot tools and outcome signals. 1. Systematic Manual Sampling (Your Core Reliable Method) This is the gold standard at zero cost. Answers vary between runs, so treat it like A/B testing. Write 10–15 real buyer prompts (never put your brand name in them). Examples: “Best [category] for a solo founder / small team on a budget” “[Competitor] alternatives” “How do I solve [specific pain your product fixes]” “Top [category] tools in 2026 for [use case]” Comparison or feature-specific versions Test cleanly: Use temporary/incognito chats or log out. Fresh chat for every prompt. Enable search/browse where available (especially ChatGPT). Run each prompt 3–5 times on both ChatGPT and Grok. Log everything in a Google Sheet: Date, prompt, engine Mentioned? (Y/N) Position (1st, listed, etc.) Sources cited Competitors named Reason given Calculate a simple visibility rate: percentage of runs where you appear. Track average position and which sources keep appearing. Re-run the same prompt set monthly (or weekly for your top 5). Consistency matters more than volume. 2. Free Snapshot Tools for Quick Baselines Use these for a fast first look or quarterly check, not as your only ongoing method: HubSpot AI Search / AEO Grader — Free, no (or minimal) account required. Runs prompts across ChatGPT, Perplexity, and Gemini and scores visibility, sentiment, and share of voice. GetMentioned free AI visibility report — No signup. Runs real buyer-style prompts and shows visibility score, competitors, and cited sources. Mangools AI Search Watcher (free plan) — 5 prompts, weekly updates, covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others. Useful for light ongoing tracking. AthenaHQ Essential — Free credits (around 300) across several major engines for prompt analysis and source insights. Limited free checks from Semrush or Ahrefs AI visibility tools when available. These give you a structured starting point without payment. 3. Outcome Signals (Often More Actionable Than Mentions) Mentions only matter if they drive awareness or signups. GA4 referrals: Create a custom channel group for AI assistants. Look for traffic from chatgpt.com, openai.com, grok.x.ai, x.ai, perplexity.ai, etc. Note that not all AI sessions pass a clean referrer. Google Search Console: Check generative AI / AI Overview impressions if available for your property. Bing Webmaster Tools: AI performance and citation reports (more relevant to Copilot, but useful context). Customer survey: On signup or in onboarding, ask “How did you first hear about us?” and include ChatGPT, Grok, “AI recommendation,” or “AI search.” This is the cleanest free signal of real impact. Practical Tips for Reliability Focus on non-branded discovery prompts. Branded queries only tell you the model knows you exist. Prioritize cited sources. For SaaS, third-party pages (comparison listicles, review sites, Reddit threads, competitor pages) often drive recommendations more than your own site. Knowing the sources is more useful than the mention rate alone. Expect high variance. One run means almost nothing; patterns across many runs matter. Grok leans more on social, X, YouTube, and community content than ChatGPT, so track those surfaces too if Grok is important. Re-test after you publish new comparison content, get mentioned on key third-party pages, or improve structured data — then measure the delta. Minimal Weekly/Monthly Workflow Maintain your fixed prompt list. Run the key prompts manually (or use a free tool’s 5-prompt limit). Update the sheet. Glance at GA4 AI referrals and any “AI” mentions in customer surveys. Note which sources keep appearing and whether you can influence them. This gives you a consistent, zero-budget measurement system that is more reliable than one-off checks or expensive tools. Start with the manual baseline this week — it takes 1–2 hours and tells you immediately where you stand.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M1-R2",
"slot_id": "S3-GROK-M1-R2",
"platform": "Grok",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 2,
"submitted_at": "2026-11-02T06:34:46.219Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:36:20.281Z",
"source_panel_declared_hits": 85,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-M1-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4135,
"public_answer_characters": 4135,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M1-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
No URLs recorded in this field. Completeness and source role are reported separately.
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete rendered 4,135-character answer and scoped DOM/activity capture. No external webpage queries, fact audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | Log everything in a Google Sheet:
| |
| HubSpot AEO Grader | HubSpot AI Search / AEO Grader | No | Yes | Yes | conditional | stated free | HubSpot AI Search / AEO Grader — Free, no (or minimal) account required. Runs prompts across ChatGPT, Perplexity, and Gemini and scores visibility, sentiment, and share of voice.
| |
| GetMentioned free AI visibility report | GetMentioned free AI visibility report | No | Yes | Yes | conditional | stated free | GetMentioned free AI visibility report — No signup. Runs real buyer-style prompts and shows visibility score, competitors, and cited sources.
| |
| Mangools AI Search Watcher | Mangools AI Search Watcher (free plan) | No | Yes | Yes | conditional | stated free | Mangools AI Search Watcher (free plan) — 5 prompts, weekly updates, covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others. Useful for light ongoing tracking.
| |
| AthenaHQ Essential | AthenaHQ Essential | No | Yes | Yes | conditional | mixed/limited tier | AthenaHQ Essential — Free credits (around 300) across several major engines for prompt analysis and source insights.
| |
| Semrush AI visibility tools (unspecified) | Semrush | No | Yes | Yes | conditional | mixed/limited tier | Limited free checks from Semrush or Ahrefs AI visibility tools when available.
| |
| Ahrefs AI visibility tools (unspecified) | Ahrefs AI visibility tools | No | Yes | Yes | conditional | mixed/limited tier | Limited free checks from Semrush or Ahrefs AI visibility tools when available.
| |
| Google Analytics 4 | GA4 | No | Yes | Yes | suggested | stated free | GA4 referrals: Create a custom channel group for AI assistants. Look for traffic from chatgpt.com, openai.com, grok.x.ai, x.ai, perplexity.ai, etc.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | conditional | stated free | Google Search Console: Check generative AI / AI Overview impressions if available for your property.
| |
| Bing Webmaster Tools | Bing Webmaster Tools | No | Yes | Yes | suggested | stated free | Bing Webmaster Tools: AI performance and citation reports (more relevant to Copilot, but useful context).
| |
| Reddit threads | No | Yes | No | discussed | unspecified | For SaaS, third-party pages (comparison listicles, review sites, Reddit threads, competitor pages) often drive recommendations more than your own site.
| ||
| X | X | No | Yes | Yes | conditional | unspecified | Grok leans more on social, X, YouTube, and community content than ChatGPT, so track those surfaces too if Grok is important.
| |
| YouTube | YouTube | No | Yes | Yes | conditional | unspecified | Grok leans more on social, X, YouTube, and community content than ChatGPT, so track those surfaces too if Grok is important.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | HubSpot AI Search / AEO Grader — Free, no (or minimal) account required. Runs prompts across ChatGPT, Perplexity, and Gemini and scores visibility, sentiment, and share of voice.
| |
| Gemini | Gemini | No | Yes | No | discussed | unspecified | HubSpot AI Search / AEO Grader — Free, no (or minimal) account required. Runs prompts across ChatGPT, Perplexity, and Gemini and scores visibility, sentiment, and share of voice.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | Mangools AI Search Watcher (free plan) — 5 prompts, weekly updates, covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others. Useful for light ongoing tracking.
| |
| Google AI Overviews | AI Overviews; AI Overview | No | Yes | No | discussed | unspecified | Mangools AI Search Watcher (free plan) — 5 prompts, weekly updates, covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others. Useful for light ongoing tracking.
| |
| Microsoft Copilot | Copilot | No | Yes | No | discussed | unspecified | Bing Webmaster Tools: AI performance and citation reports (more relevant to Copilot, but useful context).
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Visibility rate",
"definition_verbatim": "Calculate a simple visibility rate: percentage of runs where you appear.",
"notes": "Run-level appearance rate as defined by this answer; do not equate automatically with recommendation rate or another answer's prompt-level denominator."
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/surfaces with supplied targets and companies separate. Generic Semrush/Ahrefs tool-family names remain product-unspecified; no extra product aliases merged.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Tool-led solution",
"Conceptual explanation"
],
"evidence": "Systematic Manual Sampling (Your Core Reliable Method)",
"rationale": "The response emphasizes fixed unbranded prompts, repeated clean-session runs, logging and a recurring workflow; snapshots and outcome indicators are supplements."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Write 10–15 real buyer prompts (never put your brand name in them)."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt 3–5 times on both ChatGPT and Grok."
},
"fresh_sessions": {
"value": true,
"evidence": "Fresh chat for every prompt."
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "Enable search/browse where available (especially ChatGPT)."
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Sources cited"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Log everything in a Google Sheet:"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Calculate a simple visibility rate: percentage of runs where you appear."
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Expect high variance. One run means almost nothing; patterns across many runs matter."
},
"human_checking": {
"value": true,
"evidence": "Systematic Manual Sampling (Your Core Reliable Method)"
},
"downstream_conversions": {
"value": true,
"evidence": "Customer survey: On signup or in onboarding, ask “How did you first hear about us?”"
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "GA4 referrals: Create a custom channel group for AI assistants."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual sampling",
"evidence": "Run each prompt 3–5 times on both ChatGPT and Grok.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; exclude from discovered-tool overlap."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual sampling",
"evidence": "Run each prompt 3–5 times on both ChatGPT and Grok.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; exclude from discovered-tool overlap."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "There is no official “how often we recommend you” dashboard from OpenAI or xAI.",
"notes": "Company-level dashboard claim, not a named measurement product; not independently verified."
},
{
"entity": "xAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "There is no official “how often we recommend you” dashboard from OpenAI or xAI.",
"notes": "Company-level dashboard claim, not a named measurement product; not independently verified."
}
],
"generic_methods": [
"Unbranded buyer-prompt list",
"Fresh-chat repeated manual sampling",
"Spreadsheet logging of appearances, position, competitors and cited sources",
"Per-run visibility rate",
"Traffic/custom analytics channel checks",
"Customer acquisition surveys",
"Third-party source inspection",
"Post-change remeasurement of content and structured-data interventions"
],
"metric_definitions_observed": [
{
"label": "Visibility rate",
"definition_verbatim": "Calculate a simple visibility rate: percentage of runs where you appear.",
"notes": "Run-level appearance rate as defined by this answer; do not equate automatically with recommendation rate or another answer's prompt-level denominator."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4135,
"observed_inline_link_occurrences": 0,
"unique_DOM_answer_URLs": 0,
"observed_inline_cited_domains": 0,
"source_badge_hits": 85,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Scoped rendered-article inventory only: zero observable inline citation anchors. This is not a complete retrieval-panel inventory or proof that no sources were used.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"complete_rendered_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Native activity shows two grouped four-search controls; actual search is observed independently of the source badge.",
"The 85-source badge is not assigned as 85 answer citations. Retrieval/source-panel contents were not collected here.",
"Plain-text hostnames in GA4 examples are referral filters, not evidence URLs or own-content citations.",
"No inline cited domains is an observed feature of the scoped answer, not absence of retrieval or absence of supporting evidence.",
"All free limits, credits, signup requirements, feature coverage, reliability claims and comparison claims remain model output, not verified facts."
],
"original_capture_answer_characters": 4135,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "No inline citation anchors observed in the scoped complete article; there are no observed cited domains to normalize. Plain-text referral hostnames are not citations.",
"mapping": {}
}
}Zero cited-domain observations can enter explicitly scoped inline-domain comparisons but cannot be generalized to zero retrieval. Two empty observed inline-domain sets remain N/A for Jaccard, not perfect stability.
Do not merge generic Semrush/Ahrefs families into any specific checker or Brand Radar. Preserve their unresolved product identity.
X and YouTube are explicitly suggested for monitoring; Reddit is an explanatory source example without a specific active instruction.
The answer tracks appearances and cited sources, but does not explicitly separate an active-recommendation field or rate.
Search enablement is advised, but logging model versions or search settings is not explicitly required in the listed fields.
S3-GROK-M1-R3 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Bing Webmaster Tools, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher |
| Mentioned tools / services / surfaces | Bing AI (unspecified surface), Bing Webmaster Tools, Gemini, Google AI Mode, Google AI Overviews, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher, Microsoft Copilot, Perplexity |
| Cited domains · mapping complete | explaingeo.com, geoaura.world, gomega.ai, goodatmarketing.com, hubspot.com, shmeer.com, snezzi.com |
| Observed inline cited domains · coverage qualified | explaingeo.com, geoaura.world, gomega.ai, goodatmarketing.com, hubspot.com, shmeer.com, snezzi.com |
| Observed cited-source categories · provisional | Other / unresolved, Tool/product documentation or sales page |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I reliably measure whether ChatGPT and Grok recommend my product to potential buyers?
Captured answer body
You can measure this reliably for free with a fixed set of buyer-intent prompts, repeated manual tests on ChatGPT and Grok, and a simple spreadsheet. This is the standard low-cost approach used by solo founders and small teams for GEO (Generative Engine Optimization).Shmeer Paid AI visibility tools exist, but you do not need them for a usable signal. 1. Build a fixed prompt set (15–30 prompts) Create questions a real potential buyer would ask without naming your product. Keep the wording identical every time so you can compare results over weeks. Include a mix of: Category / “best of” queries (“best [category] for [specific use case]”, “top tools for [problem]”) Problem-solving queries (“how do I [problem your product solves]”) Comparison / alternative queries (“[competitor] alternatives”, “cheapest [category] that does X”) Evaluation queries (“what to look for in a [category] tool”) Start with 15–20 high-intent ones. Add more later if needed. These become your permanent test panel.Gomega 2. Run them manually and log everything For each prompt: Run it 3–5 times on ChatGPT and on Grok (answers vary a lot; single runs are noisy).Geoaura Prefer modes that can search the web when available, since product recommendations often rely on current information rather than pure training data. Record in a Google Sheet (or any free spreadsheet): Date Engine Prompt Mentioned? (Y/N) Position in list Cited your site? Competitors mentioned Sentiment / accuracy notes Sources cited Key metrics to calculate from the log: Mention rate — % of runs where your product appears at all Share of voice — your mentions vs. competitors Average position — where you appear when mentioned Citation rate — how often your domain is linked or referenced as a source Accuracy / sentiment — whether the description is correct and positive Do this weekly or every two weeks. Look at trends, not any single answer. One-off results fluctuate heavily.Snezzi 3. Use free one-time or limited tools for a baseline HubSpot AEO / AI Search Grader — free, no account required for the basic check. Enter your brand + category and it scores how ChatGPT, Perplexity, and Gemini characterize you (sentiment, presence, share of voice, etc.). Good for a quick snapshot.Hubspot Mangools AI Search Watcher — free plan includes 5 prompts monitored weekly across multiple engines, including ChatGPT and Grok. Useful for a small recurring automated check while you do the rest manually. Free page-level GEO checkers (various sites) — these score how “citable” a page is (structure, schema, crawler access, etc.). They do not replace prompt testing but help explain why you might or might not appear. 4. Check actual traffic impact (lagging indicator) In Google Analytics 4: Go to Reports → Acquisition → Traffic acquisition. Look for the AI Assistant channel (added in 2026) or filter by session source for chatgpt.com (and similar for other tools). Note: many AI visits arrive without a clear referrer and show as Direct. OpenAI has documented that ChatGPT can add utm_source=chatgpt.com to links.Goodatmarketing Also check: Google Search Console for AI Overviews / AI Mode impressions (when relevant). Bing Webmaster Tools AI Performance report for citations in Copilot/Bing AI. This shows whether mentions are turning into visits, not just whether the models know you exist. Practical tips for a solo founder Keep the prompt list short enough that a full cycle takes 1–2 hours. Consistency beats volume. Test both ChatGPT and Grok the same way so you can compare engines. When you get mentioned, note how (brief list item, detailed recommendation, with link or without). That tells you what to improve. Technical readiness still matters: make sure AI crawlers can reach your site, important content is in the HTML (not only behind JS), and key pages have clear structure + schema. Free GEO checkers flag the biggest blockers. Expect noise. The reliable signal is directional change over 4–8 weeks of consistent testing, not day-to-day jumps. This method is what most practical GEO guides recommend when you have zero budget: fixed buyer prompts + repeated manual logging + free graders + GA4/Search Console for confirmation.Explaingeo Start with your 15–20 prompts this week, establish a baseline, then re-run on a schedule. You’ll quickly see whether ChatGPT and Grok surface your product for real buyer questions and whether that improves as you update content or get more mentions elsewhere.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M1-R3",
"slot_id": "S3-GROK-M1-R3",
"platform": "Grok",
"prompt_id": "M1",
"intent": "M",
"wording": 1,
"repeat": 3,
"submitted_at": "2026-11-02T06:49:23.031Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:51:00.297Z",
"source_panel_declared_hits": 65,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-M1-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4493,
"public_answer_characters": 4493,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M1-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Shmeer | https://shmeer.com/blog/how-to-measure-geo | shmeer.com | Inline answer link | Rendered DOM href |
| Gomega | https://www.gomega.ai/blog/how-to-measure-geo-performance | www.gomega.ai | Inline answer link | Rendered DOM href |
| Geoaura | https://geoaura.world/blog/how-to-measure-geo-visibility | geoaura.world | Inline answer link | Rendered DOM href |
| Snezzi | https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/ | snezzi.com | Inline answer link | Rendered DOM href |
| Hubspot | https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool | www.hubspot.com | Inline answer link | Rendered DOM href |
| Goodatmarketing | https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | goodatmarketing.com | Inline answer link | Rendered DOM href |
| Explaingeo | https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/ | www.explaingeo.com | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 4,493-character rendered body, seven exact scoped DOM hrefs and existing browser activity. No external webpage queries, factual audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | Record in a Google Sheet (or any free spreadsheet):
| |
| HubSpot AEO Grader | HubSpot AEO / AI Search Grader | Yes | Yes | Yes | conditional | stated free | HubSpot AEO / AI Search Grader — free, no account required for the basic check. Enter your brand + category and it scores how ChatGPT, Perplexity, and Gemini characterize you (sentiment, presence, share of voice, etc.). Good for a quick snapshot.
| https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool |
| Mangools AI Search Watcher | Mangools AI Search Watcher | No | Yes | Yes | conditional | mixed/limited tier | Mangools AI Search Watcher — free plan includes 5 prompts monitored weekly across multiple engines, including ChatGPT and Grok. Useful for a small recurring automated check while you do the rest manually.
| |
| Google Analytics 4 | Google Analytics 4; GA4 | No | Yes | Yes | suggested | unspecified | In Google Analytics 4:
Go to Reports → Acquisition → Traffic acquisition.
| |
| Google Search Console | Google Search Console; Search Console | No | Yes | Yes | conditional | unspecified | Google Search Console for AI Overviews / AI Mode impressions (when relevant).
| |
| Bing Webmaster Tools | Bing Webmaster Tools | No | Yes | Yes | suggested | unspecified | Bing Webmaster Tools AI Performance report for citations in Copilot/Bing AI.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | HubSpot AEO / AI Search Grader — free, no account required for the basic check. Enter your brand + category and it scores how ChatGPT, Perplexity, and Gemini characterize you (sentiment, presence, share of voice, etc.). Good for a quick snapshot.
| |
| Gemini | Gemini | No | Yes | No | discussed | unspecified | HubSpot AEO / AI Search Grader — free, no account required for the basic check. Enter your brand + category and it scores how ChatGPT, Perplexity, and Gemini characterize you (sentiment, presence, share of voice, etc.). Good for a quick snapshot.
| |
| Google AI Overviews | AI Overviews | No | Yes | No | discussed | unspecified | Google Search Console for AI Overviews / AI Mode impressions (when relevant).
| |
| Google AI Mode | AI Mode | No | Yes | No | discussed | unspecified | Google Search Console for AI Overviews / AI Mode impressions (when relevant).
| |
| Microsoft Copilot | Copilot | No | Yes | No | discussed | unspecified | Bing Webmaster Tools AI Performance report for citations in Copilot/Bing AI.
| |
| Bing AI (unspecified surface) | Bing AI | No | Yes | No | discussed | unspecified | Bing Webmaster Tools AI Performance report for citations in Copilot/Bing AI.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://shmeer.com/blog/how-to-measure-geo | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
| https://www.gomega.ai/blog/how-to-measure-geo-performance | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
| https://geoaura.world/blog/how-to-measure-geo-visibility | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
| https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/ | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
| https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool | Tool/product documentation or sales page | None | provisional_from_returned_label_URL_and_answer_role_only | Own-brand AEO-grader product path linked alongside an active snapshot recommendation. | Own-content link to a conditional product recommendation |
| https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
| https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/ | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_label_URL_and_answer_role_only | Measurement, analytics how-to or free-tool guide path suggests practical explanatory content; authorship, first-person research, editorial and commercial roles are not independently established. | Supporting citation; label-only publisher not actively recommended |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Mention rate — % of runs where your product appears at all"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice — your mentions vs. competitors",
"notes": "No exact formula or denominator supplied."
},
{
"label": "Average position",
"definition_verbatim": "Average position — where you appear when mentioned"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate — how often your domain is linked or referenced as a source"
},
{
"label": "Accuracy / sentiment",
"definition_verbatim": "Accuracy / sentiment — whether the description is correct and positive"
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/features with supplied target platforms and company attribution separate. Generic page-level checkers remain methods/tool categories; citation-label-only publishers are not added as tool recommendations.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Tool-led solution",
"Conceptual explanation"
],
"evidence": "fixed buyer prompts + repeated manual logging + free graders + GA4/Search Console for confirmation.",
"rationale": "Prompt panel, repeated manual logging, metrics and cadence form the core. Free graders/monitoring and lagging analytics supplement it."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Create questions a real potential buyer would ask without naming your product."
},
"repeated_runs": {
"value": true,
"evidence": "Run it 3–5 times on ChatGPT and on Grok (answers vary a lot; single runs are noisy)."
},
"fresh_sessions": {
"value": false,
"evidence": null
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "Prefer modes that can search the web when available"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "When you get mentioned, note how (brief list item, detailed recommendation, with link or without)."
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Citation rate — how often your domain is linked or referenced as a source"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Mentioned? (Y/N)\tPosition in list\tCited your site?"
},
"raw_answer_inspection": {
"value": true,
"evidence": "When you get mentioned, note how (brief list item, detailed recommendation, with link or without)."
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Mention rate — % of runs where your product appears at all"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Expect noise. The reliable signal is directional change over 4–8 weeks of consistent testing, not day-to-day jumps."
},
"human_checking": {
"value": true,
"evidence": "Run them manually and log everything"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "This shows whether mentions are turning into visits, not just whether the models know you exist."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual sampling",
"evidence": "Run it 3–5 times on ChatGPT and on Grok (answers vary a lot; single runs are noisy).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target platform supplied by question, not a newly discovered tool."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual sampling",
"evidence": "Run it 3–5 times on ChatGPT and on Grok (answers vary a lot; single runs are noisy).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target platform supplied by question, not a newly discovered tool."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "OpenAI has documented that ChatGPT can add utm_source=chatgpt.com to links.",
"source_urls": [
"https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/"
],
"notes": "Company-level attribution in a tracking claim, with a third-party citation rather than a captured OpenAI document. Claim not verified; outside tool counts."
}
],
"generic_methods": [
"Fixed unbranded buyer-intent prompt panel",
"Repeated manual checks on both engines",
"Named or generic free spreadsheet logging",
"Mention rate, competitor share of voice, position, citation rate and accuracy/sentiment summaries",
"Generic page-level GEO readiness checkers",
"Traffic/referral diagnostics",
"Crawler access, rendered HTML and structured-data checks"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Mention rate — % of runs where your product appears at all"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice — your mentions vs. competitors",
"notes": "No exact formula or denominator supplied."
},
{
"label": "Average position",
"definition_verbatim": "Average position — where you appear when mentioned"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate — how often your domain is linked or referenced as a source"
},
{
"label": "Accuracy / sentiment",
"definition_verbatim": "Accuracy / sentiment — whether the description is correct and positive"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4493,
"observed_inline_link_occurrences": 7,
"unique_DOM_answer_URLs": 7,
"unique_cited_domains": 7,
"source_badge_hits": 65,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Seven inline hrefs from the complete rendered article, not an inventory of 65 retrieval hits.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Activity shows two grouped three-search controls. The expanded exact query strings were not captured.",
"Citation-label names for Shmeer, Gomega, Geoaura, Snezzi, Goodatmarketing and Explaingeo are supporting references, not substantive tool-use recommendations.",
"The HubSpot citation is attached to a substantive grader recommendation and qualifies as an own-content tool citation.",
"The 65-source badge is not a unique inline citation count.",
"Workspace-explored activity labels do not establish their contents or influence.",
"Prompt counts, time estimates, free/signup claims, analytics-report capabilities, tracking behavior and reliability assertions remain unverified model output."
],
"original_capture_answer_characters": 4493,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual leading-www normalization retains exact DOM hrefs/hosts. No external public-suffix lookup used.",
"mapping": {
"shmeer.com": "shmeer.com",
"www.gomega.ai": "gomega.ai",
"geoaura.world": "geoaura.world",
"snezzi.com": "snezzi.com",
"www.hubspot.com": "hubspot.com",
"goodatmarketing.com": "goodatmarketing.com",
"www.explaingeo.com": "explaingeo.com"
}
}
}Do not infer the X platform from the placeholder X in example prompts; this answer does not substantively mention the X service.
Generic page-level GEO checkers are suggested without named products; do not invent tool identities from supporting publishers.
Citation/mention/recommendation modes are discussed, but no active-recommendation-rate formula or binary recommendation log column is provided.
Model versions, search-state logging and fresh-session controls are not explicitly specified in the workflow; preferring web-search modes is narrower.
Analytics capabilities and channel/report availability are captured as advice, not verified product specifications.
S3-CHATGPT-M2-R1 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | Reddit, X |
| Cited domains · mapping complete | openai.com, x.com |
| Observed inline cited domains · coverage qualified | openai.com, x.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
## A free weekly “AI recommendation audit” Use the free versions of ChatGPT and Grok, a spreadsheet, and roughly 30 minutes per run. ChatGPT web search is available on the Free plan, while Grok is also free to start and can search both the web and public X posts. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) ### 1. Build a 10-prompt test set Choose prompts representing how buyers actually shop—**without mentioning your product**. | Intent | Prompt template | |---|---| | Category | “What are the best `[category]` tools for `[ICP]`?” | | Problem | “I need to `[job]` without `[pain]`. What tools should I consider?” | | Constraints | “Recommend five `[category]` tools for `[ICP]` under `$X/month`.” | | Alternative | “What are the best alternatives to `[market leader]` for `[use case]`?” | | Shortlist | “Which tools should a `[role]` shortlist for `[outcome]`?” | | Comparison | “Compare the leading tools for `[job]`, including price, strengths and weaknesses.” | Create variants for your two or three most important personas, use cases, or geographies. Keep the same prompts each week. ### 2. Run two versions of every prompt **A. Default recommendation** > What are the five best `[category]` tools for `[ICP/use case]`? Rank them and briefly explain each choice. This tests what a normal user may experience. **B. Web-grounded recommendation** > Search the web for current information. Recommend five `[category]` tools for `[ICP/use case]`. Include pricing, explain the ranking, and cite the sources used. This reveals whether your site—or third-party pages mentioning you—enters the retrieval layer. ChatGPT search responses can include clickable citations, although OpenAI warns that citations can still be incomplete or incorrect. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) ### 3. Keep every run clean For **ChatGPT**: - Start a new **Temporary Chat** - Choose **Unpersonalized** - Don’t use a conversation in which you previously discussed your company - Don’t mention your brand until after the unaided test OpenAI states that unpersonalized Temporary Chats do not use memories, custom instructions, or plugins for personalization. ([help.openai.com](https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt)) For **Grok**: - Start a fresh conversation - Disable Grok personalization where available - Ideally use a dedicated testing account with no company-related posting or chat history On X, Grok personalization can incorporate profile, posts, engagement, interests and previous Grok interactions, so testing through your founder account can skew results. ([help.x.com](https://help.x.com/en/using-x/about-grok)) Incognito mode alone is insufficient if you sign into a personalized account. ### 4. Repeat each test three times Model answers vary. Run each prompt: - Three times on ChatGPT - Three times on Grok - In separate fresh conversations - On the same day, if possible If free-tier limits make that excessive, rotate the set: - Week 1: five prompts - Week 2: the other five - Repeat monthly ### 5. Record results in a spreadsheet Use these columns: ```text Date Platform Prompt ID Prompt Default or web-grounded Product mentioned? Y/N Rank Description accurate? Y/N Positive / neutral / negative Our website cited? Y/N Third-party page citing us Competitors mentioned Competitor source domains Notes Screenshot/chat link ``` Track four metrics separately: 1. **Mention rate:** percentage of runs mentioning you 2. **Top-three rate:** percentage ranking you first through third 3. **Citation rate:** percentage citing your website or a page featuring you 4. **Accuracy rate:** percentage describing your positioning, pricing and features correctly Example: ```text Mention rate = mentions / total runs Top-3 rate = top-3 appearances / total runs Owned citation rate = runs citing your domain / total runs ``` Don’t combine these into one score initially. “Frequently mentioned but inaccurately described” is a different problem from “rarely mentioned but ranked highly when found.” ### 6. Add a separate branded diagnostic After the blind tests, open another clean conversation and ask: > What is `[Product]`, who is it best for, what are its main alternatives, and why might someone choose or avoid it? Search the web and cite every factual claim. This does **not** measure recommendation visibility. It diagnoses: - Whether the model can identify your company - Category confusion - Outdated pricing - Incorrect features - Which pages it treats as authoritative - Which competitors it associates with you ### 7. Turn the findings into a content queue When you’re absent, inspect what the models cited instead. Look for repeat patterns: - “Best tools” articles - Software directories - Reddit or community discussions - Competitor comparison pages - Integration directories - Documentation - Pricing pages - Public X discussions, especially for Grok Then make one measurable improvement per week: - Clarify your homepage category and ideal customer - Publish a detailed pricing page - Create a page for each important use case - Publish honest “X alternative” and “X vs. Y” pages - Add comparison tables with concrete facts - Improve documentation and integration pages - Correct inaccurate third-party listings - Earn inclusion in the specific independent pages repeatedly cited - Encourage genuine customers to publish detailed reviews and use-case discussions Avoid manufacturing reviews, fake community posts, or mass-produced “best tools” pages. The audit should tell you where real, verifiable product information is missing. ## Recommended cadence - **Weekly:** Run five highest-value prompts - **Monthly:** Run the complete matrix three times per platform - **After major launches:** Run the branded accuracy diagnostic - **Quarterly:** Replace low-value prompts using language from sales calls, support tickets and search data The key is to treat this as a **longitudinal visibility test**, not a definitive ranking checker. Results vary by model, search mode, personalization, location and time; your trend over repeated controlled runs is more meaningful than any single answer.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21303,
"input_tokens_details": {
"cache_write_tokens": 34,
"cached_tokens": 4316
},
"output_tokens": 2319,
"output_tokens_details": {
"reasoning_tokens": 963
},
"total_tokens": 23622
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1360884,
"cost_estimate_detail": {
"estimated_usd": 0.1360884,
"token_cost_usd": 0.1160884,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M2-R1",
"slot_id": "S3-CHATGPT-M2-R1",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 1,
"submitted_at": "2026-11-02T09:22:53.546659+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:23:28.342955+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21303,
"input_tokens_details": {
"cache_write_tokens": 34,
"cached_tokens": 4316
},
"output_tokens": 2319,
"output_tokens_details": {
"reasoning_tokens": 963
},
"total_tokens": 23622
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1360884,
"token_cost_usd": 0.1160884,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read complete weekly recommendation-audit workflow through final longitudinal-test paragraph. All four explicit source occurrences match annotations; complete model/effort/search metadata. No truncation or tool/output cap; advice about free plans, time cost and product behavior is collected output, not verified.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:24:23.671052+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations official help",
"site:help.openai.com ChatGPT memory temporary chat official",
"site:help.x.com Grok search sources official",
"site:docs.x.ai Grok web search citations official"
],
"query": "site:help.openai.com ChatGPT search citations official help",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9275245-chatgpt-free-tier-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-how-do-i-search-my-chat-history-in-chatgpt%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.midi"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.avif"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu%2525252525252523.iso"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com \"Searching the web with ChatGPT\" free users official",
"site:help.openai.com ChatGPT search available free users official",
"site:docs.x.ai/grok temporary chat memory personalization Grok official",
"site:help.x.com Grok personalization chat history memory official"
],
"query": "site:help.openai.com \"Searching the web with ChatGPT\" free users official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-finding-your-chats-projects-and-files-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/6783457-what-is-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9275245-chatgpt-free-tier-faq"
},
{
"type": "url",
"url": "https://docs.x.ai/build/modes-and-commands"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/skills-routines-and-automations"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/bots"
},
{
"type": "url",
"url": "https://docs.x.ai/build/settings"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M2-R1.json",
"request_file": "requests/S3-OPENAIAPI-M2-R1.json",
"response_file": "responses/S3-OPENAIAPI-M2-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 6345,
"public_answer_characters": 6273,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M2-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Temporary chat in ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | API body url_citation annotation | |
| About Grok | X Help Center | https://help.x.com/en/using-x/about-grok | help.x.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 6,345-character API answer, four citation annotations and corresponding explicit Markdown URLs. No external page access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M2-R1.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| No | Yes | No | discussed as a possible cited-source pattern | unspecified | Reddit or community discussions
| |||
| X | X; public X posts; Public X discussions | Yes | Yes | No | discussed as a Grok retrieval/personalization context and possible cited-source pattern | unspecified | Public X discussions, especially for Grok
| https://help.x.com/en/using-x/about-grok |
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned Help Center title and official help.openai.com/help.x.com URL identify platform help. Page content, free availability and personalization claims were not independently checked. | Supporting documentation citation, not a recommendation to adopt an additional measurement product | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned Help Center title and official help.openai.com/help.x.com URL identify platform help. Page content, free availability and personalization claims were not independently checked. | Supporting documentation citation, not a recommendation to adopt an additional measurement product | |
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned Help Center title and official help.openai.com/help.x.com URL identify platform help. Page content, free availability and personalization claims were not independently checked. | Supporting documentation citation, not a recommendation to adopt an additional measurement product | |
| https://help.x.com/en/using-x/about-grok | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned Help Center title and official help.openai.com/help.x.com URL identify platform help. Page content, free availability and personalization claims were not independently checked. | Supporting documentation citation, not a recommendation to adopt an additional measurement product |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.x.com/en/using-x/about-grok | help.x.com | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = mentions / total runs"
},
{
"label": "Top-3 rate",
"definition_verbatim": "Top-3 rate = top-3 appearances / total runs"
},
{
"label": "Owned citation rate",
"definition_verbatim": "Owned citation rate = runs citing your domain / total runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "percentage describing your positioning, pricing and features correctly",
"notes": "Named with a verbal percentage definition; a separate numeric formula is not supplied."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1360884,
"token_cost_usd": 0.1160884,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21303,
"input_tokens_details": {
"cache_write_tokens": 34,
"cached_tokens": 4316
},
"output_tokens": 2319,
"output_tokens_details": {
"reasoning_tokens": 963
},
"total_tokens": 23622
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2319,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named services/channels; generic spreadsheet, supplied platforms and publisher/company context are separated. Potential cited-source examples are not automatically recommendations to use or publish on those channels.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Score validation or audit checklist",
"Conceptual explanation"
],
"evidence": "A free weekly “AI recommendation audit”",
"rationale": "Seven steps define buyer prompts, default/search conditions, clean sessions, repetitions, spreadsheet metrics, a branded diagnostic and a content queue. No named third-party measurement tool is recommended."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Choose prompts representing how buyers actually shop—**without mentioning your product**."
},
"repeated_runs": {
"value": true,
"evidence": "Three times on ChatGPT"
},
"fresh_sessions": {
"value": true,
"evidence": "In separate fresh conversations"
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Default or web-grounded"
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Owned citation rate = runs citing your domain / total runs"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Product mentioned? Y/N"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Description accurate? Y/N"
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Screenshot/chat link"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Mention rate = mentions / total runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": true,
"evidence": "Don’t combine these into one score initially."
},
"uncertainty_variance": {
"value": true,
"evidence": "Results vary by model, search mode, personalization, location and time; your trend over repeated controlled runs is more meaningful than any single answer."
},
"human_checking": {
"value": true,
"evidence": "When you’re absent, inspect what the models cited instead."
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": false,
"evidence": null
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "This does **not** measure recommendation visibility. It diagnoses:"
},
"positive_negative_stance_recording": {
"value": true,
"evidence": "Positive / neutral / negative"
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual tests",
"free_budget_fit": "stated free",
"evidence": "Use the free versions of ChatGPT and Grok, a spreadsheet, and roughly 30 minutes per run.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform, excluded from discovery/tool overlap. Official OpenAI help or X Help Center about Grok is cited for platform/search/personalization behavior; source claims were not independently verified. Specific Temporary Chat/search advice is retained within this supplied-platform record rather than double-counted as a newly discovered tool."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual tests",
"free_budget_fit": "stated free",
"evidence": "Use the free versions of ChatGPT and Grok, a spreadsheet, and roughly 30 minutes per run.",
"source_urls": [
"https://help.x.com/en/using-x/about-grok"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform, excluded from discovery/tool overlap. Official OpenAI help or X Help Center about Grok is cited for platform/search/personalization behavior; source claims were not independently verified. Specific Temporary Chat/search advice is retained within this supplied-platform record rather than double-counted as a newly discovered tool."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI states that unpersonalized Temporary Chats do not use memories, custom instructions, or plugins for personalization.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
],
"notes": "Provider attribution and documentation publisher, not an additional tool recommendation. The quoted behavior claim is not independently verified."
}
],
"generic_methods": [
"Unnamed spreadsheet tracker; no named spreadsheet brand",
"Fixed unbranded buyer-intent prompt matrix",
"Separate default and web-grounded tests",
"Clean sessions and repeated sampling",
"Separate mention, top-three, owned-citation and accuracy metrics",
"Standalone branded diagnostic after blind tests",
"Source inspection and content improvement queue",
"Weekly/monthly longitudinal audit"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Mention rate = mentions / total runs"
},
{
"label": "Top-3 rate",
"definition_verbatim": "Top-3 rate = top-3 appearances / total runs"
},
{
"label": "Owned citation rate",
"definition_verbatim": "Owned citation rate = runs citing your domain / total runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "percentage describing your positioning, pricing and features correctly",
"notes": "Named with a verbal percentage definition; a separate numeric formula is not supplied."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 6273,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 4,
"explicit_body_URL_occurrences": 4,
"unique_answer_URLs": 4,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Four citation annotations match four explicit Markdown URLs; the two representations are not eight citations. Retrieved action.sources entries remain a distinct search inventory.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M2-R1.json",
"response_file": "responses/S3-OPENAIAPI-M2-R1.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"API answer is not a consumer ChatGPT Search UI observation.",
"Four separate citation URLs map to two normalized publisher domains; same article IDs with different returned slugs are retained as separate observed URLs, without asserting their destination equivalence.",
"All provider capability, free-plan, time-cost, personalization and recommended metric claims remain platform output.",
"No named spreadsheet brand is inferred from the generic spreadsheet recommendation."
],
"original_capture_answer_characters": 6345,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of official help subdomains to openai.com and x.com; exact URLs/hostnames retained. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"help.x.com": "x.com"
}
}
}C/M/R are separate and non-exclusive. X is cited and mentioned but is not actively recommended as an additional measurement tool.
Reddit and X are potential source-pattern examples in a content queue; no direct publish/use instruction for those platforms is added.
No claims about GEO effectiveness or reliability of a real visibility vendor follow from this advice answer.
Interpret wording sensitivity separately from within-cell repetition and from the API-versus-consumer-interface difference.
S3-CHATGPT-M2-R2 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Google Analytics, X |
| Mentioned tools / services / surfaces | Google Analytics, X |
| Cited domains · mapping complete | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | openai.com, x.ai |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
A good **$0 workflow** is a small monthly “AI recommendation audit” using ChatGPT, Grok, and a spreadsheet. ## 1. Build a fixed prompt bank Start with **12–20 prompts** based on how buyers actually shop. Never mention your product’s name in these discovery prompts. | Prompt type | Template | |---|---| | Category | “What are the best `[category]` tools for `[customer type]`?” | | Use case | “I need to `[job]`. Which tools should I consider?” | | Constraint | “Recommend affordable `[category]` software for a team of `[size]`.” | | Alternative | “What are good alternatives to `[major competitor]`?” | | Comparison | “Compare the leading tools for `[use case]`.” | | Persona | “What would you recommend to a `[role]` who needs `[outcome]`?” | Include qualifiers that genuinely differentiate you: bootstrapped, privacy-friendly, self-hosted, no-code, developer-focused, Shopify-specific, under $50/month, and so forth. ## 2. Test two lanes Run each prompt two ways: ### Lane A: Natural recommendation Use the prompt exactly as a buyer might: > What are the best customer-feedback tools for an early-stage B2B SaaS? This tests whether the model already associates your product with the category. ### Lane B: Web-researched recommendation Append: > Search the web, recommend five options, explain why each fits, and cite your sources. This tests whether your current web presence is discoverable. ChatGPT Search can be explicitly selected and is available on the free tier; Grok’s free plan includes limited real-time web and X search. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) Don’t ask, “Should you recommend **MyProduct**?” That measures the model’s ability to agree with you, not organic visibility. ## 3. Minimize personalization For ChatGPT, use an **Unpersonalized Temporary Chat** for every test. This prevents memories and custom instructions from influencing the result. ([help.openai.com](https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt)) For Grok: - Start a fresh chat for each prompt. - Remove any custom instructions or personalization available in your settings. - Don’t discuss your brand in that account immediately before testing. - Keep the account, model, location, and test conditions consistent between audits. Also record the date and model shown in the interface. Results can change when models or search systems change. ## 4. Run each prompt twice LLM answers are variable. For a manageable free audit: - 12 prompts - 2 platforms - 2 lanes - 2 repetitions That’s **96 answers per audit**. If free limits make that impractical, use one repetition monthly and run the second repetition only for your six highest-value prompts. Run everything within a short window—ideally one or two days—so the results are reasonably comparable. ## 5. Track results in a spreadsheet Use these columns: ```text Date Platform Model/mode Prompt ID Prompt Natural or web-researched Run number Brand mentioned? Y/N Position Top 3? Y/N Recommended positively? Y/N Own website cited? Y/N Third-party source cited Description accurate? Y/N Competitors mentioned Notes Screenshot/chat URL ``` For `Position`, count only the actual recommendations: - `1–5`: listed position - `6`: mentioned outside the main list - blank: absent Save the answer or a screenshot. URLs alone can become less useful if interfaces or responses change. ## 6. Calculate four simple metrics ### Recommendation visibility ```text Answers mentioning you ÷ total tested answers ``` ### Top-three rate ```text Answers placing you in the top 3 ÷ total answers ``` ### Researched visibility Calculate visibility separately for web-researched prompts. This tells you whether the problem is current web discoverability rather than general model awareness. ### Accuracy rate ```text Accurate descriptions ÷ answers mentioning you ``` A mention with the wrong pricing, positioning, or features is not necessarily a win. An example dashboard: | Metric | ChatGPT | Grok | |---|---:|---:| | Natural visibility | 17% | 25% | | Researched visibility | 42% | 33% | | Top-three rate | 13% | 8% | | Accuracy | 80% | 50% | | Own-site citation rate | 20% | 10% | ## 7. Diagnose *why* you are missing After the blind test, use a separate diagnostic prompt: > You did not include `[product]`. Search the web and evaluate whether it qualifies for this recommendation. If not, explain the specific evidence or positioning that is missing. Then ask: > Which sources most influenced your recommendations? Treat the explanation as a hypothesis, not an authoritative account of the model’s internal ranking process. Validate it against the actual citations. Look for recurring gaps: - Unclear category positioning - No dedicated use-case pages - Weak comparison or alternatives pages - Pricing unavailable publicly - Few independent reviews - Outdated product descriptions - Missing evidence for important claims - Site blocked from crawling - Competitors described more concretely ## 8. Check the technical basics For ChatGPT web visibility, confirm that: - `robots.txt` does not block `OAI-SearchBot` - Your CDN or firewall does not block its published traffic - Important product, pricing, comparison, and documentation pages are publicly accessible - Pages do not require JavaScript interactions or login to reveal essential facts Allowing access makes pages eligible; it does **not** guarantee placement. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) Also monitor analytics for ChatGPT referrals. OpenAI states that referral URLs include `utm_source=chatgpt.com`, making them identifiable in tools such as Google Analytics. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results)) For Grok, include a few X-centric prompts because its search capabilities can use both real-time web and X information. ([docs.x.ai](https://docs.x.ai/developers/tools/web-search)) ## 9. Repeat monthly—not daily Monthly is usually enough for a bootstrapped product: 1. Run the unchanged core prompt bank. 2. Add at most one or two new prompts. 3. Compare visibility and accuracy with the prior month. 4. Note major website, review, launch, or positioning changes. 5. Investigate only meaningful trends across multiple runs. Keep **80% of prompts unchanged**. Otherwise, you won’t know whether visibility improved or you merely changed the test. ## Recommended starter version If you want the leanest possible version: - Six buyer-intent prompts - ChatGPT and Grok - Natural and web-researched lanes - One run each - Monthly - Track mention, position, citation, and accuracy That’s only **24 answers per month**, takes roughly 30–45 minutes, and gives you a useful directional benchmark. It is not an official “AI ranking”—personalization, location, model updates, search results, and response randomness all matter—but it will reveal whether your product is becoming more or less visible over time.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 20861,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2433,
"output_tokens_details": {
"reasoning_tokens": 861
},
"total_tokens": 23294
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.136444,
"cost_estimate_detail": {
"estimated_usd": 0.136444,
"token_cost_usd": 0.116444,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M2-R2",
"slot_id": "S3-CHATGPT-M2-R2",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 2,
"submitted_at": "2026-11-02T09:28:30.430693+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:29:09.672856+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 20861,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2433,
"output_tokens_details": {
"reasoning_tokens": 861
},
"total_tokens": 23294
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.136444,
"token_cost_usd": 0.116444,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read all nine workflow sections and lean starter version through coherent final caveat. Five citation occurrences match annotations; four unique source URLs retained verbatim, including tracking and encoded FAQ strings. Model/effort and two searches match; no limits reached. Dashboard percentages are illustrative. Recommendation visibility is defined with mention-count numerator and is not treated as a validated recommendation metric.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:29:27.035936+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations recommendations official",
"site:help.openai.com ChatGPT memory personalization search official",
"site:x.ai Grok web search citations official",
"site:docs.x.ai Grok API web search official"
],
"query": "site:help.openai.com ChatGPT search citations recommendations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10408842-google-app-data-controls-faq"
},
{
"type": "url",
"url": "https://x.ai/pricing"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%23.zst"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9275245-chatgpt-free-tier-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-how-do-i-search-my-chat-history-in-chatgpt%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252523.midi"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8554407-gpts-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11780217-using-study-mode-in-chatgpt"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search publishers OAI-SearchBot official",
"site:platform.openai.com OAI-SearchBot official crawler",
"site:docs.x.ai Grok citations sources web search consumer",
"site:docs.x.ai Grok personalization memory temporary chat"
],
"query": "site:help.openai.com ChatGPT search publishers OAI-SearchBot official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/20001243-advertiser-guidance-for-allowing-openai-web-crawlers"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/skills-routines-and-automations"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://docs.x.ai/build/modes-and-commands"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/collections-search"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/bots"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M2-R2.json",
"request_file": "requests/S3-OPENAIAPI-M2-R2.json",
"response_file": "responses/S3-OPENAIAPI-M2-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7447,
"public_answer_characters": 7309,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M2-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 7,447-character API answer, five citation occurrences and four unique explicit URLs. No outside webpage access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M2-R2.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Analytics | Google Analytics | No | Yes | Yes | conditional example for referral monitoring | unspecified | making them identifiable in tools such as Google Analytics.
| |
| X | X; X-centric prompts | No | Yes | Yes | suggested for Grok-oriented prompt coverage | unspecified | For Grok, include a few X-centric prompts because its search capabilities can use both real-time web and X information.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned official help.openai.com or docs.x.ai title/hostname identifies platform documentation; contents and factual claims were not independently checked. | Supporting citation for search, controls, referral/crawler behavior, not a recommendation to adopt an extra measurement product | |
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned official help.openai.com or docs.x.ai title/hostname identifies platform documentation; contents and factual claims were not independently checked. | Supporting citation for search, controls, referral/crawler behavior, not a recommendation to adopt an extra measurement product | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned official help.openai.com or docs.x.ai title/hostname identifies platform documentation; contents and factual claims were not independently checked. | Supporting citation for search, controls, referral/crawler behavior, not a recommendation to adopt an extra measurement product | |
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned official help.openai.com or docs.x.ai title/hostname identifies platform documentation; contents and factual claims were not independently checked. | Supporting citation for search, controls, referral/crawler behavior, not a recommendation to adopt an extra measurement product | |
| https://docs.x.ai/developers/tools/web-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned official help.openai.com or docs.x.ai title/hostname identifies platform documentation; contents and factual claims were not independently checked. | Supporting citation for search, controls, referral/crawler behavior, not a recommendation to adopt an extra measurement product |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://docs.x.ai/developers/tools/web-search | docs.x.ai | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Recommendation visibility",
"definition_verbatim": "Answers mentioning you ÷ total tested answers",
"notes": "Conflates label and numerator: this counts mentions, not independently coded active recommendations."
},
{
"label": "Top-three rate",
"definition_verbatim": "Answers placing you in the top 3 ÷ total answers"
},
{
"label": "Researched visibility",
"definition_verbatim": "Calculate visibility separately for web-researched prompts.",
"notes": "No distinct numerator formula is stated."
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate descriptions ÷ answers mentioning you"
},
{
"label": "Own-site citation rate",
"definition_verbatim": "| Own-site citation rate | 20% | 10% |",
"notes": "Illustrative table entry, not an observed experimental rate and not an explicit formula."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.136444,
"token_cost_usd": 0.116444,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 20861,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2433,
"output_tokens_details": {
"reasoning_tokens": 861
},
"total_tokens": 23294
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2433,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named measurement services and prompt-input channels; supplied targets, operational crawlers, example qualifiers and citation-only publishers stay separate. Google Analytics is retained without inferring GA4; unnamed spreadsheet stays unbranded.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Score validation or audit checklist"
],
"evidence": "A good **$0 workflow** is a small monthly “AI recommendation audit” using ChatGPT, Grok, and a spreadsheet.",
"rationale": "Nine steps cover prompt construction, natural/search lanes, session controls, repeated sampling, a tracker, metrics, diagnostics, access checks and monthly cadence."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Never mention your product’s name in these discovery prompts."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt twice"
},
"fresh_sessions": {
"value": true,
"evidence": "Start a fresh chat for each prompt."
},
"model_search_version_logging": {
"value": true,
"evidence": "Also record the date and model shown in the interface."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Natural or web-researched"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Recommended positively? Y/N",
"notes": "Separate mention, recommendation and own-citation tracker fields exist, but the later metric label conflates recommendation visibility with mentions."
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "| Own-site citation rate | 20% | 10% |",
"notes": "The illustrative dashboard separates citation and visibility rates; no own-citation-rate formula is supplied."
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Own website cited? Y/N"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Description accurate? Y/N"
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Save the answer or a screenshot."
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Answers mentioning you ÷ total tested answers"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "LLM answers are variable."
},
"human_checking": {
"value": true,
"evidence": "Validate it against the actual citations."
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Also monitor analytics for ChatGPT referrals."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "After the blind test, use a separate diagnostic prompt:"
},
"universal_rank_caution": {
"value": true,
"evidence": "It is not an official “AI ranking”"
},
"metric_label_conflation": {
"value": true,
"evidence": "### Recommendation visibility\n\n```text\nAnswers mentioning you ÷ total tested answers",
"notes": "The label says recommendation visibility, but the numerator counts mentions. Preserve this as returned advice rather than silently correcting it."
},
"posthoc_source_explanation_caution": {
"value": true,
"evidence": "Treat the explanation as a hypothesis, not an authoritative account of the model’s internal ranking process."
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for natural and web-researched manual tests",
"free_budget_fit": "stated free",
"evidence": "using ChatGPT, Grok, and a spreadsheet.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; own-provider OpenAI/xAI documentation is cited. API collection is not a consumer ChatGPT Search UI observation."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for natural and web-researched manual tests",
"free_budget_fit": "stated free",
"evidence": "using ChatGPT, Grok, and a spreadsheet.",
"source_urls": [
"https://docs.x.ai/developers/tools/web-search"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; own-provider OpenAI/xAI documentation is cited. API collection is not a consumer ChatGPT Search UI observation."
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Do not block the crawler or its published traffic; access eligibility does not guarantee placement.",
"evidence": "`robots.txt` does not block `OAI-SearchBot`",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_tool_overlap": true,
"notes": "Operational search-eligibility advice, separate from recommendations to use measurement tools. Source claim was not independently verified."
}
],
"contextual_non_tool_entities": [
{
"entity": "Shopify",
"entity_type": "hypothetical_buyer_prompt_qualifier",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Shopify-specific",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "An example buyer qualifier, not a measurement product recommendation."
},
{
"entity": "OpenAI",
"entity_type": "provider_attribution",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI states that referral URLs include `utm_source=chatgpt.com`",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
],
"excluded_from_default_overlap": true,
"notes": "Documentation publisher/provider attribution, not an additional discovered measurement tool."
}
],
"generic_methods": [
"Unnamed spreadsheet tracker; no Google Sheets brand inferred",
"Fixed unbranded buyer-intent bank with natural and web-researched lanes",
"Clean sessions and two-repetition sampling",
"Separate mention/recommendation/citation tracker fields",
"Appearance-based visibility and accuracy formulas",
"Post-blind-test diagnostic with source explanations treated as hypotheses",
"Crawler/public-page eligibility and referral monitoring",
"Monthly fixed-core benchmark"
],
"metric_definitions_observed": [
{
"label": "Recommendation visibility",
"definition_verbatim": "Answers mentioning you ÷ total tested answers",
"notes": "Conflates label and numerator: this counts mentions, not independently coded active recommendations."
},
{
"label": "Top-three rate",
"definition_verbatim": "Answers placing you in the top 3 ÷ total answers"
},
{
"label": "Researched visibility",
"definition_verbatim": "Calculate visibility separately for web-researched prompts.",
"notes": "No distinct numerator formula is stated."
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate descriptions ÷ answers mentioning you"
},
{
"label": "Own-site citation rate",
"definition_verbatim": "| Own-site citation rate | 20% | 10% |",
"notes": "Illustrative table entry, not an observed experimental rate and not an explicit formula."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 7309,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 5,
"explicit_body_URL_occurrences": 5,
"unique_answer_URLs": 4,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Five citation occurrences match five Markdown links, collapsing to four exact source URLs and two publisher domains. Returned retrieval sources do not expand the citation set.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M2-R2.json",
"response_file": "responses/S3-OPENAIAPI-M2-R2.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Full nine sections, starter version and final paragraph reviewed.",
"Same ChatGPT search URL appears twice; annotation and Markdown representations are not separate sources.",
"Two returned search actions are separate from the count of queries within actions and citation occurrences.",
"No observed tool/output cap; time/free-tier claims and illustrative statistics remain unverified advice."
],
"original_capture_answer_characters": 7447,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of help.openai.com to openai.com and docs.x.ai to x.ai; exact source URLs and hostnames retained. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"docs.x.ai": "x.ai"
}
}
}The recommendation-visibility metric counts answers mentioning the product; preserve and flag this C/M/R conflation.
Dashboard percentages and sample audits are illustrative returned advice, not Stage 3 findings.
Google Analytics is version-unspecified and not automatically merged with Google Analytics 4.
Generic spreadsheet is not a Google Sheets mention. Shopify is only a qualifier example.
X-centric prompt advice is input coverage, not a content-distribution recommendation.
Source explanations are expressly framed as hypotheses, not a causal account of ranking.
Encoded source URL and tracking parameters remain verbatim in local data; no external page validity/claim verification occurred.
S3-CHATGPT-M2-R3 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | Reddit, X |
| Cited domains · mapping complete | openai.com, x.com |
| Observed inline cited domains · coverage qualified | openai.com, x.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
## A free “AI recommendation visibility” workflow Treat this like lightweight rank tracking—not a one-time vanity search. Results vary by prompt wording, personalization, location, search mode, and run. ### 1. Build a fixed prompt matrix Create 15–25 prompts based on how real buyers describe their problem. **Do not mention your product name** in these discovery prompts. Use five prompt families: | Family | Template | |---|---| | Category | “What are the best `[category]` tools for `[audience]`?” | | Use case | “Recommend software for `[specific task]`.” | | Pain point | “I’m struggling with `[problem]`. What tools could help?” | | Alternatives | “What are good alternatives to `[major competitor]`?” | | Constraints | “What’s the best `[category]` for a bootstrapped team under `$X/month`?” | Example for an email-validation SaaS: - “What are the best email verification tools for indie hackers?” - “Recommend an affordable tool for cleaning a 10,000-contact email list.” - “What are good alternatives to ZeroBounce?” - “Which email verifier has a simple API and no annual contract?” - “I need to reduce email bounce rates. What tools should I consider?” Add two **branded diagnostic prompts**, but score them separately: - “What is `[Product]`, and who is it for?” - “Compare `[Product]` with `[Competitor]`.” These reveal whether the system understands your product accurately; they do not measure organic discovery. --- ### 2. Run two separate tests per prompt #### A. Model-recall test Ask: > Answer from your existing knowledge without searching the web. Recommend up to 10 products, with one sentence explaining each choice. This tests whether your brand appears in the model’s existing knowledge. #### B. Current-web test Ask: > Search the web for current information. Recommend up to 10 products and cite the sources used for each recommendation. This tests whether your site and third-party coverage are discoverable now. ChatGPT can search automatically or when Search is selected, and its answers may include citations and a Sources panel. OpenAI notes that search results can be incomplete or incorrect, so inspect the cited pages rather than merely recording a mention. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) Grok can decide to search public X posts and the broader web; its web-search capability is designed to retrieve current pages and sources. ([help.x.com](https://help.x.com/en/using-x/about-grok)) --- ### 3. Minimize personalization Run every prompt in a **fresh conversation**. For ChatGPT: - Use **Temporary Chat** - Choose **Unpersonalized** - Avoid custom instructions and prior context - Keep the same location/network when comparing results over time An unpersonalized Temporary Chat does not use memory, custom instructions, or plugins. This matters because ChatGPT memory can influence both responses and rewritten web-search queries; approximate location can also affect search results. ([help.openai.com](https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt)) For Grok: - Use **Private Chat**, if available - Turn off “Personalize Grok using X” - If testing through X, disable Grok personalization in X’s privacy settings Grok’s results can be personalized using profile, posts, engagement, interests, and prior interactions. Its official controls allow users to disable this personalization. ([help.x.com](https://help.x.com/en/using-x/about-grok)) **Important:** Don’t alternate between personalized and unpersonalized sessions in the same benchmark. Pick a baseline and keep it consistent. --- ### 4. Run each prompt three times LLM responses are variable. For each prompt: 1. Start a fresh, unpersonalized chat. 2. Paste the exact prompt. 3. Record the result. 4. Repeat twice more. 5. Do this on both ChatGPT and Grok. If free-plan limits make that too laborious: - Run every prompt once weekly. - Run your five highest-value prompts three times. - Perform the full three-run audit monthly. Don’t regenerate an answer in the same conversation; a regeneration is not as clean a test as a fresh chat. --- ### 5. Track results in a spreadsheet Use one row per run: ```text Date Platform Search mode: Recall / Web Prompt ID Exact prompt Product mentioned: Yes / No Position in list Recommended positively: Yes / Neutral / No Description accurate: Yes / Partly / No Product website cited: Yes / No Third-party source cited Competitors mentioned Screenshot or shared-chat reference Notes ``` Calculate four simple metrics: ### Mention rate ```text Runs mentioning your product ÷ total eligible runs ``` ### Top-three rate ```text Runs where you rank 1–3 ÷ total eligible runs ``` ### Citation rate ```text Web-search runs citing your domain ÷ total web-search runs ``` ### Accuracy rate ```text Accurate product descriptions ÷ runs mentioning your product ``` Report recall and web-search visibility separately. A product may have poor model recall but strong current-web visibility—or the reverse. A compact monthly dashboard could look like: | Platform/mode | Mention rate | Top-three rate | Domain citation rate | Accuracy | |---|---:|---:|---:|---:| | ChatGPT recall | 20% | 7% | N/A | 80% | | ChatGPT web | 53% | 27% | 33% | 94% | | Grok recall | 13% | 0% | N/A | 75% | | Grok web | 47% | 20% | 27% | 89% | --- ### 6. Diagnose *why* you’re absent For prompts where you don’t appear, follow up in a separate diagnostic chat: > What criteria did you use to choose these products? Then: > What publicly available evidence would a lesser-known product need to be included? Also inspect: - Which competitors recur? - Which pages and publications get cited? - Are results driven by listicles, comparison pages, directories, reviews, Reddit, X posts, or first-party sites? - Does the model misunderstand your category? - Does it know your pricing, integrations, target user, and differentiators? - Are your strongest claims independently corroborated? For ChatGPT web visibility specifically, ensure your site permits **OAI-Searchbot** and that your CDN or firewall is not blocking OpenAI’s published crawler traffic. Eligibility does not guarantee placement. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) --- ### 7. Repeat on a stable cadence A practical bootstrapped schedule: - **Weekly, 20 minutes:** Run your five money prompts once on each platform. - **Monthly, 60–90 minutes:** Run the full matrix, with three runs for important prompts. - **After major launches:** Re-run relevant web-search prompts after your new pages and third-party coverage have had time to be discovered. - **Quarterly:** Update the matrix based on customer calls, search queries, competitor changes, and new use cases. Change prompts only at the start of a new benchmark period. Otherwise, you won’t know whether movement came from improved visibility or altered wording. ## The key rule **Optimize for repeated, unprompted inclusion—not whether ChatGPT or Grok can recognize your product after you name it.** Your strongest signal is something like: > Across three runs of “best invoicing tools for solo consultants,” the product appeared twice, ranked fourth and sixth, and its official comparison page was cited once. That is much more actionable than “I asked ChatGPT about my startup and it knew who we were.”
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21065,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2390,
"output_tokens_details": {
"reasoning_tokens": 715
},
"total_tokens": 23455
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1364,
"cost_estimate_detail": {
"estimated_usd": 0.1364,
"token_cost_usd": 0.1164,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-M2-R3",
"slot_id": "S3-CHATGPT-M2-R3",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 3,
"submitted_at": "2026-11-02T09:34:08.010230+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:34:41.017034+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21065,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2390,
"output_tokens_details": {
"reasoning_tokens": 715
},
"total_tokens": 23455
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1364,
"token_cost_usd": 0.1164,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read complete seven-section recall/web workflow and final example through coherent ending. Five source occurrences match annotations, three distinct URLs. Correct model/effort and two actual searches; no caps. ZeroBounce is an illustrative buyer-prompt competitor; dashboard figures and final two-of-three example are illustrative, not observed results of this experiment.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:35:21.688356+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations temporary chat memory recommendations official",
"site:help.openai.com ChatGPT search how search works official",
"site:docs.x.ai Grok search citations official",
"site:help.x.com Grok search sources recommendations official"
],
"query": "site:help.openai.com ChatGPT search citations temporary chat memory recommendations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12677804-what-is-chatgpt-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9275245-chatgpt-free-tier-faq"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-how-do-i-search-my-chat-history-in-chatgpt%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.midi"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.avif"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai/grok private chat history personalization Grok official",
"site:help.x.com/en/using-x Grok personalization history private chat official",
"site:docs.x.ai/grok web search sources citations Grok official",
"site:x.ai/legal/privacy-policy Grok chat personalization data official"
],
"query": "site:docs.x.ai/grok private chat history personalization Grok official",
"sources": [
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/x-timeline"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/account-suggestions"
},
{
"type": "url",
"url": "https://help.x.com/en/rules-and-policies/recommendations"
},
{
"type": "url",
"url": "https://help.x.com/en/personalization-data-settings"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-chat"
},
{
"type": "url",
"url": "https://help.x.com/en/managing-your-account/accessing-your-x-data"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://x.ai/legal/privacy-policy"
},
{
"type": "url",
"url": "https://help.x.com/en/resources/recommender-systems"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-12-31-xai-frontier-artificial-intelligence-framework.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/ccpa_personnel_privacy_notice.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-M2-R3.json",
"request_file": "requests/S3-OPENAIAPI-M2-R3.json",
"response_file": "responses/S3-OPENAIAPI-M2-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7521,
"public_answer_characters": 7431,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-M2-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| About Grok | X Help Center | https://help.x.com/en/using-x/about-grok | help.x.com | API body url_citation annotation | |
| Temporary chat in ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | API body url_citation annotation | |
| About Grok | X Help Center | https://help.x.com/en/using-x/about-grok | help.x.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant review of the complete 7,521-character API answer, five citation annotations and matching explicit Markdown URLs. No outside webpage access, factual audit or independent human double-coding.
coding/S3-OPENAIAPI-M2-R3.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| No | Yes | No | discussed as a possible cited-source pattern | unspecified | Are results driven by listicles, comparison pages, directories, reviews, Reddit, X posts, or first-party sites?
| |||
| X | public X posts; X posts; X personalization | Yes | Yes | No | discussed as retrieval, personalization and source-pattern context | unspecified | Are results driven by listicles, comparison pages, directories, reviews, Reddit, X posts, or first-party sites?
| https://help.x.com/en/using-x/about-grok |
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI/X Help Center title and official help hostname identify platform documentation; content and capability claims were not independently verified. | Supporting citation for search/control/eligibility advice, not an extra measurement-product recommendation | |
| https://help.x.com/en/using-x/about-grok | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI/X Help Center title and official help hostname identify platform documentation; content and capability claims were not independently verified. | Supporting citation for search/control/eligibility advice, not an extra measurement-product recommendation | |
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI/X Help Center title and official help hostname identify platform documentation; content and capability claims were not independently verified. | Supporting citation for search/control/eligibility advice, not an extra measurement-product recommendation | |
| https://help.x.com/en/using-x/about-grok | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI/X Help Center title and official help hostname identify platform documentation; content and capability claims were not independently verified. | Supporting citation for search/control/eligibility advice, not an extra measurement-product recommendation | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | OpenAI/X Help Center title and official help hostname identify platform documentation; content and capability claims were not independently verified. | Supporting citation for search/control/eligibility advice, not an extra measurement-product recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.x.com/en/using-x/about-grok | help.x.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.x.com/en/using-x/about-grok | help.x.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Runs mentioning your product ÷ total eligible runs"
},
{
"label": "Top-three rate",
"definition_verbatim": "Runs where you rank 1–3 ÷ total eligible runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Web-search runs citing your domain ÷ total web-search runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate product descriptions ÷ runs mentioning your product"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1364,
"token_cost_usd": 0.1164,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21065,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4350
},
"output_tokens": 2390,
"output_tokens_details": {
"reasoning_tokens": 715
},
"total_tokens": 23455
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2390,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap reached. Returned usage supports an estimated charge, not reconciled provider billing; total budget remains a cumulative collection constraint."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantive named channels, separate from supplied target platforms, operational crawlers, hypothetical buyer-prompt products, citation publishers and unnamed spreadsheet methods. Source-pattern examples are not active platform-use or publishing recommendations.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Score validation or audit checklist"
],
"evidence": "A free “AI recommendation visibility” workflow",
"rationale": "Seven steps provide fixed buyer prompts, recall/web tracks, clean sessions, three runs, spreadsheet metrics, diagnosis and a repeated cadence. No named third-party measurement product is suggested."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "These reveal whether the system understands your product accurately; they do not measure organic discovery."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt three times"
},
"fresh_sessions": {
"value": true,
"evidence": "Run every prompt in a **fresh conversation**."
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Search mode: Recall / Web"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Recommended positively: Yes / Neutral / No"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Web-search runs citing your domain ÷ total web-search runs"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Product website cited: Yes / No"
},
"raw_answer_inspection": {
"value": true,
"evidence": "inspect the cited pages rather than merely recording a mention."
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Screenshot or shared-chat reference"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Runs mentioning your product ÷ total eligible runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Results vary by prompt wording, personalization, location, search mode, and run."
},
"human_checking": {
"value": true,
"evidence": "Are your strongest claims independently corroborated?"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": false,
"evidence": null
},
"paid_tool_dependence": {
"value": false,
"evidence": null
},
"branded_diagnostic_separated": {
"value": true,
"evidence": "Add two **branded diagnostic prompts**, but score them separately:"
},
"stable_benchmark_cadence": {
"value": true,
"evidence": "Change prompts only at the start of a new benchmark period."
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for fresh repeated recall/web audits",
"free_budget_fit": "stated free",
"evidence": "Do this on both ChatGPT and Grok.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform. OpenAI/X Help Center platform documentation is cited; advice about Temporary/Private modes is not a new discovered tool."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for fresh repeated recall/web audits",
"free_budget_fit": "stated free",
"evidence": "Do this on both ChatGPT and Grok.",
"source_urls": [
"https://help.x.com/en/using-x/about-grok"
],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform. OpenAI/X Help Center platform documentation is cited; advice about Temporary/Private modes is not a new discovered tool."
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"observed_name": "OAI-Searchbot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Permit the crawler and do not block published traffic; eligibility is not a placement guarantee.",
"evidence": "ensure your site permits **OAI-Searchbot**",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_tool_overlap": true,
"notes": "Capitalization variant normalized with observed spelling retained. Operational access advice, not a recommended measurement tool; claim not independently verified."
}
],
"contextual_non_tool_entities": [
{
"entity": "ZeroBounce",
"entity_type": "hypothetical_buyer_prompt_product",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "What are good alternatives to ZeroBounce?",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Example competitor in a hypothetical email-validation prompt, not a measurement recommendation or an actual seeded target."
},
{
"entity": "OpenAI",
"entity_type": "provider_attribution",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI notes that search results can be incomplete or incorrect, so inspect the cited pages rather than merely recording a mention.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt",
"https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Documentation publisher/provider attribution, not another discovered tool."
}
],
"generic_methods": [
"Unnamed spreadsheet tracker",
"Fixed unbranded prompt matrix and separate branded diagnostics",
"Independent model-recall and current-web tests",
"Fresh unpersonalized repeated conversations with no regeneration",
"Separate mention/top-three/citation/accuracy metrics",
"Cited-source pattern and corroboration review",
"Crawler eligibility checks",
"Weekly/monthly fixed benchmark cadence"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Runs mentioning your product ÷ total eligible runs"
},
{
"label": "Top-three rate",
"definition_verbatim": "Runs where you rank 1–3 ÷ total eligible runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Web-search runs citing your domain ÷ total web-search runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accurate product descriptions ÷ runs mentioning your product"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 7431,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 5,
"explicit_body_URL_occurrences": 5,
"unique_answer_URLs": 3,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Five citation occurrences match five Markdown links and deduplicate to three exact source URLs, two publisher domains. The same X help and ChatGPT-search links recur; retrieval action.sources stay separate.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-M2-R3.json",
"response_file": "responses/S3-OPENAIAPI-M2-R3.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Full seven-step workflow and key-rule/final illustrative example reviewed.",
"Citation annotations and explicit Markdown references are duplicate representations, not ten source occurrences.",
"Two returned search actions are distinct from citation occurrences and from the count of queries inside actions.",
"No observed tool/output cap; all model recommendations and example rates remain unverified output."
],
"original_capture_answer_characters": 7521,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping of help.openai.com to openai.com and help.x.com to x.com, retaining exact observed source URLs and hostnames. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"help.x.com": "x.com"
}
}
}The dashboard percentages and final two-of-three appearance example are illustrative advice, not this experiment's measured recommendations.
Generic spreadsheet is not a named spreadsheet brand. ZeroBounce is only a hypothetical buyer-prompt competitor.
Reddit/X source-pattern questions do not establish an active publishing recommendation.
The suggested follow-up diagnostics are collected advice. This collection itself sent only the fixed original prompt; it did not perform those follow-ups.
No claim that citations reveal actual ranking causality is independently validated.
All platform capabilities, free-plan cadence/time estimates and crawler claims remain unverified returned output.
S3-GROK-M2-R1 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs Brand Radar, Bing Webmaster Tools, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher, Reddit, Semrush AI Visibility Checker, X |
| Mentioned tools / services / surfaces | Ahrefs Brand Radar, Bing Webmaster Tools, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AEO Grader, Mangools AI Search Watcher, Reddit, Semrush AI Visibility Checker, X |
| Cited domains · mapping complete | Unavailable · no eligible answers |
| Observed inline cited domains · coverage qualified | seozest.io, solcrys.com, stork.ai |
| Observed cited-source categories · provisional | Other / unresolved |
{ "cited_domains": "Citation mapping is not confirmed complete; use separately labeled observed-link comparisons." }
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
**Use a fixed set of buyer-intent prompts, run them manually in fresh ChatGPT and Grok sessions on a schedule, and log results in a Google Sheet.** This is the most reliable free, repeatable method. Answers are probabilistic (they change often), so consistency and multiple runs matter more than any single screenshot. ### 1. Build a fixed prompt set (do this once) Create 8–15 category-level, buying-intent prompts your actual customers would ask. Avoid pure branded queries (“Is [YourProduct] good?”) because those almost always surface you. Focus on discovery and recommendation language: - “Best [your category] software for [target user or use case]” - “Top [category] tools for small teams / startups / bootstrapped founders” - “[Category] alternatives to [main competitor]” - “How do I choose a [category] tool?” - “[Category] tools that [key feature or outcome you offer]” - “Best [category] for [specific pain point]” Keep the exact wording stable so you can compare over time. Add a couple of comparison or “why not” prompts later if you want deeper sentiment data. ### 2. Set up a simple tracker (Google Sheets is free) Columns that work well: - Date - Engine (ChatGPT / Grok) - Prompt - Appeared? (Yes/No) - Position (if it was listed: 1st, 2nd, etc., or “mentioned but not ranked”) - Competitors named - Sources/citations (if shown) - Notes (sentiment, accuracy, anything odd) - Optional: link to screenshot or chat Track **share of voice** = % of prompts where you appear. Also note which competitors show up most often and which sources the models trust. ### 3. The repeatable check (weekly or bi-weekly, ~30–45 min) 1. Open a fresh/incognito session on **chat.openai.com** (free tier is enough) and on **Grok** (grok.com or via X). 2. Run your fixed prompts. Note whether web search was used on ChatGPT if the option appears. 3. For your most important prompts, run each twice in separate chats and record both results—answers vary significantly by session, date, and search setting. 4. Log immediately while the response is in front of you. 5. Calculate your appearance rate and glance at competitor patterns. Grok often pulls more from X/Twitter signals, so social mentions and activity can matter more there than on ChatGPT. ### 4. Free supplements that reduce pure manual work - **Mangools AI Search Watcher free tier**: 5 prompts, 1 monitor, weekly updates, includes both ChatGPT and Grok (and several others). Each prompt is checked twice. No credit card. Useful for the highest-priority prompts while you do broader manual checks. - One-off free checkers (HubSpot AI Search Grader, Semrush free AI Visibility Checker, Ahrefs Brand Radar, and others) for occasional baseline or competitor snapshots. - Downstream signals: Google Analytics 4 AI Assistants channel (shows referral traffic from ChatGPT, Grok, etc.) and Google Search Console / Bing Webmaster Tools generative AI reports. These don’t tell you whether you were *recommended*, only what happened after a click or impression. ### 5. Make the loop useful When you’re missing: - Note which competitors appear and which third-party sources are cited. - Prioritize getting mentioned in those sources (review sites, directories, comparison content, Reddit, relevant publications, X for Grok). - Confirm AI crawlers can reach your site (`robots.txt` should not block GPTBot, OAI-SearchBot, etc.). **Time cost**: 30–45 minutes every 1–2 weeks once the prompt set and sheet exist. That’s the practical free ceiling. Paid GEO tools automate sampling and scoring, but this manual workflow gives you the same core signal (do you appear in real buyer prompts?) without spending money. Start with 8–10 prompts this week, log a baseline, and re-run the same ones in two weeks. The trend over a few cycles is more useful than any single check.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M2-R1",
"slot_id": "S3-GROK-M2-R1",
"platform": "Grok",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 1,
"submitted_at": "2026-11-02T06:24:29.470Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:28:47.228Z",
"source_panel_declared_hits": 55,
"capture_method": "Complete rendered answer and full Copy response corroborated using offline local buffer.",
"search_evidence": "Expanded native activity saved separately; query/result badge counts are not citation counts.",
"raw_file": "records/S3-GROK-M2-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 3821,
"public_answer_characters": 3821,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M2-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Does ChatGPT Recommend Your Product? How to Check | Stork.AI (Stork) | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | Inline answer link | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| Does ChatGPT Recommend Your Product? How to Check | Stork.AI (Stork) | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | Inline answer link | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| Does ChatGPT Recommend Your Product? How to Check | Stork.AI (Stork) | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | Inline answer link | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| How to Check If ChatGPT, Perplexity, Grok, or Claude AI Is Recommending Your Business (A Free 10-Minute Audit) - Seozest (Seozest) | https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/ | seozest.io | Inline answer link | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| Free ChatGPT Visibility Tools (2026): What's Actually Free (Solcrys) | https://solcrys.com/free-chatgpt-visibility-tools/ | solcrys.com | Inline answer link | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete captured body, matching 3,821-character copy and existing native complete-rendered/activity evidence. No external webpage queries or fact verification; no independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet; Google Sheets | No | Yes | Yes | suggested | stated free | Set up a simple tracker (Google Sheets is free)
| |
| Mangools AI Search Watcher | Mangools AI Search Watcher free tier | No | Yes | Yes | conditional supplement | stated free | Mangools AI Search Watcher free tier**: 5 prompts, 1 monitor, weekly updates, includes both ChatGPT and Grok (and several others). Each prompt is checked twice. No credit card. Useful for the highest-priority prompts while you do broader manual checks.
| |
| HubSpot AEO Grader | HubSpot AI Search Grader | No | Yes | Yes | conditional supplement | stated free | One-off free checkers (HubSpot AI Search Grader, Semrush free AI Visibility Checker, Ahrefs Brand Radar, and others) for occasional baseline or competitor snapshots.
| |
| Semrush AI Visibility Checker | Semrush free AI Visibility Checker | No | Yes | Yes | conditional supplement | stated free | One-off free checkers (HubSpot AI Search Grader, Semrush free AI Visibility Checker, Ahrefs Brand Radar, and others) for occasional baseline or competitor snapshots.
| |
| Ahrefs Brand Radar | Ahrefs Brand Radar | No | Yes | Yes | conditional supplement | stated free | One-off free checkers (HubSpot AI Search Grader, Semrush free AI Visibility Checker, Ahrefs Brand Radar, and others) for occasional baseline or competitor snapshots.
| |
| Google Analytics 4 | Google Analytics 4 AI Assistants channel | No | Yes | Yes | suggested | stated free | Downstream signals: Google Analytics 4 AI Assistants channel (shows referral traffic from ChatGPT, Grok, etc.) and Google Search Console / Bing Webmaster Tools generative AI reports. These don’t tell you whether you were *recommended*, only what happened after a click or impression.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | suggested | stated free | Downstream signals: Google Analytics 4 AI Assistants channel (shows referral traffic from ChatGPT, Grok, etc.) and Google Search Console / Bing Webmaster Tools generative AI reports. These don’t tell you whether you were *recommended*, only what happened after a click or impression.
| |
| Bing Webmaster Tools | Bing Webmaster Tools generative AI reports | No | Yes | Yes | suggested | stated free | Downstream signals: Google Analytics 4 AI Assistants channel (shows referral traffic from ChatGPT, Grok, etc.) and Google Search Console / Bing Webmaster Tools generative AI reports. These don’t tell you whether you were *recommended*, only what happened after a click or impression.
| |
| No | Yes | Yes | suggested | unspecified | Prioritize getting mentioned in those sources (review sites, directories, comparison content, Reddit, relevant publications, X for Grok).
| |||
| X | X/Twitter; X for Grok | No | Yes | Yes | suggested | unspecified | Prioritize getting mentioned in those sources (review sites, directories, comparison content, Reddit, relevant publications, X for Grok).
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://stork.ai/blog/does-chatgpt-recommend-your-product | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_title_and_URL_only | Returned how-to title and blog path suggest a practical tutorial; commercial publisher, authorship and first-person procedure are not established by metadata alone. | supporting citation, not an explicit recommendation to read or use the publisher |
| https://stork.ai/blog/does-chatgpt-recommend-your-product | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_title_and_URL_only | Returned how-to title and blog path suggest a practical tutorial; commercial publisher, authorship and first-person procedure are not established by metadata alone. | supporting citation, not an explicit recommendation to read or use the publisher |
| https://stork.ai/blog/does-chatgpt-recommend-your-product | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_title_and_URL_only | Returned how-to title and blog path suggest a practical tutorial; commercial publisher, authorship and first-person procedure are not established by metadata alone. | supporting citation, not an explicit recommendation to read or use the publisher |
| https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/ | Other / unresolved | Practitioner tutorial or experiment | provisional_from_returned_title_and_URL_only | Returned title describes a free 10-minute audit tutorial. It does not establish an original experiment or the author's practitioner status. | supporting citation, not an explicit recommendation to read or use the publisher |
| https://solcrys.com/free-chatgpt-visibility-tools/ | Other / unresolved | Editorial or industry reporting | provisional_from_returned_title_and_URL_only | Returned title compares tools claimed to be free. The metadata does not establish primary product documentation versus editorial/practitioner advice. | supporting citation, not an explicit recommendation to read or use the publisher |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "share of voice",
"definition_verbatim": "share of voice** = % of prompts where you appear.",
"notes": "Retain the answer's definition rather than harmonizing it silently with competitor-relative share-of-voice definitions from other answers. It operationalizes prompt-level appearance here."
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named measurement tools, analytics services and distribution channels. Generic methods, supplied target platforms, citation-only publishers and crawler compatibility entities are separate.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Tool-led solution"
],
"evidence": "Use a fixed set of buyer-intent prompts, run them manually in fresh ChatGPT and Grok sessions on a schedule, and log results in a Google Sheet.",
"rationale": "The main structure gives prompt selection, spreadsheet logging, scheduled repeated checks and a feedback loop. Named free tools are supplements."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Avoid pure branded queries (“Is [YourProduct] good?”) because those almost always surface you."
},
"repeated_runs": {
"value": true,
"evidence": "For your most important prompts, run each twice in separate chats and record both results—answers vary significantly by session, date, and search setting."
},
"fresh_sessions": {
"value": true,
"evidence": "Open a fresh/incognito session"
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Note whether web search was used on ChatGPT if the option appears."
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"raw_answer_inspection": {
"value": true,
"evidence": "Optional: link to screenshot or chat"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "share of voice** = % of prompts where you appear."
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Answers are probabilistic (they change often)"
},
"human_checking": {
"value": true,
"evidence": "Log immediately while the response is in front of you."
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "These don’t tell you whether you were *recommended*, only what happened after a click or impression."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"observed_name": "ChatGPT; chat.openai.com",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual checks",
"evidence": "Open a fresh/incognito session on **chat.openai.com** (free tier is enough) and on **Grok** (grok.com or via X).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; textual consumer-site addresses are not treated as inline supporting citations or newly discovered tools."
},
{
"entity": "Grok",
"observed_name": "Grok; grok.com",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual checks",
"evidence": "Open a fresh/incognito session on **chat.openai.com** (free tier is enough) and on **Grok** (grok.com or via X).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Supplied target platform; textual consumer-site addresses are not treated as inline supporting citations or newly discovered tools."
}
],
"operational_entities": [
{
"entity": "GPTBot",
"entity_type": "crawler_agent",
"citation": false,
"mention": true,
"recommendation": false,
"recommended_action": "Do not block crawler access",
"evidence": "Confirm AI crawlers can reach your site (`robots.txt` should not block GPTBot, OAI-SearchBot, etc.).",
"excluded_from_default_tool_overlap": true,
"notes": "Crawler compatibility instruction, not a recommendation to adopt this crawler as a measurement tool. The safety or applicability of this instruction was not independently assessed."
},
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": false,
"mention": true,
"recommendation": false,
"recommended_action": "Do not block crawler access",
"evidence": "Confirm AI crawlers can reach your site (`robots.txt` should not block GPTBot, OAI-SearchBot, etc.).",
"excluded_from_default_tool_overlap": true,
"notes": "Crawler compatibility instruction, not a recommendation to adopt this crawler as a measurement tool. The safety or applicability of this instruction was not independently assessed."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Fixed unbranded buyer-intent prompt set",
"Manual repeated checks on a weekly/biweekly schedule",
"Spreadsheet logging of appearances, positions, competitors and citations",
"Appearance-rate and competitor-pattern summaries",
"Earning third-party mentions in cited sources",
"AI crawler and robots.txt access checks"
],
"metric_definitions_observed": [
{
"label": "share of voice",
"definition_verbatim": "share of voice** = % of prompts where you appear.",
"notes": "Retain the answer's definition rather than harmonizing it silently with competitor-relative share-of-voice definitions from other answers. It operationalizes prompt-level appearance here."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 3821,
"copy_file_matches_record": true,
"observed_inline_link_occurrences": 5,
"unique_standardized_answer_URLs": 3,
"unique_displayed_answer_domains": 3,
"source_badge_hits": 55,
"retrieval_inventory_complete": false,
"citation_mapping_complete": null,
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"complete_rendered_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Earlier accessibility snapshots were partial, but later complete native rendering and matching full copied body corroborate the coherent final ending. Do not score a historical partial view as the current answer body.",
"Three Stork link occurrences collapse to one URL/domain. Five captured inline occurrences yield three unique URLs/domains, not five independent sources.",
"Native activity exposes grouped three-search and two-search controls and a browsed-link control. Exact query strings were not expanded in the supplied activity capture.",
"The 55-hit source badge is not a citation count or a complete retrieval inventory.",
"Workspace-explored activity is visible; its underlying contents and influence are unresolved.",
"Captured URL targets omit scheme and may omit www; HTTPS standardization does not independently establish exact hrefs.",
"Free tiers, engine coverage, feature names, crawler advice, time estimates and claims of reliability remain model output, not independently verified facts."
],
"original_capture_answer_characters": 3821,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping retains the three displayed base domains. Display-standardized URL schemes and any omitted www are not treated as independently recovered exact hrefs; no external public-suffix lookup used.",
"mapping": {
"stork.ai": "stork.ai",
"seozest.io": "seozest.io",
"solcrys.com": "solcrys.com"
}
}
}Review the provisional HubSpot AI Search Grader / AEO Grader alias before cross-answer aggregation. Mangools Watcher versus Grader and Ahrefs Brand Radar versus AI Visibility Checker remain separate products.
Distribution-channel recommendations are not equivalent to measurement-tool recommendations; retain role labels.
The answer records appearances and citations but does not calculate citation rate or operationalize active recommendation rate separately.
Model/version logging is not explicitly requested by this answer; logging the engine/date and search use is narrower.
GA4/GSC/Bing reporting claims and the term share of voice must remain as observed advice, not validated tool capabilities or harmonized metric definitions.
No source-publisher census is folded into the tool counts; no own-content tool citations appear among captured inline links.
S3-GROK-M2-R2 · Grok Expert · consumer interface · completed
Source panel groups results by web-search query; 65 is the sum of reported result hits, not unique answer citations. Complete body has no inline citation anchors.
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs Brand Radar, Google People also ask, Google Search Console, Google Sheets, Notion, Reddit, X |
| Mentioned tools / services / surfaces | Ahrefs Brand Radar, Capterra, G2, Google People also ask, Google Search Console, Google Sheets, Notion, Reddit, X |
| Cited domains · mapping complete | None observed within this eligible set |
| Observed inline cited domains · coverage qualified | None observed within this eligible set |
| Observed cited-source categories · provisional | None observed within this eligible set |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
A free, repeatable workflow is a fixed set of buyer-intent prompts + manual sampling in fresh sessions + a simple spreadsheet. This is the standard DIY approach used by bootstrapped founders and small teams for ChatGPT and Grok (and other engines). Paid trackers exist, but they are unnecessary for a baseline and trend view. 1. Build a fixed prompt set (do this once) Create 8–15 prompts that real buyers would type before they know your brand name. Freeze the exact wording so you can compare over time. Core types: Category discovery: “What’s the best [your category] for [specific use case / company size / audience]?” Alternatives: “Best alternatives to [leading competitor]” Comparisons: “[Competitor A] vs [Competitor B] for [use case]” Problem / job-to-be-done: “How do I [the outcome your product delivers] without [common pain]?” Pull language from customer calls, support tickets, Google’s “People also ask,” Reddit, or your own Search Console queries. Avoid pure branded questions (“Is [your product] good?”) for discovery checks—they inflate results. Keep branded prompts only for accuracy checks (pricing, features, downsides). 2. Run the checks cleanly ChatGPT (chatgpt.com): Use a Temporary Chat and choose the unpersonalized option (or log out / use a private window). This reduces memory and personalization bias. Prefer sessions with web search enabled when available. Run each prompt 2–3 times in fresh chats. Grok (grok.com or via X): Start a fresh conversation each time. Grok draws heavily from X posts in addition to the web, so its recommendations can differ from ChatGPT. Same 2–3 runs per prompt. For every answer, note: Did your product appear? (Y/N) Position / order if it was in a list Competitors named Sources cited (especially useful) Brief framing (how it described you or why it preferred someone else) 3. Log it in a spreadsheet One row per run. Columns that matter: | Date | Engine | Prompt | Mentioned (Y/N) | Position | Competitors | Citations | Notes | At the end of each cycle, calculate simple metrics: Mention rate (how often you appear on unbranded prompts) Share of voice vs the 2–3 competitors that keep showing up Citation rate (how often your site or a trusted third-party page about you is linked) A Google Sheet or Notion table is enough. No special tools required. 4. Cadence Run a full baseline now. Repeat monthly (or every 2 weeks on your 5–6 highest-priority prompts). Extra spot-checks after major content launches, new reviews (G2/Capterra/etc.), or big X activity. Time cost is typically 20–40 minutes per cycle once the prompts are set. Answers are non-deterministic—the same prompt can produce different shortlists across sessions, models, and dates—so the trend across multiple runs matters more than any single screenshot. Practical notes for bootstrapped SaaS Free one-shot checkers (Ahrefs Brand Radar free preview, various “AI visibility” free tools) can give a quick extra data point, but they don’t replace your controlled, repeated prompt set. Ensure your site is crawlable (don’t block major AI crawlers in robots.txt). For Grok specifically, an authentic presence on X helps because of its heavier weighting of X content. This measures whether you appear in recommendations. Improving it usually means earning consistent third-party coverage (listicles, reviews, comparisons) plus clear, accurate pages on your own site—not paid placements (OpenAI has stated product results are not ads). This workflow is free, fully under your control, and gives a clear before/after view of whether ChatGPT and Grok are recommending your product. Start with the prompt list and a spreadsheet this week; the rest is just consistent sampling.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M2-R2",
"slot_id": "S3-GROK-M2-R2",
"platform": "Grok",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 2,
"submitted_at": "2026-11-02T06:37:26.314Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:38:35.514Z",
"source_panel_declared_hits": 65,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"search_query_labels": [
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcfree method to check if product appears in ChatGPT recommendations15",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edchow to monitor SaaS product visibility in Grok and ChatGPT AI search10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcrepeatable workflow AI SEO for bootstrapped SaaS ChatGPT Grok10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edc\"ChatGPT\" \"appears in\" OR \"mentioned in\" recommendations SaaS free tool OR method10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcGrok AI recommendations how to check product visibility free10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcChatGPT temporary chat OR private for AI visibility check10"
],
"raw_file": "records/S3-GROK-M2-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 3717,
"public_answer_characters": 3717,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M2-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
No URLs recorded in this field. Completeness and source role are reported separately.
Retrieval-panel sources · not matched answer citations
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| stork.ai/blog/does-chatgpt-recommend-your-product | https://www.stork.ai/blog/does-chatgpt-recommend-your-product | www.stork.ai | Browsed link in search-activity panel; not an inline answer citation | |
| arahi.ai/blog/how-to-check-brand-visibility-in-chatgpt | https://arahi.ai/blog/how-to-check-brand-visibility-in-chatgpt | arahi.ai | Browsed link in search-activity panel; not an inline answer citation |
Assistant review of the complete 3,717-character rendered answer and recorded native web-query/browsed-link evidence; no outside webpage queries, factual verification or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google People also ask | Google’s People also ask | No | Yes | Yes | suggested | stated free | Pull language from customer calls, support tickets, Google’s “People also ask,” Reddit, or your own Search Console queries.
| |
| No | Yes | Yes | suggested | stated free | Pull language from customer calls, support tickets, Google’s “People also ask,” Reddit, or your own Search Console queries.
| |||
| Google Search Console | Search Console | No | Yes | Yes | suggested | stated free | Pull language from customer calls, support tickets, Google’s “People also ask,” Reddit, or your own Search Console queries.
| |
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | A Google Sheet or Notion table is enough. No special tools required.
| |
| Notion | Notion table | No | Yes | Yes | suggested | stated free | A Google Sheet or Notion table is enough. No special tools required.
| |
| G2 | G2 | No | Yes | No | discussed | unspecified | Extra spot-checks after major content launches, new reviews (G2/Capterra/etc.), or big X activity.
| |
| Capterra | Capterra | No | Yes | No | discussed | unspecified | Extra spot-checks after major content launches, new reviews (G2/Capterra/etc.), or big X activity.
| |
| X | X | No | Yes | Yes | conditional | unspecified | For Grok specifically, an authentic presence on X helps because of its heavier weighting of X content.
| |
| Ahrefs Brand Radar | Ahrefs Brand Radar free preview | No | Yes | Yes | conditional | mixed/limited tier | Free one-shot checkers (Ahrefs Brand Radar free preview, various “AI visibility” free tools) can give a quick extra data point, but they don’t replace your controlled, repeated prompt set.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "Mention rate (how often you appear on unbranded prompts)"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice vs the 2–3 competitors that keep showing up",
"notes": "No explicit formula is supplied."
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate (how often your site or a trusted third-party page about you is linked)",
"notes": "This includes third-party pages about the brand and is broader than the experiment's entity-own-content C definition. Do not silently harmonize it with own-site-only citation advice."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"retrieval_sources": [
{
"domain": "www.stork.ai",
"kind": "Browsed link in search-activity panel; not an inline answer citation",
"text": "stork.ai/blog/does-chatgpt-recommend-your-product",
"url": "https://www.stork.ai/blog/does-chatgpt-recommend-your-product",
"source_category": "Other / unresolved",
"candidate_category": "Practitioner tutorial or experiment",
"classification_status": "provisional_from_activity_label_and_URL_only",
"classification_evidence": "The how-to blog path suggests a practical guide. Authorship, original empirical research and commercial provenance were not established from this captured metadata.",
"content_verified": false,
"role_in_analysis": "Browsed retrieval source only; exclude from cited-domain and answer-citation statistics."
},
{
"domain": "arahi.ai",
"kind": "Browsed link in search-activity panel; not an inline answer citation",
"text": "arahi.ai/blog/how-to-check-brand-visibility-in-chatgpt",
"url": "https://arahi.ai/blog/how-to-check-brand-visibility-in-chatgpt",
"source_category": "Other / unresolved",
"candidate_category": "Practitioner tutorial or experiment",
"classification_status": "provisional_from_activity_label_and_URL_only",
"classification_evidence": "The how-to blog path suggests a practical guide. Authorship, original empirical research and commercial provenance were not established from this captured metadata.",
"content_verified": false,
"role_in_analysis": "Browsed retrieval source only; exclude from cited-domain and answer-citation statistics."
}
]
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/features, with supplied target platforms and company context separate. Measurement tools, prompt inputs, review examples and content-channel advice retain distinct roles.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Conceptual explanation",
"Tool-led solution"
],
"evidence": "A free, repeatable workflow is a fixed set of buyer-intent prompts + manual sampling in fresh sessions + a simple spreadsheet.",
"rationale": "Fixed prompt construction, clean-session sampling, per-run logging, metrics and cadence dominate; named trackers and a checker preview are supplements."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Keep branded prompts only for accuracy checks (pricing, features, downsides)."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt 2–3 times in fresh chats."
},
"fresh_sessions": {
"value": true,
"evidence": "Use a Temporary Chat and choose the unpersonalized option (or log out / use a private window)."
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "Prefer sessions with web search enabled when available."
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Citation rate (how often your site or a trusted third-party page about you is linked)"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "| Date | Engine | Prompt | Mentioned (Y/N) | Position | Competitors | Citations | Notes |"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Brief framing (how it described you or why it preferred someone else)"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": false,
"evidence": null
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "Answers are non-deterministic—the same prompt can produce different shortlists across sessions, models, and dates"
},
"human_checking": {
"value": true,
"evidence": "manual sampling in fresh sessions"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": false,
"evidence": null
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for fresh-session repeated checks",
"evidence": "ChatGPT (chatgpt.com):",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied in prompt and explicitly used in workflow; not a newly discovered tool. Prose addresses are not inline citations."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for fresh-session repeated checks",
"evidence": "Grok (grok.com or via X):",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied in prompt and explicitly used in workflow; not a newly discovered tool. Prose addresses are not inline citations."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "OpenAI has stated product results are not ads",
"notes": "Company attribution in a product-results/advertising claim, not a measurement product; unverified platform output."
}
],
"generic_methods": [
"Frozen unbranded buyer prompts plus separate branded accuracy checks",
"Repeated fresh unpersonalized ChatGPT and fresh Grok sessions",
"One-row-per-run tracking table",
"Mention rate, competitor share of voice and citation rate",
"Baseline and monthly/fortnightly cadence",
"Event-triggered spot checks",
"Crawler access checks",
"Third-party coverage and clear on-site content"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "Mention rate (how often you appear on unbranded prompts)"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice vs the 2–3 competitors that keep showing up",
"notes": "No explicit formula is supplied."
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate (how often your site or a trusted third-party page about you is linked)",
"notes": "This includes third-party pages about the brand and is broader than the experiment's entity-own-content C definition. Do not silently harmonize it with own-site-only citation advice."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 3717,
"observed_inline_link_occurrences": 0,
"unique_DOM_answer_URLs": 0,
"observed_inline_cited_domains": 0,
"retrieval_links_recorded": 2,
"source_badge_hits": 65,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Scoped rendered answer has zero observable inline anchors. This does not establish absence of retrieval or a complete source-panel inventory.",
"search_observed": true,
"search_enabled_UI": null,
"search_query_labels": [
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcfree method to check if product appears in ChatGPT recommendations15",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edchow to monitor SaaS product visibility in Grok and ChatGPT AI search10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcrepeatable workflow AI SEO for bootstrapped SaaS ChatGPT Grok10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edc\"ChatGPT\" \"appears in\" OR \"mentioned in\" recommendations SaaS free tool OR method10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcGrok AI recommendations how to check product visibility free10",
"\\u5df2\\u641c\\u7d22\\u7f51\\u7edcChatGPT temporary chat OR private for AI visibility check10"
],
"search_hit_counts_reported": [
15,
10,
10,
10,
10,
10
],
"notes": [
"Six recorded native web-query labels carry hit counts 15 plus five groups of 10. Their sum is 65 result hits, not 65 unique sources or citations.",
"Preserve query labels verbatim, including UI text and concatenated result counts; they are not asserted as parsed exact provider-query strings.",
"Two browsed links are explicitly retrieval/activity sources, not answer citations. Their domain/category fields are separate from the empty inline-cited-domain set.",
"All tool pricing/feature claims, model weighting, time estimates and reliability assertions remain unverified Grok output."
],
"original_capture_answer_characters": 3717,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "No inline answer citation domains to normalize. Separate retrieval hosts normalize www.stork.ai to stork.ai while retaining observed hrefs; arahi.ai unchanged. No external public-suffix lookup used.",
"cited_mapping": {},
"retrieval_mapping": {
"www.stork.ai": "stork.ai",
"arahi.ai": "arahi.ai"
}
}
}Recommended prompt inputs, spreadsheet/table tools and content channels are different roles; avoid presenting all seven as dedicated GEO measurement tools.
G2/Capterra are examples of review-change events, not explicit adoption or posting recommendations.
Citation rate in this answer includes trusted third-party pages about the brand. This is a substantive metric-definition difference, not a change to the experiment's C coding.
Active recommendation rate is not operationalized separately from appearance rate.
Search enablement is advised, but logging actual search use/model versions is not explicit in the tracking columns.
Zero observed inline citation domains does not imply no evidence retrieval. Preserve both separate lists and their coverage limits.
S3-GROK-M2-R3 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs AI Visibility Checker, Capterra, G2, Google Search, Google Sheets, Indexly Grok Visibility Checker, Mangools AI Search Watcher, Product Hunt, Ranked AI Visibility Checker, Reddit, Snoika brand checker, X |
| Mentioned tools / services / surfaces | Ahrefs AI Visibility Checker, Capterra, Claude, G2, Gemini, Google AI Overviews, Google Search, Google Sheets, Indexly Grok Visibility Checker, Llama, Mangools AI Search Watcher, Perplexity, Product Hunt, Ranked AI Visibility Checker, Reddit, Snoika brand checker, X |
| Cited domains · mapping complete | 01net.com, amicited.com, indexly.ai, mangools.com, ranked.ai, snoika.com |
| Observed inline cited domains · coverage qualified | 01net.com, amicited.com, indexly.ai, mangools.com, ranked.ai, snoika.com |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved, Tool/product documentation or sales page |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what's a free, repeatable workflow for checking whether my product appears in ChatGPT and Grok recommendations?
Captured answer body
Here’s a free, repeatable workflow that bootstrapped SaaS founders actually use to check whether ChatGPT and Grok recommend their product.01net It takes ~30–45 minutes once set up, then 20–40 minutes on a weekly or bi-weekly cadence. Total cost: $0. 1. One-time setup (30–60 min) Define 8–15 buyer prompts (the questions real prospects ask, not your brand name): Category: “best [your category] for [audience/use case]” Alternatives: “best alternatives to [top competitor]” Problem: “tools to [problem your product solves]” Comparison: “[competitor] vs [other competitor]” Direct (secondary): “what is [your product]” / “is [your product] worth it” Create a simple Google Sheet with columns: Date | Prompt | Platform | Mentioned (Y/N) | Position (if listed) | Sentiment | Competitors named | Sources cited | Notes/quote Run free snapshot tools once for a baseline: ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain. Snoika brand checker — includes Grok; distinguishes “knows you” vs “recommends you” + community/trust signals. Indexly Grok Visibility Checker — free, no login, paste your URL. Ahrefs AI Visibility Checker (brand name, no signup) for a broader ChatGPT-heavy view. 2. Repeatable routine (weekly or every 2 weeks) Clean sessions only ChatGPT: Temporary chat or incognito + web search enabled. Grok: Fresh chat on grok.x.ai (or the X app). Grok pulls real-time web + X data, so results can shift faster. Run your core prompts (start with your top 5–8). Do each once or twice — answers are non-deterministic. Log results immediately in the sheet. Note exact wording when you appear and which competitors/sources win instead. Optional free automation layer Set up Mangools AI Search Watcher free plan (5 prompts, 1 monitor, weekly updates). It covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others. Perfect for a small recurring sample without manual work every week. Monthly deeper look Re-run the free snapshot tools, check “yourbrand -site:yourdomain.com” on Google for third-party mentions, and glance at G2/Capterra/Reddit/Product Hunt presence. Those sources heavily influence both ChatGPT and Grok recommendations.Amicited What the data usually shows Recommendation visibility is driven more by third-party mentions (reviews, Reddit, comparison listicles, community discussion) than by your own blog alone. ChatGPT and Grok often cite different sources; tracking both is worthwhile. One check is noisy. Trends over 4–8 weeks are what matter. Quick improvement loop (informed by the checks) When you’re missing from answers, the usual high-leverage free/low-cost fixes are: Honest comparison / alternatives pages on your site More genuine reviews on G2/Capterra Useful answers in relevant Reddit threads (disclose you’re the founder) Clear, structured product pages that answer buyer questions directly This workflow stays free, stays repeatable, and gives you the same signal paid GEO tools charge for — just with a bit of discipline on the logging side. Start with the prompt list and the Google Sheet this week; the rest compounds from there.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-M2-R3",
"slot_id": "S3-GROK-M2-R3",
"platform": "Grok",
"prompt_id": "M2",
"intent": "M",
"wording": 2,
"repeat": 3,
"submitted_at": "2026-11-02T06:45:00.609Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:46:38.810Z",
"source_panel_declared_hits": 75,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-M2-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 3174,
"public_answer_characters": 3174,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-M2-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| 01net | https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/ | www.01net.com | Inline answer link | Rendered DOM href |
| ranked.ai free checker | https://www.ranked.ai/free-tools/ai-visibility-checker | www.ranked.ai | Inline answer link | Rendered DOM href |
| Snoika brand checker | https://snoika.com/free-tools/brand-ai-visibility-checker | snoika.com | Inline answer link | Rendered DOM href |
| Indexly Grok Visibility Checker | https://indexly.ai/grok-visibility-checker | indexly.ai | Inline answer link | Rendered DOM href |
| Mangools AI Search Watcher free plan | https://mangools.com/ai-search-watcher | mangools.com | Inline answer link | Rendered DOM href |
| Amicited | https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/ | www.amicited.com | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 3,174-character rendered answer, six scoped DOM hrefs and existing activity capture. No external webpage visits, feature/pricing audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | Create a simple Google Sheet with columns:
| |
| Ranked AI Visibility Checker | ranked.ai free checker | Yes | Yes | Yes | suggested | stated free | ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain.
| https://www.ranked.ai/free-tools/ai-visibility-checker |
| Snoika brand checker | Snoika brand checker | Yes | Yes | Yes | suggested | stated free | Snoika brand checker — includes Grok; distinguishes “knows you” vs “recommends you” + community/trust signals.
| https://snoika.com/free-tools/brand-ai-visibility-checker |
| Indexly Grok Visibility Checker | Indexly Grok Visibility Checker | Yes | Yes | Yes | suggested | stated free | Indexly Grok Visibility Checker — free, no login, paste your URL.
| https://indexly.ai/grok-visibility-checker |
| Ahrefs AI Visibility Checker | Ahrefs AI Visibility Checker | No | Yes | Yes | suggested | stated free | Ahrefs AI Visibility Checker (brand name, no signup) for a broader ChatGPT-heavy view.
| |
| Mangools AI Search Watcher | Mangools AI Search Watcher free plan | Yes | Yes | Yes | conditional | mixed/limited tier | Set up Mangools AI Search Watcher free plan (5 prompts, 1 monitor, weekly updates). It covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others.
| https://mangools.com/ai-search-watcher |
| Google Search | No | Yes | Yes | suggested | stated free | Re-run the free snapshot tools, check “yourbrand -site:yourdomain.com” on Google for third-party mentions, and glance at G2/Capterra/Reddit/Product Hunt presence.
| ||
| G2 | G2 | No | Yes | Yes | suggested | unspecified | More genuine reviews on G2/Capterra
| |
| Capterra | Capterra | No | Yes | Yes | suggested | unspecified | More genuine reviews on G2/Capterra
| |
| No | Yes | Yes | suggested | unspecified | Useful answers in relevant Reddit threads (disclose you’re the founder)
| |||
| Product Hunt | Product Hunt | No | Yes | Yes | suggested | unspecified | Re-run the free snapshot tools, check “yourbrand -site:yourdomain.com” on Google for third-party mentions, and glance at G2/Capterra/Reddit/Product Hunt presence.
| |
| X | X app | No | Yes | Yes | conditional | unspecified | Grok: Fresh chat on grok.x.ai (or the X app).
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain.
| |
| Gemini | Gemini | No | Yes | No | discussed | unspecified | ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain.
| |
| Llama | Llama | No | Yes | No | discussed | unspecified | ranked.ai free checker — live answers from ChatGPT, Grok, Claude, Gemini, Perplexity, Llama (no signup). Paste one of your prompts + domain.
| |
| Google AI Overviews | AI Overviews | No | Yes | No | discussed | unspecified | Set up Mangools AI Search Watcher free plan (5 prompts, 1 monitor, weekly updates). It covers ChatGPT, Grok, Gemini, Claude, AI Overviews, and others.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/ | Editorial or industry reporting | provisional_from_returned_label_URL_and_answer_role_only | SEO/tools how-to article on a publication-style domain. Provisional classification from URL only; content and author not independently inspected. | Introductory supporting citation, not a creator/resource recommendation | |
| https://www.ranked.ai/free-tools/ai-visibility-checker | Tool/product documentation or sales page | provisional_from_returned_label_URL_and_answer_role_only | Displayed product recommendation links directly to the same-brand checker/monitor path; classification provisional without inspecting page content. | Own-content link attached to an active/optional product recommendation | |
| https://snoika.com/free-tools/brand-ai-visibility-checker | Tool/product documentation or sales page | provisional_from_returned_label_URL_and_answer_role_only | Displayed product recommendation links directly to the same-brand checker/monitor path; classification provisional without inspecting page content. | Own-content link attached to an active/optional product recommendation | |
| https://indexly.ai/grok-visibility-checker | Tool/product documentation or sales page | provisional_from_returned_label_URL_and_answer_role_only | Displayed product recommendation links directly to the same-brand checker/monitor path; classification provisional without inspecting page content. | Own-content link attached to an active/optional product recommendation | |
| https://mangools.com/ai-search-watcher | Tool/product documentation or sales page | provisional_from_returned_label_URL_and_answer_role_only | Displayed product recommendation links directly to the same-brand checker/monitor path; classification provisional without inspecting page content. | Own-content link attached to an active/optional product recommendation | |
| https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/ | Other / unresolved | provisional_from_returned_label_URL_and_answer_role_only | Branded how-AI-recommends-software blog path suggests explanation/tutorial content. Original research, authorship and commercial role unresolved. | Supporting citation about recommendation sources; no instruction to read/use the publisher |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[]
Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/surfaces with supplied target platforms separate. Product recommendations, monitoring channels, coverage mentions and source-only citation labels retain distinct roles.",
"answer_type": {
"primary": "Manual measurement workflow",
"secondary": [
"Tool-led solution"
],
"evidence": "Here’s a free, repeatable workflow that bootstrapped SaaS founders actually use to check whether ChatGPT and Grok recommend their product.",
"rationale": "Fixed prompts, a Google Sheet, repeated clean-session checks and a cadence form the core. Snapshot checkers and an optional monitor are actively suggested supplements."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Direct (secondary): “what is [your product]” / “is [your product] worth it”"
},
"repeated_runs": {
"value": true,
"evidence": "Do each once or twice — answers are non-deterministic."
},
"fresh_sessions": {
"value": true,
"evidence": "Clean sessions only"
},
"model_search_version_logging": {
"value": false,
"evidence": null
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "ChatGPT: Temporary chat or incognito + web search enabled."
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Date | Prompt | Platform | Mentioned (Y/N) | Position (if listed) | Sentiment | Competitors named | Sources cited | Notes/quote"
},
"awareness_vs_recommendation_distinction": {
"value": true,
"evidence": "distinguishes “knows you” vs “recommends you”"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Note exact wording when you appear and which competitors/sources win instead."
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": false,
"evidence": null
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": false,
"evidence": null
},
"uncertainty_variance": {
"value": true,
"evidence": "One check is noisy. Trends over 4–8 weeks are what matter."
},
"human_checking": {
"value": true,
"evidence": "Log results immediately in the sheet."
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": false,
"evidence": null
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for clean repeated manual checks",
"evidence": "ChatGPT: Temporary chat or incognito + web search enabled.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied in the question. Prose access URLs are not additional supporting citations."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for clean repeated manual checks",
"evidence": "Grok: Fresh chat on grok.x.ai (or the X app).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied in the question. Prose access URLs are not additional supporting citations."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Fixed category/alternative/problem/comparison buyer prompts",
"Secondary branded accuracy prompts",
"Per-run spreadsheet logging",
"Weekly/fortnightly manual sampling",
"Optional free recurring monitor",
"Monthly third-party-mention search and source presence monitoring",
"Comparison content, genuine reviews, disclosed community participation and structured pages"
],
"metric_definitions_observed": [],
"capture_scope": {
"body_complete": true,
"answer_characters": 3174,
"observed_inline_link_occurrences": 6,
"unique_DOM_answer_URLs": 6,
"unique_cited_domains": 6,
"source_badge_hits": 75,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Six exact rendered-article hrefs; not all 75 retrieval hits.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [
"Mangools AI Search Watcher free plan Grok tracking"
],
"notes": [
"Activity shows grouped searches of four and two plus a separate displayed web query with 10 results, with browsed-link controls. Exact query coverage is partial.",
"Introductory 01net and later Amicited strings are rendered citation labels, not substantive recommendations to use or follow creators.",
"The 75-source badge is not a unique inline citation count.",
"Workspace-explored activity labels do not establish their underlying contents or influence.",
"Claimed free tiers, signup requirements, engine coverage, times, reliability and relative source influence remain platform output, not verified facts."
],
"original_capture_answer_characters": 3174,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual normalization removes leading www from 01net.com, ranked.ai and amicited.com; exact observed DOM hrefs/hostnames remain preserved. No external public-suffix lookup used.",
"mapping": {
"www.01net.com": "01net.com",
"www.ranked.ai": "ranked.ai",
"snoika.com": "snoika.com",
"indexly.ai": "indexly.ai",
"mangools.com": "mangools.com",
"www.amicited.com": "amicited.com"
}
}
}Run free snapshot tools once is an active instruction for all four named checkers; their recommendations are not inferred merely from source labels.
Four own-domain product links support C=true for Ranked, Snoika, Indexly and Mangools. Ahrefs is recommended without an own-content href.
01net is a supporting citation label adjacent to the introduction, not a substantive creator recommendation.
Google search, review/community monitoring, community contribution and measurement software are distinct roles.
Model/version/search-state logging is not required by the listed columns, despite advice to enable ChatGPT search.
A claimed checker distinction between knowing and recommending a brand is recorded; the manual log itself does not explicitly define a separate active-recommendation field/rate.
S3-CHATGPT-V1-R1 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | xAI Web Search, xAI X Search |
| Cited domains · mapping complete | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | openai.com, x.ai |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
**Short answer:** don’t trust the number itself. Trust it only when you can audit the prompts, settings, raw answers, and calculation behind it. Generative answers vary even when the input is unchanged, so a single query—or a mysterious 0–100 score—is not a reliable measurement. OpenAI’s own evaluation guidance recommends task-specific tests, logging everything, continuous evaluation, and combining numerical metrics with human judgment. ([platform.openai.com](https://platform.openai.com/docs/guides/evaluation-best-practices)) ## An 8-point trust test Give the tool one point for each item: 1. **Exact prompts are visible** You should see every query used. Most should be unbranded buyer questions, such as “best invoicing tool for freelancers,” not questions containing your company name. 2. **The prompt set stays fixed** If prompts change every scan, score changes may reflect different questions rather than improved visibility. 3. **Raw answers are available** You should be able to inspect the full response, citations, timestamp, competitors mentioned, and your position—not just a dashboard percentage. 4. **The product surface is identified** “ChatGPT” is insufficient. Was it the consumer app or API? Was search enabled? Was the session personalized? What location was used? ChatGPT can automatically search, rewrite queries, and use approximate location or memory, all of which can affect results. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) 5. **Grok’s retrieval mode is identified** The vendor should disclose whether Grok used no retrieval, web search, X search, or both. xAI documents these as separate tools, and current information is unavailable through its API models unless search tools are enabled. ([docs.x.ai](https://docs.x.ai/docs/guides/live-search)) 6. **Exact model/version and date are logged** Avoid results labeled merely “latest.” xAI’s aliases can automatically migrate to newer versions, while dated model IDs are intended for consistent workflows. ([docs.x.ai](https://docs.x.ai/docs/models)) 7. **Prompts are run repeatedly** The report should show a denominator such as “mentioned in 7 of 30 runs,” not merely “23% visibility.” One run per prompt is a snapshot, not a dependable trend. 8. **The formula is public and decomposable** You should separately see: - Unprompted mention rate - Citation rate - Recommendation position - Factual accuracy - Competitor share of voice A blended score should show its weights. Otherwise, a high score could hide zero citations or inaccurate descriptions. ### Interpretation - **7–8 points:** trustworthy for tracking trends - **5–6 points:** directional only - **0–4 points:** primarily a marketing metric Even an 8/8 score estimates visibility for its chosen prompt panel—not visibility across all real ChatGPT or Grok users. ## Free DIY validation Create ten prompts based on actual buying situations: - Four category-discovery prompts - Three “alternatives to…” prompts - Two problem-to-solution prompts - One branded prompt for factual accuracy Run each prompt **three times in ChatGPT and three times in Grok**, using fresh, unpersonalized conversations where possible. That is 60 responses per monthly check. Track this in a spreadsheet: | Date | Engine | Search mode | Prompt | Run | Mentioned? | Position | Own domain cited? | Other source cited? | Accurate? | |---|---|---|---|---:|---|---:|---|---|---| Calculate: ```text Unprompted mention rate = unbranded runs mentioning you / unbranded runs Citation rate = runs citing your domain / total runs Top-3 rate = runs listing you among first 3 recommendations / total runs Accuracy rate = audited claims that are correct / claims audited ``` Keep ChatGPT and Grok separate. Their retrieval mechanisms and product surfaces differ, so averaging them can hide useful information. ## How to spot-check a vendor Ask for five randomly selected observations, then rerun those prompts yourself with matching settings. You should not expect identical wording. You should expect roughly similar: - Brand inclusion frequency - Competitors named - Sources cited - General recommendation position If the vendor cannot provide the **prompt, raw answer, timestamp, model, search mode, and location/account context**, consider the score unverifiable. ## Biggest red flags - “Proprietary methodology” with no formula - One run per prompt - Prompts containing your brand mixed with unbranded prompts - API results presented as consumer-app visibility - No distinction between search-on and search-off answers - No raw responses or citations - Score changes without sample counts - Comparing your score across unrelated industries - Recommendations optimized to increase the vendor’s own score - Claims that a small decimal movement proves an improvement **Bottom line:** with no budget, your fixed 10-prompt spreadsheet is more useful than a sophisticated opaque score. Track stable, auditable rates monthly and treat referral traffic, trials, and customer-reported discovery as the ultimate validation.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 46785,
"input_tokens_details": {
"cache_write_tokens": 4349,
"cached_tokens": 0
},
"output_tokens": 2974,
"output_tokens_details": {
"reasoning_tokens": 1871
},
"total_tokens": 49759
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.270969,
"cost_estimate_detail": {
"estimated_usd": 0.270969,
"token_cost_usd": 0.250969,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R1",
"slot_id": "S3-CHATGPT-V1-R1",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T09:18:30.426126+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:19:27.136515+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 46785,
"input_tokens_details": {
"cache_write_tokens": 4349,
"cached_tokens": 0
},
"output_tokens": 2974,
"output_tokens_details": {
"reasoning_tokens": 1871
},
"total_tokens": 49759
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.270969,
"token_cost_usd": 0.250969,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Assistant read full returned body, checked coherent final paragraph, all four explicit Markdown source URLs match url_citation annotations, completed live-search actions and requested model/effort. No truncation or tool/output cap reached. Claims and the proposed 8-point scoring thresholds are collected output, not externally verified.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:20:55.760328+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:platform.openai.com/docs evaluations best practices reproducibility model outputs OpenAI",
"site:docs.x.ai Grok API models search citations system prompt reproducibility",
"site:openai.com index ChatGPT search citations how search works official",
"AI visibility score methodology ChatGPT Grok benchmark prompts citations official documentation"
],
"query": "site:platform.openai.com/docs evaluations best practices reproducibility model outputs OpenAI",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://llm-visibility-index.com/methodology/"
},
{
"type": "url",
"url": "https://citations.io/docs"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://openai.com/academy/search-and-deep-research/"
},
{
"type": "url",
"url": "https://visibilityradar.ai/methodology"
},
{
"type": "url",
"url": "https://www.generaldataworks.com/ai-visibility/methodology"
},
{
"type": "url",
"url": "https://www.temso.ai/rankings/methodology"
},
{
"type": "url",
"url": "https://www.theaivisibilityindex.ai/faq"
},
{
"type": "url",
"url": "https://recosignal.com/methodology"
},
{
"type": "url",
"url": "https://sofindable.com/methodology"
},
{
"type": "url",
"url": "https://getaisnapshot.com/methodology"
},
{
"type": "url",
"url": "https://getfoundonai.com/assets/attachments/AIVO_Methodology.pdf"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a-family-guide-to-help-teens-use-ai-responsibly.pdf"
},
{
"type": "url",
"url": "https://ittc.ku.edu/~fli/papers/2024_ccs-checkgpt-full.pdf"
},
{
"type": "url",
"url": "https://futureoflife.org/wp-content/uploads/2025/12/AI-Safety-Index-Report_131225_Full_Report_Digital.pdf"
},
{
"type": "url",
"url": "https://hai.stanford.edu/assets/files/hai_ai_index_report_2025.pdf"
},
{
"type": "url",
"url": "https://www.aigl.blog/content/files/2025/07/AI-Safety-Index-Summer-2025.pdf"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SEO_LLM/comments/1vho94t/reviewcomparison_sites_pitching_a_new_ai/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1xd9m3v/what_i_learned_trying_to_measure_ai_visibility/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SaaS/comments/1wro769/how_we_measure_ai_search_visibility_the_method/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SEO/comments/1wrvs14/how_do_you_verify_an_ai_visibility_tracker_is/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/digital_marketing/comments/1ule9bi/sat_through_6_ai_search_optimization_pitches_this/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SaaS/comments/1u2al4v/i_built_an_ai_visibility_tracker_ran_it_on_my_own/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1xbgclz/i_have_a_problem_with_ai_visibility_reports_that/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SEMrush/comments/1rdk0hi/how_to_conduct_an_ai_visibility_audit_in_2026/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/aeo/comments/1uuz0yd/new_research_is_the_data_from_your_ai_visibility/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/aeo/comments/1wn1399/how_can_i_find_the_ai_prompts_where_my_website_is/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1w4ecsw/ai_visibility_tracking_needs_a_model_changelog/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1xd99cf/how_do_you_measure_brand_visibility_in_ai_answers/"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2603.25220"
},
{
"type": "url",
"url": "https://arxiv.org/abs/1810.03993"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2505.18059"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai/docs guides search Grok live search sources citations official",
"site:docs.x.ai/docs models Grok model versions official",
"site:platform.openai.com/docs/guides/evals evaluation best practices official OpenAI",
"site:help.openai.com ChatGPT search location personalization memory official"
],
"query": "site:docs.x.ai/docs guides search Grok live search sources citations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8142208-chatgpt-ios-and-android-app-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search//"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.avif"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10408842-google-app-data-controls-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/7039943-how-openai-handles-data-in-consumer-services"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/collections-search"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/models"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "open_page",
"url": "https://docs.x.ai/docs/guides/live-search"
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "find_in_page",
"pattern": "Evaluate early and often",
"url": "https://platform.openai.com/docs/guides/evaluation-best-practices"
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "open_page",
"url": "https://docs.x.ai/developers/tools/x-search"
}
}
],
"raw_file": "records/S3-OPENAIAPI-V1-R1.json",
"request_file": "requests/S3-OPENAIAPI-V1-R1.json",
"response_file": "responses/S3-OPENAIAPI-V1-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 5158,
"public_answer_characters": 5158,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V1-R1",
"historical_ui_attempt_ids": [
"S3-CHATGPT-V1-R1",
"S3-CHATGPT-V1-R1-A2",
"S3-CHATGPT-V1-R1-A3"
]
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Evaluation best practices | OpenAI API | https://platform.openai.com/docs/guides/evaluation-best-practices | platform.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Web Search | SpaceXSI Docs | https://docs.x.ai/docs/guides/live-search | docs.x.ai | API body url_citation annotation | |
| Grok Models & Pricing | SpaceXSI Docs | https://docs.x.ai/docs/models | docs.x.ai | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant semantic review of the full finalized API answer, raw response, annotations and request/response metadata. No external webpage queries, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V1-R1.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| xAI Web Search | web search | Yes | Yes | No | discussed as a methodology disclosure requirement | unspecified | The vendor should disclose whether Grok used no retrieval, web search, X search, or both. xAI documents these as separate tools, and current information is unavailable through its API models unless search tools are enabled.
| https://docs.x.ai/docs/guides/live-search |
| xAI X Search | X search | Yes | Yes | No | discussed as a methodology disclosure requirement | unspecified | The vendor should disclose whether Grok used no retrieval, web search, X search, or both. xAI documents these as separate tools, and current information is unavailable through its API models unless search tools are enabled.
| https://docs.x.ai/docs/guides/live-search |
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://platform.openai.com/docs/guides/evaluation-best-practices | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned source title and official platform documentation/help hostname identify provider documentation; page contents and claims were not independently audited. | Supporting citation for methodology or platform behavior, not a recommendation to adopt a vendor tool | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned source title and official platform documentation/help hostname identify provider documentation; page contents and claims were not independently audited. | Supporting citation for methodology or platform behavior, not a recommendation to adopt a vendor tool | |
| https://docs.x.ai/docs/guides/live-search | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned source title and official platform documentation/help hostname identify provider documentation; page contents and claims were not independently audited. | Supporting citation for methodology or platform behavior, not a recommendation to adopt a vendor tool | |
| https://docs.x.ai/docs/models | Platform documentation | provisional_from_explicit_attribution_title_and_official_domain | Returned source title and official platform documentation/help hostname identify provider documentation; page contents and claims were not independently audited. | Supporting citation for methodology or platform behavior, not a recommendation to adopt a vendor tool |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://platform.openai.com/docs/guides/evaluation-best-practices | platform.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown body URL; same source as annotation | ||
| https://docs.x.ai/docs/guides/live-search | docs.x.ai | Explicit Markdown body URL; same source as annotation | ||
| https://docs.x.ai/docs/models | docs.x.ai | Explicit Markdown body URL; same source as annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Unprompted mention rate",
"definition_verbatim": "Unprompted mention rate = unbranded runs mentioning you / unbranded runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate = runs citing your domain / total runs"
},
{
"label": "Top-3 rate",
"definition_verbatim": "Top-3 rate = runs listing you among first 3 recommendations / total runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accuracy rate = audited claims that are correct / claims audited"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI’s own evaluation guidance recommends task-specific tests, logging everything, continuous evaluation, and combining numerical metrics with human judgment.",
"source_urls": [
"https://platform.openai.com/docs/guides/evaluation-best-practices",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_tool_overlap": true,
"notes": "Source attribution, not an active recommendation to use or follow OpenAI as a newly discovered tool."
},
{
"entity": "xAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "xAI documents these as separate tools",
"source_urls": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models"
],
"excluded_from_default_tool_overlap": true,
"notes": "Substantive documentation attribution, separate from named retrieval-tool features."
}
],
"model_proposed_scoring_rule": {
"name": "8-point trust test",
"points_per_check": 1,
"thresholds": {
"7-8": "trustworthy for tracking trends",
"5-6": "directional only",
"0-4": "primarily a marketing metric"
},
"empirically_validated": false,
"notes": "These point weights and cutoffs are generated advice, not externally established experimental findings or a validated scoring instrument."
},
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.270969,
"token_cost_usd": 0.250969,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 46785,
"input_tokens_details": {
"cache_write_tokens": 4349,
"cached_tokens": 0
},
"output_tokens": 2974,
"output_tokens_details": {
"reasoning_tokens": 1871
},
"total_tokens": 49759
},
"max_tool_calls": 8,
"observed_web_actions": 5,
"max_output_tokens": 32768,
"observed_output_tokens": 2974,
"tool_limit_reached": false,
"output_limit_reached": false,
"budget_currency": "USD",
"authorized_total_budget": 10.0,
"concerns": [
"No observed tool/output cap was reached. Cost is an estimate, not reconciled billing; later request costs must still be tracked cumulatively within US$10."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named API retrieval features, supplied targets and documentation publishers are separate. No branded spreadsheet or visibility checker is inferred from generic workflow advice.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "An 8-point trust test",
"rationale": "Trustworthiness criteria, raw evidence and reproducibility dominate; a free generic spreadsheet workflow supports the checklist."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "One branded prompt for factual accuracy"
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt **three times in ChatGPT and three times in Grok**, using fresh, unpersonalized conversations where possible."
},
"fresh_sessions": {
"value": true,
"evidence": "using fresh, unpersonalized conversations where possible"
},
"model_search_version_logging": {
"value": true,
"evidence": "Exact model/version and date are logged"
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Search mode | Prompt | Run"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "Was it the consumer app or API?"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Top-3 rate = runs listing you among first 3 recommendations / total runs"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Citation rate = runs citing your domain / total runs"
},
"raw_answer_inspection": {
"value": true,
"evidence": "You should be able to inspect the full response, citations, timestamp, competitors mentioned, and your position"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Unprompted mention rate = unbranded runs mentioning you / unbranded runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "A blended score should show its weights."
},
"composite_score_caution": {
"value": true,
"evidence": "A blended score should show its weights. Otherwise, a high score could hide zero citations or inaccurate descriptions."
},
"uncertainty_variance": {
"value": true,
"evidence": "One run per prompt is a snapshot, not a dependable trend."
},
"human_checking": {
"value": true,
"evidence": "Ask for five randomly selected observations, then rerun those prompts yourself with matching settings."
},
"downstream_conversions": {
"value": true,
"evidence": "treat referral traffic, trials, and customer-reported discovery as the ultimate validation."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual validation",
"evidence": "Run each prompt **three times in ChatGPT and three times in Grok**, using fresh, unpersonalized conversations where possible.",
"source_urls": [
"https://platform.openai.com/docs/guides/evaluation-best-practices",
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied platform; documentation is first-party to its provider. API collection of this answer is not the consumer platform being advised about."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual validation",
"evidence": "Run each prompt **three times in ChatGPT and three times in Grok**, using fresh, unpersonalized conversations where possible.",
"source_urls": [
"https://docs.x.ai/docs/guides/live-search",
"https://docs.x.ai/docs/models"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied platform; documentation is first-party to its provider. API collection of this answer is not the consumer platform being advised about."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Eight-point audit checklist",
"Fixed ten-prompt buyer panel with separate branded accuracy prompt",
"Sixty repeated monthly consumer-interface responses",
"Unnamed spreadsheet logging",
"Separate per-engine mention, own-domain citation, top-three and accuracy rates",
"Five-observation vendor spot check"
],
"metric_definitions_observed": [
{
"label": "Unprompted mention rate",
"definition_verbatim": "Unprompted mention rate = unbranded runs mentioning you / unbranded runs"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate = runs citing your domain / total runs"
},
{
"label": "Top-3 rate",
"definition_verbatim": "Top-3 rate = runs listing you among first 3 recommendations / total runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "Accuracy rate = audited claims that are correct / claims audited"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 5158,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 4,
"explicit_body_URL_occurrences": 4,
"unique_answer_URLs": 4,
"unique_cited_hostnames": 3,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_scope": "Four answer url_citation annotations and the same four explicit Markdown URLs; duplicates across representations count once. Retrieval sources remain distinct and do not expand the answer-citation set.",
"source_inventory_complete_for_answer_URLs": true,
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 5,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V1-R1.json",
"response_file": "responses/S3-OPENAIAPI-V1-R1.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Five completed web actions comprise two searches, two opens and one find. Reported billable requests are two, not five.",
"API output and citation annotations are not consumer ChatGPT Search UI observations.",
"Model/effort fields are returned API metadata, not an audit of internal execution.",
"Four official-document source claims and all score thresholds remain unverified platform output."
],
"original_capture_answer_characters": 5158,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual documented mapping of official provider subdomains to openai.com and x.ai; exact annotations and hostnames retained. No external suffix lookup.",
"mapping": {
"platform.openai.com": "openai.com",
"help.openai.com": "openai.com",
"docs.x.ai": "x.ai"
}
}
}Named xAI web/X retrieval tools are methodology context, not adoption recommendations.
Source-provider documentation citations do not by themselves recommend those providers.
Generic spreadsheet advice is not a Google Sheets recommendation.
The answer is from OpenAI API + web search; comparisons against Grok additionally change the collection surface.
Eight-point scoring thresholds are unvalidated generated advice.
S3-CHATGPT-V1-R2 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | ChatGPT Search, xAI Web Search, xAI X Search |
| Cited domains · mapping complete | openai.com |
| Observed inline cited domains · coverage qualified | openai.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
**Short answer:** don’t trust a single “AI visibility score.” Trust the **underlying observations** only if they’re transparent, repeatable, and relevant to real buyer prompts. ChatGPT and Grok can produce different answers depending on whether web search is active, prompt wording, location, conversation context, and model version. ChatGPT may rewrite searches and use approximate location; OpenAI also warns that search results and citations can be incomplete, outdated, or incorrect. Grok’s API does not access real-time information unless Web Search or X Search is enabled. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) ## Ask these seven questions A trustworthy vendor should answer all seven: 1. **Which exact product and model were tested?** “ChatGPT” isn’t enough. Ask whether it was logged-in ChatGPT, Search mode, API, or another mode—and which Grok model/interface. 2. **Was live search enabled?** Search-enabled and training-knowledge answers are fundamentally different tests. Citations should be captured when available. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) 3. **Can I see the actual prompts and full responses?** Reject black-box scores that provide only a number or chart. 4. **How many independent runs were performed?** One run per prompt is essentially anecdotal. I’d want at least **three fresh-chat runs per prompt**, preferably on different days. 5. **What exactly counts as visibility?** These should be reported separately: - Brand mentioned - Brand recommended - Position in the answer - Linked or cited - Description accurate - Sentiment positive/neutral/negative 6. **Were prompts genuinely buyer-oriented?** “What is Acme?” measures recognition. “Best invoicing tool for a two-person US consulting firm” measures commercial discovery. 7. **Does the score include uncertainty?** “Mentioned in 9 of 30 runs” is useful. “Visibility score: 76” without raw counts is not. ## Red flags Be skeptical if: - The methodology is proprietary. - You cannot export prompts and responses. - Every prompt includes your brand name. - Competitor and category prompts are missing. - Results come from one run on one day. - Search mode, location, model, and date are undisclosed. - Mentions and citations are treated as equivalent. - The score moves dramatically after tiny website changes. - The vendor guarantees higher rankings. - It claims to represent “ChatGPT visibility” while testing only an API model. ## A free DIY validation test Create **15 prompts**: - 5 category prompts: “Best tools for ___” - 5 problem prompts: “How can I solve ___?” - 3 comparison prompts: “Alternatives to [competitor]” - 2 high-intent prompts: “What should a [specific customer] buy for ___?” Then: 1. Run each prompt **three times in fresh conversations** on ChatGPT and Grok. 2. Keep search settings consistent. 3. Record the date, interface, model if shown, answer, citations, and your brand’s position. 4. Calculate: - **Mention rate:** mentions ÷ total runs - **Recommendation rate:** genuine recommendations ÷ total runs - **Citation rate:** responses citing your domain ÷ search-enabled runs - **Accuracy rate:** accurate descriptions ÷ mentions 5. Repeat monthly using the same prompt set. A simple spreadsheet is enough. Your baseline might look like: | Metric | ChatGPT | Grok | |---|---:|---:| | Mention rate | 8/45 | 5/45 | | Recommendation rate | 3/45 | 2/45 | | Domain citation rate | 2/45 | 1/45 | | Accurate descriptions | 7/8 | 3/5 | That is more informative than “AI visibility: 42.” ## Best purchasing rule Only pay for a visibility tool if it saves enough manual work **and** its results substantially match your DIY sample. Compare 10–15 identical prompts. If the tool reports strong visibility while your controlled tests rarely show your company, ask the vendor to explain the discrepancy before trusting it. The most trustworthy product is not necessarily the one with the fanciest score. It is the one that lets you inspect every prompt, response, model, search setting, date, citation, and calculation.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 12913,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1470,
"output_tokens_details": {
"reasoning_tokens": 503
},
"total_tokens": 14383
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.0753956,
"cost_estimate_detail": {
"estimated_usd": 0.0753956,
"token_cost_usd": 0.0653956,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R2",
"slot_id": "S3-CHATGPT-V1-R2",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 2,
"submitted_at": "2026-11-02T09:30:13.472155+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:30:36.689352+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 12913,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1470,
"output_tokens_details": {
"reasoning_tokens": 503
},
"total_tokens": 14383
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 1
}
},
"estimated_cost": {
"estimated_usd": 0.0753956,
"token_cost_usd": 0.0653956,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read full seven-question checklist, red flags, 15-prompt DIY sample and final purchasing-rule paragraph. Two explicit source occurrences match annotations. Coherent ending, requested model/effort and actual search verified, no caps. Example rates are illustrative; citation support and product claims not externally audited.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:31:30.426176+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search how answers sources citations official",
"site:platform.openai.com docs web search tool official OpenAI",
"site:docs.x.ai Grok search sources citations official",
"site:docs.x.ai Grok API search tools official"
],
"query": "site:help.openai.com ChatGPT search how answers sources citations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%23.zst"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12677804-what-is-chatgpt-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10500283-deep-research-faq/"
},
{
"type": "url",
"url": "https://platform.openai.com/pricing"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-how-do-i-search-my-chat-history-in-chatgpt%25252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.midi"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/models"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/citations"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-V1-R2.json",
"request_file": "requests/S3-OPENAIAPI-V1-R2.json",
"response_file": "responses/S3-OPENAIAPI-V1-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4247,
"public_answer_characters": 4211,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V1-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant semantic review of complete finalized 4,247-character API answer, response status, citations and request/response metadata. No external queries, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V1-R2.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| ChatGPT Search | Search mode | Yes | Yes | No | discussed as a model/search setting to audit | unspecified | Ask whether it was logged-in ChatGPT, Search mode, API, or another mode—and which Grok model/interface.
| https://help.openai.com/en/articles/9237897-chatgpt-search https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt |
| xAI Web Search | Web Search | No | Yes | No | discussed as a model/search setting to audit | unspecified | Grok’s API does not access real-time information unless Web Search or X Search is enabled.
| |
| xAI X Search | X Search | No | Yes | No | discussed as a model/search setting to audit | unspecified | Grok’s API does not access real-time information unless Web Search or X Search is enabled.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_returned_title_and_official_help_hostname | OpenAI Help Center search documentation URL/title; content support and claims not independently checked. | Supporting citation, not an automatic vendor-tool recommendation | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_returned_title_and_official_help_hostname | OpenAI Help Center search documentation URL/title; content support and claims not independently checked. | Supporting citation, not an automatic vendor-tool recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown URL matching annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "**Mention rate:** mentions ÷ total runs"
},
{
"label": "Recommendation rate",
"definition_verbatim": "**Recommendation rate:** genuine recommendations ÷ total runs"
},
{
"label": "Citation rate",
"definition_verbatim": "**Citation rate:** responses citing your domain ÷ search-enabled runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "**Accuracy rate:** accurate descriptions ÷ mentions"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI also warns that search results and citations can be incomplete, outdated, or incorrect.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_tool_overlap": true,
"notes": "Supporting documentation attribution, not an active recommendation to adopt a provider/tool."
}
],
"illustrative_data": {
"baseline_table_is_empirical": false,
"notes": "8/45, 5/45 and the other baseline-table counts are explicitly an example of what a baseline might look like. They are not measurements made by this experiment. Minimum run counts and sample advice are generated recommendations, not validated universal thresholds."
},
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.0753956,
"token_cost_usd": 0.0653956,
"charged_searches_estimate": 1,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 12913,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1470,
"output_tokens_details": {
"reasoning_tokens": 503
},
"total_tokens": 14383
},
"max_tool_calls": 8,
"observed_web_actions": 1,
"max_output_tokens": 32768,
"observed_output_tokens": 1470,
"tool_limit_reached": false,
"output_limit_reached": false,
"authorized_total_budget_USD": 10.0,
"concerns": [
"No observed tool/output cap reached. Estimated US$0.0753956 is not reconciled billing; continue cumulative budget monitoring."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named consumer search surface and Grok API retrieval modes are coded separately from supplied targets and the supporting publisher. Generic spreadsheet advice is not converted to a branded tool.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Ask these seven questions",
"rationale": "Seven trust questions and red flags lead into a repeated manual audit, explicit rates and a purchasing check. No branded visibility-monitoring software is recommended."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "“What is Acme?” measures recognition."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt **three times in fresh conversations** on ChatGPT and Grok."
},
"fresh_sessions": {
"value": true,
"evidence": "three fresh-chat runs per prompt"
},
"model_search_version_logging": {
"value": true,
"evidence": "Search mode, location, model, and date are undisclosed."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Was live search enabled?"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "It claims to represent “ChatGPT visibility” while testing only an API model."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**Recommendation rate:** genuine recommendations ÷ total runs"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "**Citation rate:** responses citing your domain ÷ search-enabled runs"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Can I see the actual prompts and full responses?"
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "You cannot export prompts and responses."
},
"numerator_denominator_definition": {
"value": true,
"evidence": "**Mention rate:** mentions ÷ total runs"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": true,
"evidence": "“Visibility score: 76” without raw counts is not."
},
"uncertainty_variance": {
"value": true,
"evidence": "Does the score include uncertainty?"
},
"human_checking": {
"value": true,
"evidence": "Compare 10–15 identical prompts."
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual repeated checks",
"evidence": "Run each prompt **three times in fresh conversations** on ChatGPT and Grok.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Target provided by prompt, not a discovered tool. Only ChatGPT has first-party body citations in this answer."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for direct manual repeated checks",
"evidence": "Run each prompt **three times in fresh conversations** on ChatGPT and Grok.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target provided by prompt, not a discovered tool. Only ChatGPT has first-party body citations in this answer."
}
],
"contextual_non_tool_entities": [
{
"entity": "Acme",
"entity_type": "illustrative_brand_placeholder",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "“What is Acme?” measures recognition.",
"notes": "Illustrative prompt placeholder, not an observed SaaS product or tool recommendation; excluded from tool overlap."
}
],
"generic_methods": [
"Seven-question vendor audit",
"Fifteen-prompt panel and three fresh-chat repetitions",
"Matching search settings",
"Unnamed spreadsheet tracking",
"Monthly fixed-panel rates",
"Vendor-versus-manual sample comparison before purchasing"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "**Mention rate:** mentions ÷ total runs"
},
{
"label": "Recommendation rate",
"definition_verbatim": "**Recommendation rate:** genuine recommendations ÷ total runs"
},
{
"label": "Citation rate",
"definition_verbatim": "**Citation rate:** responses citing your domain ÷ search-enabled runs"
},
{
"label": "Accuracy rate",
"definition_verbatim": "**Accuracy rate:** accurate descriptions ÷ mentions"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4211,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 2,
"explicit_body_URL_occurrences": 2,
"unique_answer_URLs": 2,
"unique_cited_hostnames": 1,
"unique_normalized_cited_domains": 1,
"citation_mapping_complete": true,
"source_inventory_complete_for_answer_URLs": true,
"source_inventory_scope": "Two annotation/Markdown source URLs share one official help hostname. Different article slugs remain two exact URLs; source representations are not counted twice. Retrieval links remain separate.",
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 1,
"search_action_count": 1,
"reported_billable_search_requests": 1,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V1-R2.json",
"response_file": "responses/S3-OPENAIAPI-V1-R2.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"One completed web-search action and one reported billable search; four queries within that action are not four billed calls.",
"No xAI body citation exists despite xAI sources appearing in retrieval.",
"This API answer is not a consumer ChatGPT Search observation.",
"Source support, retrieval-capability claims and sampling advice remain unverified generated output."
],
"original_capture_answer_characters": 4247,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual official-subdomain normalization help.openai.com to openai.com; both exact URL slugs retained, without asserting a verified redirect/alias.",
"mapping": {
"help.openai.com": "openai.com"
}
}
}Both body sources are OpenAI Help articles; they do not independently verify every Grok API claim in the paragraph.
Generic unnamed spreadsheet and visibility-tool advice are not branded product recommendations.
The example baseline table is illustrative and must not enter empirical result counts.
Citation-rate denominator is explicitly search-enabled runs; preserve that definition.
Search-mode disclosure and fresh runs are requested, but no specific paid monitoring product is suggested.
S3-CHATGPT-V1-R3 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | X |
| Cited domains · mapping complete | openai.com |
| Observed inline cited domains · coverage qualified | openai.com |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
**Treat an AI visibility score as a sampled estimate—not an objective ranking.** ChatGPT and Grok can change answers based on model version, search mode, location, prompt wording, session context, and randomness. ChatGPT may rewrite searches and use approximate location; Grok’s results differ depending on whether web/X search is enabled. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) ## A trustworthy score should pass these tests 1. **Prompts are visible** - You can inspect every prompt. - They resemble real buyer questions. - Most are unbranded, such as “best invoicing software for solo consultants,” rather than prompts containing your company name. 2. **The exact environment is disclosed** - ChatGPT and Grok versions or product surfaces. - Consumer app versus API. - Web search on/off. - Country or location. - Date and frequency of collection. - Fresh conversations, with memory/personalization controlled. An API test without search is not a valid proxy for users asking a search-enabled consumer product. OpenAI and xAI both document that search changes how current information and sources are retrieved. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) 3. **Each prompt is run repeatedly** - One response per prompt is essentially anecdotal. - Look for at least **3 independent runs per prompt**, preferably spread across different days. - The report should show variance or a range, not false precision like “43.72.” For perspective, even observing a mention in 5 of 10 runs produces an approximate 95% interval of **24%–76%**. Small samples cannot support precise claims. 4. **You can inspect the raw evidence** Every score should click through to: - Full answer text - Timestamp - Model and mode - Brand mentions and competitors - Position in the answer - Citations and source URLs OpenAI explicitly recommends opening citations because search results can be incomplete, outdated, or incorrect. ([help.openai.com](https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt)) 5. **ChatGPT and Grok are reported separately** A combined “AI score” can hide “strong on ChatGPT, invisible on Grok.” Require per-engine results before any combined number. 6. **It measures more than name mentions** Useful dimensions include: - **Mention rate:** Were you named? - **Shortlist rate:** Were you presented as a viable option? - **Prominence:** First recommendation versus footnote - **Citation rate:** Did the answer cite your site? - **Accuracy:** Were claims about your product correct? - **Competitive share:** How often did you appear relative to relevant alternatives? 7. **The methodology is stable and versioned** Prompts, weights, and scoring rules should not quietly change between reports. Model changes should be annotated because score movement may come from the engine rather than anything you did. ## Free validation process Create a spreadsheet with **10–20 important buyer prompts**. For each: 1. Run it three times in fresh ChatGPT conversations. 2. Run it three times in fresh Grok conversations. 3. Keep search settings consistent. 4. Record mentions, shortlist position, competitors, citations, and factual errors. 5. Repeat monthly using the same prompt set. Your simplest useful metric is: > **Visibility rate = responses containing an unprompted, relevant mention ÷ total responses** Keep separate columns for ChatGPT and Grok. Also maintain a “commercial shortlist rate”—the percentage of responses where your SaaS is genuinely recommended, not merely mentioned. ## Fast red-flag checklist Do **not** trust the score if the provider: - Hides the prompts or raw responses - Runs each prompt only once - Cannot identify the model or search mode - Mixes branded and unbranded prompts without disclosure - Combines all engines into one unexplained number - Changes the prompt set between reporting periods - Claims tiny week-over-week movements are meaningful - Equates mentions with traffic, trials, or revenue - Uses the score mainly to generate an alarming sales audit **Bottom line:** trust the underlying prompt-level evidence and repeatability—not the headline number. For a solo founder, a small frozen prompt panel measured consistently is more useful than a sophisticated black-box “visibility score.”
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 19633,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1816,
"output_tokens_details": {
"reasoning_tokens": 872
},
"total_tokens": 21449
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 3
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1291956,
"cost_estimate_detail": {
"estimated_usd": 0.1291956,
"token_cost_usd": 0.09919560000000001,
"charged_searches_estimate": 3,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R3",
"slot_id": "S3-CHATGPT-V1-R3",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 3,
"submitted_at": "2026-11-02T09:40:11.574647+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:40:39.468143+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 19633,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1816,
"output_tokens_details": {
"reasoning_tokens": 872
},
"total_tokens": 21449
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 3
}
},
"estimated_cost": {
"estimated_usd": 0.1291956,
"token_cost_usd": 0.09919560000000001,
"charged_searches_estimate": 3,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read complete seven-test trust checklist, DIY process, red flags and coherent final paragraph. Three citation occurrences point to the same explicit URL and match annotations; correct model/effort and three actual searches, no caps. Interval example and vendor-method claims remain collected advice, not this experiment results or an external audit.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:40:56.685058+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:openai.com ChatGPT search how results sources official",
"site:x.ai Grok search sources official",
"AI visibility score methodology ChatGPT Grok brand monitoring"
],
"query": "site:openai.com ChatGPT search how results sources official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://www.theaivisibilityindex.ai/faq"
},
{
"type": "url",
"url": "https://llm-visibility-index.com/methodology/"
},
{
"type": "url",
"url": "https://visibilityradar.ai/methodology"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://sofindable.com/methodology"
},
{
"type": "url",
"url": "https://www.generaldataworks.com/ai-visibility/methodology"
},
{
"type": "url",
"url": "https://www.temso.ai/rankings/methodology"
},
{
"type": "url",
"url": "https://pondral.com/methodology"
},
{
"type": "url",
"url": "https://recosignal.com/methodology"
},
{
"type": "url",
"url": "https://promptedbrand.com/ai-visibility-index/methodology"
},
{
"type": "url",
"url": "https://ogoing.ai/methodology"
},
{
"type": "url",
"url": "https://scratchmm.com/ai-tools-buyers-guide"
},
{
"type": "url",
"url": "https://substack-post-media.s3.us-east-1.amazonaws.com/post-files/173102281/5d38c637-fe29-4d63-9fb2-41c2ee842734.pdf?response-content-disposition=attachment%3B+filename%3D%22Profound_Answer_Engine_Optimization_Guide_2025.pdf%22&x-id=GetObject"
},
{
"type": "url",
"url": "https://getfoundonai.com/assets/attachments/AIVO_Methodology.pdf"
},
{
"type": "url",
"url": "https://www.dominoeffectlabads.com/_files/ugd/4045c2_59478a2c888e40b4b6abc7e4d2736a1d.pdf"
},
{
"type": "url",
"url": "https://www.britopian.com/wp-content/uploads/2025/04/Report-Brand-Visibility-and-Reputation-in-the-Age-of-AI-Driven-Search.pdf"
},
{
"type": "url",
"url": "https://www.whitelabeliq.com/wp-content/uploads/2025/09/AI-Visibility-Audit-Report.pdf"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2610.00253"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2609.23162"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SaaS/comments/1u2al4v/i_built_an_ai_visibility_tracker_ran_it_on_my_own/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/AISearchLab/comments/1qjuas4/what_geo_metrics_actually_measure_and_what_they/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1x5i3kq/which_ai_visibility_tracking_tool_are_you_guys/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1xbgclz/i_have_a_problem_with_ai_visibility_reports_that/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/digital_marketing/comments/1ule9bi/sat_through_6_ai_search_optimization_pitches_this/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/GEO_optimization/comments/1uis2co/some_findings_from_working_on_branded_ai/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/aeo/comments/1wy7ei5/how_we_measure_ai_visibility_20_sealed_buyer/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SEO/comments/1wm7txg/how_are_you_guys_actually_measuring_ai_visibility/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SaaS/comments/1qp8uu7/how_to_track_ai_visibility_i_need_more_than_just/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SEO/comments/1t53rdl/most_ai_search_visibility_tools_are_statistically/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/SaaS/comments/1qz7exs/how_to_measure_ai_search_visibility/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/b2bmarketing/comments/1qws059/this_is_why_the_data_from_most_ai_visibility/"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2401.13138"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2605.19720"
},
{
"type": "url",
"url": "https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results"
},
{
"type": "url",
"url": "https://www.techradar.com/pro/surfer-ai-tracker-review"
},
{
"type": "url",
"url": "https://www.techradar.com/pro/ahrefs-brand-radar-review"
},
{
"type": "url",
"url": "https://www.techradar.com/pro/i-tested-the-7-best-aeo-tools"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai Grok web search citations official docs",
"site:docs.x.ai Grok API search tool official",
"site:x.ai/news Grok search web citations"
],
"query": "site:docs.x.ai Grok web search citations official docs",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/citations"
},
{
"type": "url",
"url": "https://x.ai/news/grok-business"
},
{
"type": "url",
"url": "https://x.ai/news/grok-1212?TB_iframe=true&height=512.1&width=921.6"
},
{
"type": "url",
"url": "https://x.ai/news/grok/model-card"
},
{
"type": "url",
"url": "https://x.ai/news/grok-4-1-fast"
},
{
"type": "url",
"url": "https://x.ai/news/grok-4-1"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/models"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/collections-search"
},
{
"type": "url",
"url": "https://x.ai/news/grok-collections-api"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"calculator: (0.5+1.96^2/(2*10)-1.96*sqrt((0.5*0.5+1.96^2/(4*10))/10))/(1+1.96^2/10)"
],
"query": "calculator: (0.5+1.96^2/(2*10)-1.96*sqrt((0.5*0.5+1.96^2/(4*10))/10))/(1+1.96^2/10)",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "api",
"name": "oai-calculator"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-V1-R3.json",
"request_file": "requests/S3-OPENAIAPI-V1-R3.json",
"response_file": "responses/S3-OPENAIAPI-V1-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4519,
"public_answer_characters": 4465,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V1-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | retrieval source, not automatically a body citation | ||
| https://www.theaivisibilityindex.ai/faq | www.theaivisibilityindex.ai | retrieval source, not automatically a body citation | ||
| https://llm-visibility-index.com/methodology/ | llm-visibility-index.com | retrieval source, not automatically a body citation | ||
| https://visibilityradar.ai/methodology | visibilityradar.ai | retrieval source, not automatically a body citation | ||
| https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls | help.openai.com | retrieval source, not automatically a body citation | ||
| https://sofindable.com/methodology | sofindable.com | retrieval source, not automatically a body citation | ||
| https://www.generaldataworks.com/ai-visibility/methodology | www.generaldataworks.com | retrieval source, not automatically a body citation | ||
| https://www.temso.ai/rankings/methodology | www.temso.ai | retrieval source, not automatically a body citation | ||
| https://pondral.com/methodology | pondral.com | retrieval source, not automatically a body citation | ||
| https://recosignal.com/methodology | recosignal.com | retrieval source, not automatically a body citation | ||
| https://promptedbrand.com/ai-visibility-index/methodology | promptedbrand.com | retrieval source, not automatically a body citation | ||
| https://ogoing.ai/methodology | ogoing.ai | retrieval source, not automatically a body citation | ||
| https://scratchmm.com/ai-tools-buyers-guide | scratchmm.com | retrieval source, not automatically a body citation | ||
| https://substack-post-media.s3.us-east-1.amazonaws.com/post-files/173102281/5d38c637-fe29-4d63-9fb2-41c2ee842734.pdf?response-content-disposition=attachment%3B+filename%3D%22Profound_Answer_Engine_Optimization_Guide_2025.pdf%22&x-id=GetObject | substack-post-media.s3.us-east-1.amazonaws.com | retrieval source, not automatically a body citation | ||
| https://getfoundonai.com/assets/attachments/AIVO_Methodology.pdf | getfoundonai.com | retrieval source, not automatically a body citation | ||
| https://www.dominoeffectlabads.com/_files/ugd/4045c2_59478a2c888e40b4b6abc7e4d2736a1d.pdf | www.dominoeffectlabads.com | retrieval source, not automatically a body citation | ||
| https://www.britopian.com/wp-content/uploads/2025/04/Report-Brand-Visibility-and-Reputation-in-the-Age-of-AI-Driven-Search.pdf | www.britopian.com | retrieval source, not automatically a body citation | ||
| https://www.whitelabeliq.com/wp-content/uploads/2025/09/AI-Visibility-Audit-Report.pdf | www.whitelabeliq.com | retrieval source, not automatically a body citation | ||
| https://arxiv.org/abs/2610.00253 | arxiv.org | retrieval source, not automatically a body citation | ||
| https://arxiv.org/abs/2609.23162 | arxiv.org | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/SaaS/comments/1u2al4v/i_built_an_ai_visibility_tracker_ran_it_on_my_own/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/AISearchLab/comments/1qjuas4/what_geo_metrics_actually_measure_and_what_they/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/GEO_optimization/comments/1x5i3kq/which_ai_visibility_tracking_tool_are_you_guys/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/GEO_optimization/comments/1xbgclz/i_have_a_problem_with_ai_visibility_reports_that/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/digital_marketing/comments/1ule9bi/sat_through_6_ai_search_optimization_pitches_this/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/GEO_optimization/comments/1uis2co/some_findings_from_working_on_branded_ai/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/aeo/comments/1wy7ei5/how_we_measure_ai_visibility_20_sealed_buyer/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/SEO/comments/1wm7txg/how_are_you_guys_actually_measuring_ai_visibility/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/SaaS/comments/1qp8uu7/how_to_track_ai_visibility_i_need_more_than_just/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/SEO/comments/1t53rdl/most_ai_search_visibility_tools_are_statistically/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/SaaS/comments/1qz7exs/how_to_measure_ai_search_visibility/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://www.reddit.com/r/b2bmarketing/comments/1qws059/this_is_why_the_data_from_most_ai_visibility/ | www.reddit.com | retrieval source, not automatically a body citation | ||
| https://arxiv.org/abs/2401.13138 | arxiv.org | retrieval source, not automatically a body citation | ||
| https://arxiv.org/abs/2605.19720 | arxiv.org | retrieval source, not automatically a body citation | ||
| https://www.techradar.com/pro/how-to-track-your-brands-visibility-in-ai-search-results | www.techradar.com | retrieval source, not automatically a body citation | ||
| https://www.techradar.com/pro/surfer-ai-tracker-review | www.techradar.com | retrieval source, not automatically a body citation | ||
| https://www.techradar.com/pro/ahrefs-brand-radar-review | www.techradar.com | retrieval source, not automatically a body citation | ||
| https://www.techradar.com/pro/i-tested-the-7-best-aeo-tools | www.techradar.com | retrieval source, not automatically a body citation | ||
| https://docs.x.ai/developers/tools/citations | docs.x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok-business | x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok-1212?TB_iframe=true&height=512.1&width=921.6 | x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok/model-card | x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok-4-1-fast | x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok-4-1 | x.ai | retrieval source, not automatically a body citation | ||
| https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | retrieval source, not automatically a body citation | ||
| https://docs.x.ai/cookbook/examples/docs-to-cited-answers | docs.x.ai | retrieval source, not automatically a body citation | ||
| https://docs.x.ai/developers/models | docs.x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/legal/faq | x.ai | retrieval source, not automatically a body citation | ||
| https://docs.x.ai/developers/tools/collections-search | docs.x.ai | retrieval source, not automatically a body citation | ||
| https://x.ai/news/grok-collections-api | x.ai | retrieval source, not automatically a body citation | ||
| https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf | data.x.ai | retrieval source, not automatically a body citation | ||
| https://data.x.ai/2025-08-20-grok-4-model-card.pdf | data.x.ai | retrieval source, not automatically a body citation | ||
| https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf | data.x.ai | retrieval source, not automatically a body citation | ||
| https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf | data.x.ai | retrieval source, not automatically a body citation | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | retrieval source, not automatically a body citation |
Assistant semantic review of complete finalized 4,519-character API answer, returned response, annotations and request metadata. No external queries, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V1-R3.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| X | web/X search | No | Yes | No | discussed as a retrieval-source context | unspecified | Grok’s results differ depending on whether web/X search is enabled.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_official_help_hostname_title_and_answer_role | Official OpenAI Help article about ChatGPT search; page contents/support not independently verified. | Supporting citation, not a newly recommended measurement product | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_official_help_hostname_title_and_answer_role | Official OpenAI Help article about ChatGPT search; page contents/support not independently verified. | Supporting citation, not a newly recommended measurement product | |
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | Platform documentation | provisional_from_official_help_hostname_title_and_answer_role | Official OpenAI Help article about ChatGPT search; page contents/support not independently verified. | Supporting citation, not a newly recommended measurement product |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown URL matching an annotation occurrence | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown URL matching an annotation occurrence | ||
| https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Explicit Markdown URL matching an annotation occurrence |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Visibility rate",
"definition_verbatim": "**Visibility rate = responses containing an unprompted, relevant mention ÷ total responses**"
},
{
"label": "Commercial shortlist rate",
"definition_verbatim": "the percentage of responses where your SaaS is genuinely recommended, not merely mentioned"
},
{
"label": "Citation rate",
"definition_verbatim": "**Citation rate:** Did the answer cite your site?"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI explicitly recommends opening citations because search results can be incomplete, outdated, or incorrect.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_tool_overlap": true,
"notes": "Supporting documentation attribution, not an adoption recommendation."
},
{
"entity": "xAI",
"entity_type": "platform_provider_attribution",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "OpenAI and xAI both document that search changes how current information and sources are retrieved.",
"source_urls": [],
"excluded_from_default_tool_overlap": true,
"notes": "Named in an attribution, but only OpenAI Help is cited. xAI retrieval results are not promoted to own-content body citations."
}
],
"illustrative_statistics": {
"example_observations": "5 of 10 runs",
"claimed_approximate_95_percent_interval": "24%–76%",
"is_empirical_result_of_this_experiment": false,
"method_verified": false,
"notes": "The answer gives a hypothetical interval illustration, without identifying its calculation method. It is retained as unverified generated advice, not sampled SaaS visibility data."
},
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1291956,
"token_cost_usd": 0.09919560000000001,
"charged_searches_estimate": 3,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 19633,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4349
},
"output_tokens": 1816,
"output_tokens_details": {
"reasoning_tokens": 872
},
"total_tokens": 21449
},
"max_tool_calls": 8,
"observed_web_actions": 3,
"max_output_tokens": 32768,
"observed_output_tokens": 1816,
"tool_limit_reached": false,
"output_limit_reached": false,
"authorized_total_budget_USD": 10.0,
"concerns": [
"No observed tool/output cap reached. Estimated US$0.1291956 is not reconciled billing; cumulative cost tracking remains required."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "The named X retrieval context is separate from supplied platforms and provider attributions. Generic spreadsheet or search-mode advice is not a branded tool recommendation. Source-title-only identities do not become tool mentions.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "A trustworthy score should pass these tests",
"rationale": "Seven trust tests, a small repeated spreadsheet audit and red flags form the answer. No named visibility checker or analytics tool is suggested."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Most are unbranded, such as “best invoicing software for solo consultants,” rather than prompts containing your company name."
},
"repeated_runs": {
"value": true,
"evidence": "Look for at least **3 independent runs per prompt**, preferably spread across different days."
},
"fresh_sessions": {
"value": true,
"evidence": "Fresh conversations, with memory/personalization controlled."
},
"model_search_version_logging": {
"value": true,
"evidence": "ChatGPT and Grok versions or product surfaces."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Web search on/off."
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "An API test without search is not a valid proxy for users asking a search-enabled consumer product."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "the percentage of responses where your SaaS is genuinely recommended, not merely mentioned"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "**Citation rate:** Did the answer cite your site?"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Every score should click through to:"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "**Visibility rate = responses containing an unprompted, relevant mention ÷ total responses**"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "Prompts, weights, and scoring rules should not quietly change between reports."
},
"composite_score_caution": {
"value": true,
"evidence": "Require per-engine results before any combined number."
},
"uncertainty_variance": {
"value": true,
"evidence": "The report should show variance or a range"
},
"human_checking": {
"value": true,
"evidence": "Free validation process"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"visibility_revenue_non_equivalence": {
"value": true,
"evidence": "Equates mentions with traffic, trials, or revenue"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested direct manual validation",
"evidence": "Run it three times in fresh ChatGPT conversations.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target, not new tool discovery. Only ChatGPT has a first-party body citation."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested direct manual validation",
"evidence": "Run it three times in fresh Grok conversations.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target, not new tool discovery. Only ChatGPT has a first-party body citation."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Seven-question trust audit",
"Visible frozen buyer-prompt panel",
"Exact environment and model/search disclosures",
"Repeated fresh-session multi-day tests",
"Raw-evidence inspection",
"Per-engine rather than unexplained blended scores",
"Unnamed spreadsheet logging",
"Separate visibility and commercial-shortlist rates",
"Stable versioned methodology"
],
"metric_definitions_observed": [
{
"label": "Visibility rate",
"definition_verbatim": "**Visibility rate = responses containing an unprompted, relevant mention ÷ total responses**"
},
{
"label": "Commercial shortlist rate",
"definition_verbatim": "the percentage of responses where your SaaS is genuinely recommended, not merely mentioned"
},
{
"label": "Citation rate",
"definition_verbatim": "**Citation rate:** Did the answer cite your site?"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4465,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 3,
"explicit_body_URL_occurrences": 3,
"unique_answer_URLs": 1,
"unique_cited_hostnames": 1,
"unique_normalized_cited_domains": 1,
"citation_mapping_complete": true,
"source_inventory_complete_for_answer_URLs": true,
"source_inventory_scope": "Three annotation/Markdown occurrences all cite the same OpenAI Help URL. Count one unique source; do not add retrieval URLs to the body-citation set.",
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 3,
"search_action_count": 3,
"reported_billable_search_requests": 3,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V1-R3.json",
"response_file": "responses/S3-OPENAIAPI-V1-R3.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Three completed search actions and three reported billable requests; no cap observed.",
"Only OpenAI Help is cited even though the prose attributes documentation to both OpenAI and xAI.",
"The interval example and suggested run counts are not independently verified statistical requirements.",
"This is an API + web-search answer, not a consumer ChatGPT Search observation."
],
"original_capture_answer_characters": 4519,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual official-help subdomain normalization help.openai.com to openai.com; exact tracking-bearing URL retained locally.",
"mapping": {
"help.openai.com": "openai.com"
}
}
}X appears as a search-source context, not an active platform or publication recommendation.
Provider attributions are not adopted tools; xAI has no own-content body citation here.
No branded spreadsheet or monitoring software is named.
Hypothetical sample counts and intervals are not empirical results collected in this experiment.
S3-GROK-V1-R1 · Grok Expert · consumer interface · completed
External numerical claims and tool pricing/coverage are model output, not independently fact-checked.
Copy text and rendered answer correspond; inline reference links preserved separately.
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs AI Visibility Checker, Google Analytics 4, Google People also ask, Google Search Console, HubSpot AEO Grader, Mangools AI Search Grader, Reddit, X |
| Mentioned tools / services / surfaces | Ahrefs AI Visibility Checker, Claude, Gemini, Google Analytics 4, Google People also ask, Google Search Console, HubSpot AEO Grader, Mangools AI Search Grader, Perplexity, Reddit, X, YouTube |
| Cited domains · mapping complete | Unavailable · no eligible answers |
| Observed inline cited domains · coverage qualified | ahrefs.com, businessinsider.com, cognizo.ai, higoodie.com, hubspot.com, maxaeo.ai, searchscope.com.au |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved, Tool/product documentation or sales page |
{ "cited_domains": "Citation mapping is not confirmed complete; use separately labeled observed-link comparisons." }
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
**Most AI visibility scores are only directionally useful at best.** Treat any single number (especially a blended “ChatGPT + Grok” score) as a rough snapshot, not a trustworthy KPI you can manage to. As a solo SaaS founder with no GEO budget, the practical test is simple: can you roughly reproduce the directional finding yourself with free manual checks? If not, the score is marketing more than measurement. ### Why these scores are noisy AI answers are non-deterministic. The same prompt on the same engine often produces different brands, rankings, and citations. One 2026 benchmark found average top-five shortlist overlap of about 43% when the *same* engine answered the same questions again, and only ~17% *across* engines. Edelman, for example, scored 22 on Claude and ~0.7 on ChatGPT in that study; averaging them hid the real picture. Additional reliability problems: - Many tools query provider APIs, which differ from the consumer apps people actually use (no memory, different system prompts, different retrieval). One test showed only ~25% top-five overlap between paid ChatGPT and the API. - Prompt sets are usually synthetic or auto-generated, not real buyer queries. - Single-run or tiny-sample scores are mostly noise. Research on sample sizes shows you need dozens of distinct prompts and multiple runs per prompt before the margin of error becomes useful. - Engines weight different signals. Grok leans heavily on live social/X presence, community citations, and freshness; classic search rank matters less for it than for ChatGPT or Google AI features. Correlations that *do* show up across studies (e.g., Ahrefs data on tens of thousands of brands) are stronger for third-party mentions (especially YouTube and branded web mentions) than for traditional backlinks. ### Checklist: Is a given score trustworthy enough to act on? | Signal | More trustworthy | Less trustworthy | |--------|------------------|------------------| | **Scope** | Separate scores for ChatGPT and Grok (and others) | One blended number | | **Transparency** | Exact prompts, # of runs, surface (app vs API), model version, date | Opaque “AI score” with no methodology | | **Sampling** | Multiple runs per prompt (ideally 3–7+); dozens of prompts | Single run or handful of prompts | | **Output** | Mention rate, citation rate, share of voice, examples of answers | Only a composite 0–100 with no range/error bars | | **Reproducibility** | You can test similar buyer prompts yourself and get the same direction | Claims precise rankings or revenue impact you can’t verify | | **Incentives** | Neutral or clear methodology | Vendor’s main product is “fix your low score” | If the tool can’t answer “what prompts, how many times, on which surface?” in plain language, discount it heavily. ### Free way to get a usable baseline yourself (no budget) 1. **Build 15–30 buyer-intent prompts** from real language (support tickets, sales calls, Reddit, Google Search Console, “People also ask”). Mix category (“best [your category] for [use case]”), comparison, problem-solving, and a few branded ones. Keep the set fixed. 2. **Run them manually**: - ChatGPT: private/incognito window, memory off, fresh chats. Do each prompt 3–5 times. - Grok: similar clean sessions. Note that results can shift quickly with X conversation volume and content freshness. - Log in a free spreadsheet: mentioned? position in any list? your domain cited? competitors named? sentiment/accuracy of the description? 3. **Calculate simple metrics**: - Mention rate = % of answers that name you - Citation rate = % that link your site - Share of voice vs the 2–3 competitors that appear most often 4. **Cross-check with free one-shot tools** for a second opinion (they are limited but useful for direction): - Ahrefs free AI Visibility Checker (brand name in → mentions across several engines, top topics/domains) - HubSpot AI Search / AEO Grader (one-time perception score across ChatGPT, Perplexity, Gemini; more training-data oriented) - Mangools AI Search Grader (broader engine coverage in some versions, including Grok) Re-run your manual set monthly. Small week-to-week swings are normal noise; look for sustained directional changes. ### What actually matters more than the score - **Third-party mentions** and clear, factual, citable content (stats, quotes, dated claims). Your own site is a minority of what engines cite. - **For Grok specifically**: genuine activity and discussion on X plus fresh, answer-first pages. Weak X presence is a structural handicap on Grok regardless of traditional SEO strength. - **Real outcomes** you can track for free: AI referral traffic in GA4 (from chatgpt.com, etc.), server-log hits from AI crawlers, branded search volume, and leads who say they found you via an AI chat. **Bottom line**: Use any commercial AI visibility score only as a directional hint. The trustworthy version for a solo founder is the one you generate yourself with a fixed set of real buyer prompts, multiple runs, and simple mention/citation rates on ChatGPT and Grok separately. Focus energy on earning credible third-party mentions and (for Grok) authentic X presence rather than chasing opaque composite scores.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V1-R1",
"slot_id": "S3-GROK-V1-R1",
"platform": "Grok",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T06:14:01.029Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_mode": "Expert native retrieval; no independent force-search toggle observed",
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:17:11.262Z",
"source_panel_declared_hits": 90,
"source_inventory_scope": "Seven inline links recorded separately from source badge of 90 hits; source-panel inventory not yet collected.",
"capture_method": "Native complete rendered answer plus Copy response pasted into offline buffer; coherent opening and final conclusion corroborated.",
"search_query_labels": [
"AI visibility score GEO ChatGPT Grok trustworthiness",
"\"AI visibility\" OR \"GEO score\" OR \"generative engine optimization\" tools methodology",
"how to measure brand visibility in ChatGPT Grok free methods",
"critiques AI SEO GEO visibility tools accuracy",
"Ahrefs free AI Visibility Checker OR HubSpot AEO Grader OR Mangools AI Search Grader",
"Grok AI visibility factors OR how Grok cites brands",
"\"AI visibility\" OR GEO \"sample size\" OR \"number of runs\" OR variance reliability"
],
"search_evidence": "Native activity displays web searches in groups 4, 3, then 2; only seven exact query labels expanded so far. Workspace-explored activity label also observed; its role not established.",
"raw_file": "records/S3-GROK-V1-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 5239,
"public_answer_characters": 5239,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V1-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| There Is No Single “AI Visibility” Score; Ask Four AIs, Get Four Shortlists | Markets Insider (Markets.businessinsider) | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| Are AI Visibility Tools Accurate? What They Miss (2026) (Searchscope) | https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | searchscope.com.au | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| AI Visibility Sample Size: How Many Prompts and Runs to Trust - MaxAEO Blog (Maxaeo) | https://maxaeo.ai/blog/ai-visibility-sample-size/ | maxaeo.ai | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| How to Rank on Grok: Get Your Brand Cited in X’s AI Search | Goodie AI (Higoodie) | https://higoodie.com/blog/how-to-rank-on-grok/ | higoodie.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| What Actually Moves Your AI Visibility Score? 7 Factors, Ranked by Data (Cognizo) | https://cognizo.ai/blog/ai-visibility-score | cognizo.ai | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| Free AI Visibility Checker - See Your Brand in AI Search - Ahrefs (Ahrefs) | https://ahrefs.com/free-ai-visibility | ahrefs.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| Hubspot | https://hubspot.com/aeo-grader | hubspot.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of the complete captured answer and seven recorded inline links; no external fact-check, independent human double-coding or webpage visits.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Ahrefs AI Visibility Checker | Ahrefs free AI Visibility Checker | Yes | Yes | Yes | conditional | stated free | Ahrefs free AI Visibility Checker (brand name in → mentions across several engines, top topics/domains)
| https://ahrefs.com/free-ai-visibility |
| HubSpot AEO Grader | HubSpot AI Search / AEO Grader | Yes | Yes | Yes | conditional | stated free | HubSpot AI Search / AEO Grader (one-time perception score across ChatGPT, Perplexity, Gemini; more training-data oriented)
| https://hubspot.com/aeo-grader |
| Mangools AI Search Grader | Mangools AI Search Grader | No | Yes | Yes | conditional | stated free | Mangools AI Search Grader (broader engine coverage in some versions, including Grok)
| |
| Google Analytics 4 | GA4 | No | Yes | Yes | suggested | stated free | Real outcomes** you can track for free: AI referral traffic in GA4 (from chatgpt.com, etc.), server-log hits from AI crawlers, branded search volume, and leads who say they found you via an AI chat.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | suggested | stated free | Build 15–30 buyer-intent prompts** from real language (support tickets, sales calls, Reddit, Google Search Console, “People also ask”).
| |
| No | Yes | Yes | suggested | stated free | Build 15–30 buyer-intent prompts** from real language (support tickets, sales calls, Reddit, Google Search Console, “People also ask”).
| |||
| Google People also ask | People also ask | No | Yes | Yes | suggested | stated free | Build 15–30 buyer-intent prompts** from real language (support tickets, sales calls, Reddit, Google Search Console, “People also ask”).
| |
| X | X | No | Yes | Yes | suggested | unspecified | Focus energy on earning credible third-party mentions and (for Grok) authentic X presence rather than chasing opaque composite scores.
| |
| YouTube | YouTube | No | Yes | No | discussed | unspecified | Correlations that *do* show up across studies (e.g., Ahrefs data on tens of thousands of brands) are stronger for third-party mentions (especially YouTube and branded web mentions) than for traditional backlinks.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | Edelman, for example, scored 22 on Claude and ~0.7 on ChatGPT in that study; averaging them hid the real picture.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | HubSpot AI Search / AEO Grader (one-time perception score across ChatGPT, Perplexity, Gemini; more training-data oriented)
| |
| Gemini | Gemini | No | Yes | No | discussed | unspecified | HubSpot AI Search / AEO Grader (one-time perception score across ChatGPT, Perplexity, Gemini; more training-data oriented)
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_title_and_URL_only | Title describes reporting of a benchmark; /news/stocks/ placement could be a syndicated or promotional release. Original research authorship is unresolved. | supporting citation, not an active source recommendation | |
| https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | Other / unresolved | provisional_from_returned_title_and_URL_only | Title and /ai-seo/ path suggest a practitioner or commercial explanatory article. The returned metadata does not establish authorship, original experiments or source role. | supporting citation, not an active source recommendation | |
| https://maxaeo.ai/blog/ai-visibility-sample-size/ | Other / unresolved | provisional_from_returned_title_and_URL_only | A branded blog title about sample size suggests methods advice, but original empirical research versus a tutorial is not established from the returned title/URL. | supporting citation, not an active source recommendation | |
| https://higoodie.com/blog/how-to-rank-on-grok/ | Other / unresolved | provisional_from_returned_title_and_URL_only | Goodie AI-branded blog and how-to title suggest commercial tutorial content. The metadata does not establish whether this is practitioner instruction, product documentation or original vendor research. | supporting citation, not an active source recommendation | |
| https://cognizo.ai/blog/ai-visibility-score | Other / unresolved | provisional_from_returned_title_and_URL_only | A branded blog title claims ranked data factors; whether it reports original vendor measurements or summarizes other studies is unresolved. | supporting citation, not an active source recommendation | |
| https://ahrefs.com/free-ai-visibility | Tool/product documentation or sales page | provisional_from_returned_title_and_URL_only | Returned title identifies the free AI Visibility Checker product page. | conditional tool recommendation | |
| https://hubspot.com/aeo-grader | Tool/product documentation or sales page | provisional_from_returned_title_and_URL_only | Returned URL /aeo-grader and label identify a product grader page; the body names its claimed coverage. | conditional tool recommendation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[]
Answer type, advice flags, scope and exclusions
{
"coding_scope": "All substantively named tools/services/features, including input sources and channels. Generic methods and supplied target platforms are separate. Supporting publishers are not included as discovered tool recommendations.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation",
"Tool-led solution"
],
"evidence": "Checklist: Is a given score trustworthy enough to act on?",
"rationale": "The dominant response supplies trustworthiness dimensions; a manual baseline and limited free checkers are supporting components."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Mix category (“best [your category] for [use case]”), comparison, problem-solving, and a few branded ones."
},
"repeated_runs": {
"value": true,
"evidence": "Multiple runs per prompt (ideally 3–7+); dozens of prompts"
},
"fresh_sessions": {
"value": true,
"evidence": "private/incognito window, memory off, fresh chats."
},
"model_search_version_logging": {
"value": true,
"evidence": "Exact prompts, # of runs, surface (app vs API), model version, date"
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Mention rate = % of answers that name you"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Mention rate, citation rate, share of voice, examples of answers"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Citation rate = % that link your site"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": true,
"evidence": "Only a composite 0–100 with no range/error bars"
},
"uncertainty_variance": {
"value": true,
"evidence": "AI answers are non-deterministic."
},
"human_checking": {
"value": true,
"evidence": "can you roughly reproduce the directional finding yourself with free manual checks?"
},
"downstream_conversions": {
"value": true,
"evidence": "leads who say they found you via an AI chat."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual baseline checks",
"evidence": "ChatGPT: private/incognito window, memory off, fresh chats. Do each prompt 3–5 times.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target platform supplied by the question; its manual use is explicitly advised, but it is not treated as newly discovered."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual baseline checks",
"evidence": "Grok: similar clean sessions. Note that results can shift quickly with X conversation volume and content freshness.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target platform supplied by the question; its manual use is explicitly advised, but it is not treated as newly discovered."
}
],
"contextual_non_tool_entities": [
{
"entity": "Edelman",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Edelman, for example, scored 22 on Claude and ~0.7 on ChatGPT in that study; averaging them hid the real picture.",
"notes": "Brand in an illustrative benchmark example; not a tool or recommendation."
},
{
"entity": "Google AI features",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "classic search rank matters less for it than for ChatGPT or Google AI features.",
"notes": "Generic surface-family comparison, not a specific named product recommendation; excluded from tool sets."
}
],
"generic_methods": [
"Free unnamed spreadsheet logging",
"Manual fixed buyer-prompt audit",
"Simple mention/citation rates and share of voice",
"Server-log and crawler monitoring",
"Branded search and lead-source monitoring"
],
"capture_scope": {
"body_complete": true,
"observed_inline_answer_links": 7,
"observed_inline_answer_domains": 7,
"source_badge_hits": 90,
"retrieval_inventory_complete": false,
"citation_mapping_complete": null,
"search_observed": true,
"search_enabled_UI": null,
"search_query_labels_captured": 7,
"notes": [
"Seven answer links are not the same as the 90-hit source badge or a complete retrieval-panel inventory.",
"Native activity reportedly included four, three and two searches, but only seven exact query labels were captured; the observed query list is partial.",
"A workspace-explored activity label was observed. Its source contents and influence are unestablished, so account-context isolation is not audited.",
"The plain copied body has no inline citation positions. Seven native link targets are preserved; exact claim-to-marker mapping is not exhaustively established.",
"Displayed targets lacked URL schemes and were standardized to HTTPS during capture; no destination was visited to verify the scheme.",
"All numerical thresholds, causal-sounding engine claims, tool coverage and free-status statements remain Grok claims."
],
"original_capture_answer_characters": 5239,
"answer_characters": 5239,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manually reviewed mapping; hostnames retained. Only the Markets Insider subdomain is collapsed. No public-suffix software or external lookup was used.",
"mapping": {
"markets.businessinsider.com": "businessinsider.com",
"searchscope.com.au": "searchscope.com.au",
"maxaeo.ai": "maxaeo.ai",
"higoodie.com": "higoodie.com",
"cognizo.ai": "cognizo.ai",
"ahrefs.com": "ahrefs.com",
"hubspot.com": "hubspot.com"
}
}
}Tool-set comparisons should preserve role labels: score checkers, analytics, prompt inputs and distribution channels serve different tasks.
False citation values mean no own-content citation observed among the captured answer links, not independently proven absence from all uncollected retrieval results.
The original answer distinguishes mentions and citations, but does not explicitly operationalize active recommendations as a separate third metric.
No full source-publisher C/M/R census was conducted; five supporting non-tool publishers are not counted as tools.
S3-GROK-V1-R2 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs AI visibility tools (unspecified), Google Analytics 4, Google Search Console, HubSpot grader (unspecified), Semrush AI visibility tools (unspecified) |
| Mentioned tools / services / surfaces | Ahrefs AI visibility tools (unspecified), Claude, Google Analytics 4, Google Search Console, HubSpot grader (unspecified), Perplexity, Semrush AI visibility tools (unspecified), X |
| Cited domains · mapping complete | businessinsider.com, searchenginejournal.com, visilay.com |
| Observed inline cited domains · coverage qualified | businessinsider.com, searchenginejournal.com, visilay.com |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
Most AI visibility scores for ChatGPT and Grok are only loosely trustworthy. There is no standard definition, engines disagree sharply, and answers vary even on repeats. A single blended number is especially weak. For a solo founder with zero budget, the most reliable signal is still a simple, repeatable manual test you control. Why scores are often unreliable No shared formula. “Visibility score” can mean mention rate, share of voice, weighted prominence, readiness signals on your site, or a composite. The same answers have produced scores ranging widely depending on the vendor’s math.Visilay High variance is structural. In a 2026 Crackle PR benchmark, the same engine’s top-five shortlists overlapped only about 43% on repeats, while cross-engine overlap was ~17%. Edelman, for example, scored ~20+ on Claude and Grok but near zero on ChatGPT and Perplexity; a blended score hid the split.Markets.businessinsider Opaque or synthetic methods. Many tools use invented prompt lists with no real query volume behind them, limited sample sizes, undisclosed model versions, and little accounting for personalization or session differences. Practitioners consistently cite methodology opacity and non-determinism as top reasons they distrust the platforms.Searchenginejournal Engines are not interchangeable. ChatGPT leans on training data plus search-grounded lists, reviews, and entity recognition. Grok weights social/community signals (especially X) and freshness more heavily and classic search rank less than most others. A blended score averages away the useful differences. Critical reviews of GEO studies have also found that many claimed tactics do not show stable, cross-platform causal effects on discoverability once you leave controlled setups. How to judge whether a specific score is trustworthy Ask these questions of the provider or the report: Is the methodology public? Exact prompts, number of runs per prompt, model versions, dates, and how they aggregate variance (means, ranges, or just a single run). Per-engine breakdown? Blended scores hide large gaps between ChatGPT and Grok. Multiple runs? One-off answers are noisy. Prefer tools that show distributions or run prompts 3–5+ times. Relevant prompts? Buyer-intent questions that match how your customers actually research, not generic or vendor-invented ones. Reproducible by you? Can you roughly verify key claims by asking the same prompts yourself? Linked to outcomes? Any correlation with AI referral traffic, self-reported “I found you on ChatGPT/Grok,” or branded search lift is more meaningful than the score alone. Red flags: Single-run precision claims, no methodology, huge jumps without explanation, only a blended number, or heavy upsell pressure after a free check. ChatGPT vs Grok specifics ChatGPT: Mentions often track entity strength, third-party coverage (lists, reviews, Wikipedia-style sources), and branded mentions. It cites frequently when browsing. Well-known brands have an advantage from training data. Grok: More sensitive to current conversation on X and other social/community sources, plus recency. Classic SEO signals matter less. A brand with little X presence is frequently weaker here even if its site and traditional SEO are solid. What a zero-budget solo founder should actually do The manual method is more trustworthy for your scale than most vendor scores, because you control the prompts, runs, and logging. Build a fixed list of 10–20 high-intent prompts (e.g., “best [your category] for [your ICP],” “alternatives to [competitor],” “how to [key job your product does]”). Run each prompt 3 times in ChatGPT and Grok in clean sessions (incognito or new chats, memory off if possible). Log in a spreadsheet: mentioned? cited? position/prominence, competitors named, fact accuracy, and key sources. Repeat monthly. Look for consistent trends and which prompts you win or lose, not a single percentage. Cross-check free signals you already have: GA4 AI referral traffic (chatgpt.com, grok.x.ai, etc.), Search Console branded queries, and server logs for AI crawlers. Free one-off checkers (Ahrefs, Semrush limited free checks, HubSpot grader, and others) can give a directional snapshot. Treat them as low-trust starting points and always verify the important prompts yourself. They are useful for spotting gaps, not for precise scores. Bottom line: Treat any AI visibility score as a noisy directional indicator. For ChatGPT and Grok specifically, prioritize transparent, multi-run, per-engine data—or just measure it yourself with a small fixed prompt set. That approach costs time instead of money and is usually more reliable for a solo founder than opaque vendor numbers.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V1-R2",
"slot_id": "S3-GROK-V1-R2",
"platform": "Grok",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 2,
"submitted_at": "2026-11-02T06:39:21.639Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:41:50.443Z",
"source_panel_declared_hits": 55,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-V1-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4708,
"public_answer_characters": 4708,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V1-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Visilay | https://www.visilay.com/blog/ai-visibility-score/ | www.visilay.com | Inline answer link | Rendered DOM href |
| Markets.businessinsider | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Rendered DOM href |
| Searchenginejournal | https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/ | searchenginejournal.com | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 4,708-character rendered answer, scoped DOM hrefs and existing browser snapshots/activity. No external webpage queries, fact audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Ahrefs AI visibility tools (unspecified) | Ahrefs | No | Yes | Yes | conditional | stated free | Free one-off checkers (Ahrefs, Semrush limited free checks, HubSpot grader, and others) can give a directional snapshot. Treat them as low-trust starting points and always verify the important prompts yourself. They are useful for spotting gaps, not for precise scores.
| |
| Semrush AI visibility tools (unspecified) | Semrush limited free checks | No | Yes | Yes | conditional | mixed/limited tier | Free one-off checkers (Ahrefs, Semrush limited free checks, HubSpot grader, and others) can give a directional snapshot. Treat them as low-trust starting points and always verify the important prompts yourself. They are useful for spotting gaps, not for precise scores.
| |
| HubSpot grader (unspecified) | HubSpot grader | No | Yes | Yes | conditional | stated free | Free one-off checkers (Ahrefs, Semrush limited free checks, HubSpot grader, and others) can give a directional snapshot. Treat them as low-trust starting points and always verify the important prompts yourself. They are useful for spotting gaps, not for precise scores.
| |
| Google Analytics 4 | GA4 | No | Yes | Yes | suggested | stated free | Cross-check free signals you already have: GA4 AI referral traffic (chatgpt.com, grok.x.ai, etc.), Search Console branded queries, and server logs for AI crawlers.
| |
| Google Search Console | Search Console | No | Yes | Yes | suggested | stated free | Cross-check free signals you already have: GA4 AI referral traffic (chatgpt.com, grok.x.ai, etc.), Search Console branded queries, and server logs for AI crawlers.
| |
| X | X | No | Yes | No | discussed | unspecified | Grok: More sensitive to current conversation on X and other social/community sources, plus recency. Classic SEO signals matter less.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | Edelman, for example, scored ~20+ on Claude and Grok but near zero on ChatGPT and Perplexity; a blended score hid the split.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | Edelman, for example, scored ~20+ on Claude and Grok but near zero on ChatGPT and Perplexity; a blended score hid the split.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://www.visilay.com/blog/ai-visibility-score/ | Other / unresolved | provisional_from_returned_label_and_URL_only | Score-calculation blog path and label suggest explanation, but original research, authorship and vendor versus practitioner role are not established. | supporting citation, not an active recommendation to read/follow the publisher | |
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_label_and_URL_only | Markets Insider citation supports an attributed multi-engine benchmark; stock-news path permits syndicated/promotional material. Primary research authorship unverified. | supporting citation, not an active recommendation to read/follow the publisher | |
| https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/ | Editorial or industry reporting | provisional_from_returned_label_and_URL_only | Search Engine Journal article path/title describes SEO practitioners' distrust of visibility platforms; reporting classification is provisional without reading the page. | supporting citation, not an active recommendation to read/follow the publisher |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Visibility score",
"definition_verbatim": "“Visibility score” can mean mention rate, share of voice, weighted prominence, readiness signals on your site, or a composite.",
"notes": "Describes incompatible score possibilities; supplies no numerical operational formula or fixed denominator."
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services; supplied target platforms and non-tool benchmark subjects/authors separate. Generic vendor/tool names retain unresolved product identities. Citation-label-only publishers are not substantive tool mentions.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation",
"Tool-led solution"
],
"evidence": "How to judge whether a specific score is trustworthy",
"rationale": "Trustworthiness criteria and red flags dominate. Manual replication and generic free checkers are qualified supporting suggestions."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": false,
"evidence": null
},
"buyer_intent_prompt_relevance": {
"value": true,
"evidence": "Buyer-intent questions that match how your customers actually research, not generic or vendor-invented ones."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt 3 times in ChatGPT and Grok in clean sessions (incognito or new chats, memory off if possible)."
},
"fresh_sessions": {
"value": true,
"evidence": "clean sessions (incognito or new chats, memory off if possible)"
},
"model_search_version_logging": {
"value": true,
"evidence": "Exact prompts, number of runs per prompt, model versions, dates, and how they aggregate variance (means, ranges, or just a single run)."
},
"search_mode_logging_explicit": {
"value": false,
"evidence": null
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Log in a spreadsheet: mentioned? cited? position/prominence, competitors named, fact accuracy, and key sources."
},
"raw_answer_inspection": {
"value": true,
"evidence": "Can you roughly verify key claims by asking the same prompts yourself?"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": false,
"evidence": null
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": true,
"evidence": "Per-engine breakdown? Blended scores hide large gaps between ChatGPT and Grok."
},
"uncertainty_variance": {
"value": true,
"evidence": "One-off answers are noisy. Prefer tools that show distributions or run prompts 3–5+ times."
},
"human_checking": {
"value": true,
"evidence": "Reproducible by you? Can you roughly verify key claims by asking the same prompts yourself?"
},
"downstream_conversions": {
"value": true,
"evidence": "Linked to outcomes? Any correlation with AI referral traffic, self-reported “I found you on ChatGPT/Grok,” or branded search lift is more meaningful than the score alone."
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "GA4 AI referral traffic"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual replication",
"evidence": "Run each prompt 3 times in ChatGPT and Grok in clean sessions (incognito or new chats, memory off if possible).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied by question; exclude from newly discovered tool overlap."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual replication",
"evidence": "Run each prompt 3 times in ChatGPT and Grok in clean sessions (incognito or new chats, memory off if possible).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Target supplied by question; exclude from newly discovered tool overlap."
}
],
"contextual_non_tool_entities": [
{
"entity": "Crackle PR",
"entity_type": "attributed_research_author",
"citation": null,
"mention": true,
"recommendation": false,
"evidence": "In a 2026 Crackle PR benchmark, the same engine’s top-five shortlists overlapped only about 43% on repeats, while cross-engine overlap was ~17%.",
"source_urls": [
"https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461"
],
"notes": "Answer attributes the benchmark to Crackle PR, but citation destination is Markets Insider. Authorship/publication provenance is not independently established; C remains unresolved rather than treated as an own-content citation."
},
{
"entity": "Edelman",
"entity_type": "illustrative_brand",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "Edelman, for example, scored ~20+ on Claude and Grok but near zero on ChatGPT and Perplexity; a blended score hid the split.",
"notes": "Illustrative brand benchmark value, not a tool recommendation. Numerical claim unverified."
}
],
"generic_methods": [
"Public prompt/run/model/date methodology inspection",
"Per-engine breakdown and variance checks",
"Manual fixed buyer-prompt replication",
"Unnamed spreadsheet logging",
"Monthly trend tracking",
"Traffic, branded queries and crawler logs",
"Source types such as Wikipedia-style material, lists and reviews"
],
"metric_definitions_observed": [
{
"label": "Visibility score",
"definition_verbatim": "“Visibility score” can mean mention rate, share of voice, weighted prominence, readiness signals on your site, or a composite.",
"notes": "Describes incompatible score possibilities; supplies no numerical operational formula or fixed denominator."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4708,
"observed_inline_link_occurrences": 3,
"unique_DOM_answer_URLs": 3,
"unique_cited_domains": 3,
"source_badge_hits": 55,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Three observed rendered-article hrefs, not the complete 55-hit retrieval/source-panel inventory.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"complete_rendered_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [
"free AI visibility checkers ChatGPT Grok 2026"
],
"notes": [
"Activity snapshot shows a grouped four-search control, browsed links, and a separately displayed web query with 10 results. Exact query coverage is partial; search is observed independently of the 55-source badge.",
"Workspace-explored activity is visible but its underlying contents and influence remain unresolved.",
"Three publisher names embedded in article innerText are citation labels, not automatically recommendations or substantive tool mentions.",
"No consumer tool prices, score statistics, research attributions, engine-weighting claims or reliability assertions were fact-checked."
],
"original_capture_answer_characters": 4708,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping preserves observed DOM hostnames/hrefs; www.visilay.com normalizes to visilay.com and markets.businessinsider.com to businessinsider.com. No external public-suffix lookup used.",
"mapping": {
"www.visilay.com": "visilay.com",
"markets.businessinsider.com": "businessinsider.com",
"searchenginejournal.com": "searchenginejournal.com"
}
}
}Generic Ahrefs/Semrush/HubSpot grader names do not establish specific product identities.
Low-trust, directional snapshot suggestions remain conditional recommendations rather than explicit exclusions.
X is explanatory engine-behavior context here; claimed importance alone is not an active instruction to use it.
Wikipedia-style sources describes a source style, not a direct Wikipedia tool recommendation.
The answer asks for methodology, versions and dates but does not explicitly demand search-state logging or a numerator/denominator formula.
S3-GROK-V1-R3 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Ahrefs AI visibility tools (unspecified), Google Analytics 4, Google Search Console, Google Sheets, HubSpot AI Search Grader, Semrush AI visibility tools (unspecified), X |
| Mentioned tools / services / surfaces | Ahrefs AI visibility tools (unspecified), Crunchbase, Google AI Overviews, Google Analytics 4, Google Search Console, Google Sheets, HubSpot AI Search Grader, LinkedIn, Semrush AI visibility tools (unspecified), Wikidata, X, YouTube |
| Cited domains · mapping complete | amicited.com, businessinsider.com |
| Observed inline cited domains · coverage qualified | amicited.com, businessinsider.com |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved |
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
Treat any single AI visibility score for ChatGPT or Grok as a directional estimate, not a trustworthy absolute number. These scores are estimates built by prompting models (often via API or consumer interfaces), and the underlying systems are highly non-deterministic. Engines frequently disagree with each other, and the same prompt on the same engine often produces different brand lists and sources.Markets.businessinsider Why most scores are only partially trustworthy Independent studies and benchmarks show significant variance: Same-engine shortlist overlap on repeated runs is often only around 40–55% (or lower for exact matches). Cross-engine overlap is typically much lower (e.g., ~17% in one multi-engine test).Markets.businessinsider A single run of a prompt captures only a fraction of the brands that appear across multiple runs. “Ghost” brands that appear once and disappear are common. No vendor has access to real OpenAI or xAI user conversation logs. Scores are synthetic samples of what the models can say under controlled conditions, not what actual users see (personalization, location, history, and model updates add more noise). The biggest red flag is a single blended “AI Visibility Score” (e.g., 73/100) with no per-engine breakdown, no prompt list, and no disclosure of how many times each prompt was run.Amicited How to evaluate a score’s trustworthiness Ask these questions (or check the methodology page): Per-engine breakdowns — Separate results for ChatGPT and Grok (and ideally others). A blended number hides large differences. Sampling depth — How many times is each prompt run? One run is nearly useless. Look for at least 3–5 (better tools go higher and report ranges or confidence intervals). Prompt transparency — Exact prompts used? Are they non-branded buyer-intent questions (“best tool for X use case,” “Y alternatives,” “how to solve Z”) rather than “what is [your brand]?” Raw data access — Can you see the actual answers, cited sources, competitors mentioned, position, and sentiment/accuracy notes? Methodology details — Models/versions, whether web search was enabled, timing between runs, and any caveats about variance. No overclaiming — Tools that call the number “accurate” or claim it reflects real user visibility without heavy caveats are less credible. Good signals include published methodology, multi-run sampling, engine-level views, and the ability to inspect answers. Opaque proprietary scores are low-trust. Zero-budget way to check it yourself (recommended) You can build a more reliable baseline than most vendor scores by doing controlled manual testing: Build 10–15 real buyer prompts for your SaaS category. Focus on commercial-intent, non-branded queries your customers actually type. Run them on ChatGPT and Grok: ChatGPT: Use a private/incognito window, turn off Memory/personalization if possible, and enable web search. Grok: Use a fresh session on x.com or the app. Run each prompt 3 times (or 5 if you can), spaced out. Log every result. Track in a simple Google Sheet: Prompt | Engine | Run # | Mentioned? | Position | Competitors | Sources cited | Notes on accuracy/sentiment. Calculate simple rates per engine (e.g., mention rate = % of runs where you appear). Retest monthly to see trends. Cross-check free signals: Google Search Console Generative AI / AI Overviews report. GA4 AI Assistant channel (referrals from ChatGPT, Grok, etc.). Free one-off checkers from Ahrefs, Semrush (limited free checks), or HubSpot’s AI Search Grader for a quick snapshot. This takes a few hours the first time and gives you data you control. Variance will still exist, but averaging multiple runs makes the signal usable. Practical takeaway for a solo founder Don’t optimize for a vendor score. Use manual multi-run checks (or free snapshots) as your source of truth for ChatGPT and Grok. Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully. A trustworthy score is transparent about its limitations, shows per-engine multi-run data, and lets you verify the underlying answers. Anything else is marketing. Start with your own spreadsheet baseline—it will be more useful than most paid dashboards at your stage.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V1-R3",
"slot_id": "S3-GROK-V1-R3",
"platform": "Grok",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 3,
"submitted_at": "2026-11-02T06:51:16.980Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:53:30.954Z",
"source_panel_declared_hits": 75,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-V1-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4491,
"public_answer_characters": 4491,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V1-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Markets.businessinsider | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Rendered DOM href |
| Markets.businessinsider | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Rendered DOM href |
| Amicited | https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/ | www.amicited.com | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 4,491-character rendered answer, three scoped DOM href occurrences and existing browser snapshots/activity; no external webpage visits, claims audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Sheets | Google Sheet | No | Yes | Yes | suggested | stated free | Track in a simple Google Sheet: Prompt | Engine | Run # | Mentioned? | Position | Competitors | Sources cited | Notes on accuracy/sentiment.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | suggested | stated free | Google Search Console Generative AI / AI Overviews report.
| |
| Google Analytics 4 | GA4 | No | Yes | Yes | suggested | stated free | GA4 AI Assistant channel (referrals from ChatGPT, Grok, etc.).
| |
| Ahrefs AI visibility tools (unspecified) | Ahrefs | No | Yes | Yes | conditional | stated free | Free one-off checkers from Ahrefs, Semrush (limited free checks), or HubSpot’s AI Search Grader for a quick snapshot.
| |
| Semrush AI visibility tools (unspecified) | Semrush (limited free checks) | No | Yes | Yes | conditional | mixed/limited tier | Free one-off checkers from Ahrefs, Semrush (limited free checks), or HubSpot’s AI Search Grader for a quick snapshot.
| |
| HubSpot AI Search Grader | HubSpot’s AI Search Grader | No | Yes | Yes | conditional | stated free | Free one-off checkers from Ahrefs, Semrush (limited free checks), or HubSpot’s AI Search Grader for a quick snapshot.
| |
| X | x.com | No | Yes | Yes | conditional | unspecified | Grok: Use a fresh session on x.com or the app.
| |
| Google AI Overviews | AI Overviews | No | Yes | No | discussed | unspecified | Google Search Console Generative AI / AI Overviews report.
| |
| YouTube | YouTube | No | Yes | No | discussed | unspecified | Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully.
| |
| No | Yes | No | discussed | unspecified | Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully.
| |||
| Crunchbase | Crunchbase | No | Yes | No | discussed | unspecified | Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully.
| |
| Wikidata | Wikidata | No | Yes | No | discussed | unspecified | Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_label_and_URL_only | Benchmark-supporting Markets Insider stock-news URL; syndicated/promotional provenance and primary research authorship remain unresolved. | supporting citation, not an active recommendation to read/follow the publisher | |
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_label_and_URL_only | Benchmark-supporting Markets Insider stock-news URL; syndicated/promotional provenance and primary research authorship remain unresolved. | supporting citation, not an active recommendation to read/follow the publisher | |
| https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/ | Other / unresolved | provisional_from_returned_label_and_URL_only | Evaluation-scorecard how-to page suggests guidance. Authorship, vendor/product documentation versus practitioner guidance are not established by the available label and URL. | supporting citation, not an active recommendation to read/follow the publisher |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "mention rate = % of runs where you appear"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"content_strategy_observations": [
{
"strategy": "Third-party mentions, consistent entity signals, quotable content and crawler access",
"recommended": true,
"evidence": "Focus effort on things that consistently correlate with mentions across studies: third-party mentions (YouTube, reviews, industry sites, LinkedIn), entity consistency (schema, consistent naming, Crunchbase/LinkedIn/Wikidata signals), clear quotable content with stats and citations, and allowing relevant AI crawlers while managing robots.txt carefully.",
"notes": "General strategy is explicitly encouraged; named services within parenthetical examples are not automatically recommendations to adopt those services."
}
]
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/features, with supplied targets and companies separate. Correlation examples and general content/identity strategies retain distinct roles from active tool recommendations. Citation-only publishers are not included as discovered tool mentions.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation",
"Tool-led solution"
],
"evidence": "How to evaluate a score’s trustworthiness",
"rationale": "Transparency, sampling, per-engine reporting, underlying answers and limitations dominate; the zero-budget manual workflow and snapshots are supporting components."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Are they non-branded buyer-intent questions (“best tool for X use case,” “Y alternatives,” “how to solve Z”) rather than “what is [your brand]?”"
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt 3 times (or 5 if you can), spaced out. Log every result."
},
"fresh_sessions": {
"value": true,
"evidence": "Grok: Use a fresh session on x.com or the app."
},
"model_search_version_logging": {
"value": true,
"evidence": "Models/versions, whether web search was enabled, timing between runs, and any caveats about variance."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "whether web search was enabled"
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "enable web search"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "often via API or consumer interfaces"
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Mentioned? | Position | Competitors | Sources cited"
},
"awareness_vs_recommendation_distinction": {
"value": false,
"evidence": null
},
"raw_answer_inspection": {
"value": true,
"evidence": "Can you see the actual answers, cited sources, competitors mentioned, position, and sentiment/accuracy notes?"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "mention rate = % of runs where you appear"
},
"composite_score_weighting_disclosed_or_checked": {
"value": false,
"evidence": null
},
"composite_score_caution": {
"value": true,
"evidence": "Separate results for ChatGPT and Grok (and ideally others). A blended number hides large differences."
},
"uncertainty_variance": {
"value": true,
"evidence": "better tools go higher and report ranges or confidence intervals"
},
"human_checking": {
"value": true,
"evidence": "controlled manual testing"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "GA4 AI Assistant channel (referrals from ChatGPT, Grok, etc.)."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual repeated sampling",
"evidence": "ChatGPT: Use a private/incognito window, turn off Memory/personalization if possible, and enable web search.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target, not a newly discovered tool."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual repeated sampling",
"evidence": "Grok: Use a fresh session on x.com or the app.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target, not a newly discovered tool."
}
],
"contextual_non_tool_entities": [
{
"entity": "OpenAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "No vendor has access to real OpenAI or xAI user conversation logs.",
"notes": "Company-level data-access claim, not a measurement-product recommendation; unverified platform output."
},
{
"entity": "xAI",
"citation": false,
"mention": true,
"recommendation": false,
"evidence": "No vendor has access to real OpenAI or xAI user conversation logs.",
"notes": "Company-level data-access claim, not a measurement-product recommendation; unverified platform output."
}
],
"generic_methods": [
"Per-engine methodological transparency checks",
"Repeated spaced non-branded buyer-prompt sampling",
"Clean-session and personalization-reduction advice",
"Google Sheet per-run logging",
"Monthly mention-rate trend checks",
"Generic free-signal cross-checks",
"Third-party mentions and consistent entity signals",
"Quotable accurate content and crawler access management"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "mention rate = % of runs where you appear"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4491,
"observed_inline_link_occurrences": 3,
"unique_DOM_answer_URLs": 2,
"unique_cited_domains": 2,
"source_badge_hits": 75,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Three inline link occurrences collapse to two unique URL/domain sources. This is not a complete 75-hit retrieval inventory.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Recorded activity exposes grouped four-search and three-search controls. Exact query strings are not expanded in the supplied snapshot.",
"The Markets Insider URL appears twice; deduplicate it for source-set comparisons, retaining occurrence count separately.",
"The 75-source badge is not a count of unique inline citations.",
"Workspace-explored activity labels do not establish their underlying contents or influence.",
"Numerical overlap ranges, log-access claims, scores/reliability claims, free/signup features and report availability are unverified platform output."
],
"original_capture_answer_characters": 4491,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping retains exact DOM hrefs/hosts; markets.businessinsider.com normalizes to businessinsider.com and www.amicited.com to amicited.com. No external public-suffix lookup used.",
"mapping": {
"markets.businessinsider.com": "businessinsider.com",
"www.amicited.com": "amicited.com"
}
}
}Specific Ahrefs/Semrush product identities are unresolved. HubSpot's observed AI Search Grader label is retained without an extra product alias merge.
YouTube/LinkedIn/Crunchbase/Wikidata occur as examples in a general content/entity-consistency strategy, not service-specific use or publishing instructions.
X is explicitly suggested as a Grok access path, not as a content-distribution strategy in this record.
Mention and citation fields are logged, but active positive-recommendation judgment/rate is not separately operationalized.
The methodology checklist asks for model versions, search state and timing; it does not explicitly demand scoring weights or a full composite-score formula.
S3-CHATGPT-V2-R1 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | ChatGPT Search, ChatGPT deep research, ChatGPT shopping, X, xAI X Search |
| Cited domains · mapping complete | arxiv.org, openai.com, x.ai |
| Observed inline cited domains · coverage qualified | arxiv.org, openai.com, x.ai |
| Observed cited-source categories · provisional | Academic or independent methods research, Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
Treat any “AI visibility score” as a **vendor-defined estimate—not an industry-standard metric**—until you can audit how it was produced.
## Checks to perform
1. **Confirm exactly what was tested**
- ChatGPT standard chat, Search, shopping, or deep research?
- Grok on grok.com, X, or API?
- Which model, mode, subscription tier, device, and date?
- Were web search and, for Grok, X search enabled?
These surfaces are not interchangeable. ChatGPT may search automatically, rewrite queries, use approximate location, and incorporate memory; Grok can use both real-time web and X search. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))
2. **Demand the actual prompt set**
Check whether prompts:
- Represent real customer questions
- Include unbranded discovery queries, not just your company name
- Cover alternatives, comparisons, use cases, and buying intent
- Use your target geography and language
- Were selected before results were observed
A score based on 20 favorable prompts can be less useful than one based on 200 poorly targeted prompts.
3. **Check for repeated runs**
One response per prompt is inadequate. Ask for:
- Runs per prompt
- Number of distinct days tested
- Fresh chats versus continued conversations
- Variance or confidence intervals
- Prompt-level consistency
LLM evaluations can vary across repeated runs and small wording changes, so aggregate rankings without uncertainty can look more definitive than they are. ([arxiv.org](https://arxiv.org/abs/2410.03492))
4. **Inspect the scoring formula**
Determine whether “visibility” means:
- Merely mentioned
- Listed among alternatives
- Positively recommended
- Ranked in the top three
- Linked or cited
- Accurately described
Also inspect weights, denominators, competitor selection, and treatment of refusals or failed searches. A mention and a first-choice recommendation should not receive the same credit.
5. **Require raw evidence**
For every scored result, the vendor should retain:
- Exact prompt
- Full response
- Timestamp
- Product/model/mode
- Search status
- Citations and destination URLs
- Screenshot or API output
OpenAI itself warns that search results and citations may be incomplete, outdated, or incorrect and recommends opening sources to verify support. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))
6. **Verify citations separately from mentions**
Track at least three different metrics:
- **Mention share:** your product appears
- **Recommendation share:** it is actually suggested
- **Citation share:** your domain is used as a source
A model may mention your brand based on another site, cite your site without recommending you, or hallucinate product capabilities.
7. **Audit entity matching**
Look for:
- Products with similar names
- Parent company versus product confusion
- Old product names or domains
- Misspellings and abbreviations
- Multiple products from your company being merged
Manually inspect a sample of both positive and negative matches.
8. **Check personalization and geographic controls**
Results can change with location, saved memory, prior conversation, workspace sources, and account state. ChatGPT explicitly says location and memory may affect search-query rewriting. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))
Prefer clean sessions with memory disabled, no prior context, and explicitly recorded locales—unless personalization is intentionally part of the test.
9. **Check whether the vendor is measuring the consumer product**
API calls with custom system prompts, fixed temperatures, or manually enabled tools may not reproduce what prospects see in ChatGPT or Grok. Ask the vendor to explain any difference between its testing harness and the actual consumer interface.
10. **Audit automated grading**
If another LLM decides whether your product was “recommended,” require:
- The grading rubric and judge model
- Human-labeled validation samples
- Agreement rates
- Manual review of ambiguous outputs
- Randomized ordering in competitor comparisons
xAI’s own evaluation guidance warns that model judges can exhibit position bias, favor longer answers, and drift from human judgment. ([docs.x.ai](https://docs.x.ai/developers/model-capabilities/text/best-practices))
11. **Check crawl and source availability**
Determine whether:
- Important pages are indexable
- Content is current and server-rendered/accessibly rendered
- Robots or CDN rules block relevant crawlers
- The model cites obsolete third-party pages instead of your canonical pages
OpenAI states that eligibility requires allowing OAI-Searchbot, but inclusion and placement are not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))
12. **Compare scores with business evidence**
Cross-check reported improvement against:
- ChatGPT/Grok referral traffic
- AI-cited landing-page visits
- Branded search growth
- “How did you hear about us?” responses
- Demo or signup attribution
- Actual citation URLs
Visibility without qualified traffic, accurate positioning, or conversions may have little commercial value.
## A cheap independent audit
Before buying or trusting a platform:
1. Select **20–30 commercially meaningful prompts**.
2. Run each prompt **at least five times** in fresh sessions.
3. Test on **two or three different days**.
4. Record mention, recommendation position, sentiment, factual accuracy, citation, and cited domain separately.
5. Compare your prompt-level results—not merely the final average—with the vendor’s raw data.
**Strongest trust signal:** the vendor lets you export prompts, raw responses, timestamps, model/mode metadata, citations, and its complete scoring formula.
**Biggest red flag:** a precise score such as “73/100 visibility” with no raw outputs, repeated-run statistics, or explanation of what the 73 represents.API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 19168,
"input_tokens_details": {
"cache_write_tokens": 32,
"cached_tokens": 4316
},
"output_tokens": 2115,
"output_tokens_details": {
"reasoning_tokens": 808
},
"total_tokens": 21283
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1234664,
"cost_estimate_detail": {
"estimated_usd": 0.1234664,
"token_cost_usd": 0.1034664,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V2-R1",
"slot_id": "S3-CHATGPT-V2-R1",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 1,
"submitted_at": "2026-11-02T09:20:55.790227+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:21:29.569815+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 19168,
"input_tokens_details": {
"cache_write_tokens": 32,
"cached_tokens": 4316
},
"output_tokens": 2115,
"output_tokens_details": {
"reasoning_tokens": 808
},
"total_tokens": 21283
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1234664,
"token_cost_usd": 0.1034664,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read full 12-check audit, cheap independent audit and final red-flag sentence. Coherent ending; all six Markdown source occurrences match annotations, three distinct source URLs. Requested model/effort and two actual searches verified. No limits reached; advice and cited claims not externally audited.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:22:34.394550+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search how search works citations official",
"site:docs.x.ai Grok web search citations official",
"site:platform.openai.com/docs models snapshots behavior official",
"LLM evaluation variance repeated prompts benchmark paper arxiv"
],
"query": "site:help.openai.com ChatGPT search how search works citations official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12677804-what-is-chatgpt-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth%3F.pls"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2609.30074"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2410.03492"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results"
},
{
"type": "url",
"url": "https://www.turing.ac.uk/sites/default/files/2025-11/towards_reproducible_llm_evaluation.pdf"
},
{
"type": "url",
"url": "https://aclweb.org/anthology/2024.acl-long.560.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10056348-finding-your-chats-projects-and-files-in-chatgpt"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2405.17202"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10984597-chatgpt-generated-links/"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2609.32547"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10500283-deep-research-faq/"
},
{
"type": "url",
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC12637745/"
},
{
"type": "url",
"url": "https://proceedings.neurips.cc/paper_files/paper/2024/file/28236482f64a72eec43706b6f3a6c511-Paper-Conference.pdf"
},
{
"type": "url",
"url": "https://www.paperpeel.com/paper/2610.30512"
},
{
"type": "url",
"url": "https://proceedings.nips.cc/paper_files/paper/2024/file/28236482f64a72eec43706b6f3a6c511-Paper-Conference.pdf"
},
{
"type": "url",
"url": "https://en.wikipedia.org/wiki/Language_model_benchmark"
},
{
"type": "url",
"url": "https://en.wikipedia.org/wiki/LLM-as-a-Judge"
},
{
"type": "url",
"url": "https://www.reddit.com/r/AIEval/comments/1s3835s/i_built_a_tool_for_deterministic_llm_evaluation/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/u_TheAIResearchCenter/comments/1v96o85/how_to_read_an_ai_evaluation_report_without_being/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/LLMDevs/comments/1qh7oxg/confused_about_llm_evaluation_approaches/"
},
{
"type": "url",
"url": "https://www.livescience.com/technology/artificial-intelligence/being-mean-to-chatgpt-increases-its-accuracy-but-you-may-end-up-regretting-it-scientists-warn"
},
{
"type": "url",
"url": "https://www.reddit.com/r/u_TheAIResearchCenter/comments/1v9jww1/when_benchmark_scores_hide_the_conditions_that/"
},
{
"type": "url",
"url": "https://www.reddit.com/r/LocalLLaMA/comments/1g5uksu"
},
{
"type": "url",
"url": "https://www.reddit.com/r/MachineLearning/comments/1w9llr4/measuring_llm_performance_drift_observations_and/"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai/docs/guides tools web search xAI Grok",
"site:docs.x.ai Grok search web X posts citations",
"site:x.ai/news Grok search web real-time information"
],
"query": "site:docs.x.ai/docs/guides tools web search xAI Grok",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/web-search"
},
{
"type": "url",
"url": "https://docs.x.ai/build/settings"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/progressive-tool-disclosure"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/computer-and-apps"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/tool-usage-details"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/overview"
},
{
"type": "url",
"url": "https://x.ai/news/grok-build-mode"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/advanced-usage"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/prompt-injection-defenses"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-xai-risk-management-framework.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-12-31-xai-frontier-artificial-intelligence-framework.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://arxiv.org/abs/2410.03492"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-V2-R1.json",
"request_file": "requests/S3-OPENAIAPI-V2-R1.json",
"response_file": "responses/S3-OPENAIAPI-V2-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 6264,
"public_answer_characters": 6156,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V2-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores | https://arxiv.org/abs/2410.03492 | arxiv.org | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Best Practices Guide | SpaceXSI Docs | https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant semantic review of the full finalized API answer, returned response, annotations and request metadata. No external visits, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V2-R1.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| ChatGPT Search | Search | Yes | Yes | No | discussed as a surface/configuration to identify | unspecified | ChatGPT standard chat, Search, shopping, or deep research?
| https://help.openai.com/en/articles/9237897-chatgpt-search |
| ChatGPT shopping | shopping | No | Yes | No | discussed as a surface/configuration to identify | unspecified | ChatGPT standard chat, Search, shopping, or deep research?
| |
| ChatGPT deep research | deep research | No | Yes | No | discussed as a surface/configuration to identify | unspecified | ChatGPT standard chat, Search, shopping, or deep research?
| |
| xAI X Search | X search | No | Yes | No | discussed as a surface/configuration to identify | unspecified | Were web search and, for Grok, X search enabled?
| |
| X | X | No | Yes | No | discussed as a surface/configuration to identify | unspecified | Grok on grok.com, X, or API?
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_title_official_domain_and_answer_role | Official provider help/developer hostname and returned title identify platform documentation; claims not externally audited. | Supporting citation, not automatic active recommendation to use/read/follow the publisher | |
| https://arxiv.org/abs/2410.03492 | Academic or independent methods research | provisional_from_title_official_domain_and_answer_role | arXiv paper title explicitly concerns reproducible LLM evaluation and uncertainty; methods, authorship and findings not independently reviewed. | Supporting citation, not automatic active recommendation to use/read/follow the publisher | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_title_official_domain_and_answer_role | Official provider help/developer hostname and returned title identify platform documentation; claims not externally audited. | Supporting citation, not automatic active recommendation to use/read/follow the publisher | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_title_official_domain_and_answer_role | Official provider help/developer hostname and returned title identify platform documentation; claims not externally audited. | Supporting citation, not automatic active recommendation to use/read/follow the publisher | |
| https://docs.x.ai/developers/model-capabilities/text/best-practices | Platform documentation | provisional_from_title_official_domain_and_answer_role | Official provider help/developer hostname and returned title identify platform documentation; claims not externally audited. | Supporting citation, not automatic active recommendation to use/read/follow the publisher | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_title_official_domain_and_answer_role | Official provider help/developer hostname and returned title identify platform documentation; claims not externally audited. | Supporting citation, not automatic active recommendation to use/read/follow the publisher |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching a url_citation occurrence | ||
| https://arxiv.org/abs/2410.03492 | arxiv.org | Explicit Markdown URL matching a url_citation occurrence | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching a url_citation occurrence | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching a url_citation occurrence | ||
| https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | Explicit Markdown URL matching a url_citation occurrence | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching a url_citation occurrence |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention share",
"definition_verbatim": "**Mention share:** your product appears"
},
{
"label": "Recommendation share",
"definition_verbatim": "**Recommendation share:** it is actually suggested"
},
{
"label": "Citation share",
"definition_verbatim": "**Citation share:** your domain is used as a source"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI itself warns that search results and citations may be incomplete, outdated, or incorrect and recommends opening sources to verify support.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_tool_overlap": true,
"notes": "Supporting source attribution, not a newly discovered measurement tool."
},
{
"entity": "xAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "xAI’s own evaluation guidance warns that model judges can exhibit position bias, favor longer answers, and drift from human judgment.",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"excluded_from_default_tool_overlap": true,
"notes": "Attribution to model-judge guidance; not an adoption recommendation or independent verification of those claims."
}
],
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1234664,
"token_cost_usd": 0.1034664,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 19168,
"input_tokens_details": {
"cache_write_tokens": 32,
"cached_tokens": 4316
},
"output_tokens": 2115,
"output_tokens_details": {
"reasoning_tokens": 808
},
"total_tokens": 21283
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2115,
"tool_limit_reached": false,
"output_limit_reached": false,
"authorized_total_budget_USD": 10.0,
"concerns": [
"No observed output/tool cap reached. Estimated cost remains unreconciled; monitor cumulative actual requests against the authorized total."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named consumer/API surfaces and retrieval modes are distinct from adopted measurement tools. Supplied targets, documentation publishers and a crawler agent remain separate. No named visibility checker or spreadsheet is inferred.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Checks to perform",
"rationale": "Twelve audit dimensions cover methods, repeated runs, raw evidence, matching, controls, consumer/API equivalence and grading; a cheap manual audit is the supporting workflow."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Include unbranded discovery queries, not just your company name"
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt **at least five times** in fresh sessions."
},
"fresh_sessions": {
"value": true,
"evidence": "Prefer clean sessions with memory disabled, no prior context, and explicitly recorded locales"
},
"model_search_version_logging": {
"value": true,
"evidence": "Which model, mode, subscription tier, device, and date?"
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Search status"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "API calls with custom system prompts, fixed temperatures, or manually enabled tools may not reproduce what prospects see in ChatGPT or Grok."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**Recommendation share:** it is actually suggested"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "**Citation share:** your domain is used as a source"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Full response"
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "For every scored result, the vendor should retain:"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "inspect weights, denominators, competitor selection, and treatment of refusals or failed searches"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "inspect weights, denominators, competitor selection, and treatment of refusals or failed searches"
},
"composite_score_caution": {
"value": true,
"evidence": "Inspect the scoring formula"
},
"uncertainty_variance": {
"value": true,
"evidence": "Variance or confidence intervals"
},
"human_checking": {
"value": true,
"evidence": "Human-labeled validation samples"
},
"downstream_conversions": {
"value": true,
"evidence": "Demo or signup attribution"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "manual real-interface validation is suggested",
"evidence": "Manually inspect a sample of both positive and negative matches.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target. The cheap independent audit calls for fresh-session tests and consumer-product matching rather than new platform discovery."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "manual real-interface validation is suggested",
"evidence": "Manually inspect a sample of both positive and negative matches.",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target. The cheap independent audit calls for fresh-session tests and consumer-product matching rather than new platform discovery."
}
],
"operational_entities": [
{
"entity": "OAI-Searchbot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Check relevant crawler access while noting inclusion is not guaranteed",
"evidence": "OpenAI states that eligibility requires allowing OAI-Searchbot, but inclusion and placement are not guaranteed.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"excluded_from_default_tool_overlap": true,
"notes": "Crawler-eligibility claim, not a recommendation to adopt a measurement tool. API output has not been externally fact-checked."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Twelve-part score audit",
"Prompt relevance and selection checks",
"Repeated multi-day fresh-session audit",
"Separate mention/recommendation/citation metrics",
"Entity alias/matching validation",
"Human validation of automated judges",
"Crawl/source availability checks",
"Business-evidence cross-checks"
],
"metric_definitions_observed": [
{
"label": "Mention share",
"definition_verbatim": "**Mention share:** your product appears"
},
{
"label": "Recommendation share",
"definition_verbatim": "**Recommendation share:** it is actually suggested"
},
{
"label": "Citation share",
"definition_verbatim": "**Citation share:** your domain is used as a source"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 6156,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 6,
"explicit_body_URL_occurrences": 6,
"unique_answer_URLs": 3,
"unique_cited_hostnames": 3,
"unique_normalized_cited_domains": 3,
"citation_mapping_complete": true,
"source_inventory_complete_for_answer_URLs": true,
"source_inventory_scope": "Six annotated/explicit Markdown occurrences map to three unique source URLs. OpenAI Help repeats four times; repeated representations do not add unique sources. Retrieval remains distinct.",
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V2-R1.json",
"response_file": "responses/S3-OPENAIAPI-V2-R1.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Two completed search actions and two reported billable searches observed.",
"Consumer/API surface differences are explicitly discussed in the answer and also apply to this experiment's collection arms.",
"Source wording, study claims, crawler eligibility and the proposed sampling workflow are unverified generated output.",
"API metadata reports model/effort; it does not audit internal execution."
],
"original_capture_answer_characters": 6264,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual mapping help.openai.com to openai.com and docs.x.ai to x.ai; arxiv.org unchanged. Exact annotations including tracking parameters retained locally. No external suffix lookup.",
"mapping": {
"help.openai.com": "openai.com",
"arxiv.org": "arxiv.org",
"docs.x.ai": "x.ai"
}
}
}Surfaces/modes are named to audit what a vendor measured, not actively recommended as new tools.
OpenAI/xAI source attributions are not automatic tool recommendations.
The arXiv paper is cited without a substantive author/title recommendation in the prose; it is not a tool entity.
Repeated citations to the same help article count once for source-set overlap.
Sampling quantities in the cheap audit are advice, not independently validated minimums or causal findings.
S3-CHATGPT-V2-R2 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | ChatGPT Search, Grok API, Reddit, X |
| Cited domains · mapping complete | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | openai.com, x.ai |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
Treat any “AI visibility score” as a **vendor-defined estimate**, not an official ChatGPT or Grok metric. Before paying for or acting on it, perform these checks: ### 1. Demand an exact metric definition Determine whether “visibility” means: - Your product name appears anywhere - Your website is cited - You are included in a recommendation list - You rank in the top three - The answer describes you accurately and positively - Your product appears across a percentage of tested prompts Ask for the actual numerator, denominator, weighting formula, and treatment of “not sure” or failed responses. A single composite score can otherwise hide meaningful differences between **mention, citation, recommendation, and preference**. ### 2. Inspect the prompt set Request the complete prompts—not merely prompt categories. Check that they: - Reflect questions real prospects ask - Include non-branded, problem-aware queries - Cover alternatives, comparisons, and use cases - Represent your target geography and customer segment - Aren’t artificially favorable to your product - Include both head terms and long-tail prompts Also check who selected the prompts. A vendor-generated benchmark may measure visibility for its imagined market rather than yours. ### 3. Confirm the exact test environment For every run, require: - Platform: consumer app or API - Model/version - Search-enabled versus model-only mode - Standard, research, reasoning, or other mode - Date, time, country, language, and device context - Fresh chat versus an existing conversation - Memory, custom instructions, and personalization status - Logged-in versus logged-out account This matters because ChatGPT can rewrite prompts into multiple search queries and may use approximate location or memory; Grok may decide whether to search the broader web, X, or neither. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) ### 4. Require repeated samples One run per prompt is not trustworthy. Ask for: - At least 5–10 independent runs per important prompt - New conversations for each run - Mention and citation frequency, not just a binary result - Confidence intervals or raw counts - Variance across days For example, “mentioned in 7 of 10 runs” is substantially more useful than an opaque score of 72. ### 5. Review the raw evidence The report should let you inspect: - Full prompt - Full response - Timestamp - Citations and source URLs - Model and mode - Screenshots or, preferably, machine-readable run logs Click citations and verify that they actually support the claims. OpenAI explicitly warns that search results and citations can be incomplete, outdated, or incorrect. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) For Grok API-based testing, ask whether the provider retained the actual web/X search events and returned sources—not merely links generated in the answer. xAI’s documentation supports exposing search sources for this kind of verification. ([docs.x.ai](https://docs.x.ai/cookbook/examples/x-post-fact-check)) ### 6. Separate model knowledge from live-search visibility A product can appear because: 1. It exists in model training data, 2. The assistant found its website, 3. The assistant found a third-party article, 4. It appeared in an X post, 5. The prompt effectively supplied the answer. These are different acquisition channels and require different actions. Search visibility should be reported separately from non-search recall. ### 7. Audit source quality Check what sources are driving your visibility: - Your own site - Review platforms - Software directories - Affiliates - Reddit, forums, or X - Editorial comparisons - Outdated pages - Competitor-authored content A high score built on inaccurate listicles or old pricing pages may be a liability. Also verify that your site is technically accessible: ChatGPT search eligibility depends partly on permitting OAI-SearchBot, although inclusion and placement are not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) ### 8. Check competitor normalization Ask: - Which competitors were included—and why? - Is the score absolute or relative? - Does adding a competitor change your score? - Are large incumbents and niche alternatives weighted equally? - Are generic options such as spreadsheets or “build it yourself” included? A “20% share of voice” is meaningless without the complete comparison set. ### 9. Test factual accuracy, not merely presence Manually grade answers for: - Correct positioning - Current features and pricing - Appropriate customer fit - Correct category - Accurate pros and cons - Confusion with similarly named companies - Broken or irrelevant citations A confident but inaccurate recommendation should count negatively, not as successful visibility. ### 10. Reproduce a sample independently Before trusting the dashboard: 1. Select 20–30 high-intent prompts. 2. Run them yourself in fresh ChatGPT and Grok conversations. 3. Record mentions, citations, rank, accuracy, and sentiment. 4. Repeat on several days. 5. Compare your results with the vendor’s raw data. You need not reproduce the precise composite score. You should be able to reproduce its **direction and major conclusions**. ### 11. Look for business correlation Compare reported visibility changes with: - `utm_source=chatgpt.com` referrals - Grok/X referrals where identifiable - Branded-search growth - Signup surveys asking “How did you hear about us?” - Demo or signup volume from AI-assistant users - Assisted conversions, not just last-click conversions Visibility without qualified traffic, accurate recommendations, or conversions is primarily a branding indicator—not an ROI metric. ChatGPT adds referral tracking parameters to relevant outbound links, which can help with validation. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results)) ### Minimum evidence I’d accept For a bootstrapped SaaS, I would not pay much for a score unless the provider supplies: - Complete prompt list - Metric formula - Model/mode/location details - Multiple runs per prompt - Raw responses and citations - Historical timestamps - Separate ChatGPT and Grok results - Mention, citation, rank, accuracy, and sentiment breakdowns - Exportable data - Results you can approximately reproduce **Practical rule:** Use the score to identify prompts and source gaps, not as a KPI by itself. Track a small first-party benchmark—perhaps 25–50 commercially important prompts—alongside referral and conversion data. That is usually more actionable for a bootstrapped company than a broad proprietary “visibility” number.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21174,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 1960,
"output_tokens_details": {
"reasoning_tokens": 558
},
"total_tokens": 23134
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1282432,
"cost_estimate_detail": {
"estimated_usd": 0.1282432,
"token_cost_usd": 0.1082432,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V2-R2",
"slot_id": "S3-CHATGPT-V2-R2",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 2,
"submitted_at": "2026-11-02T09:32:01.882925+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:32:29.432784+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21174,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 1960,
"output_tokens_details": {
"reasoning_tokens": 558
},
"total_tokens": 23134
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1282432,
"token_cost_usd": 0.1082432,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read all eleven checks, minimum-evidence list and complete practical-rule ending. Five source occurrences match annotations; three unique source URLs retained verbatim, including encoded FAQ destination. Correct model/effort and two actual searches; no caps. Quantitative recommendations and citation support are collected advice, not verified findings.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:32:54.066468+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations sources official",
"site:openai.com ChatGPT search official how search works citations",
"site:docs.x.ai Grok web search citations official",
"site:x.ai Grok search sources citations official"
],
"query": "site:help.openai.com ChatGPT search citations sources official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12677804-what-is-chatgpt-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10093903-chatgpt-search-for-enterprise-and-edu"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://openai.com/academy/search-and-deep-research/"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
},
{
"type": "url",
"url": "https://openai.com/index/introducing-chatgpt-search/"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/20001203-offline-web-search-for-chatgpt-workspaces"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12911370-using-shopping-r"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10500283-deep-research-faq%3F.apk"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a-family-guide-to-help-teens-use-ai-responsibly.pdf"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf"
},
{
"type": "url",
"url": "https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai web search tool Grok citations sources",
"site:docs.x.ai Grok search X posts web official documentation",
"site:x.ai/news Grok search web citations official",
"site:help.x.com Grok sources web search official"
],
"query": "site:docs.x.ai web search tool Grok citations sources",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/x-search"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://docs.x.ai/build/settings"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://x.ai/news/grok-1212?TB_iframe=true&height=512.1&width=921.6"
},
{
"type": "url",
"url": "https://x.ai/grok/use-cases/research-synthesis"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/files-and-results"
},
{
"type": "url",
"url": "https://x.ai/pricing"
},
{
"type": "url",
"url": "https://x.ai/grok?q=Tesla+ai+computer"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/troubleshooting"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-xai-risk-management-framework.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-12-31-xai-frontier-artificial-intelligence-framework.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-V2-R2.json",
"request_file": "requests/S3-OPENAIAPI-V2-R2.json",
"response_file": "responses/S3-OPENAIAPI-V2-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7061,
"public_answer_characters": 6971,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V2-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| AI Fact-Checker: Check Posts on 𝕏 with the Grok API | SpaceXSI Docs | https://docs.x.ai/cookbook/examples/x-post-fact-check | docs.x.ai | API body url_citation annotation | |
| Searching the web with ChatGPT | OpenAI Help Center | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | API body url_citation annotation | |
| Publishers and Developers - FAQ | OpenAI Help Center | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | API body url_citation annotation |
Retrieval-panel sources · not matched answer citations
Assistant semantic review of full finalized 7,061-character API answer, returned response, annotations and request metadata. No external queries, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V2-R2.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| ChatGPT Search | ChatGPT search | Yes | Yes | No | discussed as measurement context or a source category to audit | unspecified | ChatGPT search eligibility depends partly on permitting OAI-SearchBot, although inclusion and placement are not guaranteed.
| https://help.openai.com/en/articles/9237897-chatgpt-search https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results |
| Grok API | Grok API-based testing | Yes | Yes | No | discussed as measurement context or a source category to audit | unspecified | For Grok API-based testing, ask whether the provider retained the actual web/X search events and returned sources—not merely links generated in the answer.
| https://docs.x.ai/cookbook/examples/x-post-fact-check |
| No | Yes | No | discussed as measurement context or a source category to audit | unspecified | - Reddit, forums, or X
| |||
| X | X | No | Yes | No | discussed as measurement context or a source category to audit | unspecified | - Reddit, forums, or X
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_official_provider_hostname_title_and_answer_attribution | OpenAI Help material or xAI cookbook documentation. The xAI fact-checker example title is a source label, not a tool the answer asks the user to adopt. Contents/support not externally audited. | Supporting provider documentation, not automatic active product recommendation | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_official_provider_hostname_title_and_answer_attribution | OpenAI Help material or xAI cookbook documentation. The xAI fact-checker example title is a source label, not a tool the answer asks the user to adopt. Contents/support not externally audited. | Supporting provider documentation, not automatic active product recommendation | |
| https://docs.x.ai/cookbook/examples/x-post-fact-check | Platform documentation | provisional_from_official_provider_hostname_title_and_answer_attribution | OpenAI Help material or xAI cookbook documentation. The xAI fact-checker example title is a source label, not a tool the answer asks the user to adopt. Contents/support not externally audited. | Supporting provider documentation, not automatic active product recommendation | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_official_provider_hostname_title_and_answer_attribution | OpenAI Help material or xAI cookbook documentation. The xAI fact-checker example title is a source label, not a tool the answer asks the user to adopt. Contents/support not externally audited. | Supporting provider documentation, not automatic active product recommendation | |
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | Platform documentation | provisional_from_official_provider_hostname_title_and_answer_attribution | OpenAI Help material or xAI cookbook documentation. The xAI fact-checker example title is a source label, not a tool the answer asks the user to adopt. Contents/support not externally audited. | Supporting provider documentation, not automatic active product recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://docs.x.ai/cookbook/examples/x-post-fact-check | docs.x.ai | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | Explicit Markdown URL matching annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Metric-definition audit",
"definition_verbatim": "Ask for the actual numerator, denominator, weighting formula, and treatment of “not sure” or failed responses."
},
{
"label": "Share of voice",
"definition_verbatim": "A “20% share of voice” is meaningless without the complete comparison set.",
"notes": "Context warning, not an operational formula."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "OpenAI explicitly warns that search results and citations can be incomplete, outdated, or incorrect.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"excluded_from_default_tool_overlap": true,
"notes": "Supporting documentation attribution, not an active vendor/tool-use recommendation."
},
{
"entity": "xAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "xAI’s documentation supports exposing search sources for this kind of verification.",
"source_urls": [
"https://docs.x.ai/cookbook/examples/x-post-fact-check"
],
"excluded_from_default_tool_overlap": true,
"notes": "Documentation support for source/event retention; the citation title does not imply adoption of an AI fact-checker tool."
}
],
"source_title_entities_not_adopted": [
{
"label": "AI Fact-Checker: Check Posts on 𝕏 with the Grok API",
"status": "citation-title-only example",
"recommendation": false,
"notes": "No fact-checker tool is recommended or substantively described in the answer body."
}
],
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1282432,
"token_cost_usd": 0.1082432,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21174,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 1960,
"output_tokens_details": {
"reasoning_tokens": 558
},
"total_tokens": 23134
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 1960,
"tool_limit_reached": false,
"output_limit_reached": false,
"authorized_total_budget_USD": 10.0,
"concerns": [
"No observed tool/output limit reached. US$0.1282432 is estimated, not reconciled billing; cumulative budget monitoring remains required."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named surfaces and community source contexts are distinct from adopted software. Supplied targets, documentation publishers and crawler compatibility are separate. No tool identity is inferred from the fact-checker citation title.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Minimum evidence I’d accept",
"rationale": "Eleven checks emphasize definitions, prompts, test controls, sampling, evidence, source quality, normalization and outcomes; a focused manual benchmark supports validation."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Include non-branded, problem-aware queries"
},
"repeated_runs": {
"value": true,
"evidence": "At least 5–10 independent runs per important prompt"
},
"fresh_sessions": {
"value": true,
"evidence": "New conversations for each run"
},
"model_search_version_logging": {
"value": true,
"evidence": "Model/version"
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Search-enabled versus model-only mode"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "Platform: consumer app or API"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "**mention, citation, recommendation, and preference**"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Mention and citation frequency, not just a binary result"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Click citations and verify that they actually support the claims."
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "Screenshots or, preferably, machine-readable run logs"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Ask for the actual numerator, denominator, weighting formula, and treatment of “not sure” or failed responses."
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "weighting formula"
},
"composite_score_caution": {
"value": true,
"evidence": "A single composite score can otherwise hide meaningful differences"
},
"uncertainty_variance": {
"value": true,
"evidence": "Confidence intervals or raw counts"
},
"human_checking": {
"value": true,
"evidence": "Manually grade answers for:"
},
"downstream_conversions": {
"value": true,
"evidence": "Assisted conversions, not just last-click conversions"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested independent manual replication",
"evidence": "Run them yourself in fresh ChatGPT and Grok conversations.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied platform; direct checking is advised, not new tool discovery."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested independent manual replication",
"evidence": "Run them yourself in fresh ChatGPT and Grok conversations.",
"source_urls": [
"https://docs.x.ai/cookbook/examples/x-post-fact-check"
],
"excluded_from_default_overlap": true,
"notes": "Question-supplied platform; direct checking is advised, not new tool discovery."
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommended_action": "Audit technical access and crawler eligibility",
"evidence": "ChatGPT search eligibility depends partly on permitting OAI-SearchBot, although inclusion and placement are not guaranteed.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"excluded_from_default_tool_overlap": true,
"notes": "Compatibility/eligibility check, not a measurement tool adoption recommendation. Claim remains unverified."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Eleven-part methodology and evidence audit",
"Separate mention/citation/recommendation/preference definitions",
"Repeated fresh-session sampling and raw counts",
"API/live-search event and source verification",
"Source-quality and competitor-normalization audits",
"Factual-accuracy grading",
"Independent multi-day prompt replication",
"Business referrals/surveys/assisted-conversion comparison"
],
"metric_definitions_observed": [
{
"label": "Metric-definition audit",
"definition_verbatim": "Ask for the actual numerator, denominator, weighting formula, and treatment of “not sure” or failed responses."
},
{
"label": "Share of voice",
"definition_verbatim": "A “20% share of voice” is meaningless without the complete comparison set.",
"notes": "Context warning, not an operational formula."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 6971,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 5,
"explicit_body_URL_occurrences": 5,
"unique_answer_URLs": 3,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_complete_for_answer_URLs": true,
"source_inventory_scope": "Five annotation/Markdown occurrences map to three distinct URLs across two hostnames/domains; repeated search-help citation is one source in set comparisons. Retrieval URLs remain separate.",
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V2-R2.json",
"response_file": "responses/S3-OPENAIAPI-V2-R2.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Two completed search actions and two reported billable requests; no cap observed.",
"The exact long publisher-FAQ URL contains percent-encoded text in its path. It is preserved verbatim rather than silently rewritten.",
"The cookbook citation is provider documentation, not evidence that the user was advised to adopt a fact-checker tool.",
"Numerical run-count and prompt-count recommendations are generated advice, not validated universal sample-size thresholds.",
"This API collection surface differs from consumer ChatGPT Search and the Grok consumer-interface arm."
],
"original_capture_answer_characters": 7061,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual provider-subdomain normalization help.openai.com to openai.com and docs.x.ai to x.ai. Exact long FAQ URL and tracking-bearing links retained as returned; no URL repair or external lookup.",
"mapping": {
"help.openai.com": "openai.com",
"docs.x.ai": "x.ai"
}
}
}Surface-disclosure and source-quality examples are not active recommendations to adopt those tools/platforms.
The xAI fact-checker example appears only in the citation title and is not counted as a recommended tool.
Official source identity does not independently verify claims or prove that a scoring methodology is validated.
Sample-count examples and recommended ranges must not enter empirical-result counts.
S3-CHATGPT-V2-R3 · OpenAI API + web search · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | None observed within this eligible set |
| Mentioned tools / services / surfaces | ChatGPT Search, X |
| Cited domains · mapping complete | openai.com, x.ai |
| Observed inline cited domains · coverage qualified | openai.com, x.ai |
| Observed cited-source categories · provisional | Platform documentation |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
Treat any “AI visibility score” as a **vendor-defined benchmark—not an objective market metric**. Before trusting it, perform these checks: ## 1. Get the precise score definition Ask what actually counts: - A product-name mention? - Inclusion in a recommendation list? - A top-three position? - A favorable recommendation? - A citation or link to your domain? - A citation to a third party that mentions you? - How are rankings, sentiment, and citations weighted? Request the formula, denominator, weighting, and one worked example. A score such as “42% visibility” is meaningless unless you know whether that means 42% of prompts, runs, weighted impressions, or something else. ## 2. Inspect the prompt set Verify that you can export every tested prompt. Check for: - **Representative intent:** category discovery, alternatives, comparisons, use cases, pricing, implementation and problem-oriented queries. - **Branded separation:** “Best alternatives to MyProduct” should not inflate your non-branded discovery score. - **Real buyer language:** prompts should resemble customer questions, not awkward SEO keyword templates. - **Relevant segmentation:** geography, language, company size, industry and buyer persona. - **No cherry-picking:** failed prompts and zero-mention prompts must remain in the denominator. For a small SaaS, manually label the 20–50 prompts that actually matter commercially rather than relying on hundreds of generic prompts. ## 3. Confirm the exact test environment Require the report to identify: - Product: ChatGPT UI, Grok UI or API - Model/version - Search enabled or disabled - Search, research, shopping or standard-chat mode - Account tier and logged-in state - Country/location - Memory, personalization and custom-instruction settings - Test date and harness version This matters because ChatGPT may automatically search, rewrite queries, and use location or memory; Grok can decide whether to search the web or X and may personalize results using X activity. ([help.openai.com](https://help.openai.com/en/articles/8590148-memory-in-chatgpt)) Do not accept an API test as a direct proxy for what customers see in the consumer interface unless the vendor demonstrates that the configurations match. ## 4. Demand repeated runs and uncertainty A single answer to a single prompt is not a reliable measurement. Ask for: - Number of repetitions per prompt - Mention rate across runs - Mean score plus spread or confidence interval - Results by day, not just a lifetime aggregate - Consistent treatment of timeouts, refusals and truncated responses xAI’s own evaluation guidance notes that reasoning-model outputs are nondeterministic, recommends multiple runs, and advises recording model, settings, date and harness version. ([docs.x.ai](https://docs.x.ai/developers/model-capabilities/text/best-practices)) A practical bootstrap test is **30–50 important prompts × 3 runs on two different days**. Large score changes that disappear under this test are probably noise. ## 5. Review raw evidence You should be able to inspect or export: - Exact prompt - Full answer - Timestamp - Model and mode - Citations and URLs - Your detected mention - Competitor mentions - The scoring decision Randomly audit at least 10–20% of results. If a dashboard supplies only a composite score or cropped answer snippets, don’t trust it. ## 6. Separate mention, recommendation and citation Track these independently: 1. **Awareness:** the model recognizes your product. 2. **Mention:** it includes you in an answer. 3. **Prominence:** your position in the list. 4. **Recommendation:** it positively presents you as suitable. 5. **Accuracy:** pricing, features and positioning are correct. 6. **Citation:** it links to evidence. 7. **Owned citation:** the link goes to your site. 8. **Conversion potential:** the answer gives a user a reason to visit. A high mention score with inaccurate positioning or no links may be much less valuable than a lower score containing strong, correctly sourced recommendations. ## 7. Validate every citation Check whether: - The cited page actually supports the claim. - The URL is live, canonical and current. - The model is citing your website versus a directory or outdated review. - A source merely contains your name or genuinely recommends the product. - Citations were returned by the platform rather than invented by the monitoring vendor. OpenAI explicitly warns that search results and citations can be incomplete, outdated or incorrect and recommends opening the source to verify support. ([help.openai.com](https://help.openai.com/en/articles/8313428)) ## 8. Check entity resolution Look for false positives involving: - Products with similar names - Your company versus your product - Open-source packages with the same name - Old product names or acquired products - Your product appearing only in unrelated text Require exact-match rules plus human review for ambiguous entities. Also confirm that URLs, descriptions and categories correspond to your actual SaaS. ## 9. Test whether your site can be discovered Check: - Relevant pages return `200` - Important content is available without login - Titles and descriptions clearly identify the product and use case - Pricing and feature claims are current - Canonical tags are correct - Your robots/CDN configuration isn’t blocking relevant crawlers For ChatGPT search, OpenAI says eligibility for summaries and citations depends in part on allowing `OAI-SearchBot`; placement is still not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search)) Also distinguish between **model-memory visibility** and **live-search visibility**. Being known without search is different from being found and cited during a current web search. ## 10. Recalculate the score yourself From the raw export, independently calculate simple metrics: - `% of prompts mentioning you` - `% recommending you` - `% citing you` - `% containing material factual errors` - Average rank when mentioned - Share of mentions versus named competitors - Stability across repeated runs If your calculation cannot reproduce the headline number, ask the vendor to explain every transformation and weight. ## 11. Tie visibility to real outcomes Compare trends against: - ChatGPT referral sessions - Grok/X referral sessions - Sign-ups containing AI-referral attribution - “How did you hear about us?” responses - Branded-search lift - Demo or trial quality OpenAI adds `utm_source=chatgpt.com` to ChatGPT referral URLs, which can help identify inbound traffic, although visibility without a click will still be missed. ([help.openai.com](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results)) ## Fast trust checklist I would trust a report only if it provides: - [ ] Transparent formula - [ ] Exportable prompt set - [ ] Raw answers and citations - [ ] Exact model/search configuration - [ ] Multiple runs per prompt - [ ] Dates and version history - [ ] Failures included in the denominator - [ ] Branded and non-branded results separated - [ ] Mention, recommendation, accuracy and citation reported separately - [ ] A manually audited sample that you can reproduce **Bottom line:** optimize around a small, commercially meaningful prompt set and inspect the underlying answers. Use the aggregate score as a directional monitoring signal—not as proof of market share or customer exposure.
API request, response, search, tokens, and estimated cost
Requested/configured model and returned model are distinct. Transport completeness is not reviewed answer completeness. Costs are recorded estimates, not reconciled billing, unless the input explicitly establishes reconciliation.
{
"interface": "Responses API",
"configured_model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"search_enabled_api": true,
"search_observed": true,
"token_usage": {
"input_tokens": 21162,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 2251,
"output_tokens_details": {
"reasoning_tokens": 694
},
"total_tokens": 23413
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"transport_body_complete": true,
"cost_estimate_usd": 0.1340152,
"cost_estimate_detail": {
"estimated_usd": 0.1340152,
"token_cost_usd": 0.1140152,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"cost_estimate_basis": null
}Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V2-R3",
"slot_id": "S3-CHATGPT-V2-R3",
"platform": "OpenAI API + web search",
"interface": "Responses API",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 3,
"submitted_at": "2026-11-02T09:35:59.689259+00:00",
"model_label": "gpt-5.6-sol",
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"personalization": false,
"search_enabled_api": true,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T09:36:36.226321+00:00",
"source_inventory_complete": true,
"transport_body_complete": true,
"refusals": [],
"response_status": "completed",
"incomplete_details": null,
"usage": {
"input_tokens": 21162,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 2251,
"output_tokens_details": {
"reasoning_tokens": 694
},
"total_tokens": 23413
},
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 2
}
},
"estimated_cost": {
"estimated_usd": 0.1340152,
"token_cost_usd": 0.1140152,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"reserved_budget_usd": 6.6,
"http_status": 200,
"service_tier": "default",
"review_note": "Read all eleven audit checks, fast trust checklist and complete concluding advice. Five explicit citation destinations match annotations; correct model/effort and two actual searches, no limits reached. Concrete panel sizes, auditing percentages and citation-support claims are collected model advice, not verified protocol requirements or findings.",
"tool_limit_reached": false,
"body_review": {
"reviewed_at": "2026-11-02T09:37:23.020152+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
},
"search_calls": [
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:help.openai.com ChatGPT search citations search results memory personalization official",
"site:help.openai.com ChatGPT search how it works official",
"site:docs.x.ai Grok search web citations official",
"site:docs.x.ai Grok API search official"
],
"query": "site:help.openai.com ChatGPT search citations search results memory personalization official",
"sources": [
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/11128490-improved-shopping-results-from-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9260256-chatgpt-capabilities-overview"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12677804-what-is-chatgpt-faq"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-faq%2525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.avif"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/6783457-what-is-chatgpt"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12911370-using-shopping-r"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/10408842-google-app-data-controls-faq"
}
]
}
},
{
"type": "web_search_call",
"status": "completed",
"action": {
"type": "search",
"queries": [
"site:docs.x.ai/docs/guides/live-search Grok live search citations",
"site:docs.x.ai \"Live Search\" citations Grok",
"site:help.x.com Grok personalization search web official",
"site:x.ai/news Grok web search official"
],
"query": "site:docs.x.ai/docs/guides/live-search Grok live search citations",
"sources": [
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/docs-to-cited-answers"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook/examples/x-post-fact-check"
},
{
"type": "url",
"url": "https://help.x.com/en/using-x/about-grok"
},
{
"type": "url",
"url": "https://x.ai/pricing"
},
{
"type": "url",
"url": "https://x.ai/build?source=x_timeline_prompt"
},
{
"type": "url",
"url": "https://docs.x.ai/cookbook"
},
{
"type": "url",
"url": "https://docs.x.ai/grok-bot/use-cases"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/tools/x-search"
},
{
"type": "url",
"url": "https://docs.x.ai/grok/faq"
},
{
"type": "url",
"url": "https://x.ai/legal/faq"
},
{
"type": "url",
"url": "https://help.x.com/en/resources/recommender-systems/search-recommendations"
},
{
"type": "url",
"url": "https://data.x.ai/2026-04-07-grok-4-20-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-xai-risk-management-framework.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-08-20-grok-4-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025-09-19-grok-4-fast-model-card.pdf"
},
{
"type": "url",
"url": "https://data.x.ai/2025.02.20-RMF-Draft.pdf"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8590148-memory-in-chatgpt"
},
{
"type": "url",
"url": "https://docs.x.ai/developers/model-capabilities/text/best-practices"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/8313428"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/9237897-chatgpt-search"
},
{
"type": "url",
"url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
}
]
}
}
],
"raw_file": "records/S3-OPENAIAPI-V2-R3.json",
"request_file": "requests/S3-OPENAIAPI-V2-R3.json",
"response_file": "responses/S3-OPENAIAPI-V2-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 7811,
"public_answer_characters": 7721,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-OPENAIAPI-V2-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
Retrieval-panel sources · not matched answer citations
Assistant semantic review of full finalized answer, API response, annotations and request metadata. No external queries, paid calls or independent human double-coding.
coding/S3-OPENAIAPI-V2-R3.json
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| ChatGPT Search | Yes | Yes | No | discussed | unspecified | For ChatGPT search, OpenAI says eligibility for summaries and citations depends in part on allowing `OAI-SearchBot`; placement is still not guaranteed.
| https://help.openai.com/en/articles/9237897-chatgpt-search | |
| X | No | Yes | No | discussed | unspecified | Grok can decide whether to search the web or X and may personalize results using X activity.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://help.openai.com/en/articles/8590148-memory-in-chatgpt | Platform documentation | provisional_from_official_hostname_title_and_attribution | Official OpenAI Help or xAI developer guidance; exact returned title/URL retained. Page contents and support not externally audited. | Supporting documentation, not automatically a tool recommendation | |
| https://docs.x.ai/developers/model-capabilities/text/best-practices | Platform documentation | provisional_from_official_hostname_title_and_attribution | Official OpenAI Help or xAI developer guidance; exact returned title/URL retained. Page contents and support not externally audited. | Supporting documentation, not automatically a tool recommendation | |
| https://help.openai.com/en/articles/8313428 | Platform documentation | provisional_from_official_hostname_title_and_attribution | Official OpenAI Help or xAI developer guidance; exact returned title/URL retained. Page contents and support not externally audited. | Supporting documentation, not automatically a tool recommendation | |
| https://help.openai.com/en/articles/9237897-chatgpt-search | Platform documentation | provisional_from_official_hostname_title_and_attribution | Official OpenAI Help or xAI developer guidance; exact returned title/URL retained. Page contents and support not externally audited. | Supporting documentation, not automatically a tool recommendation | |
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | Platform documentation | provisional_from_official_hostname_title_and_attribution | Official OpenAI Help or xAI developer guidance; exact returned title/URL retained. Page contents and support not externally audited. | Supporting documentation, not automatically a tool recommendation |
Explicit body URLs · parallel representation, not additional citation counts
These manually reviewed Markdown/body links are preserved separately. When they represent the same annotated source occurrence, they are not added again to citation counts or domain sets.
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| https://help.openai.com/en/articles/8590148-memory-in-chatgpt | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/8313428 | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | Explicit Markdown URL matching annotation | ||
| https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | Explicit Markdown URL matching annotation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Awareness",
"definition_verbatim": "**Awareness:** the model recognizes your product."
},
{
"label": "Mention",
"definition_verbatim": "**Mention:** it includes you in an answer."
},
{
"label": "Prominence",
"definition_verbatim": "**Prominence:** your position in the list."
},
{
"label": "Recommendation",
"definition_verbatim": "**Recommendation:** it positively presents you as suitable."
},
{
"label": "Accuracy",
"definition_verbatim": "**Accuracy:** pricing, features and positioning are correct."
},
{
"label": "Citation",
"definition_verbatim": "**Citation:** it links to evidence."
},
{
"label": "Owned citation",
"definition_verbatim": "**Owned citation:** the link goes to your site."
},
{
"label": "Conversion potential",
"definition_verbatim": "**Conversion potential:** the answer gives a user a reason to visit."
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"source_publisher_observations": [
{
"entity": "OpenAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"recommendation_stance": "discussed",
"free_budget_fit": "unspecified",
"evidence": "OpenAI explicitly warns that search results and citations can be incomplete, outdated or incorrect and recommends opening the source to verify support.",
"source_urls": [
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"notes": "Supporting documentation attribution.",
"excluded_from_default_tool_overlap": true
},
{
"entity": "xAI",
"entity_type": "platform_documentation_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"recommendation_stance": "discussed",
"free_budget_fit": "unspecified",
"evidence": "xAI’s own evaluation guidance notes that reasoning-model outputs are nondeterministic, recommends multiple runs, and advises recording model, settings, date and harness version.",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"notes": "Evaluation advice attribution; not a new monitoring tool.",
"excluded_from_default_tool_overlap": true
}
],
"model_proposed_sampling_advice": {
"prompt_range": "30–50",
"repetitions": 3,
"different_days": 2,
"manual_audit_fraction": "10–20%",
"empirically_validated": false,
"notes": "Generated sampling/audit suggestions, not validated universal thresholds."
},
"cost_and_limits_review": {
"estimated_cost": {
"estimated_usd": 0.1340152,
"token_cost_usd": 0.1140152,
"charged_searches_estimate": 2,
"search_count_basis": "reported billable web_search.num_requests",
"billing_reconciled": false
},
"usage": {
"input_tokens": 21162,
"input_tokens_details": {
"cache_write_tokens": 0,
"cached_tokens": 4348
},
"output_tokens": 2251,
"output_tokens_details": {
"reasoning_tokens": 694
},
"total_tokens": 23413
},
"max_tool_calls": 8,
"observed_web_actions": 2,
"max_output_tokens": 32768,
"observed_output_tokens": 2251,
"tool_limit_reached": false,
"output_limit_reached": false,
"authorized_total_budget_USD": 10.0,
"concerns": [
"No observed tool/output cap reached. Estimated US$0.1340152 remains unreconciled; cumulative budget monitoring required."
]
}
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Named search surface and X context are separate from supplied targets, documentation publishers and crawler compatibility. Generic search/research/shopping modes are not invented branded tools.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Fast trust checklist",
"rationale": "Eleven sections and a final checklist emphasize definitions, uncertainty, raw evidence, source/identity checks, recalculation and outcomes; no named monitoring software is suggested."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "“Best alternatives to MyProduct” should not inflate your non-branded discovery score."
},
"repeated_runs": {
"value": true,
"evidence": "A practical bootstrap test is **30–50 important prompts × 3 runs on two different days**."
},
"fresh_sessions": {
"value": false,
"evidence": null
},
"model_search_version_logging": {
"value": true,
"evidence": "Model/version"
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "Search enabled or disabled"
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "Do not accept an API test as a direct proxy for what customers see in the consumer interface unless the vendor demonstrates that the configurations match."
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Separate mention, recommendation and citation"
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "**Owned citation:** the link goes to your site."
},
"raw_answer_inspection": {
"value": true,
"evidence": "Randomly audit at least 10–20% of results."
},
"raw_answer_archiving_explicit": {
"value": true,
"evidence": "You should be able to inspect or export:"
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Request the formula, denominator, weighting, and one worked example."
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "explain every transformation and weight"
},
"composite_score_caution": {
"value": true,
"evidence": "If a dashboard supplies only a composite score or cropped answer snippets, don’t trust it."
},
"uncertainty_variance": {
"value": true,
"evidence": "Mean score plus spread or confidence interval"
},
"human_checking": {
"value": true,
"evidence": "Require exact-match rules plus human review for ambiguous entities."
},
"downstream_conversions": {
"value": true,
"evidence": "Sign-ups containing AI-referral attribution"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"entity_type": "supplied_target_platform",
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual validation",
"free_budget_fit": "unspecified",
"evidence": "A practical bootstrap test is **30–50 important prompts × 3 runs on two different days**.",
"source_urls": [
"https://help.openai.com/en/articles/8590148-memory-in-chatgpt",
"https://help.openai.com/en/articles/8313428",
"https://help.openai.com/en/articles/9237897-chatgpt-search",
"https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results"
],
"notes": "Supplied target. Exact UI/API configurations must be logged; not newly discovered.",
"supplied_in_prompt": true,
"excluded_from_default_overlap": true
},
{
"entity": "Grok",
"entity_type": "supplied_target_platform",
"citation": true,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual validation",
"free_budget_fit": "unspecified",
"evidence": "A practical bootstrap test is **30–50 important prompts × 3 runs on two different days**.",
"source_urls": [
"https://docs.x.ai/developers/model-capabilities/text/best-practices"
],
"notes": "Supplied target. Exact UI/API configurations must be logged; not newly discovered.",
"supplied_in_prompt": true,
"excluded_from_default_overlap": true
}
],
"operational_entities": [
{
"entity": "OAI-SearchBot",
"entity_type": "crawler_agent",
"citation": true,
"mention": true,
"recommendation": false,
"recommendation_stance": "discussed",
"free_budget_fit": "unspecified",
"evidence": "For ChatGPT search, OpenAI says eligibility for summaries and citations depends in part on allowing `OAI-SearchBot`; placement is still not guaranteed.",
"source_urls": [
"https://help.openai.com/en/articles/9237897-chatgpt-search"
],
"notes": "Technical compatibility/eligibility context; unverified claim, not measurement-tool adoption.",
"recommended_action": "Audit crawler access without assuming placement",
"excluded_from_default_tool_overlap": true
}
],
"contextual_non_tool_entities": [
{
"entity": "MyProduct",
"entity_type": "illustrative_brand_placeholder",
"citation": false,
"mention": true,
"recommendation": false,
"recommendation_stance": "discussed",
"free_budget_fit": "unspecified",
"evidence": "“Best alternatives to MyProduct” should not inflate your non-branded discovery score.",
"source_urls": [],
"notes": "Illustrative placeholder; excluded from tool overlap."
}
],
"generic_methods": [
"Eleven-part score audit",
"Segmented representative prompt panel",
"Branded/non-branded separation",
"Exact environment and harness disclosure",
"Repeated multi-day sampling with failures retained",
"Manual raw-result auditing",
"Separate awareness, mention, recommendation and citation measures",
"Source validation and entity resolution",
"Crawler-access audit",
"Independent score recalculation",
"Referral and conversion cross-checks"
],
"metric_definitions_observed": [
{
"label": "Awareness",
"definition_verbatim": "**Awareness:** the model recognizes your product."
},
{
"label": "Mention",
"definition_verbatim": "**Mention:** it includes you in an answer."
},
{
"label": "Prominence",
"definition_verbatim": "**Prominence:** your position in the list."
},
{
"label": "Recommendation",
"definition_verbatim": "**Recommendation:** it positively presents you as suitable."
},
{
"label": "Accuracy",
"definition_verbatim": "**Accuracy:** pricing, features and positioning are correct."
},
{
"label": "Citation",
"definition_verbatim": "**Citation:** it links to evidence."
},
{
"label": "Owned citation",
"definition_verbatim": "**Owned citation:** the link goes to your site."
},
{
"label": "Conversion potential",
"definition_verbatim": "**Conversion potential:** the answer gives a user a reason to visit."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 7721,
"transport_body_complete": true,
"response_status": "completed",
"incomplete_details": null,
"coherent_ending": true,
"url_citation_occurrences": 5,
"explicit_body_URL_occurrences": 5,
"unique_answer_URLs": 5,
"unique_cited_hostnames": 2,
"unique_normalized_cited_domains": 2,
"citation_mapping_complete": true,
"source_inventory_complete_for_answer_URLs": true,
"source_inventory_scope": "Five annotated and explicit source URLs, all unique: four OpenAI Help pages and one xAI page. Retrieval URLs remain separate.",
"retrieval_inventory_complete": false,
"search_observed": true,
"search_enabled_API": true,
"web_action_count": 2,
"search_action_count": 2,
"reported_billable_search_requests": 2,
"requested_model": "gpt-5.6-sol",
"returned_model": "gpt-5.6-sol",
"requested_reasoning_effort": "medium",
"returned_reasoning": {
"context": "all_turns",
"effort": "medium",
"mode": "standard",
"summary": null
},
"model_and_effort_match": true,
"request_file": "requests/S3-OPENAIAPI-V2-R3.json",
"response_file": "responses/S3-OPENAIAPI-V2-R3.json",
"request_context_check": {
"exact_registered_prompt": true,
"store": false,
"tool_choice": "required"
},
"notes": [
"Two completed searches and two reported billable requests; no cap observed.",
"Generic citation and owned citation are explicitly different concepts.",
"No explicit fresh-chat/reset requirement appears in this answer, despite extensive account-context logging advice.",
"Provider behavior and generated sample-size/audit advice are not independently verified.",
"The collection surface is OpenAI API + web search, not consumer ChatGPT Search."
],
"original_capture_answer_characters": 7811,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Manual provider subdomain normalization help.openai.com to openai.com and docs.x.ai to x.ai. Exact long FAQ URL remains as returned; no external lookup or URL repair.",
"mapping": {
"help.openai.com": "openai.com",
"docs.x.ai": "x.ai"
}
}
}Search eligibility and X context are not active recommendations to adopt new measurement tools.
Supporting source identity does not imply a tool recommendation.
Awareness, generic citation and owned citation are separate definitions.
No branded spreadsheet, analytics platform or visibility checker is named.
Fresh-session advice is not inferred from controls logging.
S3-GROK-V2-R1 · Grok Expert · consumer interface · completed
Actual search activity needs reopening; final badge alone does not establish search tool use.
Saved HTML-only export did not include the rendered answer DOM; it is not treated as a complete answer or citation capture.
URL display normalization may omit www; exact href not independently recovered. Observed target strings are retained.
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | GitHub, Google Analytics 4, Google Search Console, Reddit, X |
| Mentioned tools / services / surfaces | Bing, Claude, GitHub, Google AI Mode, Google AI Overviews, Google Analytics 4, Google Search Console, Reddit, X |
| Cited domains · mapping complete | Unavailable · no eligible answers |
| Observed inline cited domains · coverage qualified | amicited.com, businessinsider.com, searchscope.com.au, visilay.com |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved |
{ "cited_domains": "Citation mapping is not confirmed complete; use separately labeled observed-link comparisons." }
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
**Treat reported ChatGPT and Grok visibility scores as directional hypotheses, not reliable facts, until you can independently verify them.** There is no standard “AI visibility score.” Tools use different formulas, denominators, prompt sets, sampling methods, models, and surfaces (API vs. consumer interface). The same brand can score anywhere from single digits to the high 60s on identical underlying answers depending on how the number is calculated. ### 1. Demand full methodology details Before trusting any number, get clear answers on: - Exact formula and weights (mention rate, share of voice, position/prominence, citations, sentiment, adversarial resistance, etc.). - Denominator (all answers vs. only answers that mention at least one brand). - The specific prompt set and how it was chosen (real buyer-intent questions vs. branded or generic ones). - Sampling: how many runs per prompt, which models/versions, web search on or off, date of collection, and whether it used the consumer web interface or an API. - Per-engine breakdowns rather than a single blended score. Different tools produce materially different scores from the same data. One test of identical ChatGPT answers produced scores ranging from ~8.5 to 67 across calculation methods for the same brand. A single blended number can also hide large engine differences (e.g., strong on Grok/Claude but near-zero on ChatGPT). ### 2. Replicate the core results yourself This is the highest-value check for a bootstrapped builder because it costs only time. - Extract or recreate the key prompts (category queries, “best X for Y,” alternatives, comparisons, pricing, problem-solution questions relevant to your SaaS). - Run each prompt 3–5 times in the actual consumer interfaces your buyers use: ChatGPT with search/browsing enabled, and Grok. - Log consistently: whether your product is mentioned, its position, whether your domain is cited, accuracy of the description, and which competitors appear. - Compare your results to the reported score. Large mismatches indicate the score is measuring a different surface, prompt set, or calculation method than what real users experience. AI answers are stochastic. In one controlled study, the same engine asked the same questions again produced only ~43% top-five shortlist overlap; cross-engine overlap was far lower (~17%). A single run or small sample is noisy. ### 3. Account for engine-specific behavior and surface mismatch ChatGPT and Grok do not measure the same thing: - ChatGPT relies more on training data plus web retrieval (historically Bing-derived) and favors established third-party authority, directories, reviews, and consistent entity coverage. - Grok has native access to real-time X (Twitter) data and weights recent engagement heavily. Visibility can shift in hours based on X activity; traditional web authority matters less relatively. Many tools query APIs. These do not perfectly match consumer apps (different memory/context, personalization, exact retrieval, and prompt rewriting). Studies have shown large gaps between API-reported citations and what ChatGPT’s consumer interface actually cites. ### 4. Check relative standing, trends, and independent signals - Compare your score to the same tool’s scores for direct competitors on the identical prompt set. Absolute numbers are less useful than relative position. - Prefer multi-week or multi-month trends over one-off snapshots. Short-term noise is high. - Cross-check with signals you control or can observe for free: - GA4 referrals from chatgpt.com, grok-related domains, or similar. - Google Search Console data on AI Overviews / AI Mode impressions (where available). - Community and third-party presence (Reddit, review sites, GitHub, directories, X engagement)—these heavily influence what the models say. - Direct customer attribution (“How did you hear about us?”). ### 5. Evaluate the vendor and the score’s practical value Prefer tools that publish methodology, raw answers, or scoring code, or that let you inspect the underlying responses. Be cautious of black-box scores that primarily support upsells for optimization services. Ask whether the score is a presence measure (based on actual answers) or a readiness/proxy measure (based on site signals). The latter does not directly measure what buyers see. ### Practical low-cost protocol Create a simple spreadsheet with 10–20 high-value buyer prompts. Run them weekly or bi-weekly on ChatGPT (search on) and Grok. Track mention rate, citation rate, position, accuracy, and share of voice versus 3–5 competitors. This gives you a grounded baseline you control and lets you validate any paid tool’s numbers against reality. In short: treat the reported scores as a starting hypothesis. Verify the methodology, replicate key prompts yourself on the consumer interfaces, separate ChatGPT from Grok results, and cross-check against traffic and community signals. That combination is far more reliable for a bootstrapped SaaS than any single vendor number.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V2-R1",
"slot_id": "S3-GROK-V2-R1",
"platform": "Grok",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 1,
"submitted_at": "2026-11-02T06:18:15.333Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:23:03.851Z",
"source_panel_declared_hits": 45,
"source_inventory_scope": "Inline displayed links retained; expanded Visilay +1 contained Visilay and Markets Insider, both already listed.",
"capture_method": "Native complete rendered response corroborated against full Copy response in offline buffer.",
"search_evidence": "Expanded native activity shows web searches and browsed links. Badge count is not used as search evidence.",
"raw_file": "records/S3-GROK-V2-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 5038,
"public_answer_characters": 5038,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V2-R1",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| AI visibility score: what it is and how it's calculated | Visilay (Visilay) | https://visilay.com/blog/ai-visibility-score/ | visilay.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| AI visibility score: what it is and how it's calculated | Visilay (Visilay) | https://visilay.com/blog/ai-visibility-score/ | visilay.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| There Is No Single “AI Visibility” Score; Ask Four AIs, Get Four Shortlists | Markets Insider (Markets.businessinsider) | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| There Is No Single “AI Visibility” Score; Ask Four AIs, Get Four Shortlists | Markets Insider (Markets.businessinsider) | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| Grok Is Citing Brands Too: The Data Behind Its Citation Logic | Am I Cited (Amicited) | https://amicited.com/blog/grok-is-citing-brands-too/ | amicited.com | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| Are AI Visibility Tools Accurate? What They Miss (2026) (Searchscope) | https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | searchscope.com.au | Inline answer link | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete copied answer and existing native UI/activity/citation-group captures. No external webpage access or fact verification; no independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Analytics 4 | GA4 | No | Yes | Yes | suggested | stated free | GA4 referrals from chatgpt.com, grok-related domains, or similar.
| |
| Google Search Console | Google Search Console | No | Yes | Yes | conditional | stated free | Google Search Console data on AI Overviews / AI Mode impressions (where available).
| |
| No | Yes | Yes | suggested | stated free | Community and third-party presence (Reddit, review sites, GitHub, directories, X engagement)—these heavily influence what the models say.
| |||
| GitHub | GitHub | No | Yes | Yes | suggested | stated free | Community and third-party presence (Reddit, review sites, GitHub, directories, X engagement)—these heavily influence what the models say.
| |
| X | X (Twitter); X engagement | No | Yes | Yes | suggested | stated free | Community and third-party presence (Reddit, review sites, GitHub, directories, X engagement)—these heavily influence what the models say.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | A single blended number can also hide large engine differences (e.g., strong on Grok/Claude but near-zero on ChatGPT).
| |
| Bing | Bing-derived | No | Yes | No | discussed | unspecified | ChatGPT relies more on training data plus web retrieval (historically Bing-derived) and favors established third-party authority, directories, reviews, and consistent entity coverage.
| |
| Google AI Overviews | AI Overviews | No | Yes | No | discussed | unspecified | Google Search Console data on AI Overviews / AI Mode impressions (where available).
| |
| Google AI Mode | AI Mode | No | Yes | No | discussed | unspecified | Google Search Console data on AI Overviews / AI Mode impressions (where available).
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://visilay.com/blog/ai-visibility-score/ | Other / unresolved | provisional_from_returned_title_URL_and_captured_UI_card_only | Title and expanded card describe an explanatory article about visibility-score formulas. Vendor documentation, practitioner tutorial or original vendor research is not established from the available metadata. | supporting citation; no active recommendation to use or follow the publisher | |
| https://visilay.com/blog/ai-visibility-score/ | Other / unresolved | provisional_from_returned_title_URL_and_captured_UI_card_only | Title and expanded card describe an explanatory article about visibility-score formulas. Vendor documentation, practitioner tutorial or original vendor research is not established from the available metadata. | supporting citation; no active recommendation to use or follow the publisher | |
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_title_URL_and_captured_UI_card_only | Returned title reports a multi-engine shortlist study. The /news/stocks/ path also permits syndicated/promotional material; primary research authorship is unresolved. | supporting citation; no active recommendation to use or follow the publisher | |
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_title_URL_and_captured_UI_card_only | Returned title reports a multi-engine shortlist study. The /news/stocks/ path also permits syndicated/promotional material; primary research authorship is unresolved. | supporting citation; no active recommendation to use or follow the publisher | |
| https://amicited.com/blog/grok-is-citing-brands-too/ | Other / unresolved | provisional_from_returned_title_URL_and_captured_UI_card_only | A branded blog title claims data about Grok citation logic; original empirical study versus commercial explanatory content is unresolved. | supporting citation; no active recommendation to use or follow the publisher | |
| https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | Other / unresolved | provisional_from_returned_title_URL_and_captured_UI_card_only | Title and /ai-seo/ path suggest explanatory/practitioner or commercial content, without establishing author or original research. | supporting citation; no active recommendation to use or follow the publisher |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[]
Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools, services and AI surfaces. Generic methods and supplied platforms separate. Citation-only publishers excluded from default tool overlap.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Demand full methodology details",
"rationale": "Five numbered audit/validation dimensions dominate; manual replication and corroborating analytics support the checks. No named AI visibility checker is recommended."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "The specific prompt set and how it was chosen (real buyer-intent questions vs. branded or generic ones)."
},
"repeated_runs": {
"value": true,
"evidence": "Run each prompt 3–5 times in the actual consumer interfaces your buyers use: ChatGPT with search/browsing enabled, and Grok."
},
"fresh_sessions": {
"value": false,
"evidence": null
},
"model_search_version_logging": {
"value": true,
"evidence": "Sampling: how many runs per prompt, which models/versions, web search on or off, date of collection, and whether it used the consumer web interface or an API."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "web search on or off"
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Track mention rate, citation rate, position, accuracy, and share of voice versus 3–5 competitors."
},
"raw_answer_inspection": {
"value": true,
"evidence": "Prefer tools that publish methodology, raw answers, or scoring code, or that let you inspect the underlying responses."
},
"numerator_denominator_definition": {
"value": true,
"evidence": "Denominator (all answers vs. only answers that mention at least one brand)."
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "Exact formula and weights (mention rate, share of voice, position/prominence, citations, sentiment, adversarial resistance, etc.)."
},
"composite_score_caution": {
"value": true,
"evidence": "Per-engine breakdowns rather than a single blended score."
},
"uncertainty_variance": {
"value": true,
"evidence": "AI answers are stochastic."
},
"human_checking": {
"value": true,
"evidence": "Replicate the core results yourself"
},
"downstream_conversions": {
"value": true,
"evidence": "Direct customer attribution (“How did you hear about us?”)."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual replication",
"evidence": "Run each prompt 3–5 times in the actual consumer interfaces your buyers use: ChatGPT with search/browsing enabled, and Grok.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Explicitly suggested for checks, but supplied in the question and excluded from discovered tool overlap by default."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for manual replication",
"evidence": "Run each prompt 3–5 times in the actual consumer interfaces your buyers use: ChatGPT with search/browsing enabled, and Grok.",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Explicitly suggested for checks, but supplied in the question and excluded from discovered tool overlap by default."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Unnamed spreadsheet with fixed buyer prompts",
"Competitor-relative comparisons within identical prompt sets",
"Weekly/biweekly repeated consumer-interface runs",
"Review sites and directories as corroborating presence signals",
"Direct customer attribution",
"Inspecting vendor methodology, underlying raw answers and score code"
],
"capture_scope": {
"body_complete": true,
"observed_inline_link_occurrences": 6,
"unique_standardized_answer_URLs": 4,
"unique_displayed_answer_domains": 4,
"source_badge_hits": 45,
"retrieval_inventory_complete": false,
"citation_mapping_complete": null,
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"citation_group_evidence_file": "[Private capture retained locally.]",
"native_browsed_link_domains": [
"markets.businessinsider.com",
"visilay.com",
"searchscope.com.au"
],
"captured_exact_query_labels": [
"Grok AI visibility scoring how Grok differs from ChatGPT in brand mentions"
],
"notes": [
"Visilay and Markets Insider each occur twice in the recorded inline list. Deduplicate for domain/URL set comparisons; retain six link occurrences separately.",
"Expanding Visilay +1 revealed Visilay and Markets Insider, both already recorded. This does not add two unique sources.",
"Captured displayed targets omit the scheme and may omit www. The standardized https URLs are not independently recovered exact hrefs. The citation-group capture separately displays https://www.visilay.com/blog/ai-visibility-score/. Domain comparison is robust to this leading-www difference; exact-URL equivalence beyond that is not asserted.",
"Native activity directly shows a grouped three-search control, a separate web-search query label and browsed-link controls. Actual search is observed independently of the 45-hit badge. Exact query-label coverage remains partial.",
"The raw notes retain an earlier statement that activity needed reopening; the later search_observed=true and explicit activity capture supersede that pending status without changing the historical note.",
"Workspace-explored activity is visible, but contents and influence are unestablished; isolation is not audited.",
"Saved HTML lacking the rendered answer DOM is not treated as an independent complete body/citation capture.",
"All research statistics, relative engine behavior, analytics capabilities and costs are platform claims, not verified facts."
],
"original_capture_answer_characters": 5038,
"answer_characters": 5038,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping, retaining displayed hostnames. Market Insider hostname collapsed to businessinsider.com; searchscope.com.au retained. Leading www variants treated as same normalized domain, not as proof of exact href. No public-suffix software or external lookup used.",
"mapping": {
"visilay.com": "visilay.com",
"markets.businessinsider.com": "businessinsider.com",
"amicited.com": "amicited.com",
"searchscope.com.au": "searchscope.com.au"
}
}
}Recommended corroborating channels are counted separately by role from score-checking software; monitoring existing community presence is not a publication recommendation.
Google AI Overviews and AI Mode are contextual coverage mentions. Coding the GSC recommendation as a recommendation of every named covered surface would overcount tools.
Citation labels name Visilay, Markets Insider, Am I Cited and Searchscope, but the substantive prose does not name or recommend them. No complete source-publisher C/M/R census is added to default tool counts.
C=false means no own-content link observed in the captured answer links; missing full retrieval inventory is not automatically a missing citation.
The answer separates mention/citation metrics but does not explicitly operationalize active recommendation rate as a third metric.
S3-GROK-V2-R2 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Google Analytics 4, X |
| Mentioned tools / services / surfaces | Claude, Google AI (unspecified surface), Google Analytics 4, Perplexity, Reddit, X, YouTube |
| Cited domains · mapping complete | aeotrack.io, businessinsider.com, loudface.co, paprik.ai, sparktoro.com |
| Observed inline cited domains · coverage qualified | aeotrack.io, businessinsider.com, loudface.co, paprik.ai, sparktoro.com |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved, Tool/product documentation or sales page, Vendor research or case study |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
Treat reported ChatGPT and Grok visibility scores as directional signals with wide error bars, not precise measurements. They are useful for spotting trends and gaps, but many factors make single numbers misleading. For a bootstrapped SaaS builder, prioritize low-cost manual checks over paid dashboards until the data holds up under scrutiny. Core Problems with These Scores Visibility scores (typically a 0–100 index or mention rate) measure how often or prominently your product appears in AI answers to tracked prompts. Common definitions include share of answers that name you, position-weighted mentions, or blends of presence + citation + sentiment.Aeotrack They are unreliable for several documented reasons: High variance / non-determinism: The same prompt rarely produces the same brand list. SparkToro research (nearly 3,000 runs across ChatGPT, Claude, and Google AI) found less than a 1-in-100 chance of identical brand lists and roughly 1-in-1,000 for the same list in the same order. Top brands appear more consistently (e.g., 55–97% inclusion rates in some categories), but exact positions and shortlists fluctuate heavily.Sparktoro Engine disagreement: ChatGPT, Grok, Claude, and Perplexity often recommend different shortlists. One 2026 benchmark of B2B tech PR questions found only ~17% average top-five overlap across engines versus ~43% for repeated asks to the same engine. Blended scores can hide that you are strong on one and invisible on another.Markets.businessinsider API vs. consumer interface gaps: Many tools query APIs. These frequently differ from what users see in the ChatGPT or Grok web/app interfaces (search behavior, sources, brand sets, response length, location/personalization). Studies have reported brand or source overlaps as low as ~15–25% in some comparisons.Paprik Prompt and methodology dependence: Scores depend on the exact prompts (ideally unbranded buyer-intent questions), number of runs, models, whether web search is enabled, scoring formula (mention rate vs. position-weighted vs. proprietary), and whether raw answers are available. Formulas and weights vary widely across vendors.Loudface Grok additionally weights freshness and its own ecosystem (especially X/Twitter citations and social signals) more heavily than ChatGPT, which leans more on training data plus search and entity authority. Practical Checks Before Trusting a Score Do these in roughly this order. Most are free or low-effort. Demand (or review) the methodology Ask the tool or agency: exact scoring formula, number of prompts and runs per prompt, models used, web search on/off, API vs. UI/scraping, date range, and whether scores are engine-specific or blended. Prefer tools that show raw answers, citations, and per-engine breakdowns. Opaque or frequently retroactively adjusted scores are red flags. Separate ChatGPT and Grok (and ignore pure blends) Get engine-specific numbers. A strong blended score can mask weakness on the platform your buyers actually use most (ChatGPT currently has far higher usage share). Manually verify on the real interfaces Use fresh, logged-out sessions (or clear memory/history). Enable web search/research mode. Test 8–15 realistic, unbranded buyer prompts relevant to your SaaS (e.g., “best [category] tools for [ICP/use case]”, “[competitor] alternatives”, “how to [solve problem your product addresses]”, comparison or feature-specific questions). Avoid only branded queries like “What is [your product]?”. Run each key prompt 3–5 times and note: mention (yes/no), position if listed, citation of your domain, accuracy of description, sentiment, and which competitors appear. Record date, exact prompt, and model if visible. Compare your notes to the reported score. Large discrepancies mean the tool’s setup does not match real user experience. Check prompt relevance and coverage Confirm the tracked prompts match how your actual buyers research and evaluate tools. Synthetic or keyword-derived prompts can inflate or deflate scores. Also note prompt coverage—how many of your important buyer questions you appear in at all. Distinguish mentions, citations, position, and accuracy A mention without a citation or with incorrect facts is weaker. Hallucinations about features, pricing, or positioning are common; flag them. Position in a shortlist matters more than a passing mention. Sentiment scoring is often noisy—spot-check the actual language. Look at sources and Grok-specific signals Review which domains or posts the engines cite. For Grok, check recent activity and mentions on X, as it prioritizes that ecosystem. For ChatGPT, entity strength, third-party coverage (reviews, listicles, Reddit, YouTube, etc.), and content quality/freshness matter more. Assess sample size and trends over absolutes Single runs or tiny prompt sets are mostly noise. Prefer tools that average many runs and show ranges or trends. Track relative movement (your share vs. key competitors) more than any absolute number. Correlate with any AI referral traffic in GA4 or similar if available. Cross-check free or lightweight options Manually as above, or try limited free tiers/audits from tools that publish methodology. Free checks give a baseline without commitment. Bottom Line for a Bootstrapped Builder Use scores for relative trends, competitor gaps, and prioritizing content or third-party mentions (YouTube, branded web mentions, and community sources often correlate more strongly with visibility than traditional backlinks alone). Do not treat a single number as truth or as a direct proxy for pipeline. Start with manual UI checks on your most important prompts; only invest in paid tracking once those match the reported data and the methodology is transparent. Focus effort on building corroborating signals (clear product descriptions, third-party coverage, structured content, and presence where the engines source information) rather than optimizing for any one dashboard.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V2-R2",
"slot_id": "S3-GROK-V2-R2",
"platform": "Grok",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 2,
"submitted_at": "2026-11-02T06:42:16.030Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:44:14.483Z",
"source_panel_declared_hits": 75,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-V2-R2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 5960,
"public_answer_characters": 5960,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V2-R2",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Aeotrack | https://www.aeotrack.io/methodology | www.aeotrack.io | Inline answer link | Rendered DOM href |
| Sparktoro | https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/ | sparktoro.com | Inline answer link | Rendered DOM href |
| Markets.businessinsider | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Rendered DOM href |
| Paprik | https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui | paprik.ai | Inline answer link | Rendered DOM href |
| Loudface | https://www.loudface.co/blog/what-is-ai-visibility | www.loudface.co | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 5,960-character rendered body, five captured DOM hrefs and existing browser snapshots/activity; no external webpage queries, research audit or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Analytics 4 | GA4 | No | Yes | Yes | conditional | unspecified | Correlate with any AI referral traffic in GA4 or similar if available.
| |
| X | X/Twitter | No | Yes | Yes | suggested | unspecified | For Grok, check recent activity and mentions on X, as it prioritizes that ecosystem.
| |
| No | Yes | No | discussed | unspecified | For ChatGPT, entity strength, third-party coverage (reviews, listicles, Reddit, YouTube, etc.), and content quality/freshness matter more.
| |||
| YouTube | YouTube | No | Yes | No | discussed | unspecified | For ChatGPT, entity strength, third-party coverage (reviews, listicles, Reddit, YouTube, etc.), and content quality/freshness matter more.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | Engine disagreement: ChatGPT, Grok, Claude, and Perplexity often recommend different shortlists.
| |
| Perplexity | Perplexity | No | Yes | No | discussed | unspecified | Engine disagreement: ChatGPT, Grok, Claude, and Perplexity often recommend different shortlists.
| |
| Google AI (unspecified surface) | Google AI | No | Yes | No | discussed | unspecified | SparkToro research (nearly 3,000 runs across ChatGPT, Claude, and Google AI)
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://www.aeotrack.io/methodology | Tool/product documentation or sales page | provisional_from_answer_attribution_and_returned_label_URL_only | Branded /methodology destination provisionally represents product/scoring documentation; content and ownership not independently inspected. | supporting citation; no explicit instruction to read/follow the source | |
| https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/ | Vendor research or case study | provisional_from_answer_attribution_and_returned_label_URL_only | Body explicitly attributes nearly 3,000-run research to SparkToro, with a same-brand new-research blog URL. Original-research/vendor classification is provisional; data, methods and commercial provenance were not independently audited. | supporting citation; no explicit instruction to read/follow the source | |
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_answer_attribution_and_returned_label_URL_only | Markets Insider stock-news URL supports a benchmark comparison; syndicated or promotional provenance remains possible and primary authorship unresolved. | supporting citation; no explicit instruction to read/follow the source | |
| https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui | Other / unresolved | provisional_from_answer_attribution_and_returned_label_URL_only | Branded blog path claims accurate API-based tracking against ChatGPT UI, but original measurement versus product explanation is unresolved from metadata. | supporting citation; no explicit instruction to read/follow the source | |
| https://www.loudface.co/blog/what-is-ai-visibility | Other / unresolved | provisional_from_answer_attribution_and_returned_label_URL_only | What-is-ai-visibility blog path suggests explanatory content; practitioner, editorial and commercial roles remain unresolved without reading the page. | supporting citation; no explicit instruction to read/follow the source |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Visibility score",
"definition_verbatim": "Visibility scores (typically a 0–100 index or mention rate) measure how often or prominently your product appears in AI answers to tracked prompts."
},
{
"label": "Possible score definitions",
"definition_verbatim": "Common definitions include share of answers that name you, position-weighted mentions, or blends of presence + citation + sentiment.",
"notes": "A conceptual share-of-answers denominator is stated; no fixed eligibility rule or complete operational numerator/denominator procedure is supplied."
},
{
"label": "Prompt coverage",
"definition_verbatim": "how many of your important buyer questions you appear in at all"
}
]Additional reviewed coding · publishers, generated rules and cost checks
Generated scoring thresholds or sampling rules are not independently validated. Source-publisher observations are separate from the default tool/entity distributions.
{
"research_entity_observations": [
{
"entity": "SparkToro",
"entity_type": "research_publisher",
"citation": true,
"mention": true,
"recommendation": false,
"evidence": "SparkToro research (nearly 3,000 runs across ChatGPT, Claude, and Google AI) found less than a 1-in-100 chance of identical brand lists and roughly 1-in-1,000 for the same list in the same order.",
"source_urls": [
"https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/"
],
"excluded_from_default_tool_overlap": true,
"notes": "Explicit substantive research attribution with a same-brand blog citation. This codes citation/mention behavior, not independent verification of the study. No instruction to use, read or follow SparkToro."
}
]
}Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/surfaces, with supplied targets and the explicitly attributed research publisher separate. Citation-label-only brands are not added as substantive tool mentions. No cross-answer comparison is performed in this record.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Practical Checks Before Trusting a Score",
"rationale": "Methodology, interface equivalence, repetition, interpretation and raw-answer checks dominate; manual consumer-interface validation is prioritized over dashboards. No specific visibility checker is actively recommended."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "Avoid only branded queries like “What is [your product]?”."
},
"repeated_runs": {
"value": true,
"evidence": "Run each key prompt 3–5 times and note: mention (yes/no), position if listed, citation of your domain, accuracy of description, sentiment, and which competitors appear."
},
"fresh_sessions": {
"value": true,
"evidence": "Use fresh, logged-out sessions (or clear memory/history)."
},
"model_search_version_logging": {
"value": true,
"evidence": "Ask the tool or agency: exact scoring formula, number of prompts and runs per prompt, models used, web search on/off, API vs. UI/scraping, date range, and whether scores are engine-specific or blended."
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "web search on/off"
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "Enable web search/research mode."
},
"API_consumer_UI_distinction": {
"value": true,
"evidence": "API vs. consumer interface gaps: Many tools query APIs."
},
"citations_mentions_recommendations_distinguished": {
"value": false,
"evidence": null
},
"citation_and_mention_rates_separate": {
"value": false,
"evidence": null
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Distinguish mentions, citations, position, and accuracy"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Prefer tools that show raw answers, citations, and per-engine breakdowns."
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "share of answers that name you"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "exact scoring formula"
},
"composite_score_caution": {
"value": true,
"evidence": "Get engine-specific numbers."
},
"uncertainty_variance": {
"value": true,
"evidence": "Prefer tools that average many runs and show ranges or trends."
},
"human_checking": {
"value": true,
"evidence": "Manually verify on the real interfaces"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Correlate with any AI referral traffic in GA4 or similar if available."
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual checks on the real consumer interface",
"evidence": "Start with manual UI checks on your most important prompts; only invest in paid tracking once those match the reported data and the methodology is transparent.",
"supporting_context": "Manually verify on the real interfaces",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "The task names both platforms and the workflow explicitly calls for their separate real-interface checks; they are not newly discovered tools."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested manual checks on the real consumer interface",
"evidence": "Start with manual UI checks on your most important prompts; only invest in paid tracking once those match the reported data and the methodology is transparent.",
"supporting_context": "Manually verify on the real interfaces",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "The task names both platforms and the workflow explicitly calls for their separate real-interface checks; they are not newly discovered tools."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Formula, sample-size and per-engine methodology audit",
"Consumer-interface versus API setup matching",
"Repeated unbranded buyer-prompt checks",
"Per-run mention, position, own-domain citation, accuracy and competitor logging",
"Prompt coverage, trends and uncertainty checks",
"Reviewing source domains and posts",
"Generic free/lightweight tiers with transparent methods",
"Third-party coverage and accurate structured content"
],
"metric_definitions_observed": [
{
"label": "Visibility score",
"definition_verbatim": "Visibility scores (typically a 0–100 index or mention rate) measure how often or prominently your product appears in AI answers to tracked prompts."
},
{
"label": "Possible score definitions",
"definition_verbatim": "Common definitions include share of answers that name you, position-weighted mentions, or blends of presence + citation + sentiment.",
"notes": "A conceptual share-of-answers denominator is stated; no fixed eligibility rule or complete operational numerator/denominator procedure is supplied."
},
{
"label": "Prompt coverage",
"definition_verbatim": "how many of your important buyer questions you appear in at all"
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 5960,
"observed_inline_link_occurrences": 5,
"unique_DOM_answer_URLs": 5,
"unique_cited_domains": 5,
"source_badge_hits": 75,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Five inline DOM hrefs from the complete rendered answer, not an inventory of all 75 retrieval hits.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"complete_rendered_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [],
"notes": [
"Recorded activity contains grouped searches of two, three and two, plus browsed links. Query strings are not expanded in the supplied snapshot.",
"The 75-source badge is not a count of unique inline citations.",
"Workspace-explored activity labels are visible, but their underlying contents and influence remain unresolved.",
"Five source labels are included in innerText; label-only brands are not automatically substantive mentions or recommendations.",
"Research sample sizes, probabilities, overlap percentages, usage-share claims, product features and engine-weighting assertions are all platform output, not independently verified facts."
],
"original_capture_answer_characters": 5960,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping retains exact DOM hrefs/hostnames; leading www removed for aeotrack.io/loudface.co and Markets Insider collapsed to businessinsider.com. No external public-suffix lookup used.",
"mapping": {
"www.aeotrack.io": "aeotrack.io",
"sparktoro.com": "sparktoro.com",
"markets.businessinsider.com": "businessinsider.com",
"paprik.ai": "paprik.ai",
"www.loudface.co": "loudface.co"
}
}
}GA4 is optional downstream analytics; X is explicitly recommended for checking recent activity, not automatically for publishing.
YouTube/Reddit are examples inside explanatory coverage/correlation claims. General advice to prioritize third-party coverage is not turned into a specific service-use recommendation.
Generic free tiers/audits are suggested without naming particular checker products; do not infer them from citation labels.
Google AI is an unspecified surface family in a research description, not a resolved Google product.
SparkToro is substantively named and cited as a research publisher but is not recommended, and is excluded from tool-only overlap.
The answer distinguishes mentions/citations/position/accuracy, but does not define an active-recommendation rate.
S3-GROK-V2-R3 · Grok Expert · consumer interface · completed
| Analysis set | Observed members |
|---|---|
| Recommended tools / services / surfaces | Google Analytics 4, X |
| Mentioned tools / services / surfaces | Claude, Google Analytics 4, X |
| Cited domains · mapping complete | brantial.ai, businessinsider.com, getmint.ai, presenc.ai |
| Observed inline cited domains · coverage qualified | brantial.ai, businessinsider.com, getmint.ai, presenc.ai |
| Observed cited-source categories · provisional | Editorial or industry reporting, Other / unresolved |
Full original prompt
Recorded submission; unchanged wording.
As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?
Captured answer body
Do not treat a reported ChatGPT or Grok “visibility score” as a reliable, comparable number until you can verify the methodology, replicate key parts of it yourself, and separate the engines. There is no industry-standard score. Different tools use different formulas, prompt sets, engines, and weightings, and the underlying answers are noisy. A recent benchmark found that the same questions produced top-five shortlist overlap of only about 17% across different AI engines (versus ~43% when the same engine was asked again), and one brand could score high on Claude/Grok while being nearly invisible on ChatGPT.Markets.businessinsider 1. Demand (or reconstruct) the methodology Ask for or find: Exact prompt list (buyer-intent, unbranded queries are far more meaningful than “what is [your brand]?”) Engines, models, and whether web search was on Formula: simple mention rate (answers naming you ÷ total answers), position-weighted, sentiment-adjusted, or something else? Sample size (number of answers, runs per prompt, date range) How competitors were chosen A score is only comparable to itself under the same conditions. Changing the prompt set, engine mix, or time window changes the number even if nothing about your product changed.Brantial 2. Replicate it yourself (the highest-ROI check for a bootstrapped builder) Build 10–20 high-intent prompts your actual customers would ask: “Best [your category] tools for [specific use case]” “[Problem] software for [team size / industry]” “Alternatives to [known competitor]” Comparison or “how to choose” questions Then: Run them in fresh ChatGPT and Grok sessions (new chats, memory cleared if possible, web search enabled). Do this across several days. Record for each answer: Was your product named? In what position / prominence? Was your site cited? Sentiment/accuracy? Which competitors appeared? This costs almost nothing and often reveals that the reported score does not match what a real user sees. Multiple runs matter because the same prompt can produce different shortlists (“weather of the day” effects of ~14 points are common).Getmint 3. Separate ChatGPT and Grok results Blended scores hide important differences. ChatGPT and Grok draw from different sources and recommendation styles; Grok is more influenced by recent X/Twitter activity and engagement. A product can look strong on one and weak on the other. Always look at engine-level numbers, not just a composite.Presenc 4. Check prompt relevance and bias Do the prompts reflect real buyer language and intent in your category? Do they favor well-known brands by design (broad category questions often do)? Are branded prompts mixed in (these inflate scores toward 100%)? Prefer unbranded, problem- or comparison-oriented prompts for meaningful measurement. 5. Evaluate reliability and noise Small samples swing wildly. Prefer hundreds of answers over time rather than a one-off snapshot. Day-to-day and run-to-run variance is normal. Look for trends over weeks, not single scores. Note whether the tool accounts for stochasticity with repeated queries. 6. Look past the headline number Useful supporting signals (often more actionable than the score itself): Citation rate — does it link to your site or pages? Position / prominence — first recommendation vs. a passing mention Share of voice versus specific competitors Sentiment and accuracy of the description Actual AI referral traffic in your analytics (GA4 or similar) Mentions without citations or positive recommendations have limited value. 7. Consider the source Is this from an independent benchmark with public methodology and data, or a tool vendor marketing its own product? Can you access raw answers or at least screenshots of key prompts? How recent is the data? AI answers and models change. Practical notes for bootstrapped SaaS Manual spot-checks on a focused prompt set are often enough to decide whether a reported score is directionally useful or mostly noise. Many early-stage products score low or zero on broad category prompts; that is common and not always a crisis if your niche prompts or direct traffic perform well. Focus next on the inputs that actually influence answers: clear entity presence, authoritative third-party mentions (reviews, docs, comparison content, communities), and (especially for Grok) relevant X activity. Track whether changes move the needle in your own repeated manual or low-cost automated checks rather than chasing a vendor’s composite score. Bottom line: Treat any reported visibility score as a hypothesis. Verify the prompts and calculation, re-run the important queries yourself on both ChatGPT and Grok over a few days, and judge the result by whether the answers would actually help a buyer discover and choose your product.
Model, search, time, capture and recovery evidence
{
"id": "S3-GROK-V2-R3",
"slot_id": "S3-GROK-V2-R3",
"platform": "Grok",
"prompt_id": "V2",
"intent": "V",
"wording": 2,
"repeat": 3,
"submitted_at": "2026-11-02T06:47:07.073Z",
"model_label": "Exact backend version undisclosed",
"mode_label": "Expert",
"reasoning_effort": "Not independently selectable",
"private": true,
"personalize_using_x": false,
"personalize_using_history": false,
"search_enabled_ui": null,
"search_observed": true,
"status": "completed",
"body_complete": true,
"captured_at": "2026-11-02T06:49:02.867Z",
"source_panel_declared_hits": 35,
"source_inventory_scope": "Observed inline answer links; grouped citations expanded; source badge and retrieval activity are separate.",
"capture_method": "Complete rendered Grok article innerText; UI heading/duration and source badge removed. Exact inline hrefs extracted from scoped rendered article.",
"raw_file": "records/S3-GROK-V2-R3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 4797,
"public_answer_characters": 4797,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
},
"selected_attempt_id": "S3-GROK-V2-R3",
"historical_ui_attempt_ids": []
}[]
Actual answer links · not the retrieval badge
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Markets.businessinsider | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | Inline answer link | Rendered DOM href |
| Brantial | https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/ | brantial.ai | Inline answer link | Rendered DOM href |
| Getmint | https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt | www.getmint.ai | Inline answer link | Rendered DOM href |
| Presenc | https://presenc.ai/compare/grok-vs-chatgpt-for-brands | presenc.ai | Inline answer link | Rendered DOM href |
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Assistant review of complete 4,797-character rendered answer, four scoped exact DOM hrefs and existing browser snapshots/activity. No external webpage visits, research verification or independent human double-coding.
C / M / R coding and evidence
| Canonical entity | Observed name | C | M | R | Stance | Budget claim | Evidence / note | Supporting URLs |
|---|---|---|---|---|---|---|---|---|
| Google Analytics 4 | GA4 | No | Yes | Yes | conditional | unspecified | Actual AI referral traffic in your analytics (GA4 or similar)
| |
| X | X/Twitter; X activity | No | Yes | Yes | suggested | unspecified | Focus next on the inputs that actually influence answers: clear entity presence, authoritative third-party mentions (reviews, docs, comparison content, communities), and (especially for Grok) relevant X activity.
| |
| Claude | Claude | No | Yes | No | discussed | unspecified | one brand could score high on Claude/Grok while being nearly invisible on ChatGPT.
|
Cited-page category coding · provisional
| URL | Assigned primary category | Candidate only · not aggregated | Classification status | Evidence | Role |
|---|---|---|---|---|---|
| https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | Editorial or industry reporting | provisional_from_returned_label_and_URL_only | Stock-news article supports a benchmark description. Classification is provisional; syndicated/promotional provenance and primary authorship are unresolved. | supporting citation, not an active source/resource recommendation | |
| https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/ | Other / unresolved | provisional_from_returned_label_and_URL_only | The branded score-calculation, measurement or platform-comparison blog path suggests explanatory/practitioner or commercial content; authorship and original empirical research are not established from the metadata. | supporting citation, not an active source/resource recommendation | |
| https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt | Other / unresolved | provisional_from_returned_label_and_URL_only | The branded score-calculation, measurement or platform-comparison blog path suggests explanatory/practitioner or commercial content; authorship and original empirical research are not established from the metadata. | supporting citation, not an active source/resource recommendation | |
| https://presenc.ai/compare/grok-vs-chatgpt-for-brands | Other / unresolved | provisional_from_returned_label_and_URL_only | The branded score-calculation, measurement or platform-comparison blog path suggests explanatory/practitioner or commercial content; authorship and original empirical research are not established from the metadata. | supporting citation, not an active source/resource recommendation |
Metric definitions returned by the model · not adopted instruments
The definitions, scores, sampling quantities and example dashboards below describe model advice. They do not become this experiment’s scoring rules or measured outcomes. Mention-based numerators remain distinct from active-recommendation judgments.
[
{
"label": "Mention rate",
"definition_verbatim": "simple mention rate (answers naming you ÷ total answers)"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate — does it link to your site or pages?"
},
{
"label": "Position / prominence",
"definition_verbatim": "Position / prominence — first recommendation vs. a passing mention"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice versus specific competitors",
"notes": "Named without a formula."
}
]Answer type, advice flags, scope and exclusions
{
"coding_scope": "Substantively named tools/services/surfaces; supplied targets remain separate. Supporting citation-label brands are not converted into substantive tool mentions or resource recommendations.",
"answer_type": {
"primary": "Score validation or audit checklist",
"secondary": [
"Manual measurement workflow",
"Conceptual explanation"
],
"evidence": "Demand (or reconstruct) the methodology",
"rationale": "Seven sections prioritize methodology, replication, per-engine results, prompt bias, noise, supporting signals and provenance; no specific visibility-checker product is suggested."
},
"content_flags": {
"branded_and_unbranded_buyer_prompts": {
"value": true,
"evidence": "buyer-intent, unbranded queries are far more meaningful than “what is [your brand]?”"
},
"repeated_runs": {
"value": true,
"evidence": "Note whether the tool accounts for stochasticity with repeated queries."
},
"fresh_sessions": {
"value": true,
"evidence": "Run them in fresh ChatGPT and Grok sessions (new chats, memory cleared if possible, web search enabled)."
},
"model_search_version_logging": {
"value": true,
"evidence": "Engines, models, and whether web search was on"
},
"search_mode_logging_explicit": {
"value": true,
"evidence": "whether web search was on"
},
"search_mode_enablement_advice": {
"value": true,
"evidence": "web search enabled"
},
"citations_mentions_recommendations_distinguished": {
"value": true,
"evidence": "Mentions without citations or positive recommendations have limited value."
},
"citation_and_mention_rates_separate": {
"value": true,
"evidence": "Citation rate — does it link to your site or pages?"
},
"citation_and_mention_observations_separate": {
"value": true,
"evidence": "Was your product named? In what position / prominence? Was your site cited?"
},
"raw_answer_inspection": {
"value": true,
"evidence": "Can you access raw answers or at least screenshots of key prompts?"
},
"raw_answer_archiving_explicit": {
"value": false,
"evidence": null
},
"numerator_denominator_definition": {
"value": true,
"evidence": "simple mention rate (answers naming you ÷ total answers)"
},
"composite_score_weighting_disclosed_or_checked": {
"value": true,
"evidence": "Formula: simple mention rate (answers naming you ÷ total answers), position-weighted, sentiment-adjusted, or something else?"
},
"composite_score_caution": {
"value": true,
"evidence": "Always look at engine-level numbers, not just a composite."
},
"uncertainty_variance": {
"value": true,
"evidence": "Day-to-day and run-to-run variance is normal. Look for trends over weeks, not single scores."
},
"human_checking": {
"value": true,
"evidence": "Replicate it yourself (the highest-ROI check for a bootstrapped builder)"
},
"downstream_conversions": {
"value": false,
"evidence": null
},
"downstream_traffic_or_impressions": {
"value": true,
"evidence": "Actual AI referral traffic in your analytics (GA4 or similar)"
},
"paid_tool_dependence": {
"value": false,
"evidence": null
}
},
"supplied_entities": [
{
"entity": "ChatGPT",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual checks",
"evidence": "Run them in fresh ChatGPT and Grok sessions (new chats, memory cleared if possible, web search enabled).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target platform; not treated as a new discovered tool."
},
{
"entity": "Grok",
"supplied_in_prompt": true,
"citation": false,
"mention": true,
"recommendation": true,
"recommendation_stance": "suggested for repeated manual checks",
"evidence": "Run them in fresh ChatGPT and Grok sessions (new chats, memory cleared if possible, web search enabled).",
"source_urls": [],
"excluded_from_default_overlap": true,
"notes": "Question-supplied target platform; not treated as a new discovered tool."
}
],
"contextual_non_tool_entities": [],
"generic_methods": [
"Methodology, formula, model/search-state and sampling audit",
"Fixed high-intent unbranded buyer prompts",
"Multi-day fresh-session manual replication",
"Per-answer mention, position, own-site citation, sentiment/accuracy and competitor logging",
"Per-engine rather than blended scores",
"Prompt-bias and variance checks",
"Raw-answer/screenshot inspection",
"Referral analytics",
"Third-party coverage and clear entity/content signals"
],
"metric_definitions_observed": [
{
"label": "Mention rate",
"definition_verbatim": "simple mention rate (answers naming you ÷ total answers)"
},
{
"label": "Citation rate",
"definition_verbatim": "Citation rate — does it link to your site or pages?"
},
{
"label": "Position / prominence",
"definition_verbatim": "Position / prominence — first recommendation vs. a passing mention"
},
{
"label": "Share of voice",
"definition_verbatim": "Share of voice versus specific competitors",
"notes": "Named without a formula."
}
],
"capture_scope": {
"body_complete": true,
"answer_characters": 4797,
"observed_inline_link_occurrences": 4,
"unique_DOM_answer_URLs": 4,
"unique_cited_domains": 4,
"source_badge_hits": 35,
"retrieval_inventory_complete": false,
"citation_mapping_complete": true,
"citation_mapping_scope": "Four inline rendered-article hrefs, not a complete 35-hit retrieval inventory.",
"search_observed": true,
"search_enabled_UI": null,
"activity_evidence_file": "[Private capture retained locally.]",
"complete_rendered_evidence_file": "[Private capture retained locally.]",
"captured_exact_query_labels": [
"\"visibility score\" ChatGPT OR Grok SaaS OR brand"
],
"notes": [
"Activity contains a grouped two-search control, a separately displayed web query with 10 results, and three browsed-link controls. Exact query coverage is partial.",
"The 35-hit source badge is not a unique citation count.",
"Workspace-explored activity labels do not establish the underlying contents or their influence.",
"Benchmark overlap, near-14-point weather effects, brand-score differences, model/engine behavior and reliability claims remain unverified platform output."
],
"original_capture_answer_characters": 4797,
"public_copy_note": "Body length and source hash describe this redacted public record. Original citation position indexes are omitted; semantic judgments are retained."
},
"registrable_domain_mapping": {
"method": "Explicit manual mapping retains exact DOM hrefs/hostnames; markets.businessinsider.com normalizes to businessinsider.com and www.getmint.ai to getmint.ai. No external public-suffix lookup used.",
"mapping": {
"markets.businessinsider.com": "businessinsider.com",
"brantial.ai": "brantial.ai",
"www.getmint.ai": "getmint.ai",
"presenc.ai": "presenc.ai"
}
}
}GA4 is an analytics example and X is content/community advice; neither is a named visibility-scoring checker here.
The answer explicitly distinguishes mentions, citations and positive recommendations, but does not define a separate recommendation-rate formula.
The four cited publishers are not substantively recommended merely because their citation labels are rendered in the answer.
Avoid treating model-generated numerical claims as findings independently established by this experiment.
Every actual submission and retained failure
The attempts ledger already includes selected submissions and historical ChatGPT consumer-UI diagnostics. Do not add it to the 24 planned positions. Recovery captures of an existing answer are not extra submissions. Technical and body-capture failures do not establish absence of tools or sources. Historical UI attempts keep their original platform label and are not API answers.
| Attempt | Position | Actual platform | Interface | Status | Selection | Submitted · UTC+8 | Public record |
|---|---|---|---|---|---|---|---|
| S3-CHATGPT-V1-R1 | S3-CHATGPT-V1-R1 | ChatGPT Search | Not recorded | technical_failure | Historical / not selected | 2026-11-02 14:05:01 +0800 | records/S3-CHATGPT-V1-R1.json |
| S3-CHATGPT-V1-R1-A2 | S3-CHATGPT-V1-R1 | ChatGPT Search | Not recorded | capture_incomplete | Historical / not selected | 2026-11-02 14:09:35 +0800 | records/S3-CHATGPT-V1-R1-A2.json |
| S3-GROK-V1-R1 | S3-GROK-V1-R1 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:14:01 +0800 | records/S3-GROK-V1-R1.json |
| S3-GROK-V2-R1 | S3-GROK-V2-R1 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:18:15 +0800 | records/S3-GROK-V2-R1.json |
| S3-GROK-M2-R1 | S3-GROK-M2-R1 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:24:29 +0800 | records/S3-GROK-M2-R1.json |
| S3-GROK-M1-R1 | S3-GROK-M1-R1 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:29:21 +0800 | records/S3-GROK-M1-R1.json |
| S3-GROK-M1-R2 | S3-GROK-M1-R2 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:34:46 +0800 | records/S3-GROK-M1-R2.json |
| S3-GROK-M2-R2 | S3-GROK-M2-R2 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:37:26 +0800 | records/S3-GROK-M2-R2.json |
| S3-GROK-V1-R2 | S3-GROK-V1-R2 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:39:21 +0800 | records/S3-GROK-V1-R2.json |
| S3-GROK-V2-R2 | S3-GROK-V2-R2 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:42:16 +0800 | records/S3-GROK-V2-R2.json |
| S3-GROK-M2-R3 | S3-GROK-M2-R3 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:45:00 +0800 | records/S3-GROK-M2-R3.json |
| S3-GROK-V2-R3 | S3-GROK-V2-R3 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:47:07 +0800 | records/S3-GROK-V2-R3.json |
| S3-GROK-M1-R3 | S3-GROK-M1-R3 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:49:23 +0800 | records/S3-GROK-M1-R3.json |
| S3-GROK-V1-R3 | S3-GROK-V1-R3 | Grok Expert · consumer interface | Not recorded | completed | Selected evidence | 2026-11-02 14:51:16 +0800 | records/S3-GROK-V1-R3.json |
| S3-CHATGPT-V1-R1-A3 | S3-CHATGPT-V1-R1 | ChatGPT Search | Not recorded | capture_incomplete | Historical / not selected | Not recorded | records/S3-CHATGPT-V1-R1-A3.json |
| S3-OPENAIAPI-V1-R1 | S3-CHATGPT-V1-R1 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:18:30 +0800 | records/S3-OPENAIAPI-V1-R1.json |
| S3-OPENAIAPI-V2-R1 | S3-CHATGPT-V2-R1 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:20:55 +0800 | records/S3-OPENAIAPI-V2-R1.json |
| S3-OPENAIAPI-M2-R1 | S3-CHATGPT-M2-R1 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:22:53 +0800 | records/S3-OPENAIAPI-M2-R1.json |
| S3-OPENAIAPI-M1-R1 | S3-CHATGPT-M1-R1 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:24:46 +0800 | records/S3-OPENAIAPI-M1-R1.json |
| S3-OPENAIAPI-M1-R2 | S3-CHATGPT-M1-R2 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:26:37 +0800 | records/S3-OPENAIAPI-M1-R2.json |
| S3-OPENAIAPI-M2-R2 | S3-CHATGPT-M2-R2 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:28:30 +0800 | records/S3-OPENAIAPI-M2-R2.json |
| S3-OPENAIAPI-V1-R2 | S3-CHATGPT-V1-R2 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:30:13 +0800 | records/S3-OPENAIAPI-V1-R2.json |
| S3-OPENAIAPI-V2-R2 | S3-CHATGPT-V2-R2 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:32:01 +0800 | records/S3-OPENAIAPI-V2-R2.json |
| S3-OPENAIAPI-M2-R3 | S3-CHATGPT-M2-R3 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:34:08 +0800 | records/S3-OPENAIAPI-M2-R3.json |
| S3-OPENAIAPI-V2-R3 | S3-CHATGPT-V2-R3 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:35:59 +0800 | records/S3-OPENAIAPI-V2-R3.json |
| S3-OPENAIAPI-M1-R3 | S3-CHATGPT-M1-R3 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:37:52 +0800 | records/S3-OPENAIAPI-M1-R3.json |
| S3-OPENAIAPI-V1-R3 | S3-CHATGPT-V1-R3 | OpenAI API + web search | Responses API | completed | Selected evidence | 2026-11-02 17:40:11 +0800 | records/S3-OPENAIAPI-V1-R3.json |
S3-CHATGPT-V1-R1 · ChatGPT Search · technical_failure
First submission prioritized for capture preflight; actual order differs from planned order. No extra prompt or follow-up.
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
No readable answer body captured. This is not a negative discovery observation.
Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R1",
"slot_id": "S3-CHATGPT-V1-R1",
"platform": "ChatGPT Search",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T06:05:01.633Z",
"model_label": "GPT-5.6 Sol",
"reasoning_effort": "Medium",
"temporary": true,
"personalization": false,
"search_enabled_ui": true,
"search_observed": false,
"status": "technical_failure",
"body_complete": false,
"captured_at": "2026-11-02T06:05:30.178Z",
"error": "cloudflare_challenge",
"raw_file": "records/S3-CHATGPT-V1-R1.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 0,
"public_answer_characters": 0,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
}
}[]
Actual answer links · not the retrieval badge
No URLs recorded in this field. Completeness and source role are reported separately.
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Not coded. No entity or source absence is inferred.
S3-CHATGPT-V1-R1-A2 · ChatGPT Search · capture_incomplete
Authorized supplement; original Cloudflare failure retained. Same prompt/model/reasoning/search settings; no follow-up.
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
No readable answer body captured. This is not a negative discovery observation.
Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R1-A2",
"slot_id": "S3-CHATGPT-V1-R1",
"platform": "ChatGPT Search",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 1,
"submitted_at": "2026-11-02T06:09:35.703Z",
"model_label": "GPT-5.6 Sol",
"reasoning_effort": "Medium",
"temporary": true,
"personalization": false,
"search_enabled_ui": true,
"search_observed": true,
"status": "capture_incomplete",
"body_complete": false,
"captured_at": "2026-11-02T06:14:27.928Z",
"source_inventory_complete": false,
"source_panel_scope": "Five source-panel entries recovered through visible UI; body citation matching unresolved. Not a complete answer.",
"error": "Answer completed but rendered body and Copy expose only reply-loading error",
"copy_result": "Unable to load this reply (UI original: \\u65e0\\u6cd5\\u52a0\\u8f7d\\u6b64\\u56de\\u590d)",
"attempt_number": 2,
"replacement_for": "S3-CHATGPT-V1-R1",
"recovery_history": [
{
"method": "Native Chrome accessibility tree",
"status": "reply_loading_error"
},
{
"method": "Copy answer, paste into offline local buffer",
"status": "same_error_only"
},
{
"method": "Native read_thread for original temporary ID",
"status": "No readable thread found; no save or new prompt"
},
{
"recorded_at": "2026-11-02T08:34:00.263Z",
"method": "User manually viewed the same failed page",
"status": "same_reply_loading_error_confirmed"
},
{
"recorded_at": "2026-11-02T08:34:00.263Z",
"method": "Reload existing temporary conversation; no resubmission",
"status": "redirected_to_empty_temporary_homepage",
"url_after": "https://chatgpt.com/?temporary-chat=true"
}
],
"searched_websites_reported": 12,
"thought_duration_ui": "44s",
"raw_file": "records/S3-CHATGPT-V1-R1-A2.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 0,
"public_answer_characters": 0,
"answer_privacy_edits": {
"body_changed": false,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
}
}[]
Actual answer links · not the retrieval badge
No URLs recorded in this field. Completeness and source role are reported separately.
Retrieval-panel sources · not matched answer citations
| Label | Public URL | Recorded host | Role | Capture provenance |
|---|---|---|---|---|
| Miraqo How we measure AI Visibility — methodology Total lines: 132 | https://miraqo.io/en/ai-visibility-methodology/ | miraqo.io | Source-panel entry; citation attribution unresolved because answer body unavailable | |
| OpenAI Help Center Searching the web with ChatGPT Total lines: 213 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | Source-panel entry; citation attribution unresolved because answer body unavailable | |
| SpaceXSI Docs Web Search Total lines: 550 | https://docs.x.ai/developers/tools/web-search | docs.x.ai | Source-panel entry; citation attribution unresolved because answer body unavailable | |
| SpaceXSI Docs Citations Total lines: 699 | https://docs.x.ai/developers/tools/citations | docs.x.ai | Source-panel entry; citation attribution unresolved because answer body unavailable | |
| AI Visibility Snapshot Methodology: How We Measure AI Visibility Total lines: 108 | https://getaisnapshot.com/methodology | getaisnapshot.com | Source-panel entry; citation attribution unresolved because answer body unavailable |
Not coded. No entity or source absence is inferred.
S3-CHATGPT-V1-R1-A3 · ChatGPT Search · capture_incomplete
Prepared preflight independently recorded the exact V1 question and all selected UI controls. User reported clicking Send. No follow-up prompt.
Submission time is bounded by preflight capture and first post-send observation; not asserted to be an exact click timestamp.
Visible article recovered 11,393 characters, but both DOM and OS Copy stop at the same unfinished final clause. This is partial evidence, not a complete answer.
Zero observed body hrefs does not establish absence of citations: five citation-button occurrences are visible. A2 source URLs are not reused for A3.
Manual-send versus automated-send differences do not establish an automation cause; this single manual control still failed full-body capture.
User authorized a subsequent formal Responses API collection; this UI attempt remains historical and excluded from the API formal answer count.
Full original prompt
Recorded submission; unchanged wording.
I'm a solo SaaS founder with no GEO budget. How can I tell whether an AI visibility score for ChatGPT and Grok is trustworthy?
Captured answer body
How to tell whether an AI visibility score is trustworthy (without a GEO budget) My short answer: Don't trust the score. Trust the measurement process behind the score. As a solo SaaS founder, you don't need an expensive GEO platform to evaluate whether ChatGPT or Grok recommends your product. You need to know: What questions were asked? How many times were they asked? What exactly counts as visibility? Can you reproduce the results yourself? AI visibility is not like traditional Google rankings. The same AI assistant can give different answers to the same question depending on the session, search behavior, location, and time. Research published in 2026 also highlights how single-run visibility measurements can produce misleading rankings. arXiv +1 Here's the practical framework I'd use. 1. The six tests of a trustworthy AI visibility score Test Trustworthy measurement Red flag Prompt transparency Shows the exact questions Says it uses proprietary prompts Repeatability Runs each prompt multiple times One answer per prompt Engine separation Separate ChatGPT and Grok scores One blended AI score Evidence Stores actual answers and URLs Only displays a number Scoring clarity Explains its formula and denominator Proprietary score from 0–100 Uncertainty Shows variability over time Claims a 3-point increase is significant A particularly important distinction: a tool that reports a score of 72/100 hasn't necessarily measured 72% visibility. It might have assigned arbitrary weights to: Brand mentions Citation frequency Position in the answer Sentiment Competitor comparisons Unless you can see the formula, you cannot interpret the score. 2. Check whether the score measures the right thing There are three different outcomes that GEO tools often combine. Metric 1 Brand mention rate Does ChatGPT or Grok mention your SaaS at all? Example: 8 out of 30 answers mention your product = 26.7%. Metric 2 Recommendation rate Does the AI actually recommend your product as a solution? Being mentioned in a list of ten tools is not the same as being recommended as the best fit. Metric 3 Citation rate Does the AI cite your website or another source that identifies your product? This helps you understand whether your content is being surfaced as a source, rather than merely whether your brand name appears. My recommendation: Track all three separately. For a small SaaS, recommendation rate may be more commercially useful than a generic visibility score. For example: 70% mention rate, but only 5% recommendation rate: you're known, but not necessarily competitive. 20% mention rate, but 15% recommendation rate: you're less frequently mentioned, but often a strong fit when you appear. 0% official-site citation rate: investigate whether your content is being discovered and cited, even if the brand is mentioned elsewhere. Also record your position in the answer (first recommendation, top three, or mentioned later). 3. Make sure ChatGPT and Grok are actually being tested correctly This is an important technical detail that many founders overlook. ChatGPT ChatGPT can automatically search the web when a question benefits from current information. Its search behavior can also involve location information and external search providers. OpenAI Help Center +1 A trustworthy report should disclose whether it tests: ChatGPT's consumer-facing interface. ChatGPT with web search enabled. An OpenAI API model without web search. An API model with web search. These are different measurement environments. Grok The same distinction applies to Grok. xAI documents a web-search tool for its API, including the ability to retrieve citations from sources encountered during searches. SpaceXSI Docs +1 Ask whether the vendor tests: The actual Grok user experience at grok.com. An xAI API model. An API model with web search enabled. An API model without web search. An API-based Grok result should not automatically be presented as proof of visibility in the consumer-facing Grok experience. A useful vendor question: "Are you testing the actual ChatGPT and Grok interfaces, or querying their APIs? What model versions, search settings, and account conditions do you use?" If they can't answer, downgrade your confidence in the score. 4. Test the score yourself for free You can do a surprisingly useful audit without purchasing any GEO software. I'd start with 10 real buyer questions, three independent runs per question, on each platform. That's 60 responses in total. Build your prompt list Don't just ask whether AI knows your brand. Test whether it recommends you when a potential customer needs your solution. Here are example prompt patterns for a SaaS business: Category Example prompt Category discovery Best [category] software for small businesses Problem solving How can I solve [customer problem] without hiring a developer? Use case What tools help automate [specific workflow]? Alternatives What are good alternatives to [competitor]? Comparison Compare [your product] with [competitor] Budget Best affordable [category] tools for startups Integration Which [category] tools integrate with [popular platform]? Use actual buyer language wherever possible. I'd allocate your initial ten prompts roughly as follows: 6 non-branded discovery questions. 2 competitor/alternative questions. 2 questions explicitly mentioning your brand. Keep branded questions separate when calculating discovery visibility. Otherwise, you're rewarding the system for already being told your name. Run the experiment Open ChatGPT and Grok separately. Start a fresh conversation for every run. Use the exact same prompt each time. Record the answer, date, model/interface, and citations. Repeat each prompt three times, ideally across different days. Don't change the prompt list halfway through the experiment. Keep the region, language, and search settings consistent. If you sell internationally, run separate tests for important markets rather than mixing them. 5. Use a simple free tracking spreadsheet Google Sheets is enough. You don't need a dashboard. I'd use these columns: Column What to record Date When you ran the test Platform ChatGPT / Grok Prompt ID P01, P02, etc. Exact prompt Original question Brand mentioned? Yes / No Recommended? Yes / No Position 1, 2, 3, or absent Official website cited? Yes / No Competitors mentioned Names of alternatives Citation URLs Actual source links Notes Context or inaccuracies Calculate three simple metrics: Mention Rate = Responses mentioning you Valid responses × 100 Mention Rate= Valid responses Responses mentioning you ×100 Recommendation Rate = Responses recommending you Valid responses × 100 Recommendation Rate= Valid responses Responses recommending you ×100 Citation Rate = Responses citing your website Valid responses × 100 Citation Rate= Valid responses Responses citing your website ×100 For example, if ChatGPT mentions your product in 8 of 30 responses, report 8/30 (26.7%), not simply 26.7%. Keep Grok and ChatGPT separate. Example tracking dashboard Illustrative numbers, not actual platform measurements Metric ChatGPT Grok Mentions 8/30 5/30 Recommendations 5/30 3/30 Website citations 4/30 2/30 Interpretation: ChatGPT currently produces more visibility in this sample. This is a directional observation, not proof that ChatGPT is inherently a better acquisition channel. One statistical warning: 30 responses per platform is a useful starting experiment, not a robust market-wide estimate. A 2026 research paper found substantial citation variability, with apparently different visibility figures sometimes falling within the statistical noise. arXiv +1 Repeat the experiment weekly before interpreting small changes as improvements. 6. Five questions to ask any GEO vendor before paying Copy these questions and send them to a vendor offering an AI visibility score. Questions about your AI visibility measurement methodology \u53d1\u9001 \u6536\u4ef6\u4eba Hi, I'm evaluating AI visibility tracking for my SaaS product, particularly across ChatGPT and Grok. Before considering a subscription, I'd like to understand your measurement methodology. Could you clarify: What exact prompts do you use to calculate my visibility score? How many times is each prompt tested per platform? Do you test the actual consumer-facing ChatGPT and Grok interfaces, or their APIs? Can I access the raw AI responses, timestamps, and citation URLs? How do you calculate the score, and do you provide separate mention, recommendation, and citation rates? How do you account for response variability, model updates, and geographic differences? Can you demonstrate that a change in my score is statistically meaningful rather than normal output variation? I'm particularly interested in reproducibility and the ability to independently verify the reported numbers. Thanks! If the vendor answers these questions transparently, that's a positive sign. If it only shows an attractive dashboard with a proprietary score, be cautious. 7. What I'd prioritize as a zero-budget SaaS founder Don't spend your limited time chasing a score from 35 to 65. I'd prioritize these activities: Priority Activity Why 1 Test 10–20 buyer-intent prompts Establish your baseline 2 Identify which competitors get recommended Find your competitive gaps 3 Inspect cited sources Discover where AI gets its information 4 Improve your comparison and use-case pages Make your positioning easier to understand 5 Repeat the same tests weekly Check whether changes persist One additional technical check: make sure your website isn't accidentally blocking OpenAI's OAI-SearchBot. OpenAI documents that this crawler is used to surface websites in ChatGPT search results. Allowing it doesn't guarantee a recommendation, but blocking it can prevent your pages from appearing in those search results. OpenAI Developers +1 My final verdict: a simple trust scale Level 1 — Marketing metric Low trust Score from 0–100, no disclosed prompts, no raw answers. Level 2 — Directional metric Moderate trust Disclosed prompts, platform breakdown, actual responses, but limited repeat sampling. Level 3 — Reproducible metric Higher trust Exact prompts, multiple runs, raw outputs, citations, scoring formula, model/interface details, and uncertainty reporting. The most important principle: AI visibility is an observation, not a business outcome. A high score doesn't prove that customers are discovering you, clicking through, or buying. Eventually, connect your observations to referral traffic, signups, and conversions. Don't confuse visibility with revenue. One thing I'd like to help you with next If you share your SaaS website URL, I can help build a free ChatGPT + Grok visibility audit around your actual product: 20 realistic buyer-intent prompts. A competitor comparison framework. A manual scoring rubric. A Google Sheets tracking template. A prioritized list of content gaps to investigate. That gives you a baseline you
Model, search, time, capture and recovery evidence
{
"id": "S3-CHATGPT-V1-R1-A3",
"slot_id": "S3-CHATGPT-V1-R1",
"platform": "ChatGPT Search",
"prompt_id": "V1",
"intent": "V",
"wording": 1,
"repeat": 1,
"submitted_at": null,
"submitted_time_reported": "User confirmed manually sent before first observation; exact click time unavailable.",
"submission_time_bounds": {
"after": "2026-11-02T08:33:13.068Z",
"before": "2026-11-02T08:36:03.977Z"
},
"first_observed_at": "2026-11-02T08:36:03.977Z",
"model_label": "GPT-5.6 Sol",
"reasoning_effort": "Medium",
"temporary": true,
"personalization": false,
"search_enabled_ui": true,
"search_observed": null,
"status": "capture_incomplete",
"body_complete": false,
"source_inventory_scope": "Five visible citation-button occurrences are recorded as labels, not resolved sources. No observed hrefs in the scoped body capture; this does not mean no citations. No A2 source-panel entries are attributed to A3.",
"source_inventory_complete": false,
"capture_method": "Partial visible rendered article plus OS Copy corroboration of the same unfinished ending; no resubmission during capture.",
"search_evidence": "Native Web Search chip selected before submission. Completion snapshot shows citation buttons and a completed/thought-duration label, but no independently captured expanded search-tool activity or query trace for this attempt. Actual search is not inferred solely from citation controls.",
"error": "Visible body and native Copy both end mid-sentence: That gives you a baseline you",
"attempt_number": 3,
"replacement_for": "S3-CHATGPT-V1-R1-A2",
"operator": "User clicked Send; agent prepared controls and prompt",
"diagnostic_role": "Single manual-send control; compare with automated submissions, not a separate planned repetition",
"recovery_history": [
{
"method": "Rendered article DOM observation",
"status": "partial_body_mid_sentence",
"body_complete": false
},
{
"method": "OS Copy response into native local capture",
"status": "same_unfinished_ending",
"body_complete": false
},
{
"method": "Completion accessibility snapshot",
"status": "completed_UI_badge_with_partial_body_and_citation_buttons",
"body_complete": false
},
{
"method": "Browser clipboard read",
"status": "empty_clipboard_capture",
"body_complete": false
}
],
"truncation_evidence": {
"rendered_characters": 11393,
"ends_with": "That gives you a baseline you",
"native_copy_same_unfinished_ending": true,
"completion_UI_status": "Answer completed",
"interpretation": "A completion badge does not establish full-body capture; original-generation versus display/capture truncation is unresolved."
},
"thought_duration_ui": "1m 27s",
"raw_file": "records/S3-CHATGPT-V1-R1-A3.json",
"public_copy": true,
"publication_scope": "Redacted transcript and observation record; private originals, account UI, opaque provider IDs and local captures are not published.",
"original_capture_answer_characters": 11393,
"public_answer_characters": 11418,
"answer_privacy_edits": {
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
}
}[]
Actual answer links · not the retrieval badge
No URLs recorded in this field. Completeness and source role are reported separately.
Retrieval-panel sources · not matched answer citations
No URLs recorded in this field. Completeness and source role are reported separately.
Not coded. No entity or source absence is inferred.
Hostname mapping and source provenance
| Answer | Public source URL | Recorded hostname | Analysis domain | Mapping basis | Precision / provenance |
|---|---|---|---|---|---|
| S3-CHATGPT-M1-R1 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R1 | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R1 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R2 | https://help.openai.com/en/articles/7730893-data-controls-in-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R2 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R2 | https://help.x.com/en/using-x/about-grok | help.x.com | x.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R3 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R3 | https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R3 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R3 | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M1-R3 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-GROK-M1-R1 | https://shmeer.com/blog/how-to-measure-geo | shmeer.com | shmeer.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R1 | https://www.gomega.ai/blog/how-to-measure-geo-performance | www.gomega.ai | gomega.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R1 | https://geoaura.world/blog/how-to-measure-geo-visibility | geoaura.world | geoaura.world | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R1 | https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | goodatmarketing.com | goodatmarketing.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://shmeer.com/blog/how-to-measure-geo | shmeer.com | shmeer.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://www.gomega.ai/blog/how-to-measure-geo-performance | www.gomega.ai | gomega.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://geoaura.world/blog/how-to-measure-geo-visibility | geoaura.world | geoaura.world | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://snezzi.com/blog/how-to-measure-geo-visibility-complete-step-by-step-guide/ | snezzi.com | snezzi.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://www.hubspot.com/aeo-grader/generative-engine-optimization-tool | www.hubspot.com | hubspot.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://goodatmarketing.com/how-to-track-chatgpt-traffic-in-google-analytics/ | goodatmarketing.com | goodatmarketing.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M1-R3 | https://www.explaingeo.com/free-geo-tools-for-small-businesses-get-ai-visibility-without-the-enterprise-price-tag/ | www.explaingeo.com | explaingeo.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-CHATGPT-M2-R1 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R1 | https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R1 | https://help.x.com/en/using-x/about-grok | help.x.com | x.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R2 | https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R2 | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%2520automatically%2520includes%2520the%2520UTM%2520parameter%2520utm_source%253Dchatgpt.com%2520in%2520referral%2520URLs%252C%2520enabling%2520clear%2520tracking%2520and%2520analysis%2520of%2520inbound%2520traffic%2520from%2520ChatGPT%2520search%2520results | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R2 | https://docs.x.ai/developers/tools/web-search | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R3 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R3 | https://help.x.com/en/using-x/about-grok | help.x.com | x.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R3 | https://help.openai.com/en/articles/8914046-temporary-chat-in-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R3 | https://help.x.com/en/using-x/about-grok | help.x.com | x.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-M2-R3 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-GROK-M2-R1 | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | stork.ai | Manual mapping recorded in coding | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| S3-GROK-M2-R1 | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | stork.ai | Manual mapping recorded in coding | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| S3-GROK-M2-R1 | https://stork.ai/blog/does-chatgpt-recommend-your-product | stork.ai | stork.ai | Manual mapping recorded in coding | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| S3-GROK-M2-R1 | https://seozest.io/how-to-check-if-chatgpt-perplexity-grok-or-claude-ai-is-recommending-your-business-a-free-10-minute-audit/ | seozest.io | seozest.io | Manual mapping recorded in coding | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| S3-GROK-M2-R1 | https://solcrys.com/free-chatgpt-visibility-tools/ | solcrys.com | solcrys.com | Manual mapping recorded in coding | Native accessibility target display; HTTPS standardized, www may be omitted by UI; exact href unverified |
| S3-GROK-M2-R3 | https://www.01net.com/en/seo/ai-visibility-tools/how-to-track-brand-mentions/ | www.01net.com | 01net.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M2-R3 | https://www.ranked.ai/free-tools/ai-visibility-checker | www.ranked.ai | ranked.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M2-R3 | https://snoika.com/free-tools/brand-ai-visibility-checker | snoika.com | snoika.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M2-R3 | https://indexly.ai/grok-visibility-checker | indexly.ai | indexly.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M2-R3 | https://mangools.com/ai-search-watcher | mangools.com | mangools.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-M2-R3 | https://www.amicited.com/blog/how-ai-recommends-software-appearing-in-best-tool-lists/ | www.amicited.com | amicited.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-CHATGPT-V1-R1 | https://platform.openai.com/docs/guides/evaluation-best-practices | platform.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R1 | https://docs.x.ai/docs/guides/live-search | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R1 | https://docs.x.ai/docs/models | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R2 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R3 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R3 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V1-R3 | https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-GROK-V1-R1 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | searchscope.com.au | searchscope.com.au | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://maxaeo.ai/blog/ai-visibility-sample-size/ | maxaeo.ai | maxaeo.ai | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://higoodie.com/blog/how-to-rank-on-grok/ | higoodie.com | higoodie.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://cognizo.ai/blog/ai-visibility-score | cognizo.ai | cognizo.ai | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://ahrefs.com/free-ai-visibility | ahrefs.com | ahrefs.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R1 | https://hubspot.com/aeo-grader | hubspot.com | hubspot.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS |
| S3-GROK-V1-R2 | https://www.visilay.com/blog/ai-visibility-score/ | www.visilay.com | visilay.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V1-R2 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V1-R2 | https://searchenginejournal.com/the-geo-trust-gap-seos-want-the-data-but-not-the-platforms-selling-it/585161/ | searchenginejournal.com | searchenginejournal.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V1-R3 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V1-R3 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V1-R3 | https://www.amicited.com/how-to/ai-visibility-tools-evaluation-scorecard/ | www.amicited.com | amicited.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-CHATGPT-V2-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R1 | https://arxiv.org/abs/2410.03492 | arxiv.org | arxiv.org | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R1 | https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R1 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R2 | https://docs.x.ai/cookbook/examples/x-post-fact-check | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R2 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R2 | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R3 | https://help.openai.com/en/articles/8590148-memory-in-chatgpt | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R3 | https://docs.x.ai/developers/model-capabilities/text/best-practices | docs.x.ai | x.ai | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R3 | https://help.openai.com/en/articles/8313428 | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R3 | https://help.openai.com/en/articles/9237897-chatgpt-search | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-CHATGPT-V2-R3 | https://help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext%3DChatGPT%20automatically%20includes%20the%20UTM%20parameter%20utm_source%3Dchatgpt.com%20in%20referral%20URLs%2C%20enabling%20clear%20tracking%20and%20analysis%20of%20inbound%20traffic%20from%20ChatGPT%20search%20results | help.openai.com | openai.com | Manual mapping recorded in coding | None |
| S3-GROK-V2-R1 | https://visilay.com/blog/ai-visibility-score/ | visilay.com | visilay.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R1 | https://visilay.com/blog/ai-visibility-score/ | visilay.com | visilay.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R1 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R1 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R1 | https://amicited.com/blog/grok-is-citing-brands-too/ | amicited.com | amicited.com | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R1 | https://searchscope.com.au/ai-seo/how-accurate-are-ai-visibility-tools/ | searchscope.com.au | searchscope.com.au | Manual mapping recorded in coding | Native accessibility target; displayed targets omit scheme, standardized to HTTPS; www may be omitted by display |
| S3-GROK-V2-R2 | https://www.aeotrack.io/methodology | www.aeotrack.io | aeotrack.io | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R2 | https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/ | sparktoro.com | sparktoro.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R2 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R2 | https://paprik.ai/blog/accurate-api-based-ai-visibility-tracking-chatgpt-ui | paprik.ai | paprik.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R2 | https://www.loudface.co/blog/what-is-ai-visibility | www.loudface.co | loudface.co | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R3 | https://markets.businessinsider.com/news/stocks/there-is-no-single-ai-visibility-score-ask-four-ais-get-four-shortlists-1036643461 | markets.businessinsider.com | businessinsider.com | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R3 | https://brantial.ai/blog/what-is-an-ai-visibility-score-how-is-it-calculated/ | brantial.ai | brantial.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R3 | https://www.getmint.ai/blog/stop-reading-the-weather-how-to-measure-your-brand-s-visibility-in-chatgpt | www.getmint.ai | getmint.ai | Manual mapping recorded in coding | Rendered DOM href |
| S3-GROK-V2-R3 | https://presenc.ai/compare/grok-vs-chatgpt-for-brands | presenc.ai | presenc.ai | Manual mapping recorded in coding | Rendered DOM href |
URLs shown from native target displays retain their capture qualification; hostname normalization does not verify the original scheme or exact destination. Source URLs retain their recorded precision limits. External checking is bounded by the source-audit record; unreviewed claims remain unverified.
findings.json · analysis-plan.md · protocol.json · runs.json · attempts.json
Publication notes and downloads
This is the separately authorized public edition of Stage 3. Private credentials, account context, conversation links, opaque provider IDs and local-only captures are excluded. Citation URLs were cleaned of tracking parameters; annotation offsets are not guaranteed after public-copy edits. These edits preserve the study’s coded observations and declared limitations.
Download public data and findings · publication-notes.json · settings-summary.json
Publication notes
{
"edition": "Redacted public edition",
"published_on": "2026-11-02",
"preserved": [
"24 complete selected answers and exact registered prompts",
"All 27 prompt submissions, including three historical ChatGPT UI failures/partial captures",
"All 24 assistant semantic coding records, source categories, C/M/R judgments and methodological limits",
"Model labels, timing, actual returned search actions/queries/sources, citations and token/tool usage",
"Twelve allowlisted API requests and response copies, cost estimates and non-generation setup history",
"Post-hoc peer-review analyses, explicit normalization rules, pending human-review packet, limited source audit and Prompt Bank implications"
],
"removed": [
"Private conversation/share URLs, opaque provider request/response/search-call identifiers and internal operation references",
"Original screenshots, DOM/HTML, native dumps, browser/account interface details and unrelated navigation",
"Local filesystem/capture paths, credential setup details, collector/key-entry software, headers and reasoning-trace output items",
"Original account preference values; restoration completion remains visible",
"Tracking, session/client/routing and signed-access parameters in actual source URLs; original annotation position indexes"
],
"transcript_policy": "Original answer wording preserved apart from necessary privacy edits and actual URL tracking removal. Analytics parameter examples outside URLs remain. Every record states whether its public body changed. The incomplete A3 prefix remains incomplete and excluded from absence-based analysis.",
"source_precision": "Original native target-display precision limits on three Grok answers remain. Retrieval sources/search-hit badges are not automatically body citations.",
"citation_positions": "Original start/end/output/content indexes omitted because redaction changes text and output-item locations. Annotated URLs and their roles remain; public record hashes and body lengths are recalculated.",
"signed_URLs": "Returned retrieval URLs can contain signed-cloud or session parameters. These parameters are removed, preserving the host/path and non-sensitive query data. Unsigned copies may not be directly fetchable; original signed URLs remain private. No destination repair or independent URL-content verification is claimed.",
"condition_notice": "OpenAI API + web search, not consumer ChatGPT Search. These results do not represent ChatGPT web app performance.",
"external_claims": "A separately recorded six-source assistant spot-check assesses access and one selected claim per source. It does not audit the full corpus, commercial accuracy, prices, tool operation or generated scoring rubrics. Independent human coding review remains pending; original source categories are historical/provisional and selected qualifications are explicit in source-audit.json.",
"evidence_boundary": "Only public copies are transformed. Complete private originals and all original failed captures remain unchanged locally.",
"language_policy": "Documentation and annotations are English; captured non-English UI tokens are represented with literal Unicode escapes."
}Public settings summary
{
"platform": "Grok",
"test_personalization_using_x": false,
"test_personalization_using_history": false,
"test_answer_count": 12,
"restoration_completed": true,
"restored_at": "2026-11-02T06:54:34.518Z",
"original_preferences": "Omitted; settings restored to their pre-test values.",
"evidence_scope": "Summary of observed controls; original settings screenshots and account navigation remain private. Not proof of audited backend isolation."
}Author: Helen Bird · Experiment date: 2026-11-02 · First published: 2026-11-02 · Last revised: 2026-11-06 · Version: v1.5
Technical maintenance: consistent canonical URLs, page-specific descriptions and social metadata, truthful author/series markup, and links to individual original-answer pages. Original answers, coding, data downloads and research outcomes are unchanged. This bundled update has no measured AI visibility effect. November 6 navigation update: linked the new technical learning record; original experiment data and findings remain unchanged.
Version metadata · Revision history · Cross-report review decisions · Stage 4 review decisions · Technical maintenance