GEO Lab #03 · S3-CHATGPT-V2-R1 · Original response

This page presents one selected response from GEO Lab #03. It adds a smaller reading and citation entry point; it does not replace the full archive or collect a new answer.

The OpenAI arm uses the Responses API with web search and does not represent consumer ChatGPT Search. Grok records identify their actual consumer-interface mode and available model label.

Recorded conditions

Record ID
S3-CHATGPT-V2-R1
Platform
OpenAI API + web search
Question / condition
V2
Repeat
1
Model label
gpt-5.6-sol
Visible mode
Not independently recorded
Reasoning
medium
Submitted / first recorded time
2026-11-02T09:20:55.790227+00:00
Completeness
Recorded body complete
Search
Actual search activity observed; exact controls and evidence remain in the raw record.

Original prompt

As a bootstrapped SaaS builder, what checks should I perform before trusting reported ChatGPT and Grok visibility scores for my product?

The agreed prompt is reproduced exactly. Any recorded difference in the actual submitted text or search UI is retained in the raw record and full appendix.

Original answer

Treat any “AI visibility score” as a **vendor-defined estimate—not an industry-standard metric**—until you can audit how it was produced.

## Checks to perform

1. **Confirm exactly what was tested**
   - ChatGPT standard chat, Search, shopping, or deep research?
   - Grok on grok.com, X, or API?
   - Which model, mode, subscription tier, device, and date?
   - Were web search and, for Grok, X search enabled?

   These surfaces are not interchangeable. ChatGPT may search automatically, rewrite queries, use approximate location, and incorporate memory; Grok can use both real-time web and X search. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))

2. **Demand the actual prompt set**
   Check whether prompts:
   - Represent real customer questions
   - Include unbranded discovery queries, not just your company name
   - Cover alternatives, comparisons, use cases, and buying intent
   - Use your target geography and language
   - Were selected before results were observed

   A score based on 20 favorable prompts can be less useful than one based on 200 poorly targeted prompts.

3. **Check for repeated runs**
   One response per prompt is inadequate. Ask for:
   - Runs per prompt
   - Number of distinct days tested
   - Fresh chats versus continued conversations
   - Variance or confidence intervals
   - Prompt-level consistency

   LLM evaluations can vary across repeated runs and small wording changes, so aggregate rankings without uncertainty can look more definitive than they are. ([arxiv.org](https://arxiv.org/abs/2410.03492))

4. **Inspect the scoring formula**
   Determine whether “visibility” means:
   - Merely mentioned
   - Listed among alternatives
   - Positively recommended
   - Ranked in the top three
   - Linked or cited
   - Accurately described

   Also inspect weights, denominators, competitor selection, and treatment of refusals or failed searches. A mention and a first-choice recommendation should not receive the same credit.

5. **Require raw evidence**
   For every scored result, the vendor should retain:
   - Exact prompt
   - Full response
   - Timestamp
   - Product/model/mode
   - Search status
   - Citations and destination URLs
   - Screenshot or API output

   OpenAI itself warns that search results and citations may be incomplete, outdated, or incorrect and recommends opening sources to verify support. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))

6. **Verify citations separately from mentions**
   Track at least three different metrics:
   - **Mention share:** your product appears
   - **Recommendation share:** it is actually suggested
   - **Citation share:** your domain is used as a source

   A model may mention your brand based on another site, cite your site without recommending you, or hallucinate product capabilities.

7. **Audit entity matching**
   Look for:
   - Products with similar names
   - Parent company versus product confusion
   - Old product names or domains
   - Misspellings and abbreviations
   - Multiple products from your company being merged

   Manually inspect a sample of both positive and negative matches.

8. **Check personalization and geographic controls**
   Results can change with location, saved memory, prior conversation, workspace sources, and account state. ChatGPT explicitly says location and memory may affect search-query rewriting. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))

   Prefer clean sessions with memory disabled, no prior context, and explicitly recorded locales—unless personalization is intentionally part of the test.

9. **Check whether the vendor is measuring the consumer product**
   API calls with custom system prompts, fixed temperatures, or manually enabled tools may not reproduce what prospects see in ChatGPT or Grok. Ask the vendor to explain any difference between its testing harness and the actual consumer interface.

10. **Audit automated grading**
    If another LLM decides whether your product was “recommended,” require:
    - The grading rubric and judge model
    - Human-labeled validation samples
    - Agreement rates
    - Manual review of ambiguous outputs
    - Randomized ordering in competitor comparisons

    xAI’s own evaluation guidance warns that model judges can exhibit position bias, favor longer answers, and drift from human judgment. ([docs.x.ai](https://docs.x.ai/developers/model-capabilities/text/best-practices))

11. **Check crawl and source availability**
    Determine whether:
    - Important pages are indexable
    - Content is current and server-rendered/accessibly rendered
    - Robots or CDN rules block relevant crawlers
    - The model cites obsolete third-party pages instead of your canonical pages

    OpenAI states that eligibility requires allowing OAI-Searchbot, but inclusion and placement are not guaranteed. ([help.openai.com](https://help.openai.com/en/articles/9237897-chatgpt-search))

12. **Compare scores with business evidence**
    Cross-check reported improvement against:
    - ChatGPT/Grok referral traffic
    - AI-cited landing-page visits
    - Branded search growth
    - “How did you hear about us?” responses
    - Demo or signup attribution
    - Actual citation URLs

    Visibility without qualified traffic, accurate positioning, or conversions may have little commercial value.

## A cheap independent audit

Before buying or trusting a platform:

1. Select **20–30 commercially meaningful prompts**.
2. Run each prompt **at least five times** in fresh sessions.
3. Test on **two or three different days**.
4. Record mention, recommendation position, sentiment, factual accuracy, citation, and cited domain separately.
5. Compare your prompt-level results—not merely the final average—with the vendor’s raw data.

**Strongest trust signal:** the vendor lets you export prompts, raw responses, timestamps, model/mode metadata, citations, and its complete scoring formula.

**Biggest red flag:** a precise score such as “73/100 visibility” with no raw outputs, repeated-run statistics, or explanation of what the 73 represents.

Recorded sources

These are the source roles recorded in the public dataset. A retrieval or source-panel item is not automatically an answer-body citation. A URL supplied in the prompt and repeated by an answer does not prove content access. No destination has been repaired or independently validated here.

Recorded answer-body links (6)

Retrieval-only / source-panel inventory (49)

Evidence and qualifications

This is the existing redacted public record. It has not been rewritten for this page. Earlier privacy edits and collection qualifications continue to apply. Editorial text is English; a few original captures retain non-English UI badges or source-panel date labels. An answer’s empirical claims are platform output, not verified findings of this archive.

Body review

{
"reviewed_at": "2026-11-02T09:22:34.394550+00:00",
"reviewer": "assistant",
"complete_body_read": true,
"coherent_ending": true,
"annotation_and_explicit_url_coverage_checked": true,
"external_claims_verified": false,
"independent_human_double_coding": false
}

Existing public-text privacy policy

{
"body_changed": true,
"policy": "Only actual URL tracking, signed/session URL parameters and necessary identifying information are removed; original wording, including analytics parameter examples outside URLs, remains.",
"annotation_positions": "Original position indexes omitted because URL/privacy edits can change character locations."
}

Prompt SHA-256: dc6b4040b40506792643c6eaf1954ddcbeac5dc12124b266cea68dab4e663c49
Answer SHA-256: b5a67b7aa8243cb94672ad6da1213a30f6cbfd6f530ef858a249c36d01d12909