# Public edition of the Stage 3 analysis plan

> Published on November 2, 2026. The following dated checkpoints preserve collection history. Final outcome: 24 complete selected answers and 27 prompt submissions. The OpenAI condition is Responses API + web search and does not represent the ChatGPT web app. Earlier local-only restrictions were superseded by this approved redacted release.

# Stage 3 analysis plan: sensitivity to question design

Prepared on November 2, 2026, before examining Stage 3 answers. This plan specifies descriptive comparisons; it does not register a GEO intervention or a test of any visibility vendor's accuracy.

## Design and intended comparisons

The design crosses two intents (M: measurement workflow; V: validation of visibility scores) with two wording variants per intent. Four prompts are each submitted to ChatGPT and Grok three times: **8 platform-by-prompt cells, 24 planned response positions**. A repeated capture of an existing answer is not a new submission or repetition.

The user's four English prompts remain verbatim. Each submission uses a fresh conversation and sends only its assigned prompt: no product name, added examples, requested answer format, steering, or follow-up. Register execution order before collection, interleave variants and platforms where feasible, and retain actual chronological order and interruptions. Do not change the order because of emerging answers.

Analyze these contrasts separately:

1. **Identical-prompt repeats:** compare the three complete selected answers within each platform-by-prompt cell. This describes observed within-cell variation under the recorded conditions.
2. **Wording within intent:** compare M1 with M2 and V1 with V2 within each platform. These express similar intentions but also change contextual wording, such as "no GEO budget" versus "bootstrapped". Differences describe sensitivity to these complete prompt variants, rather than a clean effect of a single phrase.
3. **Intent across M and V:** compare information-source categories and tool recommendation sets, pooling the two variants only in a separately labeled view. The requested task changes, so differences cannot be assigned wholly to wording.

Platform contrasts are secondary descriptive comparisons. Three repeats per cell cannot establish stable rates, population-level platform superiority, or independence from account and temporal effects.

## Collection conditions and provenance

- Record actual interface, displayed model label, reasoning setting, local date/time and time zone, fresh-conversation evidence, visible personalization or memory controls, search selection, and search activity for every submission. Unknown values remain unknown.
- Keep model and reasoning fixed within each platform's collection. Grok may expose only a mode rather than an exact backend version. A shared UI label is not proof of identical backend execution. Log any change and obtain authorization before changing agreed conditions; do not silently treat affected runs as equivalent.
- At initial registration, the computer-use tool blocked ChatGPT desktop access and the collection interface and model were unresolved. The original rule required user confirmation before a browser substitution. The dated amendment below records the subsequently authorized interface and observed settings; the initial uncertainty is historical, not the current collection state.
- Separate **search enabled in the interface**, **search actually observed**, and **search claimed in the answer**. An absent activity capture does not prove that search was disabled. Record visible query labels verbatim, distinguishing attached hit counts from query text. Tool-hit totals are not citation totals.
- Record account-context evidence and visible control settings, without claiming audited backend isolation. Existing conversation history, saved recovery copies, elapsed time, provider changes and model updates can limit comparability.

This work collects consumer-interface answers; it does not implement subscription-backed model inference in a software project. The shared model gateway is therefore not invoked for this collection.

## Collection amendment: November 2, 2026

The pre-analysis definitions and comparison rules above remain the baseline. The following authorized collection changes and observations are recorded without revising the original prompts or treating technical failures as answers:

- **ChatGPT interface and settings established:** after the desktop-app denial, the user explicitly approved the ChatGPT web fallback. Before the first V1 repetition-1 submission, the UI explicitly selected **GPT-5.6 Sol, Medium reasoning**, a **non-personalized temporary conversation**, and the native **Web Search** chip. These are observed interface conditions, not an independently audited backend identity or isolation guarantee.
- **First submission failed technically:** the actual V1 repetition-1 submission encountered a Cloudflare challenge and produced no captured answer. Retain it as an actual failed attempt; it does not establish a substantive answer, absence of recommendations, or a completed repetition.
- **Supplements and pacing authorized:** the user approved additional submissions to replace missing answers while retaining all failures, and requested longer intervals. Register a **minimum of 90 seconds between actual submissions on the same platform**, including supplementary submissions. Retain actual timestamps and any longer interruptions; this pacing rule is not evidence that it prevents access or rendering failures.
- **Earlier Grok checkpoint:** original account preferences were recorded privately; permission to change them was then pending. No Grok prompt had been submitted at that checkpoint. Original preference values are omitted from this public edition.

The initial amendment authorized the stated ChatGPT fallback, supplements and pacing; it did not authorize bypassing a CAPTCHA or imply approval of the then-pending Grok setting changes.

### Subsequent authorized control update: November 2, 2026

The user subsequently approved disabling Grok's X-based and conversation-history personalization for testing and restoring the original values afterward. Both test switches were actually set OFF, and the completed Grok records retain those observed control values. This establishes visible UI controls, not audited backend isolation. Post-test restoration remains a required later action; approval to restore is not proof it has already happened.

At this collection checkpoint, three Grok answer bodies are complete. ChatGPT has technical-failure and body-capture-failure records. Preserve their distinct statuses; neither establishes a negative recommendation result, and a rendering/capture failure does not prove server-side generation failed. In particular, the earlier partial accessibility view of Grok M2 repetition 1 was superseded by a complete native rendered answer and matching 3,821-character copied body with a coherent ending; its historical partial capture is not treated as a separate submission or as evidence that the selected body remains incomplete.

### Capture-channel update: November 2, 2026

During recovery of the existing Grok M1 repetition-1 submission, the browser connection became available for the same browser profile/account. Collection could therefore move from native Chrome accessibility/copy capture to read-only rendered browser DOM capture. This is a change in the evidence-capture channel, not a change of consumer platform, account, Expert mode, prompt or model setting. All earlier native captures remain preserved. No prompt was resubmitted to make this capture change.

The M1 selected body is the complete 4,126-character rendered answer, reviewed from its opening through its final paragraph, with four observed DOM hrefs, including the Shmeer/Gomega citation group. An empty browser clipboard read is retained as a failed capture event, not as an empty answer. The first three Grok answers' source URLs remain subject to their original accessibility-display qualifications, including potentially omitted schemes or leading www; later exact DOM hrefs do not retroactively establish those earlier exact destinations. Compare normalized domains with provenance intact, and keep URL-level precision differences explicit.

### Grok collection and restoration checkpoint: November 2, 2026

All 12 planned Grok answer positions are now complete, covering all four prompts and three repetitions. There have been 14 actual submissions across both platforms at this checkpoint: the 12 completed Grok submissions and two ChatGPT attempts with technical/body-capture failures. ChatGPT still has **zero complete selected answer bodies**; its 12 planned positions remain outstanding rather than negative findings. This is an interim collection checkpoint, not completion of the 24-answer design.

After Grok collection, both personalization settings were actually restored to their pre-test values (original preferences omitted). The final rendered controls are recorded in `[Private capture retained locally.]`; this is observed restoration rather than a planned action. The completed Grok records retain their test-time OFF/OFF values. Earlier pending-restoration statements describe earlier checkpoints and are superseded by this observation.

An evidence-reference audit also found that `[Private capture retained locally.]` captured the user prompt through a failed global selector. That original is preserved as non-supporting evidence. The complete M1 repetition-1 body is supported by `[Private capture retained locally.]`, while the article and four exact hrefs are corroborated by `[Private capture retained locally.]`. The coding reference was corrected without submitting another query or changing the selected answer.

### Manual UI control and approved API replacement: November 2, 2026

After the preceding 14-submission checkpoint, a third ChatGPT consumer-web submission was sent manually by the user, with the agent preparing the unchanged V1 prompt and the selected GPT-5.6 Sol / Medium, non-personalized temporary-chat and native-search controls. The exact click time is unavailable and is represented by its recorded time bounds. This is a diagnostic supplement for the same planned V1 repetition-1 position, not a fourth repetition.

The manual control recovered an 11,393-character visible prefix. Both the rendered body and OS Copy stop at **"That gives you a baseline you"**. The UI says the answer is complete, but the shared unfinished ending prevents a complete-body classification. Status is `capture_incomplete`, `body_complete=false`; retain the whole available prefix without coding whole-answer absences or using it as a complete formal answer. The native completion and copy artifacts contain private account/email-related controls and remain private evidence. No email or sharing operation was performed.

Five citation-button occurrences are visible, but the scoped body capture contains no resolved hrefs. Zero observed hrefs does not mean no citations. Do not reuse A2 source-panel URLs as A3 sources. The search chip is independently observed as selected, while actual search remains unconfirmed for A3 because no expanded search-tool trace was captured; citation buttons and a thought-duration/completion label alone do not establish that trace. Manual sending produced more visible text than the preceding loading-error-only capture, but still failed the complete-body criterion; this single control cannot identify an automation cause.

The user explicitly approved replacing the consumer-ChatGPT formal collection arm with **12 new OpenAI Responses API answers using native `web_search`, model `gpt-5.6-sol`, reasoning effort `medium`, within a total US$10 budget**. The current formal platform label is **OpenAI API + web search** and its interface is **Responses API**. This API configuration is not consumer ChatGPT Search and must never be presented as if the two surfaces were equivalent. Model availability and collection success are not established merely by this authorization.

The formal design still has 24 planned answer positions: 12 completed Grok consumer-interface answers and 12 planned independent API answers. Original `S3-CHATGPT-{prompt}-R{repeat}` position identifiers are retained for traceability, with explicit API platform/interface metadata. Planned new actual attempt identifiers are `S3-OPENAIAPI-{prompt}-R{repeat}`. No API attempt is selected or recorded as submitted yet. All three historical consumer-UI attempts remain in the ledger, excluded from formal API-answer counts. There are now **15 actual submissions, 12 complete selected bodies, three historical UI failures/partial captures, and zero API submissions**.

For the API arm, send exactly the registered prompt in each stateless independent request, without prior response IDs, conversation history, personalization context, earlier answers, added examples or output-format directions. Configure native search as required and record actual returned search calls and citation annotations separately. Keep the authorized model and effort fixed and retain the **minimum 90-second same-platform submission interval**. Do not silently substitute an unavailable model, effort, provider or search configuration. Credential setup and request-limit configuration are still pending validation; protocol status is `awaiting_api_credentials`. This amendment does not set unapproved API request limits or claim any API expenditure.

Within-API repetition and wording contrasts remain descriptive. Any comparison with Grok additionally changes interface, retrieval configuration and collection timing, so it cannot isolate a consumer-platform or wording effect. Preserve these amended conditions and the historical design rather than pooling the failed consumer-UI attempts into the formal API arm. Public release still requires its separate authorization and privacy review.

## Submission ledger, completeness and selection

Maintain a ledger of every actual submission, including failed attempts. Use a planned-position ID and a separate attempt ID. A capture or recovery event belongs to its existing attempt and does not increase the query count.

Preserve the original answer body, prompt, returned links, source evidence, capture method, completeness assessment and errors. A technical failure is not a negative recommendation result. A partial prefix cannot establish that an entity or source was absent from the complete answer. Pagination ending, a completion badge or a coherent introductory paragraph alone does not establish full-body completeness.

Use the earliest complete eligible answer for each planned position. Any additional submission to replace a failed or incomplete attempt requires the applicable authorization and remains an additional attempt in the ledger. Never choose between eligible answers based on their recommendations or source overlap. If no complete eligible answer exists, keep the position unresolved and show the available partial evidence separately.

Report planned positions, actual submissions, complete selected answers, failures and partial captures separately. Completeness of text and completeness of citation mapping are separate fields. Preserve all superseded captures and corrections.

## Source extraction and categories

For each answer, retain every observable answer URL with its displayed label, original URL, hostname and role. Distinguish:

- Explicit body links or inline answer citations.
- Citation-panel entries matched to an answer marker.
- Retrieved sources with no observed citation match.
- Unscoped page-anchor inventories or navigation links, which are not citation inventories.

Do not invent a missing citation URL from an article title. Mark unresolved citation markers and the coverage limit. Keep tracking-bearing originals in local evidence; public copies may remove tracking parameters with a documented transformation. Do not merge unrelated article paths simply because they share a domain.

For domain comparisons, preserve the exact hostname and use a documented registrable-domain normalization for the analysis, retaining the mapping. Do not approximate a registrable domain by blindly taking the last two hostname labels. Count each normalized cited domain once per answer, regardless of repeated links. Keep URL-level counts as a separate descriptive view.

Code the cited page by its substantive role, not just by its domain. Use one primary category and optional secondary labels:

| Source category | Coding rule |
| --- | --- |
| Platform documentation | Official AI/search platform help, product documentation or measurement documentation |
| Academic or independent methods research | Original studies, benchmarks or methodological work with identifiable procedures |
| Vendor research or case study | Original measurements published by a visibility, SEO or marketing vendor; commercial provenance retained |
| Tool/product documentation or sales page | Capabilities, pricing, tutorials or marketing for a named product |
| Practitioner tutorial or experiment | First-person workflow or experiment by a practitioner or builder |
| Editorial or industry reporting | Reporting, comparison or explanatory material, rather than primary experimental evidence |
| Community or social discussion | Forum posts, social posts and discussion threads; add a research label if an original experiment is actually presented |
| Other / unresolved | Insufficient information, inaccessible page role, or a source outside these categories |

A source's category can be unresolved when only its label or URL is available. Platform descriptions of studies as "verified" and their numerical claims remain platform output, unless an explicitly separate fact-check establishes otherwise.

## Tools, entities and C/M/R

Record substantively named tools, products and services, including spreadsheets or free utilities when explicitly suggested. Preserve the original name and a manually reviewed canonical name. Document alias decisions before aggregation; do not merge products merely because they have similar names. Record generic methods, such as manual prompt audits or writing a script, separately from branded tools.

For each coded tool/entity, retain evidence excerpts and assess the following independently:

| Field | Rule |
| --- | --- |
| Citation (C) | The answer cites content published or authored by that entity. A third-party page about the tool is not automatically its own-content citation. |
| Mention (M) | The answer explicitly names the entity in its substantive text. A URL hostname alone is insufficient. |
| Recommendation (R) | The answer actively suggests using, reading or following it for the task. A comparison, caveat, paid alternative or explicit exclusion is not automatically a recommendation. |

Use true, false or unresolved, with the reason. Citation may accompany rejection; recommendation may lack an own-content citation. Do not derive a recommendation from a source-panel appearance. Code recommendation stance (suggested, conditional, discussed, rejected) and free-budget fit (stated free, stated paid, mixed/limited tier, unspecified) separately. Do not verify current prices merely from a platform's claim.

ChatGPT and Grok are supplied in all prompts. Contextual mentions of those target platforms are tagged as supplied entities and excluded from discovery/recommendation-set overlap by default; an explicit recommendation to use a particular feature can still be recorded separately. Apply this same rule to all answers.

## Answer-type codebook

Assign one primary answer type based on its dominant response to the request, plus secondary labels where appropriate. Retain an explanatory excerpt rather than force-fitting borderline answers.

| Answer type | Defining behavior |
| --- | --- |
| Manual measurement workflow | Gives a repeatable sequence for selecting buyer prompts, collecting outputs and tracking recommendations |
| Score validation or audit checklist | Explains provenance, denominators, sampling, scoring definitions, repeatability or checks against underlying answers |
| Tool-led solution | Primarily directs the user to one or more products or services |
| Conceptual explanation | Mainly explains GEO, visibility metrics or conceptual distinctions without an executable workflow |
| General advice / mixed | Broad advice or a mixture with no dominant workflow, validation or tool focus |
| Refusal / insufficient access or information | Explicitly cannot answer or requires information absent from the fixed prompt |

Also record nonexclusive content flags: branded versus unbranded buyer prompts; repeated runs; fresh sessions; model/search/version logging; citations versus mentions versus recommendations; raw-answer inspection; numerator/denominator definitions; composite-score weighting; uncertainty/variance; human checking; downstream conversions; paid-tool dependence. These describe returned advice, not independently validated best practices.

## Descriptive calculations

Build separate sets for cited domains, mentioned tools and recommended tools. Do not add C, M and R counts as though they were mutually exclusive. Count an entity at most once per answer for each field.

- For each identical-prompt cell, report all three pairwise Jaccard overlaps, the range and mean, alongside set sizes and entities shared across all three answers. Jaccard is intersection divided by union. If both sets are empty, report **N/A: both empty**, not perfect stability; if only one is empty and both captures are complete, overlap is zero.
- For wording contrasts, report each variant's union and per-answer prevalence (0/3 to 3/3 when all three are eligible), then pooled-set overlap. Optionally show the nine cross-variant answer-pair overlaps, clearly stating that the nine values reuse six answers and are not nine independent trials. Interpret wording differences against the observed repeat variation rather than attributing every differing item to phrasing.
- For M-versus-V contrasts within a platform, report source-category prevalence per eligible answer and recommendation lists with denominators. A category appearing twice in one answer counts once in prevalence. Show the two variants separately before a six-answer-per-intent pooled view. Categories may overlap, so percentages need not sum to 100%.
- Report answer-type counts and actual-search evidence by platform and prompt. Separate a source's inclusion/exclusion role from whether it was cited.

Every proportion displays its numerator and eligible denominator. Use complete selected bodies for absence-based comparisons. If citation mapping is incomplete, exclude that answer from fully observed citation-domain overlap or report explicitly labeled observed-link overlap; never present partial mapping as an exhaustive domain set. Missing judgments are not false values.

## Reporting boundaries

Lead with observed findings, then tables, per-answer evidence and the complete attempt history. State deviations before interpreting affected comparisons. Keep model/interface labels and timestamps attached to each result. Document who coded the answers and whether any independent review occurred; do not describe assistant coding as independent human double-coding.

This experiment measures sensitivity of observed AI outputs to question design under specific conditions. It cannot establish GEO optimization effectiveness, whether any product has good or bad GEO, the accuracy of a visibility provider's score, or stable recommendation probabilities. No intervention, fixed target product, pre-optimization baseline or control group is present. The four prompts also cannot isolate every contextual or semantic difference between their wordings.

Prepare local evidence and analysis first. Public release is a separate action requiring its applicable authorization and a redacted edition; do not expose private conversation/share links, account UI, local paths or internal identifiers merely to support reproducibility.


## Final collection and coding checkpoint — November 2, 2026

All 24 planned positions have complete selected bodies and assistant semantic coding: 12 Grok consumer-interface answers and 12 OpenAI Responses API + web search answers. The 27 prompt submissions include the three preserved historical ChatGPT UI failures/partial captures. Non-generation TLS/model-access checks are separate setup records.

The API condition requested and returned the undated `gpt-5.6-sol` ID with `medium` reasoning. Each independent request contained exactly the registered user prompt, with required live web search and no conversation/history/system additions. All 12 returned completed actual searches; none reached the fixed eight-tool/32,768-output-token limits. The API sequence had a minimum 103.041462-second submission interval. Its 23 reported billable searches and token usage yield an estimated US$1.667876, not a reconciled invoice, within the US$10 authorized budget. No further submissions are planned.

All API requests occurred after the completed Grok sequence. Interface, model and time are confounded in cross-platform summaries; exact backend snapshots are not independently established. The registered wording variants also differ in budget/action cues (for example, V1 says “no GEO budget,” V2 “bootstrapped”). Treat within-intent contrasts as sensitivity to those phrase packages rather than a synonym-only intervention.

Sparse API tool recommendations require care: nine of twelve answers recommended no new named tool/service/feature/channel, while still giving substantive methods. Both-empty Jaccard pairs remain N/A. C/M/R summaries use entity-by-answer rows excluding question-supplied targets and source-publisher-only rows; they do not measure market share or causal GEO effectiveness. Model-proposed scoring rubrics, example dashboards and sample sizes are retained as generated advice, not adopted experimental metrics or verified observations.


## Post-hoc peer-review analysis amendment — November 2, 2026

This amendment was written after the 24 answers and their original coding existed. It does not pre-register or add model submissions. Original strict entity judgments, recorded answers and technical failures remain the baseline. New analyses compare six within-wording answer pairs with nine cross-wording pairs per intent and execution environment using the same Jaccard definition. Those pairs reuse six responses and are dependent descriptive comparisons; no causal effect, statistical significance or independent-trial claim is made. Both-empty sets remain N/A and numeric denominators are displayed.

An explicit 52-name entity-normalization mapping adds a product-line family lens, coarse category lens and strict-name analyses filtered by entity category. Grouping versions/unspecified names in a sensitivity lens does not identify them as the same product in the strict codebook. Google Analytics and GA4 share a family only in that secondary lens; Google Sheets and Search Console remain distinct families. Category collapse can mechanically increase overlap.

URL-level comparisons use the declared privacy/tracking/session/signature-cleaned URL string, preserving paths, meaningful query parameters and fragments without redirect resolution, decoding repair or article-identity inference. Exact original strings remain private evidence. Complete-map and qualified observed-link cohorts remain separate; the early three Grok precision limits still apply. Domain overlap is not URL or article overlap.

A purposive eight-case packet prepares independent human spot-checking of ambiguous C/M/R and identity decisions. No human judgments have yet been supplied; assistant review is not independent human validation and no agreement score is claimed. A separate limited six-source assistant audit checks access and one relevant claim per selected source, without validating all external claims or commercial scores. The public reader is shortened while all original evidence, C/M/R conditional-denominator tables and source composition remain in an expanded technical appendix.
