GEO Lab Interim Review · Part 1

Historical replication and sample-size evidence

Helen Bird · Published 2026-11-10 · v1.0

Methods and collection

Collection took place November 7–10, 2026, Beijing time. OpenAI used gpt-5.6-sol, medium reasoning and a 16-call native web-search cap. The maximum recorded number of web-search calls was 15. Grok used fresh ordinary Expert chats with an explicit web-search chip and both personalization settings disabled for collection. Expert is the visible model label. An exact backend version is unavailable.

The ledger retains 305 submitted attempts and five preparations without a submission. Of the submissions, 300 were selected, four OpenAI outcomes remained unresolved, and one complete Grok response was excluded for mode deviation.

Set overlap is intersection size divided by union size. Unknown sets and comparisons of two observed empty sets are handled separately. Citation domains represent final answer sources. A source panel item is retrieval evidence and is recorded separately.

Across the 150 planned pairs, the median absolute submission gap was 2.35 hours and the maximum was 52.15 hours. Historical interface, tool-cap and source availability differences limit comparisons across batches. The historical ChatGPT consumer interface and the current OpenAI Responses API are separate environments.

Historical claims

Historical claim IDDecisionObserved evidence
H01-Q3-OVERLAPSUPPORTED105/105 matched n=2 subsets below 0.50 and 0.25. Full panel 3 shared of 65 units.
H01-REPEAT-VARIATIONOBSERVEDEvery Q1–Q3 condition had unequal evaluable recommendation sets. Q3 had 105/105 unequal pairs per environment. Citation-domain variation is recorded separately.
H01-CMR-SEPARATIONREPRODUCEDTyped discordant C/M/R patterns observed, including 42 citation-only API Q3 entity-answer rows out of 319.
H02-DISCOVERY-GRADIENTMIXEDGrok A−B and A−C both +100 pp. API A−B −73.3 pp and A−C 0 pp. The API B matches used the public archive.
H02-ACCESS-NOT-DISCOVERYOBSERVED_IN_GROKGrok read the original in 15/15 A answers and did not find the original in 15/15 C answers. API had zero direct A successes.
H03-REPEAT-VARIATIONOBSERVED_IN_EVALUABLE_CELLSRepeat tool lists differed in evaluable cells. OpenAI V2 has 15 known empty strict sets. Pairwise coverage and empty unions are tracked.
H03-WORDING-VS-NOISEHISTORICAL_DIRECTION_UNVERIFIEDStrict complete-block D was positive in three contrasts, +2.6 to +20.1 pp. API V1 versus V2 is N/A. Historical signed D under this projection is unverified.
H03-INTENTRETAINEDWorkflow dominated M in all 60 answers. Checklist dominated V in 59/60 answers.
H03-SOURCE-MIXOBSERVEDSource categories, named tools and domain overlaps differed in the observable output. API 60/60 M/V answers and Grok 1/60 had final URLs. Grok search panels are separate.
H04-Q2-ABSENCEREPRODUCEDAll three candidate families had 0/15 for each of C/M/R in each Q2 environment. Wilson upper limit 20.4% per cell.
H04-ASSET-PERSON-GAPREPRODUCEDGrok Q1 recommended The GEO Lab in 15/15 with Artur named in 0/15. Q3 work-without-name cases were 1/15 API and 2/15 Grok.
H04-QUESTION-DEPENDENCEOBSERVEDQuestion-specific recommendation rates differ. For example Kevin or Growth Memo occurred in 13/15 API Q3 answers and 0/15 in each of Q1 and Q2.

Subset results

MetricAnswers keptRate rangeMore than 20 pp from the full rate
Exact quote, OpenAI API30%–100%235/455 (51.65%)
Exact quote, OpenAI API1060%–100%11/3,003 (0.37%)
Artur or his work, Grok Q330%–100%235/455 (51.65%)
Artur or his work, Grok Q3100%–40%11/3,003 (0.37%)

Subsets reuse answers and their distribution describes this batch. The 95% Wilson intervals for 11/15 and 4/15 are about 48%–89% and 11%–52%. The interval model assumes approximately independent binary observations.

Data and reproduction

Reproduce the headline metrics offline with python3 scripts/build_interim_review.py from the repository root. See the GitHub experiment record for the data guide. Cite a commit permalink and an answer ID to point to a fixed record.

Public records omit private conversation links, unverified document-sharing links, credentials, account details, local paths, opaque provider IDs, signed and tracking parameters, and private interface captures. Search evidence unavailable from the provider remains unknown. Exported bodies may differ from the private originals where redaction was necessary.