GEO Lab Interim Review · Part 1
Historical replication and sample-size evidence
Methods and collection
Collection took place November 7–10, 2026, Beijing time. OpenAI used gpt-5.6-sol, medium reasoning and a 16-call native web-search cap. The maximum recorded number of web-search calls was 15. Grok used fresh ordinary Expert chats with an explicit web-search chip and both personalization settings disabled for collection. Expert is the visible model label. An exact backend version is unavailable.
The ledger retains 305 submitted attempts and five preparations without a submission. Of the submissions, 300 were selected, four OpenAI outcomes remained unresolved, and one complete Grok response was excluded for mode deviation.
Set overlap is intersection size divided by union size. Unknown sets and comparisons of two observed empty sets are handled separately. Citation domains represent final answer sources. A source panel item is retrieval evidence and is recorded separately.
Across the 150 planned pairs, the median absolute submission gap was 2.35 hours and the maximum was 52.15 hours. Historical interface, tool-cap and source availability differences limit comparisons across batches. The historical ChatGPT consumer interface and the current OpenAI Responses API are separate environments.
Historical claims
| Historical claim ID | Decision | Observed evidence |
|---|---|---|
| H01-Q3-OVERLAP | SUPPORTED | 105/105 matched n=2 subsets below 0.50 and 0.25. Full panel 3 shared of 65 units. |
| H01-REPEAT-VARIATION | OBSERVED | Every Q1–Q3 condition had unequal evaluable recommendation sets. Q3 had 105/105 unequal pairs per environment. Citation-domain variation is recorded separately. |
| H01-CMR-SEPARATION | REPRODUCED | Typed discordant C/M/R patterns observed, including 42 citation-only API Q3 entity-answer rows out of 319. |
| H02-DISCOVERY-GRADIENT | MIXED | Grok A−B and A−C both +100 pp. API A−B −73.3 pp and A−C 0 pp. The API B matches used the public archive. |
| H02-ACCESS-NOT-DISCOVERY | OBSERVED_IN_GROK | Grok read the original in 15/15 A answers and did not find the original in 15/15 C answers. API had zero direct A successes. |
| H03-REPEAT-VARIATION | OBSERVED_IN_EVALUABLE_CELLS | Repeat tool lists differed in evaluable cells. OpenAI V2 has 15 known empty strict sets. Pairwise coverage and empty unions are tracked. |
| H03-WORDING-VS-NOISE | HISTORICAL_DIRECTION_UNVERIFIED | Strict complete-block D was positive in three contrasts, +2.6 to +20.1 pp. API V1 versus V2 is N/A. Historical signed D under this projection is unverified. |
| H03-INTENT | RETAINED | Workflow dominated M in all 60 answers. Checklist dominated V in 59/60 answers. |
| H03-SOURCE-MIX | OBSERVED | Source categories, named tools and domain overlaps differed in the observable output. API 60/60 M/V answers and Grok 1/60 had final URLs. Grok search panels are separate. |
| H04-Q2-ABSENCE | REPRODUCED | All three candidate families had 0/15 for each of C/M/R in each Q2 environment. Wilson upper limit 20.4% per cell. |
| H04-ASSET-PERSON-GAP | REPRODUCED | Grok Q1 recommended The GEO Lab in 15/15 with Artur named in 0/15. Q3 work-without-name cases were 1/15 API and 2/15 Grok. |
| H04-QUESTION-DEPENDENCE | OBSERVED | Question-specific recommendation rates differ. For example Kevin or Growth Memo occurred in 13/15 API Q3 answers and 0/15 in each of Q1 and Q2. |
Subset results
| Metric | Answers kept | Rate range | More than 20 pp from the full rate |
|---|---|---|---|
| Exact quote, OpenAI API | 3 | 0%–100% | 235/455 (51.65%) |
| Exact quote, OpenAI API | 10 | 60%–100% | 11/3,003 (0.37%) |
| Artur or his work, Grok Q3 | 3 | 0%–100% | 235/455 (51.65%) |
| Artur or his work, Grok Q3 | 10 | 0%–40% | 11/3,003 (0.37%) |
Subsets reuse answers and their distribution describes this batch. The 95% Wilson intervals for 11/15 and 4/15 are about 48%–89% and 11%–52%. The interval model assumes approximately independent binary observations.
Data and reproduction
- Original prompts and condition identifiers
- 300 redacted answer records with links, visible queries and source panels
- 310 attempt histories with recorded submission and failure statuses
- Normalized outcomes and sets used for calculations
- All twelve claim decisions · Recomputed summary · Data archive
Reproduce the headline metrics offline with python3 scripts/build_interim_review.py from the repository root. See the GitHub experiment record for the data guide. Cite a commit permalink and an answer ID to point to a fixed record.
Public records omit private conversation links, unverified document-sharing links, credentials, account details, local paths, opaque provider IDs, signed and tracking parameters, and private interface captures. Search evidence unavailable from the provider remains unknown. Exported bodies may differ from the private originals where redaction was necessary.