GEO Lab Interim Review · Part 1
Which GEO findings held up after 300 new answers?
1. What did we expect?
An interim review is a checkpoint in the ongoing GEO Lab. Here we revisit earlier findings and use the evidence to shape the next experiments.
- Several historical findings would recur. These include low cross-platform recommendation overlap and recommendations of a work without a named author.
- Three answers would give unstable estimates of some rates. We expected larger subsets to be less sensitive to which runs were included.
2. How did we test it?
We reran ten original prompts from GEO Labs #01–#04 fifteen times per environment, for 300 new answers. We compared them with 60 historical answers. The two environments were OpenAI Responses API with web search and Grok Expert in the web interface with web search.
For each condition we checked all 455 ways to keep three answers and all 3,003 ways to keep ten. For cross-environment comparisons, we used the same scheduled block numbers on both sides.
Labs #01 and #04 share the new answers for three identical prompts. Lab #05 covered technical discoverability and is outside this answer replication. Matching uses scheduled block numbers; actual submission times differ.
3. What did we find?
- Low recommendation overlap held up. On the tutorial prompt, overlap was below 25% in all 105 matched comparisons using two answers per environment. Median overlap was 4.5%. The full lists shared 3 of 65 distinct recommendation units.
- Direct access and natural discovery differed. Grok read the original X post in 15/15 direct-link runs and found it in 0/15 natural-search runs. The OpenAI API identified it in 11/15 exact-quote runs through a public archive that became available after the original experiment.
- Work and author visibility diverged. On the free-learning prompt, Grok recommended The GEO Lab in 15/15 answers and named Artur Ferreira in 0/15. Measurement prompts usually elicited workflows and score-verification prompts usually elicited checklists. Evidence remains insufficient to establish whether wording changes consistently outweigh repeat variation.
- Three answers could give very different rates. On the tutorial prompt, Grok recommended Artur or his work in 4/15 answers. Three-answer subsets ranged from 0% to 100%. For two example rates, the share of subsets more than 20 percentage points from the full-panel rate fell from 51.6% at three runs to 0.37% at ten.
Three runs can show that an outcome occurred. Ten to fifteen can support a more informative description of a fixed prompt. Estimating a rate to a specific precision requires a separate sample-size decision.
The OpenAI arm measures the API setup. Earlier ChatGPT consumer results used a different environment. This comparison does not identify a causal effect of GEO optimization.
See all twelve historical claim decisions and the public evidence.