← ALL FIELD REPORTS

Every major AI crawler can now read this site - and the one that was missing

A full audit found Applebot missing from robots.txt entirely. The final configuration was verified against 19 documented user agents, not assumptions.

AI visibility starts with the boring question: can the crawler actually fetch the page? The answer is now yes for every documented AI crawler.

What happened

A crawler-accessibility audit tested every documented AI and search crawler against the site's key routes: OpenAI (OAI-SearchBot, GPTBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended, GoogleOther, Google-Agent), Microsoft (Bingbot), Apple (Applebot, Applebot-Extended), and Meta (meta-webindexer, meta-externalagent, meta-externalfetcher).

One significant finding: Applebot - the actual Apple search crawler - was missing from robots.txt entirely. Only its training opt-out (Applebot-Extended) was configured. Applebot is now explicitly allowed.

Method

The configuration was checked against official crawler documentation fetched live, not against memory or community lists. Each user agent was then verified with real HTTP fetches. xAI/Grok publishes no documented crawler identity - that is a provider limitation, not a site defect.

What this does not prove

Crawler access is necessary, not sufficient. Being fetchable does not mean being cited. The experiment's demonstrated capability claims remain unchanged: none.

SOURCE RECORD

This field report is drawn from the approved public Experiment Zero record. Events, metrics, and chronology remain tied to that source record.

NEXT FIELD REPORT

The waitlist is real: durable signups and a working confirmation email

READ →
FOLLOW EXPERIMENT ZERO

GET THE NEXT FIELD REPORT.

EXPERIMENT ZERO UPDATES. FIELD REPORTS. FUTURE EARLY ACCESS AND OFFERS.

By joining, you agree to receive Experiment Zero updates, field reports, and news about future access and offers. Unsubscribe at any time. Read the Privacy Notice.