AI crawler check
Enter a page's address. Geoffy fetches it as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and as a normal Chrome browser, reads your robots.txt, and tells you which AI crawlers can read the page. Free. No email. Results in under 15 seconds.
Last updated
Results for
This page has very little text in its raw HTML. The main AI crawlers do not run JavaScript, so if the page adds its text with JavaScript, they probably see very little of it.
| Crawler | Status | Size | Words | Verdict |
|---|
Some sites check the real IP address of a bot. A real AI crawler can get a different result from this check.
Can the crawlers read you? Next, see whether ChatGPT, Perplexity, Gemini and Claude name you.
What the verdicts mean
- OK. The crawler got the page, and robots.txt lets it read it.
- Blocked. The site answered 401 or 403 to the crawler, or the connection failed.
- Rate limited. The site answered 429: it limited this request, which does not mean the crawler is unwelcome. Try again in a minute.
- Site error. The site answered with another error, such as 404 or 500, so the crawler got an error page.
- Empty shell. The crawler got less than a fifth of the words a normal browser got.
- Disallowed. robots.txt tells the crawler not to read this page.
Each crawler gets one verdict, checked in that order: a crawler that is blocked or rate limited is not also reported as disallowed. Google-Extended is a robots.txt rule rather than a crawler, so it can only be OK or Disallowed.
What this check cannot tell you
Some sites check the real IP address of a bot. A real AI crawler can get a different result from this check. Firewalls such as Cloudflare's can allow a verified crawler and block anyone who only borrows its name, and the reverse.
The Chrome column is the HTML a browser receives before it runs any script. Most AI crawlers do not run scripts either, so a page that only fills in with JavaScript looks thin in every row, Chrome's included. When Chrome's copy is that thin, the check says so above the table.
FAQ
AI crawler check: questions
What people ask before they run it.
Is it free?
Yes. No account, no email and no install. Paste a page's address and the results appear on this page.
Which crawlers does it check?
GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot from Perplexity, each with the user agent its maker publishes. Google-Extended is a robots.txt rule rather than a crawler, so it is checked against robots.txt only.
Why does it fetch the page as Chrome too?
To compare. If a crawler gets far fewer words than a normal browser does, the site is serving bots a thinner page, and the crawler cannot see what your customers see.
Do you keep the address I check?
No. The check fetches the page, shows you the result and stores nothing.
More free tools
- The free GEO Score: where AI engines name your products, and where they name a competitor.
- Live AI crawler visits to geoffy.ai: the bots reading this site, as it happens.