Guides
Is my site blocking ChatGPT? A two-minute answer
Look for OAI-SearchBot, not GPTBot, then rule out the firewall and JavaScript.
Is my site blocking ChatGPT? To find out, look for one name in your robots.txt file: OAI-SearchBot. If a rule disallows it, OpenAI says your pages will not be shown in ChatGPT search answers. The quickest way to be sure is to run your domain through an AI crawler check such as the one at seogeoaeo.ai, which lists every rule that blocks an AI bot and the line to change.
ChatGPT uses more than one bot
OpenAI's crawler documentation lists separate agents, and each one is controlled on its own:
- OAI-SearchBot finds pages for ChatGPT search. This is the one that decides whether you can be cited.
- GPTBot collects pages that may be used to train OpenAI's models. Blocking it is an opt-out from training, not from search.
- ChatGPT-User loads a page when a person asks ChatGPT to. OpenAI says robots.txt rules may not apply to it and that it is not used to decide what appears in search.
So a site can block GPTBot and still show up in ChatGPT search, as long as OAI-SearchBot is allowed. A site owner who blocked GPTBot to shut ChatGPT out has only opted out of training, and the reverse happens too: a broad rule meant for scrapers can catch the search bot by accident.
Three places a block can hide
1. robots.txt. Open yoursite.com/robots.txt and look for User-agent: OAI-SearchBot or User-agent: * followed by Disallow: /. Watch the group rule: a bot obeys only the most specific group that names it. If you have a User-agent: OAI-SearchBot group, whatever sits under * no longer applies to it.
2. Your firewall or CDN. A bot-protection setting can return errors to OpenAI's bots even when robots.txt says yes. OpenAI publishes IP lists for OAI-SearchBot, GPTBot and ChatGPT-User so you can allow them on purpose. Your access logs will show whether requests from these bots got a 200 or an error.
3. JavaScript. The bot may get in and still find nothing. seogeoaeo.ai notes that most AI crawlers read the HTML the server returns without running scripts. If your text is drawn in by JavaScript, the page can look empty to them.
A quick test plan
| Check | How | What a problem looks like |
|---|---|---|
| robots.txt rules | seogeoaeo.ai Robots.txt & AI Crawler Checker, 10 credits | A Disallow that matches OAI-SearchBot |
| What the bot can read | seogeoaeo.ai AI Crawler View, 8 credits | Main text appears only after JavaScript |
| Whether the bot arrives | seogeoaeo.ai AI Bot Log Analyzer on a pasted log, 8 credits | No OpenAI bot visits, or refused requests |
The robots check also covers GPTBot, ClaudeBot, PerplexityBot and Google-Extended, so you can answer the same question for Claude, Perplexity and Gemini in one pass.
After you fix it
OpenAI says ChatGPT search can take about 24 hours to adjust after a robots.txt change. Run the check again once the new file is live to confirm the matching rule changed.
You can do the robots.txt part by hand for free if the file is short and has one group. Once there are several groups or wildcards, a checker that names the matching rule saves you from guessing.