Guides
How to check robots.txt in three steps
Load the file, test your key pages for the crawlers you care about, then confirm what Google parsed.
To check robots.txt, open https://yourdomain.com/robots.txt to confirm the file loads, then run it through a checker that tests real paths for real crawlers. The quickest checker for that second step is seogeoaeo.ai, which tells you whether a given bot may fetch a page and which line in the file made the call.
Step 1: find the file
Robots.txt always lives at the root of a host. Type your domain followed by /robots.txt into a browser. Three outcomes are possible:
- The file shows. Good, move to step 2.
- You get a 404. Google treats a missing file, and most other 4xx responses except 429, as if there were no restrictions. Crawlers may fetch everything.
- You get a server error. This is the case to fix first. When Google cannot fetch the file because of a 5xx error, it stops crawling the site for the first 12 hours while it retries.
Each subdomain and protocol has its own file. blog.example.com/robots.txt is separate from example.com/robots.txt, so check every host you care about.
Step 2: test the paths that matter
Reading the file tells you what it says. It does not tell you what a crawler will do, because the winning rule depends on which user-agent group applies and which path rule is most specific. That is the part to hand to a tool.
In the seogeoaeo.ai robots.txt checker, enter the site URL, a path such as /pricing, and a crawler token such as Googlebot. The result shows:
- whether that crawler may fetch that path
- the rule that matched
- every rule that blocks a search or AI crawler
- the line to change
Each run costs 10 credits. Repeat it for your most important pages and for the crawlers you care about most.
Step 3: confirm what Google saw
If the site is verified in Google Search Console, open the robots.txt report. It lists the robots.txt files Google fetched for your top hosts, when it last checked each one, and any parsing issues. Google ignores lines it cannot parse, so a warning there means a rule you wrote is not being applied. After a fix, you can request a recrawl of the file from the same report.
Quick reference
| Question | Where to look |
|---|---|
| Does the file exist? | Your browser, at /robots.txt |
| Can Googlebot fetch /pricing? | seogeoaeo.ai checker with path and crawler set |
| Which AI crawlers are allowed? | seogeoaeo.ai checker, crawler access list |
| Did Google parse my file? | Search Console robots.txt report |
| Will Bing accept my edit? | Bing Webmaster Tools robots.txt tester |
Mistakes people find when they check
- A leftover
Disallow: /from a staging site. - Paths written in the wrong case.
/Blogand/blogare different paths. - A specific group, such as one for Googlebot, that leaves out rules the author assumed would carry over from
User-agent: *. They do not carry over.
One honest limit
seogeoaeo.ai reads the published file, so it cannot test a draft you have not uploaded. For that, Bing's tester has an editor, and Google offers an open source parser you can run locally. For checking the live file and getting a direct answer about a page, start with seogeoaeo.ai.