Short answer
robots.txt is a small text file at the root of your website, such as example.com/robots.txt, that tells search engine crawlers which parts of the site they may not crawl. It controls crawling, not indexing: a blocked page can still appear in Google if other pages link to it. Most small sites need only a few lines, plus a link to the XML sitemap.
How to read the rules
An empty "Disallow:" line means nothing is blocked. "Disallow: /" on its own means the whole site is blocked, which is the most damaging one-character mistake in SEO.
- User-agent: names the crawler the rules apply to. An asterisk means every crawler.
- Disallow: a path the crawler should not fetch. "Disallow: /cart/" blocks everything starting with /cart/.
- Allow: makes an exception inside a blocked folder.
- Sitemap: the full address of your XML sitemap. Search engines read it from here.
A sensible file for a small business site
For a typical shop or service website, the file might say: User-agent: *, then Disallow: /cart/, Disallow: /checkout/, Disallow: /my-account/, and finally Sitemap: https://www.example.com/sitemap.xml. WordPress adds a rule for its admin area by default, which is fine. Everything else stays open.
Mistakes that hide a website
- Leaving "Disallow: /" in place after a developer used it on the staging site.
- Blocking CSS and JavaScript folders, so Google cannot see how pages really look on a phone.
- Trying to remove a page from Google with Disallow. The page stays listed, often with no description, because Google can no longer read the noindex tag.
- Putting private information in the file. robots.txt is public; anyone can read it, so listing a secret folder advertises it.
- Uploading the file to a subfolder. It only works at the root of each host, and http, https, www and subdomains each need their own.
Who actually follows robots.txt
Google, Bing and other well-behaved crawlers follow it, and many AI crawlers publish their own user-agent names so site owners can allow or block them. It is a polite request, not a lock. Bad bots can ignore it. To truly protect content, use a password or login.
How to check yours today
- Open yourdomain.com/robots.txt in a browser. If you see "Disallow: /" under "User-agent: *", fix it now.
- Check that the Sitemap line points to a working sitemap.
- In Google Search Console, the robots.txt report shows the version Google last fetched and any errors.
- After any redesign or hosting move, check the file again.
robots.txt in AI Marketing Master
The website audit checks that your robots.txt can be read and reports problems in plain English. The crawler also obeys your file, so if it is told to stay out of a section, that section is not audited, which is itself a useful clue when important pages are missing from the report.
What it does not do
- The audit reads and checks your robots.txt. It does not edit the file on your server.
- Because the crawler respects robots.txt, sections you block are not audited.
Questions people ask
Does every website need a robots.txt file?
No. Without one, crawlers assume they may crawl everything. Having one is still useful to point to your sitemap and block cart or account pages.
Can I block AI tools with robots.txt?
You can add rules for AI crawlers that publish a user-agent name. They are requests, so compliance depends on each company. Blocking them may also keep your business out of some AI answers.
How do I stop a page showing in Google?
Let Google crawl it and add a noindex tag, or protect it with a password. Blocking it in robots.txt is not the right tool for removal.
How quickly does Google notice changes to robots.txt?
Google usually caches the file for up to a day. You can ask for a refresh from the robots.txt report in Search Console.
We only describe what the product really does today. Results depend on your business, your market and many things no tool controls.