Short answer
Use noindex when you want a page kept out of search results; use robots.txt Disallow when you want crawlers to stop fetching a section. They do different jobs. Disallow does not remove a page from Google, and if you block a page that also has noindex, Google can never read the noindex, so the page may stay listed. To remove a page, let it be crawled with noindex.
Two different questions
Crawling is Google fetching a page. Indexing is Google storing it so it can appear in results. robots.txt Disallow answers "may you fetch this?" The noindex instruction answers "may you show this?" Mixing them up is one of the most common reasons a page stays in Google long after someone tried to hide it.
Side by side
- Where it lives: Disallow sits in the robots.txt file. noindex sits in the page as a meta robots tag, or in the X-Robots-Tag HTTP header for files like PDFs.
- What it stops: Disallow stops fetching. noindex stops appearing in results.
- Can the page still show in Google? With Disallow, yes, as a bare link if others link to it. With noindex, no, once Google has re-crawled it.
- Scope: Disallow is easy to apply to a whole folder. noindex is set page by page or by template.
- Visibility to people: neither hides the page from someone who has the link.
When to choose noindex
- Thank-you pages after a form, so they do not show up as a search result.
- Thin tag or author archives you do not want in search.
- Internal search result pages.
- Old offers you want to keep for existing customers but not promote.
- A staging or test site, together with a password.
When to choose Disallow
- Cart, checkout and account areas that have no search value.
- Endless filter combinations that create thousands of near-identical addresses and waste crawling.
- Folders of files that are not meant for search, as long as you accept they may still be listed if linked.
The combination that backfires
Someone adds noindex to a page and, to be safe, blocks it in robots.txt too. Google can no longer fetch the page, so it never sees the noindex. If other pages link to it, it can sit in the index for months as "Indexed, though blocked by robots.txt" in Search Console. The fix is to remove the Disallow, let Google crawl the page and see noindex, and only add a block later if you really need one.
Checking your own pages with AI Marketing Master
The website audit reads your robots.txt and respects it while crawling, so blocked sections do not appear in the crawl. It reports problems with your robots.txt and sitemap in plain English. For pages already in Google, use the page indexing report and URL Inspection in Search Console to see why each page is or is not indexed.
What it does not do
- The crawler follows robots.txt, so it does not audit pages you have blocked.
- AI Marketing Master does not add noindex tags or edit robots.txt on your site.
Questions people ask
How fast does noindex remove a page?
Only after Google re-crawls the page, which can take days or weeks. For an urgent case, Search Console's Removals tool hides it for about six months while noindex takes effect.
Can I put noindex in robots.txt?
No. Google stopped supporting noindex rules inside robots.txt in 2019. Use the meta tag or HTTP header.
Is nofollow the same as noindex?
No. nofollow is about links: it asks search engines not to pass credit through a link. It does not stop the page itself appearing in results. noindex is the one that keeps a page out of search.
How do I noindex a PDF?
PDFs have no head section for a meta tag, so the noindex goes in an X-Robots-Tag HTTP header. Your host or developer can add it for a file or a whole folder.
Should noindex pages be in my sitemap?
No. A sitemap should list only pages you want indexed. Listing a noindex page sends mixed signals.
We only describe what the product really does today. Results depend on your business, your market and many things no tool controls.