noindex vs robots.txt disallow: which one to use when

noindex and robots.txt Disallow look like they do the same thing. They do not. They do opposite things, and using the wrong one is one of the most common technical SEO mistakes I see across client sites.

Disallow tells search engine crawlers not to visit a page. noindex tells them to visit it, read it, and then remove it from the search index. One blocks access. The other grants access and then instructs the crawler to exclude the page from results.

The confusion between these two causes real damage. Pages stay indexed when they should not. Pages disappear from search when they should be visible. And the worst part is that neither failure throws an obvious error. It just silently breaks.

What robots.txt Disallow actually does

A Disallow directive in your robots.txt file tells crawlers not to request a specific URL or URL pattern. If Googlebot respects the directive, it will never fetch the page content.

But here is the critical part most people miss: blocking a page from being crawled does not remove it from the index. If Google already knows about the URL, through an inbound link, a sitemap, or a previous crawl, it can still appear in search results. It will show up as a URL with no snippet, no description, and no cached version, but it will still be there.

This is exactly what happens when someone Disallows a page they want hidden from Google. The URL stays indexed. It looks broken in search results. And because Google cannot crawl it, it cannot see any noindex tag that might also be on the page.

The Disallow directive is a crawl control, not an indexing control. Use it when you want to prevent crawl waste on pages that do not need to be crawled, like internal search results, filtered URLs, or admin panels. Do not use it when you want a page removed from search.

What noindex actually does

A noindex tag, either in the page's HTML head as a meta robots tag or as an X-Robots-Tag HTTP header, tells search engines to crawl the page but exclude it from the search index.

For noindex to work, the page must be crawlable. Google has to be able to fetch the page, read the noindex directive, and then act on it. If you block the page in robots.txt, Google never fetches it, never sees the noindex tag, and never removes it.

This is the fundamental mistake: Disallowing a page you want to noindex. The two directives conflict. Disallow prevents the crawl. noindex requires the crawl. When you use both, the Disallow wins because it fires first, and the noindex is never processed.

The decision framework

The question is not "which one should I use." It is "what outcome do I want for this page."

Use noindex when:

• You want the page completely out of search results
• The page has inbound links that might cause Google to discover it
• You want Google to crawl the page (for example, to follow its links) but not show it in results
• You are removing thin, duplicate, or outdated content from the index

Use robots.txt Disallow when:

• You want to prevent crawl waste on pages Google does not need to visit
• The pages are behind authentication and discovery is not a risk
• You have a large set of internal URLs (faceted filters, search results) that burn crawl budget
• You do not care whether the URL appears in the index, just that Google does not spend resources crawling it

Use both (carefully) when:

• You have pages behind authentication where you also want to ensure they never appear in search. But in this case, the authentication itself is doing most of the work.

Common scenarios and the right tool

Internal search results pages: Disallow. These pages are dynamically generated, they create infinite crawl paths, and they do not need to be indexed. Blocking the crawl is the right move.

Old blog posts you want removed from Google: noindex. Google already knows about them. Disallowing them will not remove them. Adding noindex and letting Google recrawl will.

Staging or development environments: noindex plus password protection. If Google somehow discovers the staging URL through a link, the noindex tag will prevent indexing. The password prevents anyone from accessing the content.

Paginated archives: This depends. If the archive pages provide genuine value and internal linking, keep them indexed. If they are thin and duplicative, noindex them. Do not Disallow them, because the links on those pages help Google discover your actual content.

E-commerce filter pages: A combination approach often works best. Canonicalize filtered pages to the main category page. If you have thousands of filter combinations, you can Disallow the ones with minimal value to save crawl budget. But for the filters that might actually rank (like "blue running shoes"), keep them crawlable and use canonical tags.

How to check your current setup

Open Google Search Console and go to the Page Indexing report. Look for pages listed under "Blocked by robots.txt" and "Excluded by noindex tag."

For pages blocked by robots.txt: ask yourself whether you intentionally blocked them. If any of those pages should not be in the index, you have the wrong tool. Switch to noindex.

For pages excluded by noindex: confirm these are pages you genuinely want out of search. If any are pages you want indexed, remove the noindex tag and check whether a plugin or theme is adding it automatically.

Also check for the worst case: pages that are both Disallowed and have a noindex tag. Google will tell you the page is blocked by robots.txt but will not report the noindex because it cannot see it. If your intent was removal from the index, remove the Disallow line so Google can reach the page and process the noindex.

The one rule that prevents most mistakes

If you want a page out of the search index, the page must be crawlable. That is the rule. Any time someone tells me "I added noindex but the page is still showing up," my first question is always "did you also Disallow it?" About half the time, the answer is yes.

The fix is simple once you understand the distinction. Disallow controls crawler access. noindex controls search visibility. They are different layers of the same system, and they work against each other when applied together.

If your indexing report shows pages in the wrong bucket, that is exactly the kind of technical cleanup I handle before any design or content work. Getting the foundations right makes everything else compound.