SEO · Published 2026-11-03

Robots.txt vs Noindex vs Canonical: Which One to Use and When

Robots.txt, the noindex directive, and the canonical tag all seem to "keep pages out of Google", so they get used interchangeably. They do different things, and combining them the wrong way produces the opposite of what you wanted: pages you blocked still appear in search, or pages you meant to keep vanish.

The short version: robots.txt controls crawling, noindex controls indexing, and canonical tells search engines which of several similar URLs to show. The rest of this guide explains what that means in practice and maps common situations to the right tool.

Robots.txt: controls crawling, not indexing

A Disallow rule in robots.txt asks compliant crawlers not to fetch a URL. It does not remove that URL from the index. If other pages link to a blocked URL, Google can still index it based on the links alone and show it with no description. Search Console reports this as "Indexed, though blocked by robots.txt".

Use robots.txt to manage crawl load: stop bots from spending time on infinite faceted URLs, internal search results, or cart and checkout steps. Do not use it to hide sensitive content. The file is public, so listing /admin-backup/ in it advertises the path.

Google has not supported a noindex rule inside robots.txt since 2019. Do not block CSS and JavaScript files either; Google renders pages and needs those resources to understand the layout.

User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?sort=
Allow: /

Sitemap: https://example.com/sitemap.xml

Noindex: controls indexing, but needs crawling

A noindex directive tells search engines not to show a page in results. It can be a meta tag in the HTML head or an X-Robots-Tag HTTP header. The header is the only option for non-HTML files such as PDFs.

The trap: a crawler has to fetch the page to see the noindex. If the same URL is also disallowed in robots.txt, the crawler never reads the directive and the page can stay indexed. When you want a page removed from search, allow crawling and add noindex. Once it has dropped out, you can leave both as they are; adding a robots.txt block afterwards is optional and only saves crawl.

Noindex is a directive, so Google follows it. Pages that remain noindexed for a long time tend to be crawled less often, and links on them may eventually stop being followed, so do not noindex pages whose main job is to pass visitors and link signals elsewhere, such as category hubs.

<!-- In the HTML head -->
<meta name="robots" content="noindex">

# As an HTTP response header (works for PDFs and other files)
X-Robots-Tag: noindex

Canonical: consolidates duplicates, as a hint

The rel="canonical" link says "this page is a copy or near-copy of that URL; show that one". Search engines then combine signals such as links onto the canonical URL. Unlike noindex, it is a hint: Google may pick a different canonical if other signals, such as internal links, sitemaps, and redirects, point elsewhere. Search Console shows this as "Duplicate, Google chose different canonical than user".

Canonical is the right tool when the duplicate pages should still be crawlable and usable by visitors: tracking parameters, sort orders, print versions, the same product in two categories, or syndicated copies of your articles on a partner site.

  • Use absolute URLs, including the protocol: https://example.com/shoes, not /shoes.
  • Add a self-referencing canonical on every indexable page so parameter variants have a clear target.
  • Point to a page that returns 200, is indexable, and is not itself canonicalised elsewhere. Chains and canonicals to redirects or 404s are ignored.
  • Keep it consistent: the canonical URL should be the one in your sitemap and the one your internal links use.
  • Do not canonicalise page 2 of a paginated list to page 1. The content is different; each page should be self-canonical.

Which one to use: common situations

Most real decisions fall into a handful of patterns. Choose by asking two questions: should visitors still reach this URL, and is its content a duplicate of another page?

  • Tracking and sort parameters (?utm_source=, ?sort=price): canonical to the clean URL.
  • Faceted navigation creating thousands of combinations: canonical for the few useful variants, robots.txt Disallow for the long tail to save crawl budget.
  • Internal site search result pages: noindex, or block in robots.txt if they are not yet indexed and you only want to stop crawling.
  • Thank-you pages, login pages, account areas: noindex.
  • Thin tag or author archives: noindex, or improve them so they deserve to be indexed.
  • PDF that duplicates an HTML page: X-Robots-Tag noindex on the PDF, or a canonical HTTP header pointing to the HTML version.
  • Staging or development site: password protection (HTTP authentication). Robots.txt and noindex are not access control, and staging copies with noindex have a habit of being deployed to production.
  • Content permanently moved: a 301 redirect, not a canonical.
  • Content deleted for good: return 404 or 410. Use the Search Console Removals tool only for urgent, temporary hiding while that takes effect.

Combinations that send conflicting signals

Problems usually come from stacking tools. Robots.txt Disallow plus noindex means the noindex is never seen. Noindex plus a canonical pointing to a different URL sends two contradictory messages: "do not index this" and "this is the same as that page, pass its signals over". Google has said it may apply the noindex to the canonical target, so pick one. Listing noindexed or canonicalised URLs in your XML sitemap tells search engines those URLs are important, which contradicts both directives.

Before a launch, audit each template once. Generate robots.txt with the Neemtools Robots.txt Generator so the syntax is correct, set robots meta tags in the Meta Tag Generator, and run key URLs through the SEO Validator to confirm what each page actually outputs. Then check the Page indexing report in Search Console a few weeks later; it will tell you which of your intentions Google has followed.