Blocker

A blocker is any mechanism that prevents search engine crawlers from reaching or indexing part of a website. It might be a rule in robots.txt, a noindex directive, HTTP password protection, or a firewall that refuses the crawler's request — each stops content from entering the search index.

Common types of blockers

• robots.txt — a Disallow rule tells compliant crawlers not to fetch matching URLs.
• noindex — a meta robots tag or X-Robots-Tag HTTP header lets a page be crawled but keeps it out of results.
• HTTP authentication — a login wall the crawler can't pass, common on staging environments.
• Firewall or bot rules — server-level filters that reject or rate-limit specific user agents.

When a blocker helps — and when it hides too much

Blockers are useful for keeping admin panels, internal search results, and duplicate parameter URLs out of the index. The danger is applying them too broadly. A common failure: a team ships a staging site with Disallow: / in robots.txt, then copies that file to production during launch. The whole site is walled off, rankings collapse within days, and nobody notices until organic traffic flatlines. The fix takes two minutes; the diagnosis often takes two weeks.

Before adding any blocker, confirm exactly which URLs it matches. For a full walkthrough of crawl directives, see our technical SEO work, and read robots.txt for the syntax that trips teams up most often.