Crawlers

Crawlers — also called bots or spiders — are automated programs search engines send across the web to fetch pages, read their content, and follow links to discover more pages. What they collect feeds the index that search results are built from, so a page that's never crawled simply can't rank.

How crawling leads to ranking

There are three stages: crawl, index, rank. Googlebot requests a URL, renders it, extracts the text and links, and queues the new links it finds. The page is then processed for indexing, and only after that can it appear in results. Break the first link in this chain — for example, by blocking the bot — and nothing downstream happens.

Guiding crawlers on your site

• robots.txt: tells bots which paths to skip (though on its own it doesn't remove a page from the index) — see robots.txt
• XML sitemap: hands crawlers a clean list of the URLs you want found
• Internal links: the strongest way to signal which pages matter and help bots reach them
• Crawl budget: on large sites, don't burn bot visits on faceted URLs, duplicates, and dead ends

Example

An e-commerce site launches 4,000 new product URLs but forgets to link them from any category page or add them to the sitemap. Googlebot has no path to reach them, so weeks pass with almost none indexed. Adding the products to category pages and the XML sitemap gets the majority crawled and indexed within days. Catching gaps like this is a routine part of technical SEO.