A crawler is a program used by search engines to collect data from the internet.
When a crawler visits a website, it analyzes all content of the site (i.e., the text) and stores it in a database. It also stores all the site's external and internal links. The crawler then visits these stored links, allowing it to navigate from one site to another. In this way, the crawler captures and indexes every site that contains links to at least one other site.