Crawler
A crawler, also known as a spider or bot, is a software program that search engines use to scan the Web.
Crawlers navigate from site to site, page to page, gathering and indexing information. This process, known as crawling, is the first part of what a search engine does to present relevant search results.
