Crawler

A crawler is a program that visits and indexes web pages. Crawlers are also called spiders. You want to set up your site so crawlers can easily find every page.

Google's crawler

For SEO, the crawler that matters most is Google's, which is called Googlebot. Googlebot is the most advanced crawler around. Many other crawlers only read the source code of your page, but Googlebot analyzes it the way a real browser would. So it also looks at where your content actually appears on the screen, and it picks up content that JavaScript adds or changes.

Crawl budget

Google gives every site a crawl budget: how often Googlebot is allowed to visit your site. Popular, high-quality sites get a bigger crawl budget than others. Errors such as broken links and duplicate content can make Googlebot burn through that budget quickly. That is why technical SEO matters so much for a website.

Instructions for crawlers

You can influence how crawlers behave. A special file, robots.txt, holds instructions for them. The robots.txt file always sits at the top level of the site (the 'root directory'), so a spider always knows where to find it.
In robots.txt you can specify, among other things, which pages may be crawled, the minimum delay between requests (some crawlers respect this, Googlebot does not) and which bots are or are not allowed on your site.