Googlebot

Googlebot is Google's crawler. It is really a whole army of crawlers that index the web around the clock. Googlebot finds new websites and pages, and it also checks regularly whether your pages have changed.

How does Googlebot work?

Googlebot crawls the pages of your website. Crawling means visiting a site, going through it and adding its information to the Google index.
Googlebot actively looks for new pages and new links. It also revisits pages often to check whether they are still the same. Make your pages easy for Googlebot to handle, so Google can index them without trouble.
Googlebot is the most advanced crawler there is, and the only major crawler that sees your page the way a real browser does. Other crawlers only see the source code. Googlebot actually renders your page, checks where every image and piece of text sits, and even notices when JavaScript changes the content.

How often does Googlebot visit your site?

How often Googlebot visits depends on your site's crawl budget. That is Google's term for how often Googlebot is allowed to visit your site. A good site with valuable content gets visited more often than a site with weaker content. And if you change your pages often, Googlebot comes by more often too.

Blocking Googlebot

You can block Googlebot if you do not want parts or pages of your site to show up in Google. There are two ways to do this.
The first is robots.txt, a small file you put on your website. In it you can tell Googlebot, per page or per folder, not to crawl it. The second is the robots meta tag, a line of code in the HTML of your page. Per page you can set whether Google may index it (index vs. noindex) and whether it may follow the links on it (follow vs. nofollow). By default the robots meta tag is 'index, follow', so every page may be indexed and followed without you having to set anything.

Googlebot and your sitemap

"Googlebot loves sitemaps" is a well-known Google quote. A sitemap is a list of all the pages on your website, and it can also say when each page last changed. When Googlebot finds a sitemap, it does not have to hunt for new or changed links on its own, so it can index your site faster and more completely.

Other instructions for Googlebot

In robots.txt you can also give the location of your sitemap, so Googlebot can find it and use it. Some crawlers also accept a crawl delay there, a pause after each request, which can help a busy, overloaded site for a while. Googlebot ignores that setting.

A warning

Be very careful when you experiment with the robots meta tag or block pages in robots.txt. You would not be the first, and certainly not the last, to remove your entire website from Google's index by accident. Always double-check your settings and keep a close eye on Google Search Console.