What does a site crawl check, and what can't it see?
A crawl sees what your links and your sitemap lead to, as a bot. It doesn't see what Google decided to do with those pages. The crawler reads the homepage, follows its links, then the links on those pages, until it runs out. That only works for real links: Google can only crawl your link if it's an <a> HTML element with an href. A menu built on a click handler is invisible to your crawler and to Googlebot alike. For every address, the crawl writes down the status code (200, 301 or 302, 404, 5xx), what robots.txt and the robots meta tag allow, the canonical, where a redirect goes, and what's on the page, down to a speed test in a lab.
What it misses is just as useful to know. A page nothing links to only turns up when the sitemap lists it, which is how a crawl finds orphan pages: in the sitemap, without a single internal link. Content that JavaScript adds in the browser stays invisible until you switch rendering on. Google itself does render: it queues all pages with a 200 HTTP status code for rendering. Pages behind a login stay out. So do pages a firewall refuses to your crawler. Blocked doesn't mean broken: Googlebot may get through where your crawler can't. And no crawl knows which pages Google indexed or which canonical it picked. Search Console's Page indexing report and URL Inspection tell you that, and the Search Console audit walks through both.
How do you crawl your own site?
You need a desktop crawler, or the crawler that comes with your SEO tool. The best-known free desktop crawler stops at 500 URLs, which is plenty for a small business site and nowhere near enough for a shop. A few hundred pages take minutes; tens of thousands can take hours.
- Start at the homepage and give the crawler your sitemap too, so pages nothing links to still show up.
- Crawl as a smartphone that follows robots.txt, because most of Google's crawl requests come from its smartphone crawler. If the site builds its pages in the browser, switch JavaScript rendering on.
- Leave out what never ends: cart and login links, calendars that link to next month forever, filter and sort parameters. Google warns that faceted navigation can generate infinite URL spaces, and the usual suspects are in crawler traps.
- Let it run. If the site sits behind a firewall like Cloudflare, let your crawler through with the rule your firewall offers for a user agent or a secret header. Never by IP address.
- Export the list and open Search Console's Page indexing report next to it. Your crawl lists what the site offers, and Search Console which of those pages Google took.
What does a crawl find?
Four kinds of things. Here's a little background on each, with the articles that go into the fixes.
Can Google reach and index the page? Status codes, robots.txt, noindex, canonicals, redirects and the sitemap. Google's minimum is short: Googlebot isn't blocked, the page returns a 200, and it has indexable content. Read more in Googlebot, the 404 page, the canonical tag and duplicate content, and use search operators for a quick look at what Google shows.
Redirects get the most questions. A 301 puts the new address in search. A 302 keeps the old address there and doesn't tell Google that the target is the canonical.
Links. Broken links, links to a redirect, orphan pages, pages more than three clicks deep, internal nofollow, anchor text and crawler traps. Google's own rule is that every page you care about should have a link from at least one other page on your site. Long redirect chains slow crawling, but crawl budget is mainly a worry for very big sites: a million pages that change every week, or 10,000 that change every day. The articles: broken links, internal nofollow, anchor text and crawler traps.
What's on the page. What Google reads on the page itself: the title, the meta description, the H1, alt text on images, structured data, and the HTML underneath.
Speed. A crawl measures speed in a lab. What your real visitors get is field data, and that's what corewebvitals.io is about. On this site: Time to First Byte, First Contentful Paint, Largest Contentful Paint, image optimization and content-visibility. Fonts, scripts and CSS have their own pages too: self-hosting Google Fonts, a webfont loading strategy, lazy-loading icon fonts, defer JavaScript, inline CSS and analytics that leave Core Web Vitals alone.
Which crawl issues do you fix first?
Start with an important page that Google can't reach or index by accident: a noindex left on, a Disallow in robots.txt, a canonical that points to another page, a 404 or a 5xx, a redirect where the page should be. That breaks one of Google's three requirements, and nothing else on the list matters as much. Check the page in Search Console's URL Inspection before you change anything.
Does the rest of the list all need fixing? That depends a lot. Often the errors sit in the template or the technical setup. Fix those first: it's simple, and it touches every page. A link in the menu, the footer or a button is on every page, so one rule can count hundreds of pages. That's why the biggest counts are often the cheapest fixes.
Then go through the most important pages, fix a little each time, and slowly the site gets a whole lot better. A problem on a page with clicks from Google goes before the same problem on a page nobody visits.
What I leave alone: notices, and flags that are the site doing its job. A cart that robots.txt keeps out, parameter URLs with a canonical to the clean page, a page that redirects on purpose. Google says as much itself: "You should not expect all URLs on your site to be indexed, only the canonical pages." For 404s, its advice is to fix "only 404 errors that you link to yourself or list in a sitemap".
The health score at the top of the report doesn't count for Google either. John Mueller said it plainly in 2023: "Once again, no, Google does not use scores from third-party SEO tools for search." The score is a counter of open work, nothing more.
In MarketingTracer's SEO audit, "Start here" sits above the full list: the three issues that pay off most, each with its number of pages. When Search Insights is on for the site, each also shows the clicks those pages got from Google in the last 28 days. Each issue carries a label for the work it takes: "Quick to do", "Needs writing" or "For your developer". You can hide a flag that is there by design with "Hide for this site", and then it stops counting. What the audit can't tell you is which pages Google indexed; for that you still open Search Console.
Demo account, sample data.
What a real crawl list turned out to say
The SEO audit of my own site, corewebvitals.io, scanned 1,000 pages and listed 28 rules that failed: 2 errors, 7 warnings and 19 notices. Sorted by pages, the top three were notices: "Link texts say little" on 900 pages, "Images without width and height" on 877, "Images larger than needed" on 861. Fourth came the warning "Links to a redirect", on 840 pages.
840 sounds like a week of work. Most of it is one template. The translated pages link, in the menu and the footer, to addresses in their own language that have no translation. In German that's /de/audit and /de/state-of-cwv in the menu, /de/privacy and /de/terms-and-conditions in the footer, and the same pattern repeats in all eight languages. Each of those answers with a 302 to the English page. Menu and footer are on every translated page, so every translated page links to a redirect. I'd change it once, in the template: link the English address where there is no translation. That clears most of the 840. None of it keeps a page out of Google.
The errors looked worse: 90 pages flagged as not indexable. The page list showed 36 of them, and I checked each one live. Sixteen were those same language addresses. Eleven were links with a ?cta= or ?service= parameter to my consultancy page, each with a canonical to the clean address. One was a JSON file, one a variant with a canonical to its main page, one an old address that now redirects to its new one, and six were normal pages again by the time I looked. Not one was an important page kept out of Google by accident.
My small shop, tenshuren.be, says the same at 13 pages. "Links to a redirect" showed up on 12 of them, from two links in the template: the "add to cart" button, which answers with a 303 to the cart, and the WhatsApp link, which answers with a 302. Its one error is that cart link, and robots.txt keeps the cart out on purpose.
My own site, corewebvitals.io, in the SEO audit on 30 October 2026, a later scan.
How often should you crawl your site?
After every release, redesign or migration, and on a schedule in between. Weekly is a good rhythm: it catches a noindex that slipped in with a deploy within days, while a yearly audit finds the same slip months later.
Compare each crawl with the one before it. What's new since last week is where a deploy broke something, and what's fixed tells you the developer's ticket worked. In the SEO audit, the Changes tab lists the new and fixed issues since last week. An opt-in mail sends them every Monday, but only when something changed.
Demo account, sample data.