How to read Googlebot in your server logs

A server log can tell you Googlebot asked for a URL. It cannot tell you Google kept that URL.

Log file analysis is the habit of reading access logs for real crawler requests: path, time, status, and response time. The useful question is narrow. Which URLs were fetched, how often, and what did the host return? Search Console will not answer that at URL level.

A log line is a fetch, not an index

A verified Googlebot hit means that URL was requested and the server answered. It does not mean the URL is canonical, selected, or shown in search.

Google’s crawling troubleshooting guide says this outright. Search Console does not give a crawl history you can filter by URL or path. Logs can show whether specific URLs were crawled. “Whether or not those crawled URLs have been indexed is another story.”

The Crawl stats announcement (November 2020) is the other half. That report totals requests, download size, and average response time, and it breaks requests down by response code, file type, crawl purpose, and Googlebot type. Host status covers the last 90 days. You get example URLs, not the full request list. Use the report for the shape of the crawl. Use logs when you need the actual paths.

Crawl stats also count requests, so one noisy URL pattern can dominate the chart. Group log lines by directory or by query-parameter pattern before you argue about “the site.”

Check the IP before you trust the user agent

The user-agent string is a label. It is easy to forge. If you are about to allowlist a bot, or tell engineering that Google is stuck in a redirect, verify the address.

Google’s verification guide, updated 20 March 2026, gives two methods. For a one-off lookup: reverse DNS, confirm the hostname is Google’s, then forward DNS back to the same IP. For bulk work, match the IP against Google’s published ranges.

Common crawlers, the set that includes Googlebot, resolve to googlebot.com (including geo.googlebot.com). They respect robots.txt on automatic crawls. Two other buckets are easy to mix in:

• Special-case crawlers, such as AdsBot, which may not respect robots.txt
• User-triggered fetchers, which ignore robots.txt because a person asked for the fetch

Filter the user agent first so the file is small enough to work with. Verify a sample of IPs, and verify all of them when the pattern looks wrong: a spike, a strange parameter, or a block you did not intend.

If the site sits behind a CDN, the origin log may show the proxy address. Read the client IP from the log that actually saw the crawler, and only trust forwarding headers from your own proxy.

Status codes that waste the next crawl

Sort verified Googlebot requests by status. The interesting rows are not the quiet 200s on the homepage.

Google’s HTTP status guide for crawlers covers the 20 status codes it sees most often. A 2xx response may be considered for later processing. A 4xx is not indexed, and a URL that used to be indexed gets dropped. 5xx and 429 make crawlers slow down for a while. Content from those responses is ignored. Already indexed URLs can stay, then eventually drop. When the host returns 2xx again, the crawl rate comes back up gradually.

Crawl budget guidance, updated 22 July 2026, is blunter about waste:

• A 404 or 410 on a page that is gone is a strong signal not to crawl that URL again. A robots.txt block keeps the URL in the queue longer, and it comes back when you lift the block.
• noindex still costs a fetch. Google requests the URL, sees the tag, and drops it.
• Soft 404s keep getting crawled. Check the Page indexing report.
• Long redirect chains have a negative effect on crawling.

Google defines a site, for this limit, as one hostname. www.example.com and shop.example.com do not share a budget. The capacity limit is shared across Google’s crawlers, so a heavy fetcher can crowd Googlebot on the same host.

Crawl budget only matters past a certain size

Most brochure sites do not have a crawl-budget problem. Google says you can skip the guide if pages are crawled the day they are published. The audience is large. Think on the order of 1 million unique pages that change about weekly, or 10,000 unique pages that change daily. Or a site with a large share of URLs stuck on “Discovered, currently not indexed.” Those counts are a rough estimate, not a threshold you fail by one page.

If you are in that range, the controllable piece is inventory. Duplicate URLs, faceted filters, and internal search are what soak the fetches. Blocking them in robots.txt does not hand the leftover budget to other pages unless Google is already at the host’s capacity limit.

Pair the log window with a crawl of your own links. A URL that Googlebot hits and your crawler never finds is usually an old sitemap entry, an external link, or a parameter trap. A URL you care about that never appears in logs is a discovery problem, not a “budget” story. How to turn that list into a backlog is covered in how to prioritize an SEO backlog.