403 Errors and SEO: When Your Firewall Blocks Googlebot

Most articles about 403 errors are written for the person who cannot get into their own site. This one is about the visitor you never see complaining: Googlebot. When a firewall answers Google with “403 Forbidden”, nothing looks broken in the browser, yet pages start dropping out of the index and rankings erode over weeks. Here is how that happens, how to spot it in Search Console, and how to let search engines through without opening the door to everyone else.

The video above explains the mechanics of a 403: a firewall, a security plugin or Cloudflare looks at a request and refuses it. The example in it is Jetpack being blocked by a country rule. Swap “Jetpack” for “Googlebot” and you have the SEO version of the same problem.

What Google does with a 403

Google treats a 403 as a sign the page is not available. A single crawl that returns 403 is forgiven, but a page that keeps answering 403 is dropped from the index, and a site that returns 403 at scale gets crawled less often. Because the response is a clean HTTP status rather than an outage, there is no alarm. Traffic drifts down and the usual suspects (a core update, a competitor, a content problem) get blamed instead.

The firewalls that cause this are rarely set up to block Google on purpose. They block a country Google happens to crawl from, challenge visitors who do not run JavaScript (crawlers do not), or rate-limit anything that requests pages quickly. Googlebot ticks every one of those boxes.

Where it shows up in Search Console

• Pages report: a “Blocked due to access forbidden (403)” reason under Not indexed. Click it to see which URLs and when they were last crawled.
• Crawl stats (Settings, Crawl stats): the “By response” breakdown. Any meaningful share of 4xx other than 404 is worth investigating, and a sudden rise in 403 lines up neatly with the day a firewall rule was added.
• URL Inspection: run a live test on an affected page. If “Page fetch” says failed with a 403, the block is confirmed for Google’s own crawler, not just for your test tools.

A related symptom: third-party SEO tools stop working. Rank trackers, site crawlers and PageSpeed tests all fetch pages from data centres, and they get the same 403 Google does. A crawler report full of 403s is a strong hint to check the firewall before blaming the tool.

Confirm it is really Googlebot

Do not allow-list on user agent alone; anyone can send “Googlebot” in a header. Take an IP from the blocked requests and run a reverse DNS lookup; genuine Google crawlers resolve to googlebot.com or google.com, and the forward lookup of that hostname returns the same IP. Google also publishes the IP ranges its crawlers use, which Cloudflare and most firewalls already recognise as verified bots.

Fixing it without weakening the firewall

1. Cloudflare: in Security, open Events and filter by the Googlebot user agent to find the rule responsible. Then either add “known bots” as an exception to that rule (the expression builder has a Verified Bot field) or add a Skip rule above it for verified bots. Check that Bot Fight Mode or Super Bot Fight Mode is not set to challenge verified bots.
2. Security plugins: look for the blocked request in the plugin’s log and allow it. Wordfence’s rate limiting has a separate setting for “If a crawler’s page views exceed”, which is the one that catches aggressive crawling; set it to throttle rather than block, or raise the threshold.
3. Hosting firewall: ask your host to confirm Google’s ranges are allowed and to show you what the server returned to Googlebot in the access log.
4. Geo-blocking: if you block or challenge whole countries, always exempt verified bots. Googlebot crawls mostly from the United States regardless of where your audience is, so a “US only” or “Australia only” rule is one of the fastest ways to de-index yourself.

After the fix

Use URL Inspection to live-test a few affected pages and request indexing on the important ones. Then watch Crawl stats for a week: the 403 share should fall to zero and the crawl rate should recover. Recovery of rankings follows recrawl, so the longer the block was in place the longer it takes. Put a note in your SEO log with the date the rule was added and the date it was fixed; it will explain the dip the next time someone asks.