WooCommerce Robots.txt Rules for Crawl Budget, Indexing and Rankings

WooCommerce stores publish URLs nobody meant to publish: add to cart links, wishlist links, sort orders, currency switches, internal search results. Crawlers follow all of them. On a modest hosting plan that crawling can overload a store, and in Search Console the same URLs turn up as noise competing with the product and category pages you actually want crawled. A considered robots.txt deals with both, as long as you know which rules help search and which can hurt it.

The video goes line by line through a robots.txt template built for a WooCommerce store that was struggling under crawler load, explaining what each rule is for and where to be careful.

The search case: crawling the pages that matter

Google does not crawl every URL on every visit. When Googlebot spends its requests on endless ?add-to-cart= and ?orderby= variations, fewer requests go to new products and updated categories. On a store with a few dozen products this barely matters. On a store with thousands of products and filter combinations, the wasted crawling adds up. If your theme links to these URLs, they also show up as problems in Search Console, which makes the reports harder to read.

A condensed version of the rules covered in the video looks like this. Adjust the paths to suit your store.

User-agent: *
Crawl-delay: 5
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /my-account/
Disallow: /*?s=
Disallow: /search/
Disallow: /*add-to-cart=
Disallow: /*add_to_wishlist=
Disallow: /*orderby=
Disallow: /*currency=
Disallow: /feed/
Disallow: /*/feed/

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap_index.xml

What each group of rules does for SEO

Login and account pages

There is no reason for a crawler to visit /wp-admin/, the login page or My Account, yet it happens when crawlers are redirected to a login page from hidden content. The admin-ajax.php exception matters: some front-end features, product variations in particular, rely on it, so it stays open. Remember that robots.txt stops crawling, not indexing. The account page should also carry a noindex tag.

Internal search results

WordPress search runs on both ?s= and /search/ URLs. Indexable search pages are a vector for negative SEO, because anyone can generate search URLs for whatever phrases they like and point crawlers at them. Blocking both patterns closes that door, and search result pages should not be crawled anyway.

Add to cart and wishlist links

On some themes these are plain links, and every crawl adds an item to a cart or wishlist. Aggressive crawlers commonly request three to five URLs per second, which is like five shoppers adding products every second. That was the problem on the store in the video: a targeted crawl was slipping past a Cloudflare rule that was not quite right. Blocking these parameters helps speed and tidies up Search Console at the same time.

Sort and filter parameters: check before you block

Sort orders and currency switches rarely deserve crawling, and your store may use other filter parameters worth adding. The warning in the video is important for SEO: some stores have built their site structure around filtered URLs instead of proper category pages. Before blocking a filter parameter, check the Performance report in Search Console for URLs containing it. If filtered URLs are earning clicks, blocking them will cost you that traffic.

Feeds, test copies and the sitemap

Feed URLs are blocked unless you use RSS for something. The template also covers the test copies that Plesk and cPanel smart update tools create, which can occasionally get crawled. Finally, always list your sitemap URL in robots.txt.

Crawl-delay: useful, with limits

A crawl delay of five means one request every five seconds, instead of several per second, for crawlers that honour the line. Plenty of hosts set a similar value by default. Two caveats. Googlebot ignores Crawl-delay entirely and sets its own pace, so this protects your server from other crawlers rather than shaping Google’s behaviour. For crawlers that do obey it, a large catalogue may take too long to crawl at five seconds per URL, so on bigger stores lower it, even to one.

Blocking SEO tools: load and competitive privacy

Ahrefs and Semrush crawl heavily. Blocking them reduces load, and it also stops those tools publishing data about your site that competitors can look up. If you extend the list with a published bad-bot list, check it does not include tools you rely on. Not every crawler respects robots.txt, so anything malicious that ignores it needs a firewall rule instead.

Measuring the change

• robots.txt report (Search Console, Settings): confirm Google has fetched the new version without errors.
• URL Inspection on a key product and category page: crawling should still be allowed. If not, a rule is too broad.
• Pages report: blocked URLs move into “Blocked by robots.txt”, which is expected. URLs that were already indexed may linger as “Indexed, though blocked by robots.txt”; for those, a noindex tag needs to be seen before the block goes in.
• Crawl stats: watch total requests and the response breakdown over the following weeks, alongside server load from your host.