Brendan Tully0 Comments
October 7, 2026
WooCommerce stores publish URLs nobody meant to publish: add to cart links, wishlist links, sort orders, currency switches, internal search results. Crawlers follow all of them. On a modest hosting plan that crawling can overload a store, and in Search Console the same URLs turn up as noise competing with the product and category pages you actually want crawled. A considered robots.txt deals with both, as long as you know which rules help search and which can hurt it.
The video goes line by line through a robots.txt template built for a WooCommerce store that was struggling under crawler load, explaining what each rule is for and where to be careful.
Google does not crawl every URL on every visit. When Googlebot spends its requests on endless ?add-to-cart= and ?orderby= variations, fewer requests go to new products and updated categories. On a store with a few dozen products this barely matters. On a store with thousands of products and filter combinations, the wasted crawling adds up. If your theme links to these URLs, they also show up as problems in Search Console, which makes the reports harder to read.
?add-to-cart=
?orderby=
A condensed version of the rules covered in the video looks like this. Adjust the paths to suit your store.
User-agent: * Crawl-delay: 5 Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /my-account/ Disallow: /*?s= Disallow: /search/ Disallow: /*add-to-cart= Disallow: /*add_to_wishlist= Disallow: /*orderby= Disallow: /*currency= Disallow: /feed/ Disallow: /*/feed/ User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / Sitemap: https://yourdomain.com/sitemap_index.xml
There is no reason for a crawler to visit /wp-admin/, the login page or My Account, yet it happens when crawlers are redirected to a login page from hidden content. The admin-ajax.php exception matters: some front-end features, product variations in particular, rely on it, so it stays open. Remember that robots.txt stops crawling, not indexing. The account page should also carry a noindex tag.
/wp-admin/
WordPress search runs on both ?s= and /search/ URLs. Indexable search pages are a vector for negative SEO, because anyone can generate search URLs for whatever phrases they like and point crawlers at them. Blocking both patterns closes that door, and search result pages should not be crawled anyway.
?s=
/search/
On some themes these are plain links, and every crawl adds an item to a cart or wishlist. Aggressive crawlers commonly request three to five URLs per second, which is like five shoppers adding products every second. That was the problem on the store in the video: a targeted crawl was slipping past a Cloudflare rule that was not quite right. Blocking these parameters helps speed and tidies up Search Console at the same time.
Sort orders and currency switches rarely deserve crawling, and your store may use other filter parameters worth adding. The warning in the video is important for SEO: some stores have built their site structure around filtered URLs instead of proper category pages. Before blocking a filter parameter, check the Performance report in Search Console for URLs containing it. If filtered URLs are earning clicks, blocking them will cost you that traffic.
Feed URLs are blocked unless you use RSS for something. The template also covers the test copies that Plesk and cPanel smart update tools create, which can occasionally get crawled. Finally, always list your sitemap URL in robots.txt.
A crawl delay of five means one request every five seconds, instead of several per second, for crawlers that honour the line. Plenty of hosts set a similar value by default. Two caveats. Googlebot ignores Crawl-delay entirely and sets its own pace, so this protects your server from other crawlers rather than shaping Google’s behaviour. For crawlers that do obey it, a large catalogue may take too long to crawl at five seconds per URL, so on bigger stores lower it, even to one.
Ahrefs and Semrush crawl heavily. Blocking them reduces load, and it also stops those tools publishing data about your site that competitors can look up. If you extend the list with a published bad-bot list, check it does not include tools you rely on. Not every crawler respects robots.txt, so anything malicious that ignores it needs a firewall rule instead.
Comments are closed.