Shopify Robots.txt: What It Does and How to Configure

If you've spent time building your Shopify store — writing product descriptions, publishing blog posts, optimising your collections — the last thing you want is Google ignoring half of it. Yet that's exactly what can happen when your robots.txt file is misconfigured. This small, often-overlooked text file tells search engine crawlers which pages to index and which to skip, and getting it wrong can quietly tank your SEO without a single error message to warn you.

The problem is that most Shopify merchants either don't know the file exists or assume Shopify handles it automatically. Shopify does generate a default robots.txt, and for many stores it works reasonably well — but it was built for a generic use case, not yours. If you run a store with a large product catalogue, a content-heavy blog, or multiple international markets, the default settings may be leaving valuable pages unindexed or wasting crawl budget on URLs that add no SEO value.

In this guide, you'll learn exactly what the Shopify robots.txt file does, why it matters for your rankings, and how to customise it safely. Whether you sell physical goods, digital products, or subscriptions, this is practical information you can act on today.

What Is a Robots.txt File and Why Does It Exist?

A robots.txt file is a plain text document hosted at the root of your domain — in Shopify's case, at yourstorename.myshopify.com/robots.txt or your custom domain equivalent. It follows a protocol called the Robots Exclusion Standard, which search engine bots like Googlebot read before they begin crawling your site. Think of it as a set of instructions posted at your store's front door, telling visitors which rooms they're welcome to enter and which are off-limits.

The file matters because search engines have a finite resource called crawl budget — the number of pages Googlebot will visit on your site within a given time window. For small stores with 50 products, this rarely causes problems. But if you're running a store with 5,000 SKUs across multiple collections, faceted navigation, and hundreds of tag-filtered pages, crawl budget becomes a genuine constraint. Without proper robots.txt configuration, Google may spend its entire budget on duplicate or low-value URLs, never reaching your most important product pages.

Beyond crawl budget, robots.txt also protects sensitive or internal pages from being indexed at all. Admin URLs, checkout pages, internal search results, and customer account pages have no business appearing in Google search results. Shopify's default file already blocks most of these, but understanding the logic helps you make smarter customisations as your store grows.

How Shopify Generates Its Default Robots.txt

Unlike most platforms where robots.txt is a static file you upload via FTP, Shopify generates its robots.txt dynamically using a Liquid template. This is actually a significant advantage — it means you can customise the file using the same templating language you use to edit your theme, and changes take effect immediately without touching server files or contacting support.

Shopify's default robots.txt blocks several categories of URL automatically: checkout pages (/checkout), customer account pages (/account), internal search results (/search), cart pages (/cart), and various admin-related paths. It also includes a sitemap directive pointing to your XML sitemap, which helps Google discover all your indexable pages efficiently. For a brand-new store, this default setup is sensible and conservative.

However, the default file was designed for a median Shopify store, not an enterprise-scale catalogue or a store with complex URL structures. If you've installed third-party apps that generate new URL patterns, run international subfolders, or created tag-filtered collection pages that produce thousands of near-duplicate URLs, you'll quickly find the default robots.txt inadequate. Customising it isn't optional at that point — it's a necessary part of your SEO strategy.

Accessing and Editing Your Shopify Robots.txt

To edit your robots.txt in Shopify, navigate to Online Store → Themes → Edit Code, then look for a file called robots.txt.liquid in your theme's root directory. If the file doesn't exist, you'll need to create it by clicking "Add a new template" and selecting robots.txt from the dropdown. Shopify introduced native robots.txt editing in 2021, so this functionality is available on all current themes without requiring any app or workaround.

Once the file exists, you'll see the default Liquid template that outputs Shopify's standard rules. The template uses a {% for group in robots.default_groups %} loop to render the default directives automatically, which means you can extend the file without rewriting the defaults from scratch. This is important — always build on top of the default output rather than replacing it entirely, because you risk accidentally unblocking sensitive pages like /admin or /payments.

A typical customisation looks like adding a Disallow directive below the default group loop. For example, if your store uses a third-party review app that creates URLs following the pattern /pages/reviews-archive, and those pages are thin on content, you'd add Disallow: /pages/reviews-archive to prevent Google from crawling them. Save the file, verify by visiting /robots.txt on your live domain, and you'll see the change reflected immediately.

Common Robots.txt Mistakes Shopify Merchants Make

The most damaging mistake is blocking pages you actually want indexed. This happens surprisingly often when merchants copy robots.txt templates from generic SEO tutorials without checking which paths apply to Shopify's URL structure. A common example is blocking /collections/, which would prevent Google from crawling every collection page on your store — a catastrophic error for any product-based business. Always audit your live robots.txt at your domain /robots.txt before assuming everything is fine.

The second common mistake is ignoring collection tag pages entirely. When customers filter a collection by tag — for example /collections/mens-shoes/running — Shopify generates a unique URL for each combination. A store with 200 products and 30 tags could generate thousands of these URLs, most of which are near-duplicate versions of parent collection pages. Without disallowing these patterns in robots.txt, you risk crawl budget dilution and potential duplicate content signals. Adding Disallow: /collections/*/ (with appropriate syntax) can address this, though test carefully before deploying.

A third error is treating robots.txt as a security tool. Disallowing a URL does not make it private or inaccessible — it simply asks polite crawlers not to index it. Anyone who knows the URL can still visit it, and some crawlers ignore robots.txt altogether. For genuinely sensitive content like wholesale price lists or draft product pages, use Shopify's password protection features or set pages to noindex via meta tags rather than relying solely on robots.txt.

Using Robots.txt Alongside Your Sitemap and Canonical Tags

Robots.txt, XML sitemaps, and canonical tags work as a system — not as independent SEO tools. Your robots.txt tells crawlers what to visit, your sitemap tells them what you want indexed, and canonical tags tell them which version of a page is the authoritative one. If these three signals contradict each other, Google will have to make its own judgement calls, which rarely favours your intended outcome.

A practical example: if you block /collections/summer-sale in robots.txt but include it in your sitemap, Google will encounter a conflict. The most likely outcome is that Google respects the robots.txt disallow and ignores the sitemap entry — meaning a page you wanted indexed won't be. Auditing your sitemap against your robots.txt rules should be a quarterly task for any store with more than 200 pages, particularly after installing new apps or adding market-specific URLs.

Canonical tags add another layer of nuance. If you allow Googlebot to crawl tag-filtered collection pages but add a canonical tag pointing back to the parent collection, you're using crawl budget on pages Google will ultimately consolidate anyway. In that scenario, disallowing the filtered pages in robots.txt is the more efficient approach. For merchants who publish regular blog content — and if you're not, our BlogFlow app makes that significantly easier — ensuring your /blogs/ path is never accidentally blocked is particularly critical.

Monitoring Your Robots.txt After Changes

Making changes to robots.txt without monitoring the impact is like adjusting your store's navigation without checking whether customers can still find checkout. After any edit, use Google Search Console's URL Inspection tool to test specific URLs and confirm they're accessible to Googlebot. The Robots.txt Tester within Search Console (found under Settings) also lets you test individual URL patterns against your live file, catching conflicts before they affect rankings.

Set a reminder to review your robots.txt whenever you make significant store changes: installing a new app, launching a new market, restructuring your URL hierarchy, or migrating themes. Each of these events can introduce new URL patterns that your current file doesn't account for. A store that launched with 100 products and a simple structure may have a very different crawl requirement two years later when it's grown to 2,000 SKUs with international routing.

Finally, watch your Google Search Console coverage reports for spikes in "Excluded by robots.txt" pages. A sudden increase after a theme update is a reliable signal that something was accidentally blocked. Catching this within a week or two minimises ranking impact — catching it six months later means rebuilding lost ground in search results, which is a much harder and slower process.

Conclusion

Your Shopify robots.txt file is a small document with outsized influence on how search engines discover and rank your store. Getting it right means balancing two goals: ensuring Google can efficiently crawl your most valuable pages while keeping it away from duplicates, thin content, and internal URLs that dilute your crawl budget. Shopify's Liquid-based robots.txt system gives you genuine flexibility to achieve both without needing developer access.

The key takeaways are straightforward: audit your current robots.txt, extend it carefully without replacing the default output, align it with your sitemap, and monitor for unintended blocks after every major store change. Treat robots.txt as a living document that evolves alongside your store.

Try BlogFlow free at saltai.app — no credit card required.