Skip to main content

GUIDE № 07 · CRAWLABILITY

How to Check If Your Shopify Store Blocks AI Crawlers

5 min read · Guide 7 of 17

Your Shopify store might be completely invisible to ChatGPT right now. Not because of your product data or your SEO — but because of a text file most merchants don't even know exists.

The file is robots.txt. And if it blocks AI crawlers, none of them can see any of your products. Not one.

What robots.txt does

robots.txt is a file that sits at the root of your website (yourstore.com/robots.txt). It tells web crawlers — including AI agents — what they can and can't access on your site.

When an AI agent like GPTBot (ChatGPT's crawler) or ClaudeBot visits your store, it checks robots.txt first. If the file says "Disallow: /", the agent stops right there. It won't crawl a single page.

Shopify's defaults are fine

Shopify's default robots.txt blocks a few internal paths:

  • /cart — the shopping cart page
  • /checkout — the checkout flow
  • /search — the search results page
  • /policies/ — policy pages (see our policies guide for more on this)

These are sensible defaults. Cart and checkout pages don't need to be crawled. The important thing is that product pages, collection pages, and your homepage are not blocked by default.

The problem: custom blocks on AI crawlers

Some stores — or apps installed on those stores — add custom rules that block specific AI crawlers. These match the AI user-agents StoreAudit checks for in its robots.txt sub-score:

  • GPTBot — OpenAI's training crawler
  • ChatGPT-User — OpenAI's live-browsing agent (ChatGPT on behalf of users)
  • Google-Extended — Google's AI training and AI Overviews crawler
  • ClaudeBot — Anthropic's training crawler
  • PerplexityBot — Perplexity's crawler
  • CCBot — Common Crawl, an open dataset many AI assistants pull from
  • anthropic-ai — legacy Anthropic user-agent still seen in the wild
  • Amazonbot — Amazon's general-purpose crawler used for AI features
  • OAI-SearchBot — OpenAI's search-product crawler

A few of these deserve disambiguation:

  • GPTBot is OpenAI's training crawler; ChatGPT-User is the live-browsing agent. Blocking GPTBot does not necessarily block ChatGPT-User — they're separate user-agents.
  • ClaudeBot is Anthropic's training crawler; claude-web is the live-browsing agent. Same story: independent user-agents, separate rules.
  • PerplexityBot is Perplexity's crawler. No formal training/browsing split; Perplexity uses one main agent.

If your robots.txt includes rules like this: 

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Then those AI agents can't access anything on your store. Every product page, every collection, every piece of content — all invisible to them.

These rules are usually added via a robots.txt.liquid template in your Shopify theme. A common mistake is writing the template with only the custom rules — that replaces Shopify's defaults entirely (kills Disallow: /admin, Disallow: /cart, the sitemap line, and so on). The canonical pattern keeps the default-groups loop alongside any custom rules:

Liquid template (robots.txt.liquid)

{% comment %}
  Customize the rules above with your real intent — this template ships
  with GPTBot allowed because that's the most common merchant goal.
  Custom rules go ABOVE the default-groups loop so they take precedence.
{% endcomment %}
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

{% comment %}
  Shopify default robots.txt rules — emit these unchanged.
  See: https://shopify.dev/docs/storefronts/themes/architecture/templates/robots-txt-liquid
{% endcomment %}
{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {%- endif %}
{% endfor %}

Rendered output

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /policies/

Sitemap: https://yourstore.com/sitemap.xml

The Liquid file is where you make changes. The rendered output is what crawlers actually see at yourstore.com/robots.txt — note that the defaults survive alongside the custom blocks. If your template only contains custom rules with no default_groups loop, you've quietly disabled Shopify's protections on cart, checkout, search, and admin.

How to check right now

  1. Open your browser and go to yourstore.com/robots.txt
  2. Look for lines that mention the AI user agents listed above
  3. If you see Disallow: / under any of those user agents, that AI agent is blocked from your entire store

If you only see Shopify's default rules (blocking /cart, /checkout, /search, /policies/), you're fine. The problem is when there are additional rules targeting AI-specific crawlers.

How to fix it

If you find AI crawler blocks in your robots.txt, here's how to remove them:

  1. Go to Online Store > Themes in your Shopify admin
  2. Click Actions > Edit code (or the three-dot menu, then "Edit code")
  3. In the Templates section, look for robots.txt.liquid
  4. If it exists, open it and look for the Disallow rules under AI user agents
  5. Remove the rules that block AI crawlers, or change Disallow: / to Allow: /
  6. Save the file

If you don't see a robots.txt.liquid file, the blocks might be coming from a Shopify app. Check your installed apps — some SEO or security apps add robots.txt rules. Look at the app settings for any crawler blocking options.

After making changes

Visit yourstore.com/robots.txt again to verify the AI crawler blocks are gone. The changes take effect immediately — there's no caching delay for robots.txt on Shopify.

Keep in mind that even after removing the blocks, AI crawlers need time to re-crawl your store. The change is instant on your end, but it may take days or weeks for AI agents to fully index your products.

Should you block any AI crawlers?

That depends on your goals. Some store owners have concerns about AI training data or content reuse. That's a valid position.

But if your goal is to have AI agents recommend your products to customers — and especially if you want to participate in Shopify's Agentic Storefronts — blocking AI crawlers works against you. You can't be recommended by agents that can't see your store.

It's worth noting that Google-Extended specifically controls AI training, not search indexing. Blocking it means your content won't be used for Google's AI training, but it won't affect your Google Search rankings. The user-agent picture for ChatGPT and Claude is more nuanced: GPTBot is OpenAI's training crawler and ChatGPT-User is the live-browsing agent; ClaudeBot is Anthropic's training crawler and claude-web is the live-browsing agent. Blocking the training crawlers does not necessarily block the live-browsing agents — they're separate user-agents. PerplexityBot is Perplexity's main crawler and serves both training and live answers.

RELATED