TL;DR: Shopify rate-limits unauthenticated crawlers, which causes 429 errors and incomplete crawls. Fix it by generating an HTTP message signature in Shopify admin (Online Store > Preferences > Crawler access) and adding the Signature, Signature-Input, and Signature-Agent values to your Web Bot Auth settings for that domain. These settings are account level, so one signature per domain covers every project that crawls that site. Lumar checks the signature at the start of every crawl and emails your account admins before it expires. Signatures last up to 3 months and cannot be renewed, so you replace them when they run out.

Shopify sites use Cloudflare as their CDN, which means standard crawl requests can be blocked or throttled before Lumar can collect meaningful data. This article explains why that happens and how to configure Lumar to crawl Shopify sites successfully.

Why Shopify sites are harder to crawl

Shopify uses URL rate limiting to protect stores from suspicious or excessive automated traffic. Without proper authentication, Lumar's crawler will hit 429 Too Many Requests or 403 Forbidden responses — and stop collecting data.

Shopify introduced Web Bot Auth in August 2025 to solve exactly this problem. It lets merchants securely authorise crawlers, scripts, and testing tools using cryptographic HTTP message signatures. Once configured, Shopify recognises Lumar as an authorised source and lets the crawl proceed without interruption.

A few things worth knowing before you start:

• Signatures apply per domain. You can only create signatures for domains already connected to your Shopify store. If you're crawling multiple domains, you'll need a separate signature for each one.
• You set a signature up once, not per project. A signature is stored at account level and applies to every project on your account that crawls that domain.
• Signatures expire. The maximum validity period is 3 months. See the Replacing an expired or invalid signature section below.
• Search engines and LLMs don't need one. Web Bot Auth is for your own authorised tools. Shopify's public indexing by search engines and large language models works independently.

No access to the signature? If your client is unable or unwilling to generate one, reduce your crawl speed as much as possible in your project's Crawler Settings. This won't guarantee success, but it reduces the chance of triggering rate limits.

Before you start. You need admin access to the Shopify store to generate the signature (Step 1), and account admin access in Lumar to add it (Step 2), because Web Bot Auth is an account-level setting.

Step 1: Create your HTTP message signature in Shopify

1. In Shopify admin, go to Online Store > Preferences
2. Scroll to the Crawler access section and click Create signature
3. Enter a descriptive Name — something like "Lumar SEO Audit" is clearer than "Crawler 1" if you're managing multiple signatures
4. In the Domain field, select the domain you want this signature to apply to
5. In the Valid for section, choose an expiration period (maximum 3 months)
6. Click Create

Shopify generates three HTTP headers. Click Copy next to each value to grab them — you'll need Signature-Input and Signature for the next step. The Signature-Agent value is always "https://shopify.com" (including the quotes).

You can view and manage all active signatures at any time under Online Store > Preferences > Crawler access.

Step 2: Add your signature to Lumar

You add a signature once per domain. Every project on your account that crawls that domain then uses it automatically, so you do not need to set it up in each project.

1. In Lumar, open the Account Management menu and select Web Bot Auth 
2. Start a new signature and select the Domain it applies to
3. Paste the Signature value from Shopify into the Signature field.
4. Paste the Signature-Input value from Shopify into the Signature-Input field.
5. Signature-Input  + Signature-Input value from Shopify
6. Paste the Signature-Agent value into the Signature-Agent field. For Shopify sites this is always "https://shopify.com" (including the quotes).
7. Save your settings

Lumar now sends the signature with every request to that domain, across all projects that crawl it.

Lumar stores your Signature and Signature-Input values securely and does not show them again after you save. If you need to change them, you replace the signature rather than edit it. See the Replacing an expired or invalid signature section below.

How Lumar checks your signature

Lumar validates your signature when a crawl is created. If it is valid, the crawl authenticates and runs normally. If it is not, Lumar does not start the crawl and shows an error naming the affected domain and the reason, so you can fix it before losing crawl data.

A signature stops being valid for one of these reasons:

• It has expired.
• Shopify has rotated its signing keys, so the stored signature no longer matches.
• The stored values are incomplete or incorrect.

The fix is the same in every case: create a new signature in Shopify and add it to Lumar.

Lumar warns you before a signature expires. Account admins get an email titled "Action Required: Web Bot Auth Signature Expiring Soon". It lists each affected domain and how long is left, and links to the page where you manage your signatures. Replace the signature before that date, because once it expires, scheduled crawls for that domain fail.

Replacing an expired or invalid signature

Shopify signatures cannot be renewed. When one expires or becomes invalid, you create a new one and add it to Lumar.

When a signature expires:

1. In Shopify admin, go to Online Store > Preferences > Crawler access and create a new signature
2. Update the Signature and Signature-Input values in your Lumar project settings
3. Save and relaunch the crawl

Because the maximum validity is 3 months, replace a signature as soon as you get the expiry email. Once a signature expires, scheduled crawls for that domain fail until you add a new one.

Troubleshooting

If crawls are still stopping early after completing the steps above, work through the checks below.

Cloudflare is blocking the crawler

Shopify sites run on Cloudflare, and some have aggressive Bot Protection Management enabled. Even with a correctly configured Shopify signature, Cloudflare may still return 429 or 403 responses.

To fix this, add the IP address you're crawling from to the allowlist in Cloudflare. This tells Cloudflare to let your crawler through regardless of its bot detection rules.

Important: Crawling with a misconfigured or expired signature can trigger a temporary IP block from Cloudflare. This typically lifts after 5–10 minutes. Don't restart the crawl immediately — wait before retrying (see below).

Try enabling JavaScript rendering

Lumar uses a Chrome-based crawler for all crawls, with JavaScript rendering either on or off. With rendering off, a crawl behaves like a lightweight HTML crawl. Shopify sites protected by Cloudflare will block those requests unless rendering is on. If your crawl hits 429 errors on the very first URL even with a valid signature, switch rendering on in your project's Crawler Settings.

You restarted the crawl too quickly after a 429

If Lumar hit a 429 error and you've since updated your settings to resolve it, wait 15–30 minutes before trying again. Shopify's servers may continue to flag your IP for unusual activity if you retry immediately.

Try enabling JavaScript rendering

Lumar uses a Chrome-based crawler for all crawls, with JavaScript rendering either on or off. When rendering is off, behaviour is similar to a lightweight HTML crawl — but Shopify sites protected by Cloudflare will block requests unless rendering is turned on. If your crawl is hitting 429 errors on the very first URL even with a valid signature, try switching rendering on in your project's Crawler Settings.

Check the Shopify troubleshooting guide

Shopify maintains a troubleshooting section in their own documentation, which lists common signature issues with step-by-step fixes. If none of the above resolves the problem, that's a good next port of call — or contact Shopify support directly.