TL;DR: Shopify rate-limits unauthenticated crawlers, which causes 429 errors and incomplete crawls. Fix it by generating an HTTP message signature in Shopify admin (Online Store > Preferences > Crawler access) and adding the Signature, Signature-Input, and Signature-Agent values to your Lumar project's Custom Header Request settings. Signatures last up to 3 months — set a reminder to replace them before they expire.

Shopify sites use Cloudflare as their CDN, which means standard crawl requests can be blocked or throttled before Lumar can collect meaningful data. This article explains why that happens and how to configure Lumar to crawl Shopify sites successfully.

Why Shopify sites are harder to crawl

Shopify uses URL rate limiting to protect stores from suspicious or excessive automated traffic. Without proper authentication, Lumar's crawler will hit 429 Too Many Requests or 403 Forbidden responses — and stop collecting data.

Shopify introduced Web Bot Auth in August 2025 to solve exactly this problem. It lets merchants securely authorise crawlers, scripts, and testing tools using cryptographic HTTP message signatures. Once configured, Shopify recognises Lumar as an authorised source and lets the crawl proceed without interruption.

A few things worth knowing before you start:

• Signatures apply per domain. You can only create signatures for domains already connected to your Shopify store. If you're crawling multiple domains, you'll need a separate signature for each one.
• Signatures expire. The maximum validity period is 3 months. An expired signature will cause crawls to fail — see Keep your signature fresh below.
• Search engines and LLMs don't need one. Web Bot Auth is for your own authorised tools. Shopify's public indexing by search engines and large language models works independently.

No access to the signature? If your client is unable or unwilling to generate one, reduce your crawl speed as much as possible in your project's Crawler Settings. This won't guarantee success, but it reduces the chance of triggering rate limits.

Step 1: Create your HTTP message signature in Shopify

1. In Shopify admin, go to Online Store > Preferences
2. Scroll to the Crawler access section and click Create signature
3. Enter a descriptive Name — something like "Lumar SEO Audit" is clearer than "Crawler 1" if you're managing multiple signatures
4. In the Domain field, select the domain you want this signature to apply to
5. In the Valid for section, choose an expiration period (maximum 3 months)
6. Click Create

Shopify generates three HTTP headers. Click Copy next to each value to grab them — you'll need Signature-Input and Signature for the next step. The Signature-Agent value is always "https://shopify.com" (including the quotes) and is already pre-filled in Lumar.

You can view and manage all active signatures at any time under Online Store > Preferences > Crawler access.

Step 2: Add your signature headers to Lumar

1. Open your project settings and navigate to Advanced Settings
2. Scroll to the Custom Request header section
3. Create a custom request header for each of the three Shopify headers:
4. Signature  + Signature value from Shopify
5. Signature-Input  + Signature-Input value from Shopify
6. Signature-Agent + "https://shopify.com" (including quotation marks)
7. Save your settings

Lumar will now send all three headers with every request to your Shopify site.

Troubleshooting

If crawls are still stopping early after completing the steps above, work through the checks below.

Cloudflare is blocking the crawler

Shopify sites run on Cloudflare, and some have aggressive Bot Protection Management enabled. Even with a correctly configured Shopify signature, Cloudflare may still return 429 or 403 responses.

To fix this, add the IP address you're crawling from to the allowlist in Cloudflare. This tells Cloudflare to let your crawler through regardless of its bot detection rules.

Important: Crawling with a misconfigured or expired signature can trigger a temporary IP block from Cloudflare. This typically lifts after 5–10 minutes. Don't restart the crawl immediately — wait before retrying (see below).

You restarted the crawl too quickly after a 429

If Lumar hit a 429 error and you've since updated your settings to resolve it, wait 15–30 minutes before trying again. Shopify's servers may continue to flag your IP for unusual activity if you retry immediately.

Try enabling JavaScript rendering

Lumar uses a Chrome-based crawler for all crawls, with JavaScript rendering either on or off. When rendering is off, behaviour is similar to a lightweight HTML crawl — but Shopify sites protected by Cloudflare will block requests unless rendering is turned on. If your crawl is hitting 429 errors on the very first URL even with a valid signature, try switching rendering on in your project's Crawler Settings.

Check the Shopify troubleshooting guide

Shopify maintains a troubleshooting section in their own documentation, which lists common signature issues with step-by-step fixes. If none of the above resolves the problem, that's a good next port of call — or contact Shopify support directly.

Keep your signature fresh

Shopify signatures cannot be renewed — once one expires, you create a new one. Because the maximum validity is 3 months, it's worth setting a reminder well before the expiry date. A mid-crawl expiry means Lumar will be sending invalid headers, which increases the risk of Cloudflare blocks and leaves you with incomplete crawl data.

When a signature expires:

1. In Shopify admin, go to Online Store > Preferences > Crawler access and create a new signature
2. Update the Signature and Signature-Input values in your Lumar project settings
3. Save and relaunch the crawl