Skip to main content

Search

Welcome to the Lumar Knowledge Base

Crawling Shopify websites with Lumar

Karolina Machalska
  • Updated

Your Lumar crawls may occasionally not return the expected results, and a 429 'Too Many Requests' error message may be one of the reasons. If you are crawling Shopify websites, you may encounter this error message as Shopify employs rate-limiting measures.

Our bots stop crawling when the server returns a 429 response, as no data can be collected from URLs with this HTTP status. To prevent these errors and allow successful crawling of your Shopify website, you’ll need to authenticate the crawler on your server using the Custom Request Header feature available in Advanced Settings. You will have to generate an HTTP message signature to set up the HTTP headers of each request, allowing Shopify to confirm that an authorised crawler is sending the requests.

Important Notes

  • Please note that the Shopify signatures have a maximum expiry of 3 months, and will not work after the expiration date. You will have to update the signature values in Advanced Settings accordingly to ensure the accurate crawl results.
  • Shopify websites are hosted behind the Cloudflare CDN, and some sites may have Cloudflare Bot Management configured with strict protection settings. As a result, Lumar may still encounter 4xx responses even when the Custom Request Header has been set up correctly. To avoid these issues, you’ll also need to add the IP address used for crawling to Cloudflare’s allowlist.
  • If you hit a 4xx error and have since added your Shopify signature to the project settings, then you will have to wait 10-25 minutes before trying again. Otherwise, you may still encounter a 4xx response.
  • Shopify have a troubleshooting section - please check through those steps, or contact their support.

Follow the steps below to configure the Custom Request Header setting to successfully crawl Shopify websites.

Create HTTP Message Signatures in Shopify

You can create and manage signatures directly in your Shopify admin.

Steps:

  1. In Shopify, go to Online Store > Preferences.
  2. In the Crawler access section, click Create signature.
  3. In the Name field, enter a descriptive name for your signature.
  4. In the Domain field, select the domain that you want to use this signature for.
  5. In the Valid for section, select an expiration period for your signature.
  6. Click Create.
  7. Please note you may copy your newly created Signature-Input and Signature values - click Copy next to each value.

Your Shopify Signature

Each signature consists of three HTTP headers, each of which is a pair: 'Header name' and a corresponding 'Header value.' This will be all required to set up a custom request header in Advanced Settings.

Add Your Signature to Your Lumar Crawl

Go to your project > Settings > Advanced Settings > Spider Settings > Custom Request Header.

Click Create Custom Request Header three times, so that three pairs of boxes appear. 

Paste the names of the HTTP headers from your Shopify signature into the boxes on the left. Please note that you have use the exact same wording.

Now, you need to enter the corresponding values for the 'Signature', 'Signature-Input', and 'Signature-Agent' headers. Please see the example below, where the value for 'Signature-Agent' was entered.

You can now go ahead and start your crawl. Lumar will send the custom HTTP headers with every request, ensuring that your Shopify website can be crawled successfully.

x