Your Lumar crawls may occasionally not return the expected results, and a 429 'Too Many Requests' error message may be one of the reasons. If you are crawling Shopify websites, you may encounter this error message as Shopify employs rate-limiting measures.
Our bots stop crawling when the server returns a 429 response, as no data can be collected from URLs with this HTTP status. To prevent these errors and allow successful crawling of your Shopify website, you’ll need to authenticate the crawler on your server using the Custom Request Header feature available in Advanced Settings. You will have to generate an HTTP message signature to set up the HTTP headers of each request, allowing Shopify to confirm that an authorised crawler is sending the requests.
Important Notes
|
Follow the steps below to configure the Custom Request Header setting to successfully crawl Shopify websites.
Create HTTP Message Signatures in Shopify
You can create and manage signatures directly in your Shopify admin.
Steps:
- In Shopify, go to Online Store > Preferences.
- In the Crawler access section, click Create signature.
- In the Name field, enter a descriptive name for your signature.
- In the Domain field, select the domain that you want to use this signature for.
- In the Valid for section, select an expiration period for your signature.
- Click Create.
- Please note you may copy your newly created Signature-Input and Signature values - click Copy next to each value.
Your Shopify Signature
Each signature consists of three HTTP headers, each of which is a pair: 'Header name' and a corresponding 'Header value.' This will be all required to set up a custom request header in Advanced Settings.
Add Your Signature to Your Lumar Crawl
Go to your project > Settings > Advanced Settings > Spider Settings > Custom Request Header.
Click Create Custom Request Header three times, so that three pairs of boxes appear.
Paste the names of the HTTP headers from your Shopify signature into the boxes on the left. Please note that you have use the exact same wording.
Now, you need to enter the corresponding values for the 'Signature', 'Signature-Input', and 'Signature-Agent' headers. Please see the example below, where the value for 'Signature-Agent' was entered.
You can now go ahead and start your crawl. Lumar will send the custom HTTP headers with every request, ensuring that your Shopify website can be crawled successfully.