Some websites will reject requests unless custom HTTP headers to be used. If required, you can add the custom headers in the Advanced Settings in Step 4 of the Crawl Setup process.

Once you've opened the Advanced Settings, scroll down to the Spider Settings section and you'll find the option for Custom Request Header. Click on the Create Custom Request Header button to add the details. If you need to add more, clicking the button again will add another row.