Lumar supports the uploading of a wide range of file types to crawl sources. This article runs through the valid file types for each source, and where to find them.
Base URL
All source uploads allow you to set a 'Base URL'. This is used only when we encounter a relative URL in your upload - for instance, if a URL in your upload starts with "/", we will prepend the base URL (http://www.example.com). If you do not set a base domain, we will use the project's primary domain instead.
Have a File Which is Not Supported?
Lumar can support any UTF-8 CSV which is not mentioned on this page. When you upload it to your project, Lumar will ask you which columns contain the required datapoints. Note that Lumar can only accept data which is aggregated to a URL, i.e. a file with each row containing a URL and the number of backlinks that each URL is perfect, but a raw backlink export containing details of every single link is not supported.
Analytics
Google Analytics
To download the data from Google Analytics, go to the relevant report and click the share icon in the top right corner. You can then click download file and choose download CSV. You will need to format the data in the correct way. We've created template files that you can use to get the data in the right format.
- Analytics - Data from standard web traffic (visits, pageviews, bouce rates, etc.)
- AI Referral Sources - Data from sessions referred by AI platforms (e.g. ChatGPT)
Adwords URLs
Adwords destination URLs can be imported into Lumar's Analytics metrics to help you ensure that you are sending users to relevant pages, and that they're not broken or orphaned. In Adwords, load the "Reports" screen Choose "Predefined Reports" > "Basic" > "Final URL" Download this report as a csv, and upload this to Lumar's Analytics tab in your Project's settings.
Backlinks
Google Search Console
Go to Google Search Console and find the ‘All Linked Pages’ report. Select “Search Traffic” > “Links To Your Site” then choose the “More >” link under “Your most linked content” Download the CSV by clicking “Download this table”
Majestic
Find your website in Majestic, choose the “Pages” report, and export this data to a CSV using the "Export" button.
Ahrefs
In Ahrefs, choose the Pages > Best by links report, and export this data to a CSV using the 'Export' button. Download the CSV "For Open Office, Libre & other (UTF-8)" and upload this to Lumar.
Open Site Explorer
In Open Site Explorer, choose the Top Pages report and export the data to CSV using the "Request CSV" link.
Default Format
If you do not have access to any of the above datasources, you can reformat your data to our default format.
Log File Summaries
Lumar supports a range of exports from your favourite log file analyser. We are unable to process raw log files, these must be summaries of the number of requests on a URL level.
Screaming Frog Log Analyser
Note that the "Screaming Frog Web Crawler" does not process log files. We support exports from the "Screaming Frog Log File Analyser". In Screaming Frog Log Analyser, open the URLs tab, and export this.
Splunk
Run the following queries to export summary statistics, you will normally need to edit these to match your setup: "host" should be the domain you're exporting data for, "useragent" is the user agent field, and "uri" is the URL field. Please contact our Support for assistance doing this.
Logz.io
In logz.io, open Kibana visualise and create a query using the Metric aggregation "count", and buckets: Split Rows > Aggregation: Terms, Field: request, Order By: "metric: Count", Order: Descending, Size: 200
Use the following queries, and export using the "Export Raw" link. Upload this file to Lumar. By default, logz.io will only export the top 200 pages using this method. You should ask your logz.io account manager to increase this limit.
Default Format
If you do not have access to any of the above datasources, you can reformat your data to our default format.