November 2023 - including Segmentation for Accessibility, Weekdays and Bi-Monthly Crawls, Data Explorer for Accessibility and Redirect Chains Processed During Crawl

Pages with Repeated Paths Report Includes Non-Indexable Pages

Release Date: November 29 2023

The Pages with Repeated Paths report identifies spider-traps caused by malformed links which creates infinite URLs by repeatedly adding paths to URLs. Non-indexable pages were previously excluded as they would not cause issues with indexing, but can still cause issues with wasted crawl budget. The report has been updated to include non-indexable pages.

Segmentation for Accessibility Accounts

Release Date: November 23 2023

To help you break down your reporting by page templates for more targeted views, we’ve added Segmentation for accessibility accounts. This allows you to segment your site into key areas (e.g. product pages, blog, etc.) and then view crawl data for each of those segments. Aside from helping you discover and monitor trends in priority areas of your site, you can also export segment data into Google Data Studio to create customized dashboards.

Find out more about Lumar Segmentation.

Screenshot of a Lumar Analyze Accessibility Overview dashboard showing the segment drop-down.

ChatGPT and Google SGE Blocked Reports

Release Date: November 22 2023

Following the release of new reports to show pages that have been blocked from search engine AIs, we’ve also created two new reports to flag pages that are blocked in ChatGPT, and pages blocked from appearing in Google’s Search Generative Experience (SGE) or appearing with a restricted snippet.

URLs which are disallowed for the GPTbot or ChatGPT-User user-agent tokens in robots.txt will be flagged as blocked in ChatGPT.

Pages with a meta nosnippet, or a max-snippet will be flagged as blocked for Google SGE.

You can find the new reports in the Indexability > Non-Indexable category.

Weekdays and Bi-Monthly Crawl Frequencies

Release Date: November 22 2023

To add greater flexibility in your crawling strategies, we’ve added a couple of new frequencies to the crawl scheduling function.

  • Monday-Friday (daily excluding Saturday and Sunday)
  • Bimonthly (every 2 months)

You’ll find the new scheduling options in step 4 of the crawl setup.

Screenshot of step 4 of the Lumar crawl setup, showing the option to schedule crawls, including bi-monthly and daily from Monday to Friday

Selected Sitemaps Shown First

Release Date: November 21 2023

We now show the selected Sitemaps in Step 2 of the project settings before the unselected Sitemaps. This makes it much easier to see which Sitemaps have been selected for inclusion in your project when you have a large number, and easier to deselect them if required.

Screenshot of step 2 of the Lumar crawl setup process, focused in on the Sitemaps source and showing the selected sitemaps at the top of the list

Duplicate Counts Exclude External URLs

Release Date: November 21 2023

The URL level Duplicate Count metrics included external URLs resulting in a mismatch with the duplicate pages, titles and description reports. The external URLs have now been removed from the duplicate count metrics.

Data Explorer for Accessibility

Release Date: November 21 2023

We’ve now enabled Data Explorer for accessibility accounts, so you can quickly summarize crawl data and communicate key issues with stakeholders. This can be a great starting point to understand the breakdown of what was crawled—the different URL pathways and which sections of the site have the most issues. Find out more about Lumar’s Data Explorer.

For accessibility accounts, crawl data can be grouped into the following relevant dimensions:

  • Path (including Path 0 to Path 9)
  • Breadcrumbs (including Breadcrumb 1 to Breadcrumb 8)
  • Segment (once available in accessibility accounts)

The data will then be available under the following columns:

  • URL Count
  • Level A Issues SUM
  • Level AA Issues SUM
  • Level AAA Issues SUM

Fix: Large Pages Reported as Empty

Release Date: November 15 2023

We fixed a bug where very large pages failed to process and were reported as Empty Pages rather than Failed Pages. The maximum size of a page and all resources can now be up to 100MB. You may see a decrease in Empty Pages if you were affected by the issue.

Alerts on Custom Extractions

Release Date: November 9 2023

We’ve expanded the functionality of Monitor alerts to include custom extractions, so now you can get notifications for any of the bespoke metrics you’ve set up in Analyze. In the alert setup, simply select your custom extraction from the Report drop-down, and then set the remaining rules as required.

Please note that custom extractions will not be included when copying alerts to another project.

Redirect Chains Processed During Crawl

Release Date: November 1 2023

Prior to this release, Lumar would crawl each step of a redirect chain separately, and any incomplete redirect chains were crawled before a crawl could be finalized. This resulted in crawls exceeding the URL limits, and delays in finalizing a crawl when triggered manually.

All steps in a redirect chain are now processed immediately, up to a limit of 15. We previously charged a credit for every URL in a redirect chain. We now only charge credits for the initial redirecting URL and the final redirect target.

Crawls which previously exceeded the URL limits will include fewer URLs than before the change.