Setting Up an Accessibility Test Suite

Step 1 – Connecting your site

From the dashboard, click on the purple ‘New test suite’ button at the top right hand corner of the screen. 

This button is also available on the ‘Test Suites’ page.

In Step 1, give your test suite a name, so you can understand the nature of the crawl once it’s run, and add in the domain/subdomain details. 

You can also add any necessary authentication in this step by expanding the ‘Authentication and Whitelisting’ section.

You can choose from authentication using Username and Password, IP Whitelisting or Custom DNS, or a combination of these methods:

• Authentication: If your site requires a username and password for the crawler to gain access, you can enter that information here.
• Whitelisting: Lumar uses the IP address 52.5.118.182 or 52.86.188.211 by default. These are the IP addresses a network administrator would need so that Lumar can bypass any firewalls or blocks you may have in place.
• Custom DNS Settings: These might be needed if you are testing a staging environment. The only caveat is that you cannot use this setting with JS rendering turned on.

Once you’ve filled in the relevant details, click ‘Save’ to move onto the next step. 

Step 2 – Choosing your crawl settings

In Step 2 you can choose your crawl settings. In the top section, choose the type of crawl. 

• Web crawl, where the crawler will start at the domain/subdomain you specified in Step 1, and follow links to crawl your site. For example, you could choose to start your crawl at a specific subdomain to test an important part of your site.
• List crawl, where you can upload a test or .CSV file with a list of specific URLs to crawl. You can use this option to upload a list of URLs built on specific templates to check for errors.

You can choose between these options, or select both. 

JavaScript rendering is enabled by default to make sure the crawler does not miss valuable code and content. For accessibility crawls, this setting cannot be disabled.

To set the crawl speed, use the slider to choose the appropriate crawl speed for your site. When choosing the crawl speed, you should consult with your web ops team to determine the best speed, based on your server capability. 

In the URL Limit field, choose the number of URLs that will be completed in the crawl. As Protect is focused on a specific purpose, you can choose a limit of up to 10,000 URLs. In Analyze (where the crawl scope is much broader) the limits are much higher. 

Finally, you can also add any custom extraction rules that you want to include.

At the bottom of this section, you can expand the ‘Advanced Settings’ section to access additional settings:

• Start URLs: To start the crawl from somewhere other than the primary domain. 
• Include and Exclude URLs: To limit your crawl to URLs that only include your specified paths, use regular expressions to match specific paths if required. Alternatively, you can add URL paths to exclude them from the crawl. When using this feature, note that exclude rules will take precedence over include rules. If you use Include or Exclude URLs, you must also include those URLs as Start URLs.
• Robots.txt Overwrite: To replace the live robots.txt for the primary domain with an alternative. 
• JavaScript Rendering Settings: Here you can add additional settings, including blocking ad or analytics scripts, blocking scripts URLs, injecting custom JS snippet, or including external JS resources.
• Test Settings: These are Lumar’s recommended levels for settings such as maximum title width, thin or empty page thresholds, or duplicate precision values. If you need to amend these, you can do so here.

Step 3 – Choosing your tests

In Step 3 you can choose up to 100 tests to include in the test suite. When Protect crawls, it will run these tests to dictate the overall success of the build. You can choose from over 90 tests related to accessibility.

At the top of step 3 you can search for the particular tests you want to include, and check the box next to the name to add it to the test suite. For accessibility, you can also check the relevant accessibility standard (e.g. Level A or Level AA) to check for all issues related to that level. 

As you check the relevant options, you’ll see them populate on the right hand side of the screen.

Once you’ve added tests to the test suite, you can then click on each one on the right hand side to choose the test settings:

• Test severity. Choose whether the threshold breach will result in a fail (which stops the build) or a warning (where the build continues). 
• Use smart thresholds. Automatically adjust thresholds based on the worst performance from the previous five crawls. Find out more about smart thresholds in Lumar Protect below. 
• Failure threshold. Choose the number of URLs that will trigger the warning or failure in Lumar Protect.
• Jira ticket creation. You can also integrate Protect with Jira, to automatically create a ticket when a Fail or Warning is triggered. Once integrated, click the slider to enable.

Step 4 – Set notifications

In Step 4 you can choose the relevant notifications for your test suites. For email, your own address will appear with notifications set for passes, passes with warnings and fails by default. If, for example, you don’t want to receive an email notification for passes, simply uncheck the box. You can also add in any other email addresses using the ‘New email’ button, and again select which notifications they will receive an email for.

Protect can also deliver notifications via Slack. This can be easily set up using a webhook, to deliver notifications to one or more Slack channels. You can also integrate Protect with Jira to automatically create issues in your backlog for fails and warnings generated when test suites run. 

Step 5 – Running your test suite

In the final step you can choose how you run your test suite:

• Run via CI/CD. With this method, when development push new elements from staging to live, Protect will run the test suites. If everything passes, publishing to the live site will complete. However, if any tests fail, it will be stopped and sent back to dev so they can fix the issues that didn’t pass. If you choose to run via CI/CD, you’ll find instructions by clicking ‘Show instructions’ under Connect to CI/CD. Instructions for the CI/CD admin can be found here.
• Run manually. If you don’t want to connect Protect to your CI/CD pipeline, you can run the test suites manually. Simply go to that option and click ‘Run test suite’. 

You can also schedule the test suite to run at a later date (see below). If you do want to set a schedule, click on ‘Go to test suites’ in the bottom right corner of the screen.

Step 6 – Viewing crawl results

The results can be seen on the Protect home page once the crawl has completed. The Builds list shows top level information about each build (crawl) and the number of tests that passed or failed.

If integrated to your CI/CD pipeline, when all tests pass or give warnings then the build will pass. If one test fails, the build will fail.

Clicking into the test suite will take you to the build detail page which shows each test that passed, failed, or gave a warning. You can see more detail about the URLs in each test by opening the Lumar report in Analyze.

Bonus Step – Scheduling a test suite run

After you’ve defined your test suite using the steps above, you can schedule your test suite to run at a particular time, or on a regular basis. Click on the ‘Scheduler’ icon (a clock) in the left hand navigation bar. You can then click ‘Add schedule’ in the top right corner.

Simply give the schedule a title, select the date and time, set any repetition you require, and choose the test suites to include in the run. 

Once you fill in the details and click save, you’ll see the schedule in the Scheduler page.