What are Smart Thresholds?

Once enabled, Smart Thresholds use the data from the previous five test runs to automatically adjust thresholds based on the worst performance from those tests. 

For example, when you start running a test suite in Protect you need to either run it manually at least 5 times, or set it up on a schedule that runs at least 5 times. Smart Thresholds will then kick in after the 5th run. If the threshold was set at 200 broken pages and you fix some of those broken pages, bringing the total down to 120 Broken Pages, Smart Thresholds will change the threshold to 121 Broken Pages without any manual intervention on your part. 

As your site continues to improve, we keep the threshold in line with it so you’re aware of any negative spikes on the site. If your Broken Pages remain over the set threshold, you will continue to be notified that this test Failed unless you manually set the threshold higher. This is to ensure that a critical alert doesn’t get missed.

Websites are obviously ever-changing, so Smart Thresholds can ensure your thresholds will be kept updated based on how your website is changing over time, without manual intervention.

When to use Smart Thresholds

Obviously you can use Smart Thresholds whenever you like. Like any automation though, it’s important to make sure it’s used efficiently. Automation is a great time saver for repetitive tasks that don’t require complex thinking, but when some errors in your site can lead to a significant loss of traffic or even revenue, your expertise is a vital factor in deciding on appropriate thresholds. For this reason, you may want to retain manual settings for the most critical tests for your site. 

Fortunately, our Smart Thresholds can be enabled on a per test basis, so you may choose to enable it for those tests that result in a ‘warning’, but keep ‘fails’ that block the build manually set. 

Of course you can also switch Smart Thresholds on and off at any time. This means you can turn them on to get a baseline for the thresholds, and then switch them off once that is set.

How to set up Smart Thresholds

In Step 3, where you add in the tests that you want to include in the test suite, you have the option to choose Smart Thresholds. 

To get started, click on the ‘Use smart thresholds’ toggle at the top of the screen.

By default, ‘apply to all tests’ will be selected. If you’d like to use Smart Thresholds for the entire suite then you’re good to go. The Example platform will use data from the previous five crawls to automatically adjust the threshold.

To enable Smart Thresholds for individual test suites, choose ‘Apply to individual tests’. You can then open the relevant tests to enable Smart Thresholds for only selected tests.

And if you change your mind, you can always disable Smart Thresholds whenever you like.