A discussion of test quality and transparency at Tenon
Tenon recently finished a period of test development the likes of which we haven’t seen since the earliest days of our product. Prior to this update, Tenon had 65 tests. Right now we’re in the process of integrating a series of changes and improvements that will give us 140 tests. A future effort we hope to complete later this year will result in a total of around 150. But total test number doesn’t accurately tell the story of how Tenon thinks of tests.
Altogether, Tenon actually tests for almost 1200 specific failure conditions across those 140 tests. We’re also careful to ensure we avoid false positives – and there are nearly an identical number of test conditions meant to ensure we’ve avoided a false positive. There are an average of 16 positive & negative test conditions for each of Tenon’s tests and some tests have as many as 288!
A fair number of our tests are informed by surveying our population of over 2000 users as well as the input of over a dozen well-respected accessibility experts. Tenon is the only company of its kind to continually, actively, and openly seek the input of its user base and other accessibility experts.
Among the most important new changes: Color-contrast testing. Admittedly, contrast testing is something any good accessibility-testing product should do, and we should have had this capability sooner. Existing users should expect that their average issue counts will rise dramatically if your site has contrast issues.
This test effort also includes dozens of improvements on existing tests, including removal of some bugs, eliminating false positive conditions, and eliminating almost a dozen overly conservative or irrelevant tests. It also includes the addition or modifications of ARIA in 70 tests.
One of the things that is just as important as the tests is the documentation of the tests themselves. To help address this we’ve modified the test information thanks to the help from plain language expert Ashley Bischoff. Throughout the rest of this summer we’ll be adding the remaining 44 tests as their documentation is finished and given the plain language treatment from Ashley.
Finally, we’ve increased the execution time limits on the Test API. The additional tests result in a threefold increase in the average issue count per page. Very large pages and very badly inaccessible pages require a longer execution time. The increased processing time significantly reduces the occurrence of “Cannot connect to Tenon’s API” errors.
We feel that the quality, breadth, and depth of our tests is an important differentiator between Tenon and its competitors and we hope you agree. If you ever come across test results that you disagree with or find buggy please let us know so that we can ensure we improve the tests or the documentation.