Over the last 15 years, I’ve done dozens of accessibility audits. I feel confident that if you hired me to do an audit, you’d be extremely happy with the results of the audit, the level of detail, and the guidance provided to fix the errors I’ve found. I take a lot of pride in the quality of my audits.
But is an audit what you really need? How many times have customers been sold an accessibility audit simply because that’s what they were told they should have?
What is the purpose of an accessibility audit?
What is the purpose of an accessibility audit?
— Karl Groves (@karlgroves) June 26, 2017
I can think of only 3 valid reasons for an audit report:
- To be used as evidence to justify (or defend) an accessibility lawsuit
- To be used as the basis for a VPAT
- To be used as justification for more budget & resources for accessibility.
In all other situations, an accessibility audit is a poor approach to solving the core problem. People want their accessibility problems solved. Audits don’t solve problems. Instead, they delay the resolution.
All software and websites have accessibility problems. An audit report is only an affirmation of the obvious and, at most, a report describing where – and how bad – the accessibility problems are. Given this reality, is an audit the solution to the problem? If what you want is a huge Word document with screenshots and lots of advice, then an audit is exactly what you want. We will be happy to oblige the request and you will be very satisfied with the quality of our work.
But is that really what you want? Or do you ultimately want to fix the accessibility problems? If that is the case, we need to have a different conversation about what is the best approach to getting things fixed. If we function under the understanding that the audit will definitely find accessibility problems, what is the shortest distance between discovery and repair? I’d argue that at best, an audit is a speed bump to progress.
The Typical Audit Process
Spoiler alert: This is how all of the big players work. How do I know? I’ve worked for them all. In some cases I defined the methodologies they use today.
Scoping
Before we give you a price for an audit, we need to see the system you want us to audit. We need to know what the system does, who uses it, how they use it, what features it has, and so on. We need to know what technologies were used to build the system, how we will access it to test it, etc. In this part of the process we’re looking for two things: First, we’re looking for anything that will bite us during the testing. Second, we’re looking for a unique subset of user interface components that will allow us to get a good glimpse at the overall system.
There should be no need to test every page or screen of a product. When the system was developed, the code to do so was based on specific practices unique to the developers. In some cases, this is closely tied to the code libraries, UI toolkits, content management systems, etc. and in other cases it is just based on the development style of the team. Either way, it is safe to say that things like forms share the same general coding patterns, as will things like tables, dynamic UI widgets, navigation, etc.
This system walkthrough informs the scoping process and the scope of the audit should take into consideration a handful of unique examples of each type of content & control without being repetitious. At the same time, there may be outliers, especially on legacy systems or content-heavy sites with a lot of contributors. Scoping an audit is more of an art than a science, but an experienced consultant can accurately scope an audit in a way that provides sufficient coverage without needless repetition.
Spoiler alert: Everyone’s hourly rates are mostly in the same range. If you get prices that are wildly less expensive or more expensive than the others, you should view those quotes pretty suspiciously. Choose based on quality, not price.
Testing & Reporting
Once you sign off on the SOW for your audit, your chosen consultant will dive in and start testing (based on schedule). Depending on the nature of the system, a tester can get through (on average) about 3-5 screens a day. Naturally this varies pretty significantly depending on the system. A screen of just text content can be as simple as 15 minutes to test. On the flipside it once took me 2 days to test a single screen of an online channel guide for a cable company, not because it had so many issues but because tracking down some issues and testing my recommendations was difficult.
The testing & reporting phase is where you’ll see some significant differences between consulting firms. The reality of accessibility consulting is that it is easy – very easy – to find accessibility problems. As I said earlier, all software and websites have accessibility problems, so even junior testers are able to find problems quite easily. Giving you solid, actionable guidance on fixing the errors is what separates the expert from the newbie.
Experts find more issues than novices when it comes to accessibility testing – owing, obviously, to experience. But what also matters as much if not more is the level of detail provided in the report. A high quality audit report will articulate not only what was found but who it impacts, why it matters, where the problem was, what the problem was, and how to fix the problem. All of these details are critical to the report’s recipients, because there may be a number of people who receive the report, each with different roles to play.
Delivery
Once the audit is done, the report is delivered. Some companies will send over the report and wish you well, some companies will hold a Q&A after they send the report, and some companies will do an overview presentation after they send the report.
One thing that isn’t done at this point: Nothing is fixed.
To review: At this point you’ve chosen a consultant and for a handsome price they’ve skillfully done a sampling of your system to determine the testing scope, followed-through with a thorough testing of your system, and (hopefully) given you a great big and informative report.
There’s certainly a value to having the knowledge contained in the report. But what you haven’t gotten so far is any tangible progress on the accessibility of your systems. There’s no doubt about it: Knowing about your system’s problems is a good first step toward getting it fixed. Unfortunately:
- Audit reports are very often not structured in a way that lends them to being directly actionable. Big Word documents are not actionable entries in an issue tracking system.
- Nobody reads the audit reports, anyway.
On point #1 above, across my 15 years in this business what happens – if we’re lucky – is that someone in the QA or development teams will go through the audit report and carve up the long prose of the document into issues to be entered into the company’s issue tracking system. This, of course, begs the question as to why accessibility consultants are shipping over Word documents. The interpretation that then needs to take place at the client’s side of things almost always leads to incomplete or inaccurate issue descriptions.
Also: Delivering the customer an Excel spreadsheet of issues is no better. Excel is a terrible format for a report and Excel is not an issue-tracking product, either.
This leads us to point #2: Nobody reads the audit reports, anyway. A substantial amount of my work over the last several years has been to do on-site training for customers. This is often part of a bigger engagement where the customer gets an audit and training. This is a very wise move. Accessibility problems happen due to ignorance, not malice, so scheduling training to follow an audit is a great idea.
Whenever I do training after an audit, I’ll ask everyone in the room who has read the audit. I’m not asking to prove a point, but because it helps during training if we can reference parts of the audit. Unfortunately, I’m usually lucky if I see more than one or two hands. Usually those one-or-two people who say they’ve read the audit report are the internal accessibility champion and another manager-or-executive level person. It is almost never any of the developers who will need to fix the system or the QA staff who will need to test those fixes.
This isn’t to say that those who don’t read the reports are bad people. Reading an audit report that could easily reach 100+ pages is the kind of thing that can quickly take a back seat to the many other things they’re asked to do each day. This alone necessitates providing the results in a much more easily absorbed format.
Rapid Discovery and Remediation
Three years ago I first said The Accessibility Consulting business is broken. This remains true. We must increase our collaboration with customers. We must deliver value more quickly, and we must do away with the idea that our customers’ problems are solved by an unwieldy audit deliverable.
In my next post, I will discuss Tenon’s Rapid Discovery and Remediation process