Efficient Web Accessibility Testing — Objective Evaluation

Summary

The objective-evaluation part aims to answer the question of how complete the web accessibility tools really are, and how they are operated. Typically, we would like to be able to say "yes, this is supported", or "no, there is no support for that". Here, we point to the individual reports, and we also explain why some checkers were disregarded.

The objective findings are discussed together with the results from the subjective evaluation in the project overview.

Testing procedure

The evaluation of accessibility checkers was divided into an objective and a subjective part. The subjective evaluation summarizes a number of personal use situations.

For the objective assessment (this part), more than 20,000 testcases were collected in a variety of testsuites, addressing topics such as HTML, CSS, DOM, and SVG, as well as ARIA and HTML 5-related APIs. We then observed the performance of each accessibility checker with the given testsuite.

Individual reports

Read the individual reports below.

• Accessibility Developer Tools (Google)
• Accessibility Extension (Firefox)
• AChecker (ATutor)
• eGovMon (Tingtun)
• SortSite
• Tawdis
• TotalValidator Pro
• Wave (Webaim)

Disregarded checkers

The following checkers have been considered but disregarded due to various reasons as detailed below.

AMP

We had tested AMP and reported our findings in a previous version of this report. In the aftermath of the publication process, SSB BART Group pointed to their license of use (they call it Master Subscription Agreement) which prohibits comparisons like these, particularly with the passus

[..] you may not access the Services for purposes of monitoring its availability, performance or functionality, or for any other benchmarking or competitive purposes.

We believe that we did check the license prior to testing. However, we are unable to prove that the license has changed in the meantime, and therefore we were forced to remove the part on AMP.

Bitv

We did not succeed in establishing an account with this web accessibility evaluation service. The test itself, being in German only, appears basically to consist of a form which simply has to be filled out in the course of an accessibility analysis which has to be conducted with other testing tools anyhow.

CynthiaSays (HiSoftware)

Unfortunately, the license for this service allows a maximum of 10 access operations per 24 hours and prohibits commercial use, which renders this offer useless for our purposes.

FireEyes (Deque)

Unfortunately, we were unable to get a copy of this software and thought it was abandoned, disregarding it for further analysis. In November 2015, however, Deque made contact and insured us that this in not the case.

Web Accessibility Inspector

Covers only WCAG 1.0.

Results

The authors of this evaluation do not want to give a clear recommendation for a single tool. Developers, testers, content producers and site owners typically have different needs and use cases and are likely to prefer different checkers depending on the context. In the table below, we therefore list if and to what degree each checker addresses a particular topic, such that developers searching for The Right Tool can make a well informed decision.

That being said, it is easy to see that SortSite currently appears to be most complete tool regarding the mentioned topics. Its use can easily be augmented with external validators and other tools as needed.

Moreover, we do want to give a recommendation against using specific tools, namely AMP, Bitv, and Tawdis. These checkers hold poor quality and/or have nothing that could not be accomplished better by other tools. In addition, the license of CynthiaSays is incompatible with commercial testing (and limited in frequency). The use of all of these tools should be avoided.

The following table gives an overview of the capabilities of checkers we evaluated.

We have put the in-depth results in a separate document.

Concluding remarks

The evaluated tools differ a lot in a number of aspects, such as mode of operation, platform, license policy, feature set, etc. However, there are some similarities common to the majority of checkers which should be noted.

Most checkers have not been able to keep up with the recent years' fast development pace regarding a number of web technologies, and appear hence nowadays as incomplete, with a limited feature set. This applies especially to XML handling in general and SVG in particular, to many HTML5 related features and APIs, particularly those associated with dynamic web services, to CSS (3), ARIA, and server headers. Also batch testing mode is very useful for monitoring purposes.

Speed appears to be an issue with many tools. This topic is important for iterative testing and frequent batch testing of, say, public websites with more than 100 pages.

Last but not least, in the context of this evaluation, it should be kept in mind that related research has shown that current automatic accessibility checking tools at their best cover about 50% of the WCAG 2.0 success criteria. This number should be considerably increased in the not too far future.