The objective-evaluation part aims to answer the question of how complete the web accessibility tools really are, and how they are operated. Typically, we would like to be able to say "yes, this is supported", or "no, there is no support for that". Here, we point to the individual reports, and we also explain why some checkers were disregarded.
The objective findings are discussed together with the results from the subjective evaluation in the project overview.
The evaluation of accessibility checkers was divided into an objective and a subjective part. The subjective evaluation summarizes a number of personal use situations.
For the objective assessment (this part), more than 20,000 testcases were collected in a variety of testsuites, addressing topics such as HTML, CSS, DOM, and SVG, as well as ARIA and HTML 5-related APIs. We then observed the performance of each accessibility checker with the given testsuite.
Read the individual reports below.
The following checkers have been considered but disregarded due to various reasons as detailed below.
[..] you may not access the Services for purposes of monitoring its availability, performance or functionality, or for any other benchmarking or competitive purposes.We believe that we did check the license prior to testing. However, we are unable to prove that the license has changed in the meantime, and therefore we were forced to remove the part on AMP.
The authors of this evaluation do not want to give a clear recommendation for a single tool. Developers, testers, content producers and site owners typically have different needs and use cases and are likely to prefer different checkers depending on the context. In the table below, we therefore list if and to what degree each checker addresses a particular topic, such that developers searching for The Right Tool can make a well informed decision.
That being said, it is easy to see that SortSite currently appears to be most complete tool regarding the mentioned topics. Its use can easily be augmented with external validators and other tools as needed.
Moreover, we do want to give a recommendation against using specific tools, namely AMP, Bitv, and Tawdis. These checkers hold poor quality and/or have nothing that could not be accomplished better by other tools. In addition, the license of CynthiaSays is incompatible with commercial testing (and limited in frequency). The use of all of these tools should be avoided.
The following table gives an overview of the capabilities of checkers we evaluated.
We have put the in-depth results in a separate document.
The evaluated tools differ a lot in a number of aspects, such as mode of operation, platform, license policy, feature set, etc. However, there are some similarities common to the majority of checkers which should be noted.
Most checkers have not been able to keep up with the recent years' fast development pace regarding a number of web technologies, and appear hence nowadays as incomplete, with a limited feature set. This applies especially to XML handling in general and SVG in particular, to many HTML5 related features and APIs, particularly those associated with dynamic web services, to CSS (3), ARIA, and server headers. Also batch testing mode is very useful for monitoring purposes.
Speed appears to be an issue with many tools. This topic is important for iterative testing and frequent batch testing of, say, public websites with more than 100 pages.
Last but not least, in the context of this evaluation, it should be kept in mind that related research has shown that current automatic accessibility checking tools at their best cover about 50% of the WCAG 2.0 success criteria. This number should be considerably increased in the not too far future.