Efficient Web Accessibility Testing

Platforms

The available tools can be grouped in three categories — online web services, browser add-ons, and compiled downloadable software. By definition, the availability of the latter will depend on the development platform. It is noted that some checkers may fall into multiple categories. E.g., AChecker can be run as a web service and installed locally.

On Microsoft Windows platforms, SortSite is a very efficient tool with the most comprehensive set of tools and the widest scope. It is however implemented on top of Internet Explorer, which has poor support for web standards relative to other browsers. Potential consumers should also be aware of the peculiar pricing policies associated with this software. See the individual report for details.

There is only one standalone, compiled tool which can be used on Unix-like and Macintosh systems: TotalValidator Pro. It is a standalone application written in Java. Still, it is primarily written for Microsoft Windows, and apart from the poor overall quality of its error reporting, it cannot be recommended due to the nature of its batch mode, on which more in the next section.

By definition, browser add-ons are tied to the browsers they were developed for. Automated checkers are available for Google Chrome/Chromium and Firefox browsers, all of which are available on all major and most minor operating systems, either as packages or in source form.

Web services can be used in any browser. Developers may need to temporarily reconfigure their servers to allow the service bots, because none of the web-only services allow for changing the user agent string of the bot.

Mode of operation

Most tools that have been evaluated, including all web services and browser add-ons, can only be used ad hoc, requiring manual activation for every document tested. In that, these semi-automated tools are neither suitable nor convenient for checking existing large sites, testsuites, and other multi-document projects of sufficient scale for conformance with the accessibility specifications. They are useful for on-the-fly development of new documents, provided they are already online. One of the browser add-ons — the Firefox Accessibility Evaluator — is capable of processing offline documents. There is no way to automatically start add-ons for a newly loaded page. It is noted, though, that for some checkers local files can be tested with the file: protocol or as http://localhost.

On-the-fly development is very different from mass assessment of existing sites, some of which can contain resources numbering in the thousands of documents. For such labor-intensive tasks, only tools which offer batch mode can be used efficiently. Among the tools tested, these are TotalValidator and SortSite. Due to the vast difference in scope and quality, only SortSite can be recommended for these tasks. TotalValidator is not capable of processing directories, and requires that all documents be interlinked, or that the developer uses text processing tools to autogenerate a document with a full site map. In addition, TotalValidator also produces one error report per document, the processing of which is tiresome and labor-intensive, particularly given the extreme number of false positives this software generates. SortSite on the other hand is fast enough to be able to download and process tens of thousands of documents per hour, generating a single, well-documented report for the entire site, directory, or an arbitrary set of documents.

For these reasons, it is recommended that mass assessment of large resources is performed with SortSite. Individual or newly created documents can be processed in any tool available for a given platform (as well as web services and browser add-ons), with the final choice depending on the analytical capabilities of the given software.

Technologies — HTTP

The most prodigious source of false positives in automated accessibility checkers is due to their failure to check and process server headers, which are the most efficient way to declare encoding, language, content type, navigation structure of auxiliary documents, and even stylesheets.

Server headers — language

One example is the following Apache server configuration:

AddLanguage no .html

This can be overridden anywhere down in the URL hierarchy, or from the top configuration file, for example through the <Files> directive. Server headers are trivially checked from within the graphical browser — see the examples from Firefox/Pentadactyl.

Server response, Norwegian version — Server responses in a graphical browser

Server response, English version — Server responses in a graphical browser

The same information can be trivially obtained from the command line, using a text browser:

% lynx -head -dump //unicus.no/ | grep Language
Content-Language: no
% lynx -head -dump //unicus.no/en/ | grep Language
Content-Language: en

Failure to check and parse server headers is a critical flaw in automated accessibility checkers:

Validating a web standards testsuite featuring well over 10.000 documents can result in thousands of bogus failures, depending on whether the root element was explicitly specified or not. This of course applies to document sets residing on a local developer machine, but due to failure of the software to parse server headers, this is also true if the testsuite resides on a remote, properly configured server. Wading through thousands of bogus errors requires enormous effort, and is counterproductive.
Most if not all documents on a properly configured server will fail validation, causing similar problems as above.
There is at least one case of a commercial company in Norway being unfairly targeted for nonexistent accessibility violations, specifically regarding alleged failure to declare document language — due to insufficient knowledge of web standards and over-reliance on automated processing by the organization issuing warnings. This event is bound to be repeated as long as there is no official list of known software bugs and inefficiencies.

Server headers — links

Stylesheets can be added to documents via link elements, but links need not reside within the documents themselves — they can be set on the server. An example is this Apache server configuration:

<Files ~ "index\.(html)$">
Header add Link '</css/define.css>; rel="stylesheet"'
</Files>

This directive injects a stylesheet link to all documents with the html extension. It may be used as a global stylesheet for the entire site, as a base stylesheet to be overridden by document-specific stylesheets, or as a way to neatly define CSS Variables, on its own a useful accessibility feature of cascading stylesheets. It is unfortunately not mentioned in the WCAG 2.0 standard, which it post-dates.

Server headers — site navigation links

Other link types can also be specified on the server, such as the site map, copyright notice, newsfeed, and many more:

<Files ~ "index\.(html)$">
Header add Link '</index/>; rel=index'
Header add Link '</legal/>; rel=copyright'
Header add Link '</atom.xml>; rel=alternate; type=application/atom+xml;
	title="site updates newsfeed"'
</Files>

These links affect accessibility in more ways than one. Some applications populate a separate navigation bar with these links, where they may appear as text and/or icon buttons. Others may add it to the text menu. Some more advanced search engine robots process server headers and may present information extracted thus in the site overview. It is also recommended that newsfeeds are appended to the documents through such a link.

Status

Developers with control of the servers where their content is stored are strongly encouraged to properly configure server headers. Unlike automated accessibility checkers, all major browsers check for, parse, and make use of most headers described in this section. Search engines will also make use of some of this information, and thus proper configuration will affect the quality of search results offered to users who may not yet have had the chance to visit the site.

The only accessibility software which checks for server headers is SortSite, but even this tool does not make proper use of these headers, instead issuing false positives. It is recommended that developers double check their configuration with console tools such as lynx (available for all platforms either as a package or in source code form) or browser add-ons such as Pentadactyl. Alternatively, a number of online services exist that can also perform this kind of check, although they are in various states of disrepair.

Technologies — HTML, MathML

While not all validation errors will be fatal or cause accessibility problems, invalid markup is very likely to make documents inaccessible, necessitating well-formedness checks on the part of the developers.

Some automated checkers have their own validators, importing the open-source code of the HTML validator offered by W3C, and then failing to keep it in sync with the source. Others only offer HTML validation as an extra option, outsourcing validation to the W3C service. In the former case, it is not possible to ascertain how old the imported code is. The output quality therefore varies from tool to tool. Furthermore, the W3C validator itself is lagging behind the development of new technologies.

For a full-featured validator which is strictly in sync with the rolling HTML5 specification, it is recommended that the developers make use of the Living Validator maintained by the specification community. It is the best validator in existence at the present time. However, it can only process one document at a time.

For MathML specifically, among the set of automated checkers tests can be performed only with SortSite.

Technologies — SVG

Unfortunately, none of the automated accessibility checkers offers any support for SVG.

Inline SVG

Another manifestation of the lack of support for SVG is that they treat vector graphics as external objects within the XML realm. In HTML5, SVG can be freely mixed within HTML documents, inline, and not as an external resource, referenced for example by an img or an object element. This is a major problem, on several levels, particularly in light of the fact that HTML parsers for SVG are under development in modern browsers. In other words, vector graphics will no longer be parsed as XML, unless the parent document of an inline is served as application/xhtml+xml, or the object is external, also served as XML: image/svg+xml. Since new parsers are under development, that may change yet, and fast. Still, even now islands of XML may be freely used, namespace-free, within HTML documents:

<div>this is
<svg width="500" height="200">
<text x="10" y="10">sample text</text>
</svg>
</div>

This snippet is perfectly valid, with text content accessible to humans, robots, graphical browsers, and also text browsers since time immemorial, see the accompanying screenshot.

Screenshot of inline vector graphics — Inline SVG in a text browser

This screenshot taken from the elinks text browser in the Unix console shows clearly that inline SVG is seamlessly mixed with the surrounding non-vector HTML content, much in the vein of image alternative text content. The text element within vector graphics is used to mark up text. Automated checkers may ignore the SVG bits, or indeed drop the entire contents of the svg element, as if it didn't exist at all. This leads to a major dysfunction, for example when vector graphics with text content are used within anchors.

<a>this is
<svg width="500" height="200">
<text x="10" y="10">all a link</text>
</svg>
</a>

Screenshot of linked inline vector graphics — Inline SVG as anchor child in a text browser

Again, text/console browsers seamlessly inject text content of inline SVG, mixing it with the inline node of the parent anchor, see the illustration.

Xlink, xpath, metadata

None of the automated checkers has support for SVG as such. SVG-specific styles are dropped or reported as invalid (which also affects color processing and contrast checking). SVG documents may be interlinked, either internally, or externally, interacting with other documents, whether HTML, XHTML, or XML. No automated assessment can be performed for document and section metadata, content labeling, content grouping, and repetitive blocks of content, for which vector graphics are particularly known, especially if they are authored using WYSIWYG software such as Inkscape or Adobe Illustrator.

Accessibility of vector graphics (and documents using vectors in general) is an area of leading research, with no definite conclusions available yet, which in turn hinders development of automated checkers, because a set of clearly identifiable guidelines does not yet exist. Therefore, analysis has to be performed manually, by an expert in the field.

Technologies — CSS

Most automated accessibility checkers do not parse or process CSS. Syntax can be checked using the W3C CSS Validator, which is offered as an add-on, outsourced tool in browser add-ons such as the Firefox Accessibility Evaluator. It is worth noting that support for CSS 2.1 in the validator is not perfect, while support for CSS 3 is incomplete, as development of validator software seriously lags behind the implementation of new modules in web browsers. See also the next section for non-syntax issues.

Color and Vision

CSS syntax aside, it is important to realize that most of styling these days is done with CSS, and therefore the usage of color is of particular importance. Some of the automated checkers, such as FAE, Wave, and Google Accessibility Developer Tools, have support for color contrast checking. It is worth noting that all these tools are browser add-ons, and utilize the same codebase for the contrast checks. In addition, SortSite performs checks for missing background or color information if only one of these was specified in the cascade, which can affect readability of documents, especially for users who require user stylesheet support because of poor vision or other health-related issues.

None of the automated checkers will perform a full color analysis however, and therefore will be unable to spot issues which are bound to negatively affect users with color blindness. For that, manual inspection must be performed, preferably augmented by the raster image analyzers which can simulate various types of color blindness.

Audio and Video

None of the automated checkers has support for aural CSS. The effect of using audio stylesheets can only verified with the usage of specialized, professional software. It is worth noting that SortSite documentation will present a list of screen readers which have problems with a certain property or element. It is the only automated checker which offers advice on this topic.

None of the automated checkers have support for rich audio/video multimedia, and therefore the fine points of level AAA accessibility compliance have to be assessed manually by a professional.

Accessibility features of HTML5 and ARIA

ARIA is not used almost anywhere on the net. One reason is that web browsers do not have support for it, and specialized page-reader software is too expensive and too unfamiliar for almost all developers. While some tools offer syntax support for the latter, only SortSite will present advice on how ARIA technologies should be used in a given context.

However, the HTML5 standard has introduced technologies which, if used correctly, obsolete the vast majority of use cases for ARIA. It is therefore recommended that the latter be used as a last resort, and that advantage is taken of the new technologies, particularly the new form and navigation elements and attributes. SortSite is the only accessibility checker with good support for these technologies, and the documentation is outstanding. At the same time, the Living Validator maintained by the specification community will assist in syntax and well-formedness assessment.

Scripting

All tools except SortSite ignore scripts. The latter parses scripts, issuing errors if it encounters incorrectly used input methods, which exclude either mouse or more likely keyboard users from accessing or processing the given construct.

Scripting should only be used as a progressive enhancement of documents, if at all possible (the increasingly common exceptions are so-called apps, or standalone web applications, more and more often targeted at mobile users). Whether replacement of spurious scripts with the relevant CSS3 and HTML5 technologies can be done should be assessed by an expert.

Text content processing

Surprisingly few automated checkers process the actual text, trying to catch problems with suspicious link text ("click here"), repetitive content, et cetera. Tools which offer such functionality are Wave, FAE, and SortSite.

Efficient Web Accessibility Testing — Recommendation details