Efficient Web Accessibility Testing — Tawdis/TAW

Versions and mode of operation

Tawdis is a very old suite of tools, available in three different flavors:

In the pre-compiled form, the software has not been updated in many years. The downloadable Java binary does not run on either BSD or Linux (tested on 4 separate systems with two latest runtime versions of Java — 1.7 and 1.8), while the Windows version crashes on startup. The last time a real update was done around 10 years ago, when Windows XP was not only supported by Microsoft, but still the dominant system. In 2014, it is neither. There are no versions for the last two major versions of Windows which are supported by Microsoft.

The runnable Windows applet dumps core on startup.

Installation of the Firefox extension is unnecessary, as it doesn't add anything that is not already present in the default web checker — it merely opens the "default" web interface.

Another indication that the web analyzer wasn't updated in many years is the fact that the WCAG 2 support is in "beta". Note that the specification reached recommendation status in 2008, more than 6 years ago, while being usable and referable several years prior to that.

In summary, whatever options the ancient binaries might have are no longer accessible to the developer. In particular, there is no batch mode, no recursion. The only usable portion of Tawdis is the online version, which can "analyze" one page at a time. This disqualifies this tool from any serious testing — even a small site with circa 100 pages would take a very long time to process, on the order of days if not weeks, while a large testsuite would necessitate what amounts to several years of cumbersome labor. Further reading of this document should make it clear as to why.

Results

Even in single-document mode, the only mode there is, the problems with this software are gravely numerous. The interface is useless and confusing in the extreme, while in reality there is no support for WCAG at all. There is no analysis performed, and every single instance of an object referenced by the specification will produce a list of links to numerous (all?) instances where that object is referenced by the specification. Such a tool cannot be taken seriously.

This is illustrated by the following topics.

No inline errors
There are no inline errors, i.e. the table of results at the top of the result document is disassociated from the context, necessitating constant switching between the table of results and the source at the bottom at the document.
No linked references
Even if there are no inline errors, navigation between the results table and the analyzed document source could have been provided. It isn't. There are no linked references to the allegedly problematic portions of the code.
Mystery meat labeling
Problem descriptions use icons, most of which are question marks, indicating ambiguity. The report table is separated from the problem-flagged source by 44 images of question marks, the meaning of which is unknown and the number of which is not related to the number of alleged errors found.
Incomplete translation
The web "interface" is only partially translated into English, with terms such as tipology and comprobation present in headers alongside line numbers.
"Tipology" seems to mean the nature, or class of the problem (e.g. Meaningful Sequence), while "comprobation" is the long description. Unfortunately, the terms used are generic enough (e.g. Text blocks aligning) to disallow immediate comprehension of the nature of the problem.
Obfuscated layout and broken navigation
The summary matrix of results is 2-dimensional and divided into problems, the meaning of which is fairly obvious, into warnings (direct quote: It is necessary a human review), and into... not reviewed (direct quote: Completely human review).
Alleged problems in each of these categories are classed into 4 subgroups: perceivable, operable, understandable, and robust. While these are labels used by the WCAG specification itself, it doesn't make much sense to categorize problems this way, especially that problems are then duplicated across categories. Wordy taxonomies are acceptable in background spec subsections, but not in a tool which is designed to find problems quickly and efficiently. These categories do not add anything to the understanding of the problem.
The same report tabs are present for all documents selected in the second part, so it is absolutely unclear what the alleged problems refer to without clicking back and forth for all 16 combinations of reports.
Following a link and returning removes focus from the last report "tab", making navigation a nightmare.
Random extraction of specification guidelines
The vast majority of alleged errors does not point to any fragment of the document at all, whether the (X)HTML parent document, or the associated resources, forcing the developer to guess "what the poet intended". Such error reports do provide a link to the relevant WCAG specification section, but it is left as an exercise for the reader to assess just where this problem might potentially apply.

In other words, the tool is incapable of teaching the developer anything that said developer didn't know before he or she used the tool, because all analysis must be performed manually anyway. Tawdis is an utter and complete failure. To illustrate the problem more clearly, an example of a single, error-free and short page will be used. Tawdis found 73 alleged (imaginary) problems. It took a very long time to slog through the bogus problem report. The time of any developer is better spent on other tools.

Longdesc

Images with proper alt are flagged as potentially needing the longdesc attribute, which has been obsolete for a long time, and replaced with a host of image- and figure-related elements in the HTML5 specification. This warning is issued regardless of the nature and content of the image, how it is coded, what kind of description accompanies it — instead, the mere lack of the longdesc attribute will trigger the warning. This is a massive source of spam.

Absolute positioning

Any usage of absolute or fixed positioning (e.g. position: absolute) will trigger a warning. Referred guidelines:

The referred guideline should never be interpreted blindly as mandating normal flow. Entire specifications such as CSS3 Flexible Box Layout were specifically developed to address the content order and so-called "skip to navigation" problems.

"Sensory characteristics"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

There is nothing that could possibly trigger this.

"Information using color"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

Automatically triggered by the use of color property. Upon encountering code related to color, the tool simply produces a list of links to guidelines related to color.

"Contrast"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

Automatically triggered by the use of color and background properties. Upon encountering this code, the tool simply produces a list of links to guidelines related to color. See also the next section to see the dumb nature of this tool (if it isn't evident already).

"Contrast for large fonts"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

See the previous section. Note also that the same specifications are referred to in two separate categories. That is of course due to blind-matching between what is found in the document and what may potentially refer to it in the set of WCAG guidelines.

Pixels versus text resize

Any usage of pixel widths generates a warning. Referred guidelines:

Tawdis also issues the following warning:

In Internet Explorer 6, Windows High Contrast Mode will resize percent-based fonts in nested tables to be too large. Firefox and later versions of IE do not resize fonts in High Contrast Mode, and don't have this issue.

(...)This will allow user agents that support text resizing to resize the text containers in line with changes in text size settings.

Tying guidelines to specific software is never good. The last generation of visual browsers which didn't resize generic containers alongside with text was produced in 2001, and none are used or supported today. Furthermore, 2014-vintage browsers may have more than one text resize mode. It remains true even if font-size is specified in pixels. There is no requirement of forcing em units for widths. The guideline remains valid for certain form control usages though, but regardless — Tawdis automatically issues this warning for any width with a pixel value. It is also unaware of the viewport specification.

"Replaceable images"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

Completely bogus associations of these guidelines, triggered by the presence of a company logo image.

"Contrast enhanced"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

Triply duplicate, dumb, automatic guideline extraction. See commentary in the "contrast" section.

"Contrast enhanced to large fonts"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

Triple duplicate, dumb, automatic guideline extraction. See immediately previous and "contrast" section.

"Background and foreground colors in text blocks"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by the use of color and background properties. Upon encountering this code, the tool simply produces a list of links to guidelines related to color and background. In addition, such automatic extraction mistakenly refers to borders used outside of CSS, even if there aren't any, again because of dumb automation.

"Width in characters of text blocks"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by the use of the width property anywhere in the stylesheet.

"Text blocks aligning"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by the use of the text-align property anywhere in the stylesheet.

"Space between lines in text blocks"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by the use of the line-height property anywhere in the stylesheet.

"Resizing of text blocks"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by the use of the font-size and width properties anywhere in the stylesheet. It doesn't matter what the value of the font-size property is.

"Images of Text (No Exception)"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is a duplicate guideline extraction, see "replaceable images" section.

"Using the pseudo-class: focus"

Any use of :focus generates a warning, regardless of styles applied.

"Automatic movement of the focus"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged as a script problem on a page without scripts, and it is triggered by CSS focus.

"Changing focus with the keyboard"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no trappable content, no plugins, no scripts, and no forms.

"Session time limit"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See the previous section, as the warnings are grouped en-masse by the automatic guideline extractor.

"Limit of time controlled by a script"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See the previous two sections.

"Reading texts on the move"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This is automatic guideline extraction based on text keywords.

"Pause, Stop, Hide"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This is automatic guideline extraction based on text keywords.

"No Timing"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is extraction duplication. See the previous section on time limits.

"Expiration of authentication information"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See the previous sections on time limits. Due to a typo in the automatic extractor database, the second guideline link is broken in the report.

"Three Flashes or Below Threshold"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is extracted en masse together with time limits, all for no purpose.

Descriptive title

In a section Page Titled (sic), a warning issued for any title content shorter than an arbitrary number of characters, regardless of the content. Referred guidelines:

"Appropriate content of headers and labels"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is issued for any heading content shorter than an arbitrary number of characters, regardless of content.

"Skip repeated blocks of content"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by a navigation list of more than one link. It is however worth noting that while these techniques, especially G1, were quite useful once, advances in layout technology in the last decade put paid to that. Entire specifications such as CSS3 Flexible Box Layout were specifically developed to address the content order and so-called "skip to navigation" problems. Any automatic failure flagging is considered harmful.

"Blocks of content"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by a navigation list of more than one link. it is worth noting here that the WCAG specification is old, and pre-dates the development and implementation of HTML5 and many CSS3 specifications. Several elements were introduced (and then vendor-implemented) in order to assist with navigation, and adding structure to navigation. Frames have been obsoleted, and support for them has been removed in some browsers, and is on the way out in others. Recommending frames is antediluvian and counter-productive. WCAG is sorely needed. Some guidelines should never be respected, or enforced.

"Logical sequence of navigation"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This leads to multiple duplicates. It is flagged on a textual page with no trappable content, no plugins, no scripts, and no forms. See previous sections on focus.

"Multiple ways of localization"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by a navigation list of more than one link. See the section on "blocks of content". It is worth noting that G185 and G126 stand in conflict with the previous techniques. A home page is not a site index, a sitemap, or a list of all possible links. Also, had it been, separate tables of contents and sitemaps would have been redundant. Navigation methods of this type should never be enforced automatically, and never all guidelines of this type at once. Sitemaps are also overkill for tiny/small sites.

"Changing focus with the event onFocus"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See previous sections on focus.

"Navigation/Ubication"

Sic! There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This leads to duplication of warnings. See the section on replaceable images. It is one more instance of a database typo producing a broken guideline link.

Server headers — language

Another source of false positives in Tawdis is due to its failure to check and process server headers. Errors are emitted on the following basis:

Headers are the most efficient way to declare encoding, language, content type, navigation structure of auxiliary documents, and even stylesheets. An example to achieve this is the Apache server configuration:

AddLanguage no .html

This can be overridden anywhere down in the URL hierarchy, or from the top configuration file, for example through the <Files> directive. Server headers are trivially checked from within the graphical browser — see the examples from Firefox/Pentadactyl:

server response, Norwegian version server response, English version
Server responses in a graphical browser

The same information can be trivially obtained from the command line, using a text browser:

% lynx -head -dump http://unicus.no/ | grep Language
Content-Language: no
% lynx -head -dump http://unicus.no/en/ | grep Language
Content-Language: en

Tawdis does not check and process server headers, and thus will complain that the language is not set for the document, with the error pointing to the doctype, where it expects a html element with the lang attribute.

That is poor advice even for small servers, let alone huge sites with thousands of pages. Using the lang attribute makes sense for standalone documents (such as testcases), and even then only for those which are shared through other means. Documents servers with a valid server header should never fail validation.

In addition, it is worth keeping in mind that the html element is optional. This algorithm is very simplistic, as even assuming away the issues of server configuration, some documents will fail, while others will pass, depending on whether the root element is explicitly specified.

Failure to check and parse server headers is a critical flaw in automated accessibility checkers:

"Reading level"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

The trigger here is unknown. Either it is the language misidentification problem (see section on server headers), or a lack of audio styles or links to audio content — or both. Most likely the former — see next section. Note that audio supplements for text are not required for any level, and are not a recommendation for the AA level.

"Pronunciation"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This is a language misidentification. See previous section.

"Changing content with the event 'onfocus'"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See previous sections on focus.

"Apertura de ventana con el evento 'onfocus'"

It is a duplicate, and untranslated. There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See previous sections on focus.

"Apertura de ventana al cambiar el foco"

Another duplicate of the same warning from the previous section, again untranslated. There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See previous sections on focus.

"Apertura de ventana con el evento 'onload'"

Untranslated, and without any reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts, with no new windows opening. This guideline is egregious and auto-culled by association. See previous sections on focus.

"Changing content"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

This is a triple duplicate within the same warning section. See previous sections on focus.

"Changes caused by the event "onchange" in a selector"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no forms or plugins or scripts. See previous sections on focus.

"Consistent navigation"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is automatically triggered by a navigation list of more than one link. See section on "repeated blocks of content".

"Consistent denomination"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no form controls, scripts, or plugins.

"Name, role and value"

There is no reference as to where the problem might be, and what its exact nature might be. Referred guidelines:

It is flagged on a textual page with no form controls, scripts, or plugins.

Basic document structure

HTML5 documents do not require as much basic structure as HTML4 documents. For a minimal document, all that is needed is the simplified doctype and a title. Even in the older specification, some basic structure was optional. The following is a valid (albeit not very useful) document:

<!doctype html>
<title>a simple document<title>

So are these:

<!doctype html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<h1>roses are blue violets are red<h1>

<!doctype html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<body>
<h1>roses are blue violets are red<h1>

<!doctype html>
<html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<h1>roses are blue violets are red<h1>

<!doctype html>
<title>a simple document<title>
<style>
html {
	color: white;
	background: black;
}
<style>
<h1>roses are blue violets are red<h1>

And so on. Tawdis is unfortunately living in the past. Such documents will be flagged as invalid, as they do not have an explicitly added root element, head element, and body element.

Parser failures such as these and subsequent validation failures have a very high impact. They affect most testsuites written by or contributed to the standards bodies, and used by browser vendors for development — they affect the exponentially growing number of HTML5 sites. In fact, a modern website is very unlikely to pass validation with Tawdis, while the number of mistakenly reported false positives will be a high multiple of the number of pages involved, hindering web development, and leading to accessibility being ignored due to too high a cost of wading through what could easily be tens or even thousands of bogus errors per site.

SVG style properties

Due to no support for vector graphics, upon encountering style properties of SVG, Tawdis will report errors, without any explanation other than line number. This applies to all vector-specific properties, such as fill and stroke, for example:

text {
	fill: white;
	stroke: black;
	stroke-width: 3;
}

"CSS Validity"

Again, no explanation whatsoever is given for allegedly invalid CSS. Instead, only line number references are given. The following valid bits were marked invalid:

There is no support for CSS3. In particular, public domain HTML validator code which couldn't parse HSL (or hue/saturation/lightness color value type) dates from around 2002, or 12 years ago. This is yet another indication that the Tawdis project is an amateur, unfinished, useless piece of abandonware.

Summary

Major issues with Tawdis are:

Tawdis is a piece of useless abandon-ware. It is considered harmful and its usage should be actively discouraged — for the sake of the sanity of the developer, if not for all technical reasons under the sun.