Efficient Web Accessibility Testing — Powermapper SortSite

Version

Versions tested: 5.5, 5.6

Potential consumers of this software should be aware that upgrading not only across major releases, but even across point releases involves shelling out the same amount as if the software was being bought anew. There are no free upgrades, unless a service contract is also purchased. Furthermore, downloads have a timeout, and must be collected within a certain timeframe; otherwise, support intervention will allow downloads of the already paid for software for 24 hours. All this is highly unusual.

Mode of operation

SortSite is a stand-alone application written for Microsoft Windows, and implemented on top of Internet Explorer. Uniquely among the set of automated checkers, it does not distinguish between a single URL and a collection of them, whether it is the entire site or a directory, including the sub-tree. The result presentation is uniform regardless of the scope. Unlike most checkers, SortSite was designed from the ground up for maximum scope testing.

Error reporting

SortSite reports are by far the best of all among the automated checkers, particularly given the fact that it supports validation of entire sites and massive testsuites in one go. The top three views are summary, issues, and pages. The summary includes three sub-views: dashboard, site map, and what was tested. The dashboard is indeed a summary, with the number of violations listed per category, plus a timeline graph should testing have been performed numerous times over time, indicating eventual progress, if any, in improving the compliance record. The site map is just a gimmick graph, showing where problems occur, in three dimensions. It is hard to imagine it being useful for anything. Finally, the what-was-tested pane includes a list of specifications, rulesets, and definitions which were used to perform evaluation. That includes HTTP codes, of which later. The pages pane is a rundown count of document or resource types, plus a document/resource-specific count of failures.

The top pane of most interest is of course the issues pane. There are three sub-views: errors, accessibility, and standards. The errors pane reports broken links, server configuration issues (Microsoft technologies and Internet RFCs). What exactly is being tested can be changed in the configuration. Accessibility and standards are self-explanatory. For each failure, priority is reported first (or accessibility level, respectively), then the description, a guideline, and relevant line. Each failure type can be customized on the fly:

The "desktop edition" allows to customize it from the options menu, while the "ondemand edition" allows for that in "edit schedule". The tested version is the standard version, with the three options listed above. Each failure by default is in summary mode. Unrolled, it shows a sample list of affected documents, and the total number of identical failures site-wide, or directory-wide. In the standard version the full list is not available, but at least it is immediately obvious what needs to be fixed, and more runs can be subsequently performed if necessary.

Snippet inspector

Each document or resource which is listed in a given error category is linked. Following this link opens a snippet page, with inline error indicators inserted in the relevant place in the source document, together with context around the place of failure. However, there is not enough context, which often requires that the resource be opened for inspection in an external editor. Furthermore, the point of insertion for HTML documents is the tag, which is broken just before the closing angle bracket. This kind of indication method takes some getting used to. On the flip side, the right side of the snippet page includes recommendations on how to fix the problem, and a rationale.

Despite some minor quirks, the entire workflow in SortSite is very friendly, with all the necessary information and reference at hand.

Skip links

The US-specific Section 508 accessibility law mandates skip links for navigation lists. If SortSite encounters a list, where list items contain anchor descendants, it fails the site for the A accessibility level, recommending that a skip link be added.

Provide a way to skip repetitive navigation links.

This is true even if the list is not a navigation list, but constitutes the actual content of the page, so the concept does not apply. This is a typical failure of automated software which requires human intervention to ascertain whether the rule really does apply to a given page. For this particular rule, SortSite should not flag it as an error, but as a warning that requires human evaluation — rather than failing the site automatically.

Server headers — language

Server headers are the most efficient way to declare encoding, language, content type, navigation structure of auxiliary documents, and even stylesheets. An example of the Apache server configuration to achieve this is:

AddLanguage no .html

This can be overridden anywhere down in the URL hierarchy, or from the top configuration file, for example through the <Files> directive. Server headers are trivially checked from within the graphical browser, see examples from Firefox/Pentadactyl:

server response, Norwegian version server response, English version
Server responses in a graphical browser

The same information can be trivially obtained from the command line, using a text browser:

% lynx -head -dump http://unicus.no/ | grep Language
Content-Language: no
% lynx -head -dump http://unicus.no/en/ | grep Language
Content-Language: en

Unlike most other automated checkers as of this writing, SortSite does check for server headers, including Content-Language, and even lists them in snippet mode, albeit hidden behind a link, in a collapsible list for clarity. Given that, it still fails documents based on the absence of the lang attribute.

Use the LANG attribute to identify the language of the page

That is poor advice even for small servers, let alone huge sites with thousands of pages. Using the lang attribute makes sense for stand-alone documents (such as testcases), and even then only for those which are shared through other means. Documents servers with a valid server header should never fail validation.

Failure to check, parse, and make use of server headers is a critical flaw in automated accessibility checkers:

Mixed XML: XHTML (+ SVG (+ MathML))

An efficient, and indeed in the case of XHTML the only correct way to define content type, is the server header. An example of the Apache server configuration to achieve this is:

AddType 'application/xhtml+xml; charset=utf-8' .html

XHTML documents served as HTML (i.e. when the server is not properly configured) are harmful. SortSite does not check headers, and thus will use HTML validation for XHTML documents, even if a relevant option is added to the command line checker, or picked from the user interface drop-down during manual testing — as long as the extension is different from .xhtml. Web browsers again do not have any problem:

server response, XHTML
server response in a graphical browser

The same can be accomplished on the command line:

% lynx -head -dump http://aappilattoq.info/tools/hsl-svg/ | grep Content-Type
Content-Type: application/xhtml+xml; charset=utf-8

XHTML documents may or may not feature a DOCTYPE. Indeed, the XHTML5 specification recommends that it be omitted, since it is not necessary, as long as the content type is properly set by the server. SortSite has problems with XML documents which do not include a doctype and/or processing instruction. A document with the following root element will be reported as blank:

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:svg="http://www.w3.org/2000/svg">

Despite claiming that the document is empty, SortSite actually does process said document. The aforecited root element is reported as invalid:

Attribute "svg" not allowed on element.

That is because SortSite does not understand namespaces if there is no doctype, and that is a failure of the checker as long as the document is sent as an application. The next source of failures is SortSite's insistence on head elements, which are not even required in HTML, let alone XHTML, sent as an application. This results in link elements being reported as disallowed.

These false positives affect XHTML 1.x and 5.x documents, even moderately complex XHTML/XML and SVG mixed documents, as well as MathML documents. While these documents are not nearly as ubiquitous as they were a decade ago due to the rise of the HTML5 specification, XML is still used widely enough to make this an issue.

Server headers — links

Stylesheets can be added to documents via link elements, but links need not reside within the documents themselves — they can be set on the server. An example of the Apache server configuration to achieve this is:

<Files ~ "index\.(html)$">
Header add Link '</css/define.css>; rel="stylesheet"'
</Files>

This directive injects a stylesheet link to all documents with the html extension. It may be used as a global stylesheet for the entire site, as a base stylesheet to be overridden by document-specific stylesheets, or as a way to neatly define CSS Variables, on its own a useful accessibility feature of cascading stylesheets. It is unfortunately not mentioned in the WCAG 2.0 standard, which it post-dates.

SortSite is aware of such links, listing them in the snippet mode, but given that it's implemented on top of IE, it doesn't make any use of these links.

Server headers — site navigation links and sitemap

Other link types can also be specified on the server, such as the site map, copyright notice, newsfeed, and many more:

<Files ~ "index\.(html)$">
Header add Link '</index/>; rel=index'
Header add Link '</legal/>; rel=copyright'
Header add Link '</atom.xml>; rel=alternate; type=application/atom+xml;
	title="site updates newsfeed"'
</Files>

These links affect accessibility in more ways than one. Some applications populate a separate navigation bar with these links, where they may appear as text and/or icon buttons. Others may add it to the text menu. Some more advanced search engine robots process server headers and may present information extracted thus in the site overview. It is also recommended that newsfeeds are appended to the documents through such a link.

SortSite is aware of these links, but again, it doesn't make any use of them. This manifests itself in another way. In entire-site mode, SortSite will fail websites due to a lack of sitemap, regardless of the size of the site:

Provide information about the general layout of a site using a site map or a table of contents".

You should provide a link labeled "site map" or "sitemap" or the equivalent in your language, on every page.

The rationale for this AA level failure is the WCAG 2.4.5 guideline. However, this guideline states that it is sufficient to provide more than one way to access a given document, with a sitemap being only one of seven example techniques, while in reality there are more. Clearly, SortSite is overly restrictive in this check.

Basic document structure

HTML5 documents do not require as much basic structure as HTML4 documents. For a minimal document, all that is needed is the simplified doctype and a title. Even in the older specification, some basic structure was optional. The following is a valid (albeit not very useful) document:

<!doctype html>
<title>a simple document<title>

So are these:

<!doctype html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<h1>roses are blue violets are red<h1>

<!doctype html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<body>
<h1>roses are blue violets are red<h1>

<!doctype html>
<html>
<title>a simple document<title>
<link rel="stylesheet" href="/css/all.css">
<h1>roses are blue violets are red<h1>

<!doctype html>
<title>a simple document<title>
<style>
html {
	color: white;
	background: black;
}
<style>
<h1>roses are blue violets are red<h1>

And so on. SortSite is unfortunately living in the past. Such documents will be flagged as invalid, as they do not have an explicitly added root element, head element, and body element. SortSite will also require end tags for elements such as html or body.

Parser failures such as these and subsequent validation failures have a very high impact. They affect most testsuites written by or contributed to the standards bodies, and used by browser vendors for development — they affect the exponentially growing number of HTML5 sites. In fact, a modern website is very unlikely to pass validation with SortSite, while the number of mistakenly reported false positives will be a high multiple of the number of pages involved, hindering web development, and leading to accessibility being ignored due to too high a cost of wading through what could easily be tens or even thousands of bogus errors per site.

CSS error reporting

Alleged CSS syntax errors are not elaborated on, merely grouped under the CSS validation error heading, with no explanation. Only the relevant line numbers are cited. In some cases, these line number references are wrong. See the next section for a particularly egregious example. SortSite should elaborate on what exactly it thinks is wrong, and provide an explanation and a document reference for the rationale (such as the relevant CSS specification).

CSS — syntax of pseudo-elements

SortSite does not support the double semicolon notation for pseudo-elements. For example:

div::before {}

In very early CSS versions, only one semicolon was valid. Later specifications, including CSS2.1, recommend a double semicolon for pseudo-elements (as opposed to pseudo-classes. The problem with this bug in SortSite is that it reports neither what it thinks is wrong, nor the correct line number for the allegedly offensive portion of the code. Instead, it may flag the last rule in the block to which this selector applies, the last rule in the previous block for the preceding selector, or the container style element itself. Unless the developer debugs the issue nailing the exact problem after many iterations, he/she will have to confront SortSite failing their documents for bogus reasons — e.g. incorrect color, background, width, height, and any property-value pair which happens to be the last one in a block associated for this selector.

On the other hand, for generated content pseudo-elements which contain non-decorative content, SortSite will correctly fail documents for this reason, per the relevant WCAG guideline. Uniquely among automated checkers, it will list screen readers which cannot cope with generated content. Very helpful, this.

CSS — missing support for modules

The pace of specification change has increased in the last five years, especially relative to the sleepy times of early 2000s when CSS 2.1 was incrementally improved upon and only non-dominant browser vendors were implementing this specification. Currently, new specifications are finalized rather quickly, and implemented even faster. It is not surprising then that automated checkers and validators are not quite catching up.

SortSite does not support some recommendations which have not reached the Recommendation status. It is a questionable decision, given that CSS 2.1 was in the Candidate Recommendation state for more than a decade. To be fair, SortSite admits the arbitrariness of the rationale for the issued warnings. The following specifications for which public testsuites exist are not supported:

CSS — linear-gradient

The linear-gradient property is not recognized:

div {
	color: hsl(232,10%,95%);
	background: linear-gradient(hsl(232,40%,22%),hsl(232,40%,8%));
}

CSS — font-weight

Every instance of the font-weight property is flagged with a warning, allegedly because it is abused to mimic the emphasis elements.

Use semantic markup like STRONG instead of using CSS font-weight property.

While the WCAG specification recommends exactly that, throwing an error for every instance of this property usage is clearly overkill. There is no way for software to guess the intention of the author. Conversely, in street html it is the semantic elements which are most often abused for presentational purposes.

CSS — text-decoration

Another source of false positives is reporting instances of text-decoration: none for anchors, which disables default underlining for links. While flagging this as an issue is very useful, the software itself suggests that if underlining is disabled, other methods which do not rely on color should be used to visually distinguish linked text from other text. The problem is that the heuristics for verifying the usage of such methods are not implemented, because there are too many ways to accomplish that task.

CSS — vendor extensions

If a vendor extension is used, it should be followed by the specification equivalent. Even so, SortSite will fail such documents merely due to the presence of the vendor extension, although this sort of check can be disabled individually and globally. Nonetheless, if it encounters a vendor extension, it should check if the rule (or a set of computed rule-value pairs) includes an unprefixed property or value which overrides the vendor extension in the cascade.

div {
	-moz-columns: 3;
	-webkit-columns: 3;
	columns: 3;
}

The above snippet should not be flagged, while this one should:

div {
	-webkit-columns: 3;
}

That is assuming of course that it is not overridden somewhere else in the cascade.

Absolute positioning

Any usage of absolute or fixed positioning (e.g. position: absolute) will trigger an accessibility error, referencing the visual order guideline, which should never be interpreted blindly as mandating normal flow. Entire specifications such as CSS3 Flexible Box Layout were specifically developed to address the content order and so-called "skip to navigation" problems.

Furthermore, the following description is neither smart nor helpful:

Absolute css positioning can make pages unreadable when style sheets are turned off

No application exists which would allow turning off stylesheets leaving absolute positioning behind and on.

SVG style properties

Due to no support for vector graphics, upon encountering style properties of SVG, SortSite will report errors. This applies to all vector-specific properties, such as fill and stroke for example:

text {
	fill: white;
	stroke: black;
	stroke-width: 3;
}

Handling of stand-alone stylesheets

If SortSite is pointed to a stand-alone stylesheet as a resource to be validated, it doesn't process it, opening the file in Notepad instead (i.e., in whatever editor is the default on a given Windows system). That is neither clever nor helpful.

Handling of in-body stylesheets

Due to the failure to recognize that the html, head, and body elements are optional in HTML5, SortSite will trip on both link rel="stylesheet" and style elements.

The former may precede all body content, in which case SortSite will merely fail the document and mention that the element is not allowed in this context. In street html however, stylesheet links may be mixed in with the body content, and appear anywhere in the document. Only then should the software flag the link as out of context. Instead, SortSite throws a non-sensical error message:

Element "link" is missing one or more required attributes. Missing "resource" "property"

The same context-and-tree commentary applies to the style element, but if it is encountered in-body, then another flavor of a non-sensical error is thrown:

Element "style" is missing one or more required attributes. Missing "scoped".

Table description — summary attribute handling

The summary attribute is a sunset component in HTML, since its contents are extractable only with CSS while the need for a more versatile mechanism was long recognized. HTML5 introduces the summary element, a child of the details element. However, for educational reasons the presence of a summary attribute should not result in a simple failure, but one accompanied by a recommendation/hint to replace it either with the new element(s), and/or with a table caption, the contents of which do not require stylesheet manipulation to be presentable to both sighted and non-sighted users.

Upon encountering a table summary, SortSite recommends removing it (or indeed the caption as well), if it suspects that the table is used for page layout. It recommends marking the table with the ARIA role="presentation" attribute/value pair, instead of insisting that the table be removed.

Multiple caption elements

SortSite flags multiple caption elements as erroneous, but in an obfuscated way.

<table>
<caption>sufficient description here</caption>
<caption>invalid caption element</caption>

Instead of simply pointing out that only one caption is allowed per table, it flags the second caption element as disallowed:

Element "caption" not allowed as child element in this context.

This is of course not exactly true for the reasons explained above, and may trick the uninitiated into removing the caption.

Inline SVG

SortSite treats vector graphics as external objects within the XML realm. In HTML5, SVG can be freely mixed within HTML documents, inline, and not as an external resource, referenced for example by an img or an object element. This is a major problem, on several levels, particularly in light of the fact that HTML parsers for SVG are under development in modern browsers. In other words, vector graphics will no longer be parsed as XML, unless the parent document of an inline is served as application/xhtml+xml, or the object is external, also served as XML: image/svg+xml. Since new parsers are under development, that may change yet, and fast.

<div>this is
<svg width="500" height="600">
<rect x="0" y="0" width="500" height="600"/>
<text x="10" y="10">sample text</text>
</svg>
</div>

Still, even now islands of XML may be freely used, namespace-free, within HTML documents, even though permissive parsing has not yet been implemented, and self-closing elements must be properly closed, as in XML.

SortSite will mark the svg element as not allowed:

Element "svg" not allowed as child element in this context.

Title length

Continuing with the theme of unnecessarily strict requirements, there is another source of failures which is best ignored — title length. SortSite will fail the document as non-compliant, citing the antiquated W3C Style Guide for justification. To wit:

The title should ideally be less than 64 characters in length. That is, many applications will display document titles in window titles, menus, etc where there is only limited room. Whilst there is no limit on the length of a title (as it may be automatically generated from other data), information providers are warned that it may be truncated if long.

Using window titles in a graphical browser as a justification is misguided, because there are many types of applications and environments (let alone resolutions), not necessarily featuring this particular widget. The primary consumers of titles are actually search engine crawlers, which truncate long titles, compensating for it with a snippet of relevant text extracted from the document. Titles are used by screen readers, and truncating them for questionable visual purposes to accommodate the aesthetics of a particular type of application would sacrifice the value of a fully descriptive title of use to assistive technologies. This "style guide" guideline is considered harmful and should not be followed.

Title content heuristics

SortSite will perform questionable heuristics on the contents of the title element.

Document title must not contain placeholder text like "Untitled" or the page filename

Whether this failure is reported or not depends on hairy details that have nothing whatsoever to do with placeholder documents. The following titles will trigger an error:

<title>
	EventTarget.addEventListener
</title>
<title>
	EventTargetaddEventListener
</title>

while the following won't:

<title>
	EventTarget addEventListener
</title>
<title>
	EventListener
</title>

This is nothing but a bogosity in the heuristic algorithm.

Placeholder text

Artificial Intelligence is a long way off yet. That is particularly true for automated accessibility checkers. SortSite will penalize innocent document for using placeholder text, such as Lorem ipsum, CHANGEME, FIXME, TODO, purely based on the presence of these bits in the text, regardless of whether they are placeholders or real content. This is obviously well-intentioned, but ultimately useless, generating false positives, while any occurrence of a stray placeholder is unlikely to end up in a public document or site. Note that no reference guideline has been provided for this experiment in AI implementation.

HTML5 reference specification

SortSite uses the static (frozen) and barely maintained W3C fork of the HTML5 specification instead of the rolling standard very actively maintained by the community that authored it in the first place (WhatWG). It is recommended that developers use the latter for their reference.

Ironically, it may be necessary to use a browser with lax security, because otherwise access will be blocked:

screenshot of Firefox protection page
Firefox blocking access to WhatWG servers due to the latter's continued usage of broken security protocols

Iframe fallback

SortSite will fail documents with plaintext iframe fallback (enclosed between tags), with the following bogus error:.

Invalid data or text not allowed

No doctype

SortSite skips validation for HTML documents without doctype, saving resources — and rightly so, since they don't claim to be valid, and it's enough to flag these documents as needing work.

Tabindex

SortSite is too eager to fail documents using tabindex to mark up non-consecutive tabbing order:

The tab order does not follow logical sequences on the page.

When the values of the TABINDEX attribute are assigned in a different order than the relationships and sequences in the content, the tab order no longer follows the relationships and sequences in the content."

It will do so even for documents which were specially crafted to showcase the correct usage of tabindex which does follow the relationships in the content. This is a source of false positives.

ARIA-(controls | describedby | flowto | labelledby | owns)

SortSite has bugs in its ARIA reference implementation. The first problem concerns several attributes, which refer to another portion of the document:

SortSite throws an error for certain documents which use these attributes, with the error being identical for all aforecited attributes, so it is sufficient to showcase just one:

The "aria-controls" attribute must point to an element in the same document

SortSite reports this if the relevant documents do contain elements with an id referenced by these ARIA attributes. A sanitized snippet from an incorrectly failed document:

<div aria-controls="one two"></div>
<ul>
<li id="one"></li>
<li id="two"></li>

Furthermore, probably due to a typo, it quotes the frozen W3C HTML5 specification instead of the WAI-ARIA specification, which is actually relevant here.

ARIA-activedescendant

SortSite will incorrectly fail documents which correctly use the ARIA aria-activedescendant attribute:

Attribute aria-activedescendant value should either refer to a descendant element, or should be accompanied by attribute aria-owns".

SortSite reports this if the relevant documents do contain elements with an id referenced by the aria-activedescendant attribute and the element also has the required aria-owns attribute. A sanitized snippet from an incorrectly failed document:

<input aria-owns="pretty" aria-activedescendant="ugly">
<ul id="pretty">
<li>some text</li>
<li id="ugly">some other text</li>

Scripts

SortSite is the only automated accessibility tool which actually parses scripts, checking them for accessibility failures. That is incredibly impressive. An example, pointing to a failure halfway down the script:

For SCRIPTs and APPLETs ensure that event handlers are input device-independent. Do not write event handlers that rely on mouse coordinates since this prevents device-independent input". "Add an equivalent event handler that accepts keyboard input via event keyCode.

It will also fail scripts which toggle visibility, creating a blinking document.

Hats off.

Summary

Major issues with SortSite are:

The benefits of using SortSite are: