Efficient Web Accessibility Testing — Tom Tester's report

Three pages from the Ruter# traffic site for the Oslo and Akershus counties were selected for analysis. Failures already identified for one type of page are not reported for other types of pages, so the results are cumulative and unique. Apart from clarity of presentation, the rationale behind this is that once a single instance of a failure is identified, the developer would audit the entire site code for similar failures. In addition, the main purpose of this exercise is to highlight the usefulness — or lack thereof — of selected automated accessibility checkers in finding failures.

The results of analysis performed are consistent with the detailed individual assessment of selected tools. TotalValidator is focused mostly in parser-type failures which are easiest to find and code for in automated software, whereas Wave and Firefox Accessibility Evaluator (FAE) are only of very limited use, but still valuable as tools due to their ability to identify problems hard to find for a human developer, such as contrast ratio failures, instances of duplicated text, or broken references.

Out of the total of 35 failures identified during analysis, the human evaluator found 31, or 88.6%, TotalValidator found 7, or 20%, Wave found 12, or 34.3%, while the FAE extension found 8, or 22.9%.

Ruter — Front page

The matrix of failures for the front page

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator Wave FAE
The lang attribute is incorrectly appended to the head element instead of the parent root element, resulting in the page not having a defined language, given that the Content-Language server header was not set yes yes yes yes
Empty anchors are present (including, but not limited to, the JavaScript menu trigger) yes yes yes no
Duplication: alt content is identical to the description surrounding the image — differentiated for mobile and desktop users, but still visible for text browser users and processed by screen readers, given the lack of aural styles yes no no no
Text search form controls are not marked up as such (input type=search) — instead, plain text inputs are used yes no no no
The input form controls (of text and radio type) do not have a label associated with them (e.g. search) yes yes yes no
The form submit button contains an encoded entity instead of action-descriptive text yes no no no
The abbr and elements are not used to mark up abbreviations (such as PDF) yes no no no
In order to provide for user agents without JavaScript support (or other user agents with scripting turned off), the contents of the option elements in the select drop-downs contents were duplicated in the form of unordered lists. These blocks of repetitive content are unskipable yes no no no
Fake JavaScript links are employed for transport type pickers yes no no no
The page has no real outline, or well-defined sections. Level-1 headings (h1) are used for the 'pane' content subsections below the main form, instead of heading the entire page. There are no subheadings, even though clearly there are distinct content groups present (e.g., the aforementioned 'panes'). Generic span wrappers with heading-imitating styles are used instead for presentational purposes yes no no yes
Not all distinct content is grouped in meaningful containers (such as main, article, or section). Generic layers (div) are used instead yes no no no
Redundant title text is used, identical to the raw anchor text content yes no yes no
section elements are not used to mark up blocks of independent content, but instead are employed as generic layer (div) substitutes in many places, for example for plain anchor containers (e.g. for tickets) yes no no no
The entire paragraph content is linked, whereas only the relevant parts with meaningful context phrases should have been linked instead yes no no no
Inline frames do not have an associated title yes no no no
Form controls are not grouped in fieldset elements with associated legend. Generic layers are employed instead yes yes yes no
The mine favoritter form has no submit button provided — instead, a fake JavaScript link is used: a href=javascript:void(0) yes yes no no
The cookie warning information is required, but is implemented as a fly-in pop-up, implying a change of context yes no no no
Images used in the cookie pop-up have no associated alt attribute text fallback yes no yes no
Empty form control label elements are used yes no yes no
Absolute values are used for font size (pixels) yes no no yes
Values denoting the bus/tram line numbers are not marked up (e.g. with the var element). Instead, a generic inline wrapper is used: span class=number yes no no no
target attributes with non-default values are used for anchors, creating an unexpected change of context yes no no no
The strong emphasis element is misused for presentational purposes (bold text) yes no no no
There are multiple instances of links with identical text yes yes yes yes
Non-focusable elements such as table cells and generic layers have event handlers associated with them in place of anchor children. These handlers are exclusively device-dependent (onclick only) no no yes yes
The for attribute for the label element does not correspond to any form control's id, resulting in the label being unassociated with any control no no yes yes
Insufficient contrast ratio on the page (below 7:1), including cases where it is below minimum acceptable level (3:1) no no no yes
Aural styles are not used; in particular, the speak: none declaration is not used in combination with the display: none declaration, wherever the latter is required yes no no no

Ruter — Traffic information search

The matrix of failures for the search page

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator Wave FAE
The time element is not used to mark up dates and times (including travel duration) yes no no no
Email form controls are not marked up as such (input type=email) — instead, plain text inputs are used yes no no no
Form controls are grouped in the fieldset element have no associated legend yes no no no

Ruter — Map

The matrix of failures for the map page

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator Wave FAE
The map is not accessible in user agents without JavaScript, or with scripting turned off. While this is expected for this kind of feature — access to the anchor leading to the map is not access-able either in these user agents. Further, the map is not access-able either in user agents with scripting on, but with referrer tracking turned off yes no no no
Multiple form label elements associated with a single form control no no yes no
Images with no alt attributes present, with text description provided only through the title attribute yes yes yes yes