Efficient Web Accessibility Testing — Tom Tester's report

Three pages from the Norwegian Air Shuttle airline site were selected for analysis. Failures already identified for one type of page are not reported for other types of pages, so the results are cumulative and unique. Apart from clarity of presentation, the rationale behind this is that once a single instance of a failure is identified, the developer would audit the entire site code for similar failures. In addition, the main purpose of this exercise is to highlight the usefulness — or lack thereof — of selected automated accessibility checkers in finding failures.

The results of analyses performed are consistent with the detailed individual assessment of selected tools. TotalValidator is focused mostly on parser-type failures which are easiest to find, whereas Wave and Firefox Accessibility Evaluator (FAE) are only of very limited use, but still valuable as tools due to their ability to identify problems which are hard to find for a human developer, such as contrast ratio failures, instances of duplicated text, or broken references. The former tool turns out to be incapable of processing large, 'street html' pages like from Norwegian, see below. As mentioned in the individual report, it simply crashes.

Out of the total of 102 failures identified during the analysis, the human evaluator found 101, or 99%, TotalValidator found 11, or 10.8%, Wave found 2, or 1.9%, while the FAE extension found 17, or 16.7%.

Norwegian Air Shuttle — Front page

The matrix of failures for the front page

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator Wave FAE
The entire page is enclosed in a form, including non-interactive, non-harvested, unrelated content yes no crashed no
Absolute values are used for font size (pixels) yes no crashed yes
There is no matching background specified for elements which have foreground color explicitly specified (e.g. headings, inline content, form controls) yes no crashed no
The page has no real outline, or well-defined sections. There are two level-1 headings but no subheadings, even though there are distinct content groups present (also marked up with generic layers, e.g. div class=headerbox or div class=sectionboxsquaredboxes) yes no crashed yes
Distinct content is not grouped in meaningful containers (such as main, article, or section). Generic layers (div) are used instead, including wrappers for hidden input controls yes no crashed yes
Tables are used for layout, nested to multiple levels, with cells interspersed with form controls and unrelated content. None of the layout tables has ARIA roles assigned to them. yes no crashed no
Tables have no description, whether in the form of captions, summaries, a details element, or descriptive text in any way associated with them yes no crashed no
In addition to tables used for layout, the form is sprinkled with meaningless layers designed to imitate table layout (div class="row") yes no crashed no
Device-dependent handlers are used in an excluding fashion — onmousedown without associated onkeypress, e.g. for area elements yes no crashed yes
Image maps are used for gratuities reasons, utilizing images of text (holiday offers) yes yes crashed yes
The form select control has options populated by a script, rendering the form unusable without JavaScript support (e.g. text browsers) yes no crashed yes
Lists are used for navigation without description, preceding heading, or meaningful markup (e.g. the nav element), buried deep in a nested layout table yes no crashed yes
Navigation lists and menus are not provided with the means to skip them. yes no crashed no
select form controls do not have a label associated with them (e.g. the language selector) yes yes crashed no
Form controls are not grouped in fieldset elements with associated legend yes yes crashed no
Invalid values are used for options (value="no;") yes no crashed no
The names of languages in the language selector are presented in their own language, but these foreign words are not marked up with a lang attribute yes no crashed no
The level-1 heading element is misused as a form legend yes no crashed no
Labels are not used to provide description for form controls — instead, generic div class=label wrappers are used yes yes crashed no
Plain text in a table cell is used in place of a legend element yes no crashed no
Punctuation is abused for presentation in form controls (<option value="">- Velg utreisested - </option>) yes no crashed no
The optgroup elements are not used to mark up groups of options — instead, punctuation within the option element is abused for presentational purposes. Examples:
<option value="">------------------------------ </option> or
<option value=""> --- Russland --- </option> 		yes no crashed no
The abbr and var elements are not used to mark up abbreviations and codes, in particular the airport codes yes no crashed no
Empty abbr elements are used instead, visually hidden with the display:none inline styles yes no crashed no
Empty anchors are present yes yes crashed no
Generic layers are used as plain text containers after the first group of form controls — layers which contain pre-selected text (a destination) yes no crashed no
Fake JavaScript links are used: a href=javascript:void(0) yes no crashed no
Presentational markup is used (e.g. b) yes no crashed no
Empty, child-less list containers are employed throughout the form, with no description provided yes no crashed no
Although there is only one form element on the page (enclosing the entire page — see above), in fact there are several distinct forms present, each with a vastly different purpose — not marked up as such, and with no description provided yes no crashed no
If label elements are used, the descriptive text follows the embedded control, instead of preceding it yes no crashed no
The absence of fieldset elements for form control groups is made up for with separators, which are then improperly marked up, with a generic layer (div class=separator and div class=line) instead of the hr element yes no crashed no
Date abbreviations are used, but not marked up as such (e.g. des.) yes no crashed no
The time element is not used to mark up dates and times yes no crashed no
The distinct forms used throughout the page (see above) have no submit button provided — instead, a fake JavaScript link (see also above) is used again. At this point the form should be terminated, but isn't yes yes crashed no
In place of subheadings, a span child of a table cell is used for visual distinction — i.e. the subheadings are not marked up yes no crashed no
Email form controls are not marked up as such (input type=email) — instead, plain text inputs are used yes no crashed no
Destinations overview is coded as a selection of images, including images of text. These images have description provided only via the alt attribute, not elsewhere yes no crashed yes
Some images of text do not have the alt attribute to mark up alternative text, and only have exclusive device-dependent event handlers associated with them yes yes crashed yes
Empty table rows and cells are used for layout purposes yes no crashed yes
The "low-price calendar" is offered as an option for the first real form (not marked up as separate — see above). Within the same giant form, another nested form, specifically for the low price calendar, is included, making it completely unclear to non-sighted user which form will be executed yes no crashed no
target attributes with non-default values are used for anchors, creating an unexpected change of context yes no crashed no
CSS is used to hide text for presentational purposes (text-indent: -9999px) yes no crashed no
There are multiple instances of links with identical text, some of those having said text hidden via CSS (see immediately above) yes no crashed no
If the alt attribute is used, there is often insufficient description provided yes no crashed no
Figures and photographs do not have description associated with them, while the figure element is never used yes no crashed no
The page footer is not marked up as such (the footer element), and its contents are unskipable — instead, multiple generic presentational layers are employed yes no crashed no
Server statistics of no interest to the user are injected into the page, e.g. 
Server:NAS-PORTALB04&nbsp; EPiServer LanguageBranch:no
(www.norwegian.no) (/)
(20:31:41.58)
(CurrentUICulture: nb-NO)
(CurrentCulture: nb-NO)
(Build number: 30.0.3.11)
yes no crashed no
Numbers of unknown provenience are not marked up (e.g. Reg. design 079837) yes no crashed no
The noscript element is used for advertisements, with its contents incorrectly encoded, mangling up the elements into a stream of meaningless characters yes no crashed no
Visual presentation on anchors relies solely on color, with underlines removed via CSS yes no crashed no
Due to the use of tables for layout and due to the lack of semantic markup, the page appears as a sea of links yes no crashed no
Empty alt attributes are present yes yes crashed yes
Color values in CSS are coded incorrectly, in hexagonal form without the pound sign, resulting in the color declaration being dropped, which can lead to color mismatch for users yes no crashed no
Obsolete markup is used for presentational purposes (background attribute) yes no crashed no
Event handlers incorrectly use camel-casing, relying on error handling in the user agent to process the handler anyway yes yes crashed no
Aural styles are not used; in particular, the speak: none declaration is not used in combination with the display: none declaration, wherever the latter is required yes no crashed no
Insufficient contrast ratio on the page (below 7:1), including cases where it is below minimum acceptable level (3:1) — including form controls and menus no no crashed yes

Norwegian Air Shuttle — Flight search results

For the sake of clarity, only issues not detected previously are listed below.

The matrix of failures for the flight search results

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator wave FAE
The page outline is destroyed due to abuse of heading elements. Unlike on the front page, subheadings are employed, but are misused for presentational purposes (bold text), with h2 immediately following h1, with the text node content indicating that it should have been a paragraph instead yes no crashed no
The natural content order is disrupted — the primary form is repeatedly injected on the results page for no reason, while the forms should have been separated, and posted below the results. The current order means that upon form execution on the front page, the first thing the user will encounter on the results page is a flight search form again. yes no crashed no
Image inputs (input type=image) are used with no alt attribute, a label, or a plain text description yes no crashed no
Tabular data is mixed up with non-tabular data in a table marked up with class=layouttable yes no crashed no
Tables including tabular data do not have a caption yes yes crashed yes
Definition list not used for a list of definitions. yes no crashed no
Spacer shims used for layout (&nbsp; as the sole content of table cells) yes no crashed no
Table header elements (th) are not used to mark real cell headers. Instead, generic wrappers as cell children are used for this purpose (span class=emphasize) yes no crashed yes
Table header elements (th) are misused — employed for non-cell-header content for presentational purposes (color) yes no crashed no
There are complex tables present with non-linear structure — i.e. where the number of cells in a row or column does not always correspond to the maximum number of row or columns in a table due to the presence of colspan or rowspan attributes. Cells in these complex tables have no headers, id, axis or scope attributes used to associate cells with their headers yes no crashed yes
In complex tables (see above), the rowspan attribute is misused (rowspan=1) yes no crashed no
Non-focusable elements such as table cells and generic layers have event handlers associated with them in place of anchor children. These handlers are exclusively device-dependent (onclick only) yes no crashed yes
Radio form controls are disassociated from their text description, with both being descendants of different table cells (e.g. in the presentation of flight pricing) yes no crashed no
Table cells are misused to enclose only hidden inputs yes no crashed no
Lists are misused for only one item. yes no crashed no
Ambiguous/insufficient text used for anchors (e.g. klikk for mer info, whereas the anchor is a fake JavaScript link yes no crashed no
&nbsp; is misused as a visual delimiter within variables — e.g. thousands of currency units, where the value is not marked up yes no crashed no
Inter-form navigation (previous day, next day) is disassociated from the results table to which it applies — encoded as a table sibling layer, with no way to skip yes no crashed no
Inter-form buttons are not marked up as buttons — fake JavaScript links are used instead yes no crashed no
Hard-coded entities are used to encode brackets for presentational purposes — a combination of Unicode and CSS should be used instead yes no crashed no
The purchase form (flight reservation) is not concluded with a submit button — instead a fake JavaScript link is used instead yes no crashed no
Special needs options for the flight are presented in the wrong logical order — after the fake flight submit JavaScript link, making the non-sighted users miss the opportunity to choose from among those options before form execution. The same applies to discount options. yes no crashed no
Misleading text content is presented to users of page reader, text browser, and visual-browser users with no JavaScript or JavaScript turned off — the page is declared as being in the process of update (Oppdatering pågår...) at all times, creating confusion about the context in which the user has found themselves yes no crashed no
Explanatory hover tooltips are implemented through scripts, but the actual content (the entire help section) is always present at the end of the search results page — it is only visually hidden and impossible to skip yes no crashed no
The strong emphasis element is misused for presentational purposes (bold text) yes no crashed no
Elements with an empty title are present (title="") yes yes crashed no
Identifiers are presented as raw text in text browsers yes no crashed no

Norwegian Air Shuttle — Baggage help

For the sake of clarity, only issues not detected previously are listed below.

The matrix of failures for the baggage help

For each accessibility failure, the table indicates whether it was found by a human evaluator, TotalValidator, Wave, and FAE, respectively.
Failure Human TotalValidator Wave FAE
Line breaks (br) are misused for presentational purposes — employed as subheading children to create the effect of visual spacing yes no no no
Obsolete, empty named anchors are used to simulate a header with an id yes no no no
Unlike on the front page and the flight search results pages, subheadings are correctly used to mark up the headings of content sections, but in the wrong, non-consecutive order yes no yes yes
Form controls are not associated with a form — the hidden inputs and the language select are used outside of a form element yes no no no
The form is purposefully disabled (form onsubmit="return false;") yes no no no
Content targeting only at screen readers (but accessible also to text browser users) is present for no reason, duplicating the submit form control text (søk), appearing out of order, immediately preceding the level-1 page heading yes no no no
Text search form controls are not marked up as such (input type=search) — instead, plain text inputs are used yes no no no
Text present on the page is hidden with CSS yes no no no
There is more than one level-1 heading (h1), but no associated section or article element is used to mark up sections of independent content yes no no no
Empty paragraphs are misused for presentational purposes (simulation of margins) yes no no no
Emphasis, when used, is not marked up with either the em or the strong emphasis elements — NB yes no no no
Abbreviated unit values (e.g. kg) are not marked up with neither the var nor the abbr elements yes no no no
button elements for the feedback form are used outside the form element yes no no no
The feedback form is visually hidden for sighted users of visual browsers through CSS, while users of other types of software and users of assistive technologies have no way to skip this content, nor associate it with the other content on the page yes no no no
The ordered list element (ol) is misapplied to an unordered navigation list yes no no no
Empty form control label elements are used yes no no no
Elements with a duplicate id attribute are used, a source of confusion for screen readers yes no yes no