Efficient Web Accessibility Testing — Firefox Accessibility Evaluator


Versions tested: various; latest:

The tool supports web developers in testing their web resources for functional accessibility features based on the iCITA HTML Best Practices.

See also the documentation and their bugtracker.

Mode of operation

The Firefox Accessibility Evaluator — henceforth abbreviated as FAE — is available in two different forms:

There are two sets of rules — FAE 1.0 and beta. The web service offers a batch mode, and requires registration. Unfortunately, it is not usable in its present state.

Batch mode
Potential users of this add-on/server combo should be aware that every resource is submitted to iCITA servers for processing (apart from registration). FAE in batch mode cannot be used anonymously. FAE should not be used for intranet projects, internal development, or anything that is of value to the company. In addition, processing of the document is accompanied during an automatic crawl of the document by an automated spider.
The results table is not intended for human consumption, featuring many cryptic abbreviations and shortcuts. It is more of a crawler dump than a presentation of results. There are no pointers whatever to help understand what is wrong where. As such, this mode can be only used for identifying/flagging the documents which allegedly need work, and then processing them one by one either manually, or in single-document mode.
Single-document mode
FAE is useful for on-the-fly development of new documents. There is, however, no way to automatically start FAE for a newly loaded page.

The single-document mode is a multi-window mode, which is decidedly unhelpful, and of which in the next section. From this point on, all comments refer to single-document mode.

Error reporting

The primary mode of operation for FAE is through multiple windows. Processing is slow, and can take up to 2 minutes for a 70-kilobyte page. Since there are two different, partially overlapping, rule sets (the previously mentioned FAE and beta), each document requires two separate testing runs. The documentation suggests that the first rule set, namely the default FAE, will be obsoleted in the future, and should have been obsoleted by Fall 2014, a deadline which seems to have passed without this option disappearing from the interface. The FAE set is notably verbose, with a much larger number of alleged errors being reported. Depreciation notwithstanding, both rule sets were tested for the purpose of this project.

The extension installs a toolbar, which is present and always on, even in the chrome-less mode such as the Pentadactyl interface for Firefox. That is good, as this way the extension does not require modification of existing software to be used. The toolbar hosts a collection of various tools, most outsourced, and in that is a fully featured developer tool set, of which the accessibility checkers are only a small part. Running either rule set opens a new window, which is very unhelpful, as it disrupts the flow and changes the context unnecessarily. Due to the slowness of operation, this window may pop up in a different workspace, or tag set in a tiled window manager. The first window that appears contains a text-unselectable list of text widgets with short issue summaries. They will be cut off depending on the window width. On a standard 21-inch monitor, this requires horizontal tiling in a wiling window manager, and window maximization otherwise.

This window contains executable buttons. The rationale button, depending on the violation chosen — will point to the iCITA documentation, which in turn contains detailed explanations relevant to the problem at hand, and usually (but not always), a pointer to either WCAG, Section 508 specification, or their own iCITA rules used as the basis for evaluation. The summary button opens yet another window with a full description of the problems, with problem count and selectable text. Often, however, even that is not enough to find the source of the problem, especially for larger pages, with result inspection requiring the execution of yet another button, the dump button, which contains a raw data dump, as promised, complete with code snippets. The summary and dump windows are auto-generated HTML pages, which are unstyled, or styled through the built-in chrome:// protocol (chrome://accessext/content/Template.html), and thus prone to user stylesheet conflict, unless a chrome-specific stylesheet is added. The dump is not quite ready for human consumption either, containing bug indicators such as this:

Abbreviations (none) undefinedundefined

Switching between all these windows is cumbersome. There is an interface problem if the two primary windows are not enough to pinpoint the issue, while the raw dump contains everything, requiring the developer to comb through mountains of data to find the problem reported in the other windows.

Server headers — language

One of the most prodigious source of false positives in FAE is due to its failure to process certain server headers, which are the most efficient way to declare encoding, language, content type, navigation structure of auxiliary documents, and even stylesheets. An example of the popular Apache server configuration to achieve this is:

AddLanguage no .html

This can be overridden anywhere down in the URL hierarchy, or from the top configuration file, for example through the <Files> directive. Server headers are trivially checked from within the graphical browser, see the examples from Firefox/Pentadactyl:

server response, Norwegian version server response, English version
Server responses in a graphical browser

The same information can be trivially obtained from the command line, using a text browser:

% lynx -head -dump http://unicus.no/ | grep Language
Content-Language: no
% lynx -head -dump http://unicus.no/en/ | grep Language
Content-Language: en

FAE will complain that the language is not set for the document, with the following error:

Each page must have a lang attribute on its html element whose value or initial sub-tag is a valid two-character language code

That is poor advice even for small servers, let alone huge sites with thousands of pages. Using the lang attribute makes sense for standalone documents (such as testcases), and even then only for those which are shared through other means. Documents servers with a valid server header should never fail validation:

Server headers — content type and character encoding

The next source of false positives is very similar, a continuation of the theme. An efficient, and indeed in the case of XHTML the only correct way to define content type is the server header. The Apache server configuration

AddType 'application/xhtml+xml; charset=utf-8' .xxx
AddType 'text/html; charset=utf-8' .html

passes the proper encoding to the client:

% lynx -head -dump http://unicus.no/en/ | grep Content-Type
Content-Type: text/html; charset=utf-8
server response, XHTML
Server response in a graphical browser

Stylesheets can be added to documents via link elements, but links need not reside within the documents themselves — they can be set on the server. Here is an example of a proper Apache server configuration:

<Files ~ "index\.(html)$">
Header add Link '</css/define.css>; rel="stylesheet"'

This directive injects a stylesheet link to all documents with the html extension. It may be used as a global stylesheet for the entire site, as a base stylesheet to be overridden by document-specific stylesheets, or as a way to neatly define CSS Variables, on its own a useful accessibility feature of cascading stylesheets. It is unfortunately not mentioned in the WCAG 2.0 standard, which it post-dates.

Firefox as such has no problem with server headers, and indeed is the only browser which honors the link server headers. Given this, and given that the Unicode-encoded pages display Unicode characters in Firefox, indicating that the headers were parsed and interpreted correctly — it is surprising that despite all this FAE will still insist that pages have incorrect encoding. This is purely a failure of FAE, and not of Firefox, due to the absence of the spurious inline meta element.

Server headers — site navigation links

Other link types can also be specified on the server, such as the site map, copyright notice, newsfeed, and many more:

<Files ~ "index\.(html)$">
Header add Link '</sitemap/>; rel=contents'
Header add Link '</index/>; rel=index'
Header add Link '</legal/>; rel=copyright'
Header add Link '</atom.xml>; rel=alternate; type=application/atom+xml;
	title="site updates newsfeed"'

These links affect accessibility in more ways than one. Some applications populate a separate navigation bar with these links, where they may appear as text and/or icon buttons. Others may add it to the text menu. Some more advanced search engine robots process server headers and may present information extracted thus in the site overview. It is also recommended that newsfeeds are appended to the documents through such a link.

FAE does not use this header information. Instead, even if contents and or index header links are specified, it complains about a missing navigation bar. See also the next section.

Navigation bar

A navigation bar may be created using links (not to be confused with anchors/hyperlinks commonly referred to as such, i.e. the a element) — either through server headers, or through link elements in the head of the document. This navigation bar, if supported, is usually implemented as an extra browser toolbar in visual GUI browsers. As explained in the previous section, FAE does not make use of these links. Almost every page in the universe triggers the following error:

Most web pages should contain at least one navigation bar.

It is unclear what this means, and what most means exactly. FAE does not support the HTML5 menu and nav elements properly, while there is no formal requirement that such elements be present on every site, let alone on every page on said site. This is reported even if there are ul or ol list elements with a site or sub-site navigation.

Continuing with the navigation bar theme, another error accompanies the above:

Each nested table, map with area, ol or ul element that appears to be a navigation bar or menu should be immediately preceded by a header element (h2..h6 -or- role=heading).

This clarifies what FAE may refer to as a navigation bar. It is a visual term. Both this and the previous error stem from the fact that iCITA issued their own mandatory requirements which are present nowhere else (and arguably do not make sense). Valid, completely accessible pages may contain no navigation "bars" of the type mentioned above. Even if present, they may be coded through aforementioned HTML5 elements. Pages may be linked sequentially, and there may be a site index. It all depends on site architecture, and there is no absolute requirement to encode navigation in any particular form.

Finally, there is no accessibility requirement to use a header element immediately before a navigation structure. Again, this can be solved in a number of ways, and only some navigation lists and constructs will necessitate such header elements. Automated warnings are a source of false positives for a vast majority of documents, and that is decidedly unhelpful and frankly counter-productive.

Title versus h1

FAE attempts to be strict in unexpected places. Reports for nearly all documents will include spam in the form of alleged discrepancy between the content of the title element and the h1 element:

The words contained in each h1 element should match a subset of the words contained in the title element. Words (%1) in h1 elements should also be in the title element.

This error is issued for pages with no title element. It is issued for pages where the content of these two elements overlap to some extent, i.e. they contains some of the same words, but not all. This is not a formal requirement, and is not useful unless reported for completely different contents of these elements.

Site name in title

FAE imposes uniquely strict (and misguided) requirements for the title element:

Title should indicate both name of the site and topic of the page.

It is clear that the authors of the iCITA specification super-set have a unique type of documents in mind, reflecting the unripe fads of decades past.

Header length

Continuing with the theme of unnecessarily strict and incorrectly implemented requirements, there is another source of failures which is best ignored — header length.

Heading content should be concise (usually 65 or fewer characters in length).

This is happily left for the reader to ascertain, flagged as a check rather than a fail. It is issued for header elements (h1 et alia rather than the header element from HTML5) which are shorter in length, even including whitespace (though not including descendant tags). Unhelpfully, the rationale button doesn't work for this particular guideline, so it is not immediately clear where the inspiration for this comes from. While such a check, or even a stricter flag, is useful to have for sites which abuse the header elements for presentational purposes and/or use them as block containers for beefier chunks of text and images — it is not helpful to have this reported for a very modest number such as 65 characters. It is debatable also whether this number should include whitespace.

The (undisclosed) rationale for this guideline check is the antiquated W3C Style Guide. To wit:

The title should ideally be less than 64 characters in length. That is, many applications will display document titles in window titles, menus, etc where there is only limited room. Whilst there is no limit on the length of a title (as it may be automatically generated from other data), information providers are warned that it may be truncated if long.

Using window titles in a graphical browser as a justification is misguided, because there are many types of applications and environments (let alone resolutions), not necessarily featuring this particular widget. The primary consumers of titles are actually search engine crawlers, which truncate long titles, compensating for it with a snippet of relevant text extracted from the document. Titles are used by screen readers, and truncating them for questionable visual purposes to accommodate the aesthetics of a particular type of application would sacrifice the value of a fully descriptive title of use to assistive technologies. This "style guide" guideline is considered harmful and should not be followed.

Linked images and figures

FAE insists that linked images are wrong, and should not be used. That stands in contrast with WCAG F89 guideline, which recommends that linked images provide at least an alt text, and optionally text node sibling of the image within an anchor.

In addition, FAE does not have support for figure and figcaption elements beyond the parser. These elements were added to the HTML5 specification precisely to address the issue of long descriptions for images, including linked images.

<figcaption>sufficient description here</figcaption>
<a href="/path/to/image">
<img src="/path/to/thumbnail" alt="image description">
--- more optional text ---

Color contrast

Like the Google Accessibility Developer Tools and Wave, FAE offers a built-in color contrast checker, based on the same codebase. Section 1.4.3 of the WCAG specification mandates that depending on the text size, sufficiently high contrast ratio of the text and background must be provided in order to make text readable. Quite sensibly, the specification is less stringent in the case of very large text, in which case the letter shapes are much more discernible regardless of color. The highest contrast is that of black text on white background, or vice versa, while the lowest contrast is that of white text on white background — or more generally, when the foreground and background colors are identical. WCAG specifies two thresholds, depending on the text size. In our case, the minimum value is 4.5 to 1.

The FAE contrast checker incorrectly reports very low contrast in certain cases:

a {
  color: hsl(232,10%,95%);
  background: linear-gradient(hsl(232,40%,22%),hsl(232,40%,38%));

This pair of foreground and background colors satisfies the WCAG minimum contrast requirement, with plenty of room to spare. In fact, FAE would report a failure even the highest contrast possible was used:

a {
  color: white;
  background: linear-gradient(black,black);
a {
  color: white;
  background: black linear-gradient(black,black);

White on black is reported as wrong not because the constituent colors are wrong, but because FAE does not have support for the linear-gradient or radial-gradient properties. This failure is not due to an incorrect heuristic as such — the colors are processed correctly, but the incomplete implementation of the CSS specification results in the entire declaration being dropped, which in turn implies that FAE is left with an incorrect pair of colors during contrast evaluation.

Link text

FAE checks for link text, analyzing it for sloppy content. These reports are very useful (what other checkers may label as suspicious):

Ensure that links that point to the same href use the same link text.

Ensure that links that point to different hrefs use different link text.

Avoid using text links that are not understandable out of context.

while the following isn't really:

Avoid using text links that are shorter than four 4 characters in length.

Again, this comes from a custom super-set, which is not enforceable, and makes little sense if applied absolutely. There are 4-letter valid, meaningful words, and there are many even shorter, valid, meaningful words. Note also that not all languages are verbose (let alone poly-synthetic), and what may constitute an undue word length is language-specific.

CSS — font-size

Due to an obvious bug, FAE misinterprets pixel widths for block elements as pixel font sizes.

div {
  font-size: 150%;
  width: 1000px;

This warning is issued for all instances of pixel widths.

Set font sizes using relative measurements (not px, pt, in, cm, mm)

Thankfully, it is a check rather than an error, but it is still a source of overly numerous false positives.

Table description — summary attribute handling

The summary attribute is a sunset component in HTML, since its contents are extractable only with CSS while the need for a more versatile mechanism was long recognized. HTML5 introduces the summary element, a child of the details element. However, for educational reasons the presence of a summary attribute should not result in a simple failure, but one accompanied by a recommendation/hint to replace it either with the new element(s), and/or with a table caption, the contents of which do not require stylesheet manipulation to be presentable to both sighted and non-sighted users.

Upon encountering a table without a summary attribute, FAE throws a fatal error:

Fail: Data tables must use summary attribute to describe the content of the table or conclusions the author intends to convey through the data in the table.

No mention is made of the new HTML5 elements:

    <summary>description definition</summary>
    <p>long description content</p>

Another problem is the inherent inability of automated checkers to associate descriptions surrounding objects such as tables and images with the latter. This is a notorious problem for figures, but in this case a long description may precede the table, marked up as a paragraph. Such context detection is beyond the capability of software, even though it is encouraged in accessibility guidelines.

FAE does not stop there though. In addition to failing the document with tables due to the absence of an obsoleted attribute, it continues complaining thus:

Warn: For each data table on the page, the summary attribute content must be unique.

It doesn't make sense to first complain that something is not there, and then remind the developer that it should be unique, as if multiple tables with identical summaries were present.


Major issues with FAE:

Benefits of using FAE: