Zero issues on zero pages is not a perfect score

At the end of October, Damdit showed a result on one of its test projects that looked like good news. The project tracked the website of a computer science department at an American university. The previous audit had found 38 issues. The latest one found none.

But nobody had fixed anything. The website just didn’t respond during the latest audit, not a single page of it. Damdit 0.7.3, a version from before the 1.0 release, still saved it as a completed audit with no findings and showed it as the current state of the website.

The bug crept in quietly. For every check, the program knew only two outcomes: it found a problem, or it didn’t. A check that never ran ended up in the second group. Zero findings on a healthy website and zero findings on a website nobody had looked at were exactly the same in the data.

A zero that isn’t a zero

If you work with web analytics, you’ve probably seen this yourself. A deployment drops the tracking code for a day, and the traffic chart shows zero, as if nobody had visited. The missing data passes for a real number. If nobody catches it, that zero ends up in the monthly report.

The same thing happened in Damdit 0.7.3. The missing result got a default value, and the default value was “fine”. Nobody designed it that way. It was simply the easiest default.

Damdit’s product spec has said from the start that missing evidence is never a clean result. The university website showed that writing the rule down isn’t enough. It has to hold everywhere an audit result goes next, and a false zero would have gone to three places.

Where a false zero would end up

First, the current state of the website. Today, Damdit doesn’t treat an audit in which no page responded as current, as long as there’s an earlier audit that actually measured something. You’ll still find it in History, with the note “Nothing could be measured: no page of the website answered.”, but the current state keeps showing the last real measurement.

Second, the score. Here the same bug had done damage before, just less visibly. In October, axe-core, the accessibility testing engine, failed on every page and so found no problems at all. The accessibility area looked flawless, and the overall score jumped by 11 to 15 points. Today a single comment in the scoring code sums it up: zero findings on zero measured pages is not 100.

So an area that measured nothing now gets no score. It’s marked Not measured, along with the reason it couldn’t be measured. If it’s one of the four core areas, URL Health, Indexability, SEO or Accessibility, Damdit doesn’t calculate an overall score either. The same goes for an audit in which at least as many pages failed as were measured. An area that was only partly measured does get a score, but always with the number of pages behind it: “Measured on X of Y pages”.

Third, the comparison between two audits. This is the most dangerous place, because the result of a comparison gets passed on, to a client or a developer. If Damdit had compared the empty university audit with the one before it, it would have reported 38 fixed issues.

Today, Damdit marks an issue as Resolved only when the new audit checked the pages where it was last found again, and they passed. It matches pages across redirects, too. If the new audit didn’t check them again, say because they didn’t respond, didn’t fit within the page limit or were blocked by bot protection, the issue stays open and the change is marked Inconclusive, with the note: “The later audit did not check again the page where it was found, so a fix is not confirmed.”

The trade-off that’s left

There’s one place this rule doesn’t fully cover, and I’ve left it visible on purpose. The score measures a single audit and counts only what that audit measured. When a new audit skips a page with an issue, the issue stays open, but without that page the score can go up.

One option was to carry unverified findings from the previous audit into the score. But then one number would mix two measurements from different days, and nobody could say what it actually means. So the gap shows up in Coverage and in the issue’s status, not in the score.

That means Damdit sometimes gives you an audit with no overall score, or keeps an issue open that a developer has already fixed. It’s less convenient. In return, every number in the report can be traced back to what was actually measured.

How to check it yourself

If you use Damdit, open the Coverage tab of your latest audit before the list of issues. You’ll see how many of the planned pages the audit actually measured and which areas are Not measured or Limited.

On the command line, damdit summary <project> shows the same. For a run that ended without a result, it also prints the reason the run recorded. damdit compare <project> lists the changes since the previous audit the same way History does, Inconclusive included.

A zero needs evidence

The university website still has that empty audit from version 0.7.3 in History. Nobody overwrote it. It just no longer pretends to be the current state. The current state is the last audit that actually measured the website, and it has 38 issues.

That empty audit shows an asymmetry. A finding is easy to prove: Damdit shows you the page, the element and what it found there. But “there’s no problem here” is a claim too, and it needs the same kind of evidence, a record that the page was actually checked. Without that, a zero tells you nothing.

That goes for any audit, not just Damdit’s. Before you trust a clean result, whatever tool it comes from, find out how many pages it’s based on and what was measured on them. If the report doesn’t say, you don’t have a clean result. You just have a zero.