About 1,300 tests were passing, and the screen still said saved when the save had failed

Summary

I built a checking tool that runs entirely in the browser, for the reviewers of a service at my own company. This tool displayed that the input had been saved even when the automatic save failed. In the final check just before I handed the tool to the reviewers, I found this defect and fixed it. Along with the fix I added a new regression test that actually runs the JavaScript inside the assembled HTML. Nine expected behaviors are now locked in by tests.

The defect had been there since the first version of the tool. For the roughly five weeks until the fix, the repository-wide test suite, about 1,300 tests, passed in full, including in the record taken right before the fix. That is because what the tests covered was only the functions that assemble the HTML. The JavaScript that runs inside the assembled HTML did not have a single test that could detect this defect. In the body I first show the code before and after the fix. Next I explain how the defect happens and how much data is lost. I also write what I found when I looked up the name for this construct in CWE, the public catalog of defect types. After that I compare what was inside the scope of the tests and what was outside. I explain how the test that runs the generated HTML in jsdom is built, and the nine behaviors it locks in. Finally I write about the remaining defects an AI review found after the fix, and the end-to-end check in a real browser.

What you can take away

The readers I have in mind are people who build tools that run entirely in the browser and hand them to others. I cover the following three things.

• Tie the display to whether the save succeeded: I show how to say saved only when the write succeeded and switch to a warning when it fails. I also show how to write test data right after the screen opens, so that whether saving works at all is confirmed first
• Check the scope your own tests cover: I explain how to judge whether your tests cover only the functions that assemble HTML, and how to build a test that runs the generated HTML in jsdom
• Learn what came out of a search for the name in primary sources: I present what I found in CWE when I looked for an established name for the defect of reporting a failure as a success, and where the related entries sit

What this article covers applies as it stands beyond this particular tool, to any tool that accumulates input in the browser’s localStorage and collects it later.

This article is a discussion based on primary records from operating a service I develop at my own company.