Which Tests Are Worth Keeping

In late September, Peter Steinberger wrote that OpenClaw had deleted around 400k lines of its own tests without much change in code coverage. Most of the code I ship now is written with Claude Code, and so are most of the tests. I had been watching my suites grow and had no good answer to a plain question: how many of these tests would catch a real bug? So I measured it. I read every test suite in 17 of my repos, scored each test case, ran mutation testing on four of them, and cut tests in two of them under a gate that would stop me if the cut weakened the suite. This post is what I found and the method I now use to decide whether a test stays.

Two of the repos are mine to name: Biblical Battle Plans and this site. The other fifteen are client projects, and they appear here only by what kind of app they are. No client names, file paths, or code from those repos appear in this post.

The numbers

Across the 17 repos there are 1,831 test files holding 26,017 test cases: 443,630 lines of test code against 647,543 lines of production code. That is 0.685 lines of test for every line of production code. The spread is wide. The largest repo, a league-management SaaS, has 1,149 test files and more test code than production code (1.092 to 1). A charter-operations app has 1,516 lines of tests for 51,545 lines of production code.

Biblical Battle Plans has 142 test files and 1,901 cases, at 0.399 lines of test per line of production code.

How much of it is AI-written. I can only give a lower bound. A commit counts as marked when it carries an AI co-author trailer or a Claude Code session link, directly or through the squash-merged PR it came from. Plenty of AI-written commits carry neither: squash merges dropped the trailers for a stretch, and some client repos ban attribution in commits outright. By that measure, at least 82.1% of the test lines in Biblical Battle Plans and at least 73.7% in the league SaaS are marked. Nine of the 17 repos are at least half marked, two are fully marked, and five show zero, which says more about their commit habits than about who wrote them. There is no human baseline here. None of these repos has a body of tests that I know a person wrote by hand, so I cannot compare the two. The split below is marked against unmarked, and that is all it is.

What a judge said about every test. I had Claude Sonnet 5.5 read every one of the 26,017 cases, one request per test file, with the production code each test imports and the titles of its sibling tests. It asked the five questions further down and gave each case a verdict. The whole run cost $48.59.

Merge means the test protects real behavior but repeats another test or copies out a boundary case that belongs in a table. Delete means it protects nothing another test does not already protect, and it has a known smell. Together those are about 14% of the cases. Marked tests were somewhat more likely to land in merge (13.6% against 11.0%) and delete (1.3% against 0.7%) than unmarked ones.

These verdicts are unaudited. The plan was for me to hand-check a sample of 100 and report how often I agreed with the judge. After a few rows I stopped. Most of these tests were written by AI, and at that volume I could not tell their value apart any better than the judge could. Copying its answers into my column would have reported perfect agreement for an audit that did not happen, so there is no agreement figure. I think that is a finding in its own right: I own these suites, and I could not audit them faster or better than a model could.

What the low-value tests looked like

The judge tagged each case with the smells it found. A case can carry more than one, and 18,843 cases carried none.

• Duplicate (2,413 cases): the same contract is already proven by another test, often one layer up or down. A route test that repeats the service test underneath it is the common shape.
• Unparameterized (1,842): five copies of one test that differ only in the input, where one table test would say the same thing in a third of the lines.
• Content pin (1,776): asserts on copy, a design token, or a label. The test fails whenever the words change on purpose.
• Implementation detail (868): checks call order, a class name, or the order of a query against a fake.
• Mock echo (856): asserts on what its own mock returned, or uses a fake database that ignores the query it was given.
• Source grep (263): reads a source file as text and runs a regex over it.
• Tautology (238): asserts a constant, or a value the code under test generated for itself.
• Schema mirror (115): checks that a table has every column it was defined with.
• Framework (70): tests the random number generator, a hash library, or the language itself.
• Test-only hook (69): needs a production export that exists only so a test can reach in.
• Dead spec (25): a spec that no runner ever picks up, or one aimed at a deployed URL.
• Throwaway target (4): tests prototype code.

What coverage said, and what mutation testing said

Coverage tells you which lines ran during the tests. It does not tell you whether any test checked what those lines did. Mutation testing asks that second question. I used Stryker (opens in a new tab), which does it for JavaScript and TypeScript.

A mutant is a copy of the production code with one small change in it. Stryker makes them from a fixed list of changes: flip a comparison, swap && for ||, replace a condition with true, empty a function body, return early. One mutant looks like this:

- if (attempts > limit) return locked();
+ if (attempts >= limit) return locked();

Killed or survived. For each mutant, Stryker runs the tests that reach the changed line. If any of them fails, the mutant is killed: some test noticed. If they all pass, it survived. A mutant on a line no test runs is counted as no coverage.

The score is the share of mutants that were caught: killed mutants (and the few that hung the tests until they timed out) divided by every mutant, uncovered ones included. The score on covered code leaves the uncovered ones out, so it measures only how well the tests check the lines they do run. The two can be far apart. The charter-operations app below scores 10.95% overall because most of its code is never run, and 73.81% on the code its tests do reach.

Why a covered line can go unchecked. A test can run a line and never look at what it did. It might assert only that the function returned something, or hand the code a fake that ignores its input. The line counts toward coverage either way. A surviving mutant on a covered line is one of those.

What it costs. Every mutant means another run of the tests that reach it. Whole repos would not fit in the three hours I allowed each run, so I mutated a sample of production files in each repo, picked in a fixed order so the run can be repeated.

The kill matrix. I set Stryker to record coverage per test and to keep running a mutant's tests after the first failure. Its report then lists, for every mutant, each test that ran it and each test that killed it. Turned around, that gives every test case a list of the mutants it kills, and which of those no other test kills. A test with a unique kill stays, whatever else is wrong with it. That is the keep rule further down, and the same report is the pilots' gate.

I ran Stryker on sampled code in four repos:

In files the tests cover completely, 223 mutants still survived across the four repos (85, 40, 25 and 73).

The clearest case was in the league SaaS. A function that reads a record by id, filtered to the record's owner, showed full line coverage. Its tests used a fake database that ignored the filter. Stryker has no mutator that drops one argument of a call, so the filter got no mutant of its own. I wrote three by hand: one that keeps only the record id in the where clause, so any user can read any owner's record; one that compares the owner column to the record id instead of the caller; and one that removes the where clause, so the first record in the table comes back. All three survived every one of the 746 related tests. That is an authorization hole that coverage reported as fully tested.

Coverage also says very little about which tests you can lose. In the league SaaS, a greedy set cover found that 546 of its 1,141 test files could go while line coverage stayed within 2%. If I had used coverage as the gate, I could have cut nearly half the files.

The judge and the mutants did not always agree. In the league SaaS sample, 285 tests the judge marked keep killed no mutant at all, and in Biblical Battle Plans two tests the judge marked delete each killed a mutant no other test did. That is why the pilots below are gated on mutation score instead of on the judge alone.

I built a small study of the ownership case: Coverage and mutants (opens in a new tab). It is a short ownership check with three tests and full line coverage. Turn on mutants and watch the coverage meter stay at 100% while the bugs get through, then add the one test that checks the query.

The bigger problem: tests that never run, or never reach the bug

Low-value tests cost time and attention. The audit turned up a larger problem in tests that are missing or never run.

Only 6 of the 17 repos ran their tests in CI in the last 30 days. Some had CI switched off, some had workflows that failed before running a single step, and some never had CI. The league SaaS is the exception that holds up: its tests do not run in CI, but its pre-commit hook runs the full suite with coverage on every commit.

That led to a fifth question the judge could not answer per test: did a bug ship in code these tests do not reach? To answer it, I took every fix commit in each repo's history, kept the ones that touched production files still in the code, and checked whether any test imports those files. Of 438 such fix commits, 70 landed in code no test reaches. By class:

The charter-operations app is the sharpest example. All of its tests exercise pure helper functions. They have not run in CI in the last 30 days, and 22 of its 24 fix commits landed in code those tests never touch: two security and access fixes, two data loss fixes, four in dates and time, five in layout, and the rest elsewhere. The judge marked 95 of its 131 tests keep. They test code that was not where the bugs were.

My own app has the same problem in a milder form. In Biblical Battle Plans, 15 of 50 fix commits landed in code no test reaches, eight of them layout and mobile fixes.

The pilots

I ran the method end to end on two suites: one services directory of the league SaaS, and the TypeScript suites of Biblical Battle Plans (its Swift tests were classified but left alone). In each, I removed or merged the cases the judge marked delete or merge, never touching a test on the retention bar below unless it was a pure duplicate. The gate was the mutation score on the same set of mutants, measured before and after. If it dropped, the cases that held the newly surviving mutants went back in.

The league SaaS lost 563 cases and 6,225 lines of test code and caught exactly the same mutants. In Biblical Battle Plans, the first cut dropped the mutation score to 57.59%. The gate pointed at three cases that had been killing mutants nothing else killed. I put them back, and the score returned to 58.18%. Coverage barely moved in either repo, which is what you would expect from removing redundant tests, and it is why coverage alone could not have caught those three.

Both pilots are merged.

The method

This is how I now decide whether a test is worth keeping. I ask five questions of each test.

1. What observable behavior, invariant, or contract does this test protect?
2. What credible regression would make it fail?
3. Is that failure already caught by another test, and which one owns it at the strongest boundary?
4. Does it need production code that exists only to support the test?
5. Did a bug ship in code this suite does not reach? This one is asked of an area of the code, by mapping fix commits to files no test imports.

The first four come from OpenClaw's test-audit skill. The fifth is mine, and it is the one that found the most.

Each test then gets one of four verdicts:

• Keep: it protects a contract, or it kills at least one mutant no other test kills.
• Merge: it tests real behavior but repeats another test, or it is one of several boundary cases that belong in one table test.
• Fix: it has a smell but still kills mutants nothing else does.
• Delete: no unique mutant kills, no unique coverage, at least one smell, and it is not on the retention bar.

The retention bar comes from OpenClaw's audit notes. Tests in these areas stay regardless of smells: authentication and authorization, money, security and cryptography, migrations and storage, protocol and API contracts, and data isolation between tenants or users.

You can try the questions yourself in Test triage (opens in a new tab). It shows sixteen real tests from the audit, renamed so no client shows. You answer the questions, pick a verdict, and then see what the judge said and why.

What OpenClaw actually did

The OpenClaw story is often retold as a target: cut 20% of the tests and keep coverage within 2%. The primary sources say something more careful.

• The first post (opens in a new tab) says OpenClaw deleted around 400k lines of tests without much change in coverage. The 20% and 2% target comes from his own reply (opens in a new tab) two minutes later. The skill (opens in a new tab) itself says never to chase deletion counts. Its four questions are for writing new tests, and deleting old ones goes through a separate audit with a preservation check that mutates the production code once and confirms the kept test goes red.
• No before-and-after coverage figure is published anywhere I could find, and a later batch (opens in a new tab) says coverage was not measured.
• The numbers kept growing. By mid-October (opens in a new tab) he wrote of about 1.2 million lines removed, and the 685 merged cleanup PRs through late October add up to about 3 million lines net. The 400k figure cannot be reproduced from the PR record.
• He also wrote that some of the deleted tests had value (opens in a new tab). One campaign, the Telegram cleanup (opens in a new tab), cut its test code from 111,122 lines to 64,651 and turned up real delivery bugs along the way.

None of that makes the work wrong. What I took from it is the questions and the preservation check.

What I changed in my own repos

This site has no unit test suite. Its checks are a design sweep that runs in PR CI, seven build-time asserts, a sweep of the story map's layouts, and 29 committed browser cases. None of that touched the contact form's API route, the design lab's inquiry checks, or the proxy that routes the lab's subdomain. Those now have route checks in a script that runs in PR CI, and each check was proven by breaking the code it guards and watching it fail.

In Biblical Battle Plans, the suites that run against a real Postgres database were skipped in CI because nothing set up a database for them. They now have one. Its CI is switched off for the moment, so they passed in a local run against the same Postgres image instead. That change is merged.

The fifth question is the one I will keep asking. The judge found about one test in seven worth merging or deleting, and the pilots showed the cut can be made without losing a single mutant kill. Seventy shipped fixes landed in code no test reached, and that is where the next tests in these repos should go.