Why tests rot, and what it costs

A test suite that has been green for three years is not evidence of anything. It's a habit.

I went looking through one org's Apex tests because a deploy had failed on a class nobody had touched, and I wanted to know whether that was bad luck or a pattern. It was a pattern. What follows is the shape of the rot, in rough order of how much it costs to leave alone.

The fixtures aged and nobody told them

Most of the suite built its data in @TestSetup, which is the right instinct. The problem is that a fixture written against the org as it was three years ago is a frozen snapshot of assumptions, and the org kept moving.

A required field gets added. A validation rule gets tightened. A record type appears. Each one breaks the fixture, not the code — and because the fixture throws before the assertions run, what you see is a failure in a test whose subject is fine.

@TestSetup
static void setup() {
  // Written when an Account needed a name and a record type.
  // Since then: two required fields, a validation rule on active records,
  // and a lookup filter on Vendor__c that this data doesn't satisfy.
  Account acc = new Account(Name = 'Test Account', RecordTypeId = RT_ID);
  insert acc;
}

The failure message points at the fixture. The ticket gets titled "fix failing test." Somebody adds the two fields, the suite goes green, and nothing has been learned.

The silent ones are the expensive ones

The loud failures get fixed because they block a deploy. The expensive rot is the tests that still pass and stopped meaning anything.

Three species, all of which I found:

• Tests with no assertions. They call the method, the method doesn't throw, the test passes. Coverage counts them.
• Tests that assert on their own fixture. Insert three records, query three records, assert there are three. The code under test could be deleted.
• Tests that run as the deploying admin. Which is nobody. Every real caller is a portal user with a permission set, and the test proves nothing about whether they can reach the class at all.

None of these ever go red. That's the whole problem: a test that fails is doing its job badly, and a test that can't fail isn't doing a job.

What it actually costs

Not the failures. The failures are an afternoon.

The cost is that nobody trusts the suite, so nobody reads it. A red build becomes a thing you re-run rather than investigate. Deploys get scheduled around test runtime instead of confidence. And the org accumulates behavior that exists only in production, because the tests that would have documented it assert nothing.

The rest of this series. 2 · Testable by accident argues that you can get most of what TDD promises by asking one question at design time, using a class where half the methods were easy to test and half were not. 3 · Data difficulty ladder is the practical companion: a tier-by-tier map of which Salesforce objects are cheap to fake, which are expensive, and which you should stop trying to build. 4 · The permission set is a dependency takes apart the one species of rot that is also a production bug. 5 · Green is not covered is about the tests that never went red and never proved anything either. 6 · Paying down the coverage debt is the scheduling post: what order to work in, and how to get a number you could defend. Each part is collected on the series page as it's published.