August 18, 2026
Five Bugs Wearing One Error Message
expect(locator).toMatchAriaSnapshot(expected) failed is the most frequent failure signature in the E2E suite I triage every morning. Over a two-month stretch I logged nine separate mornings that opened with it. Nearly every one was filed as a test problem and closed the same way: regenerate the baseline, commit, green.
That reflex is wrong more often than it’s right. Those nine failures had five distinct root causes, and regenerating the baseline is the correct fix for exactly two of them. For the other three, a regenerated baseline buys you one green run and pins the bug in place.
Why the reflex is so seductive
Aria snapshots are a genuinely good idea. Instead of asserting on a dozen brittle CSS selectors, you assert on the accessibility tree — roles, names, structure — the thing a screen reader would read out. It’s semantic, it’s readable in a diff, and it catches structural regressions that individual locator assertions sail past.
It also comes with --update-snapshots, which turns any failure into a two-second fix. And because the failure message is identical in all five cases, there’s no signal in the error itself telling you which of the five you’re looking at. The tooling makes the wrong fix cheaper than the diagnosis. So the wrong fix wins.
The five causes
1. The app genuinely changed, and added affordances
A product PR added a new column and a handful of links to a record-details view. The baseline was stale by construction — it was captured before the feature existed.
Regenerate. This is what the mechanism is for. The expected output changed once, deliberately, and the new baseline is now correct until the next deliberate change. The only discipline required is confirming the change was intentional before you bless it.
2. A feature flag flipped the rendering path
A flag flipped ON in the test environment, switching four tests from the old view to an already-shipped new one while the baselines still described the old one.
Regenerate — but only after you’ve confirmed the new state is the intended default. If the flag is still being evaluated, you don’t regenerate: you branch on the flag. Regenerating here without asking is how you delete the only remaining assertion covering the path you’re about to roll back to.
3. The snapshot pins volatile third-party data
This is where it starts going wrong. One helper captured a panel showing compliance results from an external provider — values that change every time an upstream job runs — plus availability data for an expired record. Every request in the trace returned 2xx. Nothing was broken. The markup involved was legacy server-rendered output that doesn’t live in any repo we scan, so there wasn’t even an app-side change to attribute it to.
Regenerating just resets the clock. The baseline is stale again the next time the upstream job runs. You haven’t fixed a test; you’ve scheduled the next failure. The fix is to stop asserting over the volatile region — split it out and assert it with a tolerant pattern, or exclude it and cover the value it’s meant to prove somewhere deterministic.
4. The assertion fired before an async panel settled
A helper did a flat waitForTimeout(3000) and then snapshotted a panel that fetches from a third party. At capture time the panel still read “Retrieving record decision…” with rows in an Updating state, so the in-progress tree leaked into the received snapshot — around 60 nodes against an expected 51.
The detail that stuck with me: the sibling method in the same file already polled the in-progress locator before asserting. The correct pattern existed, ten lines away, and had simply never been applied to the first check.
Regenerating this baseline pins the race. You’d be recording “sometimes it says Retrieving” as the expected output, and the test would then fail whenever the panel was fast. A fixed sleep is not a wait. Poll to a terminal state, then capture.
5. Shared fixture state drifted underneath the test
Three failures, one family — and in each, the app was innocent and so was the test.
A retention sweep ran for the first time and expired the compliance reports belonging to the long-lived E2E fixture accounts, breaking three aria-snapshot tests across three different providers simultaneously. Another: a shared fixture record moved into an opened state, so the page started rendering a minutes PDF and a bulk-download link that had never been there before. A full diff audit exonerated the only deploy in the window.
Regenerating here means your baseline now encodes whatever state the fixture happened to be in on Tuesday. The real fix is fixture isolation — a test that asserts on structure cannot share a record with anything that mutates that record’s lifecycle state.
The table we should all have on day one
| Cause | Tell | Is regeneration the fix? |
|---|---|---|
| 1. Intentional app change | An app PR in the window adds the exact nodes in the diff | Yes |
| 2. Feature flag flipped | Whole rendering path swapped; diff is wholesale, not incremental | Yes, once the default is confirmed |
| 3. Volatile third-party data | All requests 2xx; diff is values, not structure; no attributable change | No — assert tolerantly or exclude |
| 4. Async race | Diff contains in-progress text; node count inflated | No — poll to a terminal state |
| 5. Fixture drift | Several unrelated tests break at once; no deploy in the window | No — isolate the fixture |
The tell that separates the top two from the bottom three is boring and reliable: can you point at an app-side change that produced exactly this diff? If you can, the expected output changed once and the baseline should follow. If you can’t — and everything returned 2xx, and three unrelated specs broke together — you are not looking at drift. You’re looking at a test asserting over something non-deterministic, and a regenerated baseline is a fake green with a timer on it.
One of the triage notes put the durability caveat better than I could:
the baseline now pins one record as failed and another as in-progress; if either state moves again the snapshot flaps, and then the poll-to-terminal rework is the fix, not another snapshot bump.
That’s the honest version of a regeneration: not “fixed”, but “bumped, and here is the condition under which we’ve agreed to stop bumping.”
A coda: verify the merge commit, not the PR description
One morning a test failed that the previous day’s summary said had already been fixed — the snapshot refresh had supposedly landed in a specific PR. It failed anyway.
git show on the merge commit explained it. The merge contained both a commit titled “update snapshot” and a commit titled “Revert update snapshot”. The fix had landed and been rolled back inside the same PR, and the file wasn’t in the merge commit’s changed-file list at all. The same PR also reverted a locator from an inner input container back to an outer control element, restoring a pointer-events: none and turning a flaky race into a deterministic failure.
One PR, two well-intentioned reverts, two recurring failures — and a status summary that said “fixed” the whole time.
A claimed fix is not a fix until the merge commit’s file list says so. Which is the same rule as the rest of this post, pointed at the process instead of the assertion: green is a claim, not evidence. Regenerating a baseline produces green. So does a revert nobody noticed. Neither one is proof that anything works.