A client asked why four days looked weak: how to test whether a dip is a signal before you go hunting for a bug

One enquiry, worth just under ten thousand francs, accounts for roughly a quarter of everything this client's new funnel has produced since launch. Hold that fact in your head and the rest of this post is obvious. A pipeline where a single enquiry can be a quarter of the total cannot be read four days at a time.

Which is exactly what I was asked to do. A client of mine — a B2B services business — messaged to say the last few days looked weaker than the previous fortnight, and could I check that everything was still working and the advertised products were actually available. A completely reasonable request, and the kind that quietly sets the agenda for the next two hours if you let it.

Answer the statistical question first

Before opening a single page, I worked out whether there was anything to explain. Four days, two enquiries. Against the baseline rate of the preceding two and a half weeks — about 0.7 enquiries a day — the expected count for a four-day window was 2.8. Observed was 2. The probability of seeing two or fewer purely by chance is 0.46. For leads rather than enquiries, expected 7.5, observed 4, probability 0.13.

Neither number is remotely unusual. There was no dip to diagnose. What there was instead was a preceding five-day stretch that had been exceptionally good, and a human being comparing the present against the best week he'd ever had.

Meanwhile the paid side had got better, not worse: more clicks per day, a slightly lower cost per click, a higher click-through rate, more impressions. Seven days against the previous seven, the enquiry count was up and the indicative enquiry value was up. The only thing that had happened is that the four days in the middle were quiet, and quiet is what 0.7-per-day looks like a lot of the time.

This has a name, and Deming named it

Reacting to common-cause variation as though it were a special cause is called tampering, and it is the first of Shewhart's two mistakes. Deming's line on it is blunt: taking action on a stable process in response to a single bad outcome only increases future variation and costs, the exact opposite of what you wanted. In the catapult version of the experiment, the volunteers who adjusted after every shot to hit the target produced the most variance of any group.

Marketing does this constantly. Four slow days, so pause the campaign. Pause the campaign, so the learning resets. The learning resets, so the next week is genuinely worse, which confirms the original fear. The cure is to look at the data in the context the process produces over time rather than at individual points — and with small numbers, that means doing the arithmetic before you do anything else.

As Nassim Nicholas Taleb puts it in Skin in the Game: the curse of modernity is that we are increasingly populated by a class of people who are better at explaining than understanding. A four-day dip will always find someone to explain it.

This is the sibling of something I wrote about last month, when a ratio doubled because its denominator collapsed. Same discipline, opposite direction: before you celebrate or panic, establish that the movement is real.

Then run the functional check anyway

Here is the part I'd get wrong if I only took the first lesson. "Is this a signal?" and "is this working?" are two different questions, and the answer to the first does not excuse you from the second. So I ran the check he'd asked for, and it found three real defects that had nothing to do with the quiet week.

Prices on six of nine landing pages were stale, all by the same four percent. A landing page was advertising a product at CHF 482.25; the shop charged CHF 501.55 for it. Two hundred and twenty-two of the three hundred and forty prices I checked were low. My first explanation was wrong — I assumed the shop had put prices up. It hadn't; prices hadn't moved in weeks. The shop had re-imported its entire catalogue between two dates, generating new internal IDs, seventeen new URLs and slightly higher prices, and the six affected pages had been built before that import and never rebuilt since. The three pages built afterwards were perfect. A generated page is a cache, and nobody had set its expiry.

Eleven product links returned a hard 404, with another six surviving only via a redirect — the same catalogue re-import, same cause, and a direct descendant of nobody owning the destination URL.

The contact form had produced nothing for ten days, against a year-long history of one every two to four days. Everything testable checked out, which makes it the most interesting of the three and the only one still open.

Everything he actually asked about — product availability — was fine. Every live product in stock, exactly one item not orderable, and it wasn't linked from any landing page. The question he asked had a clean answer; the questions he didn't ask had three dirty ones.

What I'd keep

Two habits came out of this. First, the significance test is now the opening move on any "this looks weak" request, and it costs about five minutes. With a funnel this lumpy, a four-day window is almost never interpretable, and saying so with a number attached is far more useful to a client than a reassuring sentence. It also protects the work: every time you act on noise, you spend credibility you'll want later for a real finding.

Second, the price drift is the one that goes in the monitoring. Any page generated from a source needs a rebuild trigger tied to that source changing — and until it has one, a weekly diff between what the page claims and what the basket charges. An ad quoting a price four percent below the checkout isn't a data-quality issue. It's the first thing a new customer learns about whether you can be trusted, and it had been true for weeks.

Test whether the movement is real. Then go looking anyway. The bugs are rarely where the worry is pointing.

Sources & further reading

External
The W. Edwards Deming Institute — impact of process tampering on variation
Deming Alliance — understanding variation, the springboard for process improvement
Lean Blog — stop reacting to red-light metrics

Related posts
Our ROAS doubled. We hadn't sold a thing more.
I built a client performance dashboard in a day — the hard part was what to call the number
Forty-six live ads. Not one of them had a page to land on.