Our ROAS doubled. We hadn't sold a thing more.

An investor on the board of a home & living brand I work with sent one line into a Monday morning: thirty-day comparison, online sales roughly the same as the previous window, ROAS roughly double. What changed?

It is the right question and it has a trap in it. The instinct is to go looking for the thing that worked. A campaign, a creative, a market. Something you can point at and repeat.

There was nothing to point at. Revenue was flat. Media spend had fallen by roughly two thirds. The ratio did not improve because the numerator went up. It improved because the denominator collapsed.

A ratio is two numbers wearing one coat

Return on ad spend is the most quoted number in performance marketing and one of the least interrogated. Every time it moves, it is worth asking which half moved, and in which direction, before you decide whether to be pleased.

Doubling ROAS by selling twice as much at the same spend is a growth story. Doubling it by spending half as much at the same revenue is a margin story. Both are good. They are not remotely the same finding and they point at completely different next moves. One says pour more in. The other says you were pouring into a hole.

In this case it was the second. Same revenue, roughly a third of the media budget, and the markets that got cut hardest were exactly the ones that had been sitting below break-even in the previous window. That is not a mystery. That is someone turning off the taps that were running into the drain — the boring, unglamorous half of the job, and usually the half that actually shows up in the bank account.

The half of the spend that could never prove itself

One channel had been switched off almost entirely. It had been buying tens of thousands of clicks a month and reporting a handful of attributable purchases against them. On paper, catastrophic.

Except the reason was known and had nothing to do with the ads. The purchase event on that platform had not fired since the previous November, and there was no server-side Conversions API to carry the signal instead. The campaigns were optimising toward a conversion that never arrived. Of course they reported almost nothing — they were being measured with a broken instrument, and they were also bidding with one, which is the more expensive half of that problem. I wrote recently about diagnosing exactly that kind of thing without ever logging into the tag manager.

So what does a near-zero return on that channel actually tell you? Less than it looks like. It tells you the channel could not credit itself. It does not tell you the channel contributed nothing.

As David McRaney puts it in You Are Not So Smart: "Lack of proof neither confirms nor denies a proposition."

What we have instead is a natural experiment, and it is genuinely informative: the spend went away and revenue held. That is the strongest evidence available that most of it was not incremental. Evidence, not proof. Two months is a short window and demand has memory — a point I keep having to make when a marketplace suddenly looks brilliant after the top-of-funnel gets cut.

This is not a fringe worry. The most-cited work on the subject came out of eBay, where Blake, Nosko and Tadelis ran large-scale geographic experiments — switching search ads off in some cities, leaving them on in others — and found returns far below what attribution had been reporting, with brand-keyword ads showing no measurable short-term benefit at all, because people simply clicked the organic result instead. The write-up of the branded-search experiment is worth twenty minutes of anyone's time.

The measuring stick changed mid-comparison

Here is the part that would have been easy to leave out of the reply, and that matters more than the headline.

The other channel wasn't cut. It was rebuilt. Spend down by roughly a third, tracked value up by well over half, clicks down far more steeply than conversions — broad low-intent traffic removed, budget concentrated on brand and high-intent search. Good work, and the numbers look excellent.

But dynamic conversion value only went live on that account in early July. Before that, the tracked value of every order was a flat proxy. So part of the improvement is real behaviour change, and part of it is the platform finally being told what a basket is actually worth — which is the entire point of value-based bidding, and also means the before and the after were measured with two different rulers.

You cannot compare a ratio across a change in how that ratio is computed and call the difference performance. You can say both things happened in the same window and you cannot yet split them. That sentence is less satisfying and considerably more useful, and it survives the follow-up question in a way the confident version does not.

What I actually sent back

Not "here is what worked." Instead: revenue held, spend fell by two thirds, here is which markets and which channels, here is the portion that is measurement rather than performance, and here is the open question — does revenue hold without that channel over a longer horizon, because two months does not answer it.

Twenty years in, the number I trust least is the one that improved without anyone being able to say which of its two halves moved. AI makes it trivially fast to pull the platform data and decompose a blended cost line; that used to be a week of work and it took an afternoon. What it will not do for you is notice that the ruler changed in July.

A metric that gets better while nothing more gets sold is not a win. It is a question.

Sources & further reading

External
Blake, Nosko & Tadelis — "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment," Econometrica
CEPR / VoxEU — An experiment to estimate the effectiveness of branded search ads
Google Ads Help — Smart Bidding using value-based bidding for Search and Shopping
Meta for Developers — Conversions API

Related posts
No login to the tag manager. The container told me everything anyway.
We cut the top-of-funnel and the marketplace 'grew.' It was harvesting demand we'd already built.
The revenue climbed. I assumed we'd spent our way there. We hadn't.