I scheduled a weekly check that could never run.
An automated SEO check that needed a browser on my laptop, scheduled in the cloud. Designing a measurement window that can honestly report nothing was the real work.
I changed the wording across roughly a hundred and ten local pages on a directory site I run, then set up a weekly automated check to tell me whether it worked. The first run came back with nothing. No numbers, no partial numbers, no estimate.
It wasn't a glitch. The routine needed a logged-in browser session on my Mac, and I had scheduled it to run in the cloud, where my Mac does not exist. It was never going to fail once. It was going to fail every Monday, forever, with a slightly different error message.
Design the job for the room it will actually run in
The mistake is easy to make because building and running happen in different worlds. I built it interactively, sitting at my machine, with a browser open and my hand on the approval button. Everything worked. Then I scheduled it into an environment with none of those things.
The fallback made it worse. If the primary path failed, the routine was supposed to check the live pages directly — and that check requires an interactive approval, per URL. An unattended job cannot approve anything. It waits, then times out. A fallback that needs a human present is not a fallback; it's the same failure wearing a different coat.
So the rule I'd hand anyone automating anything: before you schedule it, list the tools it needs and ask which of them exist in the context where it will run — not the context where you wrote it. Same class of error as losing track of which machine you're standing on, and the fix is the same. Be specific about where the work happens.
Rebuilding it on the Search Analytics API instead of browser automation took an afternoon. It now runs unattended, returns reproducible numbers instead of values read off a screenshot, and writes its history to a file. That part was plumbing. The part that mattered was deciding what it should measure.
The obvious comparison would have buried the answer
The natural design is to pull the last three months every week and compare against the baseline I took on rollout day. It feels rigorous. It's wrong.
A rolling three-month window contains the pre-change period for months. Every week I'd be comparing a mostly-old window against an entirely-old baseline, diluting exactly the movement I'm trying to see, and I'd conclude the rollout did nothing right up until the point the window finally cleared — long after I'd have made a decision.
What the script does instead is hold two windows of equal length. The after window runs from the day after rollout to three days ago, because the API distinguishes finalised data from fresh data that can still change and I'd rather wait than chase revisions. The before window is exactly the same number of days, ending the day before rollout. Both grow by seven days each week. After a month it's a clean twenty-eight against twenty-eight, and it was honest at every point along the way.
Put the noise rules in the code, not in your head
This is the bit I'd defend in an argument. The script refuses to report certain things, and it refuses in code rather than in a convention I'm supposed to remember.
A position gain under three places is noise and never counts as a win. Anything with fewer than ten impressions in the window is not evidence, no matter how large the delta looks — an average position derived from three impressions is not a number, it's a rumour. A window shorter than fourteen days returns "too early" regardless of how good the figures are. Brand searches, which are most of the reach, are excluded from new-term detection entirely, or every week would look like a triumph.
There's a self-test with twenty-seven assertions covering precisely these cases. Two of them read, in effect, "twelve places gained on three impressions is not a win" and "seven days is too early even when the gains are strong."
Rules you enforce in code survive the week you badly want good news. Rules you keep in your head do not.
The run after that one duly reported no result: seven days old, too early, no judgement offered. That's the system working. Measuring before acting only helps if the measurement is allowed to come back empty.
Store everything, not the nine numbers you care about
Every run dumps the full query list to a file, not just the reference terms I'm tracking. That looked like over-collection until I worked out what the first real positive signal would be.
It isn't a better rank on the terms I already know about. It's the appearance of new generic "service plus town" queries that the site never surfaced for before — the thing the vocabulary change was supposed to unlock. And "new" is only computable if every previous week is on file. Nine tracked terms can't tell you that; the full dump can. Six impressions have told me what to write before, and they'd have been invisible in a summary.
The automation I first shipped would have run for months, failed every time, and told me nothing. The one that replaced it will mostly tell me it's too early to say. That's a real upgrade, and it's the only test of a system that counts: what it does when nobody's watching it.
Sources & further reading
External: Search Analytics: query — Search Console API · Updates to Search Console's API, Google Search Central · The Search Analytics API now supports hourly data
Related posts: The hard part isn't Git · Ten minutes of querying killed the content sprint · Six search impressions told me what to write · The real test of a system