All field notes
    Field notes

    Day 4 of 5: verification

    Day 4 of 5. Verification.

    Green CI and a merged PR are proxies. Neither tells you a customer can use the thing. The rule we work to is verify the behaviour, not the proxy.

    This fortnight, in a content studio for a pharma brand team, every test was green and the real export produced 0 of the 5 review PDFs it should have. Only running the real export in the real environment showed it. The cause was found within the hour.

    On another project, every part of a feature worked on its own on our test site. Then an agent walked a fresh case in a real browser, the way a user would, and hit a dead end. Nothing on screen let you confirm who the case was about, so it could never be signed off. No test was going to see that.

    Absence isn't evidence either. "No errors in the log" only counts once you've seen that log record a known good case.

    Every piece of work ends with a footer like this:

    Verified: what behaviour, where, with what evidence
    Unverified: what I didn't check
    Residual risk: HIGH / MED / LOW
    If the gap hides a bug, the symptom would be: …

    I read Unverified first. An agent that tells me what it didn't check is one I can work with.

    Tomorrow, deployment, and where the bottleneck has moved.

    What's the last thing your CI said was fine that turned out not to be?

    Verify the behaviour, not the proxy. Green CI is a proxy: Every test was green. The real export produced 0 of 5 review PDFs. Walk it in a real browser: The way a user would, in the real environment. A quiet log proves nothing yet: It counts once you've seen it record a known good case. End every task with this footer: Read the Unverified line first. Verified: what behaviour, where, what evidence Unverified: what I didn't check Residual risk: HIGH / MED / LOW If the gap hides a bug, the symptom would be: …
    Download card (PNG)