A/B testing for people who run experiments for a living

Spec every edge case before engineering starts

I wrote the edge cases into the acceptance criteria: entry states for one versus many variants, traffic that doesn’t sum to 100 percent, the rule that the last variant can’t be deleted, and exactly what control means.

Why: so engineering could build without coming back with questions.

Replace a workaround with a real holdout

Customers needed a slice of traffic that sees nothing, and with no real feature they were faking one with invisible variants, causing scroll-lock bugs, CSS conflicts, and support tickets. I turned that workaround into requirements for a dedicated holdout that renders no creative at all, which keeps measurement clean.

Review it live with people who test for a living

Sessions with three enterprise customers who run experiments daily confirmed the core interactions and caught one subtle issue: a setting applied at the wrong level that would have corrupted the metrics. It went to engineering as a spike before it could ship.

One variable at a time: creative or settings, never both.Customer experimentation practice, validated in session

Let the split decide the default

Customers split on whether dismissal settings belong at the campaign or variant level. Instead of imposing one opinion, it became a configurable setting, with campaign level as the default because 2 of 3 customers preferred it.