A/B TESTING AND EXPERIMENT DESIGN: Plan Valid Experiments and Make Better Data-Driven Decisions
HALSTEAD, CORIN
Synopsis "A/B TESTING AND EXPERIMENT DESIGN: Plan Valid Experiments and Make Better Data-Driven Decisions"
A statistically significant result does not automatically mean you made the right decision. A/B testing sounds simple: divide users into groups, introduce a change, compare the results, and choose the better-performing option. In practice, trustworthy experimentation is far more demanding. Was the assignment truly random? Did users actually see the treatment? Is the metric measuring real value—or just something easy to count? Was the experiment large enough to detect an effect that matters? Did repeated checking inflate the chance of a false positive? And what happens when the results are statistically significant but practically worthless? A/B Testing and Experiment Design is a practical guide to planning, running, interpreting, and scaling experiments that produce evidence you can actually trust. Rather than focusing only on formulas and significance thresholds, this book connects statistical thinking with the real decisions faced by product teams, analysts, marketers, engineers, founders, and data professionals. Inside, you'll learn how to: Turn business questions into clear, testable hypotheses Separate correlation from causal evidence Choose meaningful primary metrics, guardrails, and diagnostic measures Select the right randomization unit for users, accounts, devices, stores, regions, or time periods Distinguish assignment from actual treatment exposure Detect sample ratio mismatch and other experiment-integrity failures Plan sample size, statistical power, minimum detectable effect, and experiment duration Interpret confidence intervals, p-values, effect sizes, and practical significance correctly Handle noisy revenue metrics, ratio metrics, clustering, and repeated observations Avoid common mistakes caused by peeking, multiple comparisons, and post-hoc analysis Use variance-reduction techniques such as CUPED, blocking, and covariate adjustment Choose between fixed-horizon, sequential, factorial, cluster, switchback, geo, crossover, and holdout designs Understand when multi-armed and contextual bandits are more appropriate Apply experimentation to product development, growth, marketing, pricing, recommendation systems, machine learning, generative AI, infrastructure, and release engineering Build experimentation platforms, metric catalogs, review processes, and organizational learning systems The book emphasizes that trustworthy experiments require alignment between the decision, causal design, measurement, operations, and statistical inference—not simply a favorable dashboard result. It also moves beyond basic A/B comparisons into factorial designs, cluster randomization, switchbacks, geo experiments, long-term holdouts, crossover designs, bandits, and quasi-experimental approaches. Whether you're a product manager, data analyst, growth professional, marketer, UX researcher, software engineer, engineering manager, founder, operations professional, or student, this book provides a practical framework for turning uncertainty into disciplined evidence and better decisions. Stop asking which variant “won.” Start asking what the experiment actually proved—and what decision the evidence justifies.
