Trustworthy Online Controlled Experiments - Summary and Key Ideas

The book Trustworthy Online Controlled Experiments is a practical guide to A/B testing and online controlled experiments, written by experts from Google, LinkedIn, and Microsoft. It provides insights into accelerating innovation using trustworthy online controlled experiments, shares examples and pitfalls, and offers advice for both beginners and experienced practitioners.

The target audience for the book "Trustworthy Online Controlled Experiments" includes students and industry professionals getting started with experiments, experienced practitioners who want to improve their data-driven decision making, and a wider audience interested in online controlled experiments. It is particularly recommended for leaders, executives, data scientists, engineers, analysts, and product managers.

Buy the book
Trustworthy Online Controlled Experiments

Key ideas


Controlled experiments are crucial for data-driven decision making in organizations, enabling the testing of ideas and strategies, but must be designed properly and interpreted correctly to ensure trustworthy results.


A culture of experimentation, supported by a robust platform and leadership, accelerates innovation by facilitating reliable experiments and learning from past outcomes.

Play in App

Experiments are crucial tools for measuring performance and progress, but they must be conducted ethically, with respect for participants, and with clear, actionable metrics that align with the organization's goals.

Play in App

Research methods, including complementary techniques and observational causal studies, each have their strengths and limitations, and should be chosen based on the specific needs of the research, ensuring a well-rounded and reliable result.

Play in App

The text emphasizes the importance of high-quality instrumentation in running server-side and client-side experiments, and fostering a culture of instrumentation for reliable data-driven decision making.

Play in App

Designing and analyzing experiments requires careful consideration of the unit of randomization, gradual rollout of features, and effective data preparation and interpretation.

Play in App

Statistical concepts like hypothesis testing, p-values, confidence intervals, and statistical power are essential tools for making evidence-based decisions, akin to detective tools in solving a case.

Play in App

A/A testing is a critical process in validating an experimentation platform's accuracy and detecting potential issues, while the concept of triggering helps refine the analysis by focusing on affected users.

Play in App

Guardrail metrics, such as the Sample Ratio Mismatch, are essential tools in experimentation to ensure valid results, detect errors, and monitor for interference or leakage in both direct and indirect connections.

Play in App

Understanding and measuring the long-term impacts of product changes is complex but crucial, requiring thoughtful strategies to overcome challenges and gain valuable insights into user behavior and market dynamics.

Play in App

Play in App

Play in App
Get the App!
Access all 11 key ideas for free! 🥳

Summary & Review

"Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing" by Ron Kohavi, Diane Tang, and Ya Xu is a comprehensive guide to conducting online controlled experiments, or A/B tests. The authors, who are leaders in experimentation at Google, LinkedIn, and Microsoft, share their practical experiences and provide advice for both beginners and experienced practitioners. The book covers the basics of running online controlled experiments, common pitfalls, and how to build an experimentation platform. It also delves into advanced topics for experienced practitioners. The authors emphasize the importance of trustworthy results, encouraging readers to double-check results and run validity tests.

Ron Kohavi, Ya Xu

Ron Kohavi is a renowned computer scientist specializing in data mining, machine learning, and web analytics. Ya Xu, on the other hand, is a leading data scientist at LinkedIn, with expertise in statistical modeling, experimental design, and causal inference.

Explore more book summaries

The Glass Universe

The book The Glass Universe (2016) is about the remarkable women who worked at the Harvard College Observatory in the late 1800s and early 1900s, using their keen intellect and perseverance to revolutionize our understanding of the universe. Through their meticulous work with glass plates of stars, they made groundbreaking discoveries about the nature of galaxies, stars, and the cosmos itself.


Mindshift explores the concept of deep personal change through learning, highlighting the power of embracing new skills, interests, and career paths at any age. It shares inspiring stories and practical tips on how to overcome obstacles, broaden passions, and unlock hidden potential.

The Art of Self-Directed Learning

"The Art of Self-Directed Learning" is a guidebook that explores the concept of independent, self-motivated learning, offering practical advice and strategies to help individuals take control of their own education and personal growth.

Chip War

"Chip War" explores the history and global impact of the semiconductor industry, highlighting its role in shaping military power, global economies, and technological advancements. It delves into the intense competition between nations, particularly the U.S., China, and Russia, in controlling this critical technology.

Speed Reading

The book "Speed Reading" is about techniques and strategies to increase reading speed without compromising comprehension and retention. It provides practical exercises and tips to improve reading efficiency and productivity.

Wits Guts Grit

"Wits Guts Grit" is a book by Jena Pincott that explores natural "biohacks" for raising smart and resilient children, focusing on factors such as microbes, nutrients, hormones, neurons, and environmental influences. The book provides insights and actionable advice on how to help children adapt to a challenging and continuously changing world, aiming to strengthen their resilience, memory, self-regulation, and other key attributes.