Experiments in the context of software involve showing different groups of people different UIs or features and measuring the impact. This is one way to make informed decisions based on data rather than opinions. If experiments aren’t used, decision-making is usually subjective and prone to errors.

Humans are not good at predicting the future. We may be able to guess where the bus will stop next, but try guessing the price of Bitcoin next year. Being able to properly evaluate the outcome of decisions has real value, while relying on opinions can lead to irreversible losses.

Experiments are one way to get that data, and they’re something I’ve practiced a lot over the last two decades. I can say I have some experience, good or bad. But how far did I get with my own understanding of what a mature experimentation system should be?

The book I’m reading introduces a very interesting metric for experimentation maturity, based on the number of experiments, run per year. Metrics like this can certainly be criticized as superficial, it’s comparable to lines of code written or numbers of PRs. But having a metric is better than not having one, so here’s the maturity model from Trustworthy Online Controlled Experiments, Chapter 4. Numbers apply for large online services, like Google.

1. Crawl phase: <10 experiments/year, few metrics, no internal system, manual post-experiment analysis
2. Walk phase: up to 50/year, establishing standard metrics and some setup and data validation procedures, some internal systems and scorecards are available
3. Run phase: up to 250/year, adoption at a level where most decisions are verified; the focus is on scaling, ease of use, increased number of standard metrics (100s to 1000s)
4. Fly phase: 1,000s/year, a memory system is developed, the process is built around experimentation, shipping is easy, proper learnings and knowledge are preserved, metric drill downs (mentioned later in the book), thousands of metrics

The book isn’t exactly clear about when you’re out of Run and into Fly, given that one is around 250 experiments and the other is 1,000+, but it at least gives you a way to estimate roughly where you are if you’re running experiments.

It also doesn’t talk much about metric classification. As experimentation matures, metrics should be grouped into categories, with drill-downs for important characteristics. Some of the examples of what should be available felt very strong, like things I wish I knew sooner.

This book is great. I’m about 20 pages away from the end, and I have enough notes to write a chapter rather than a blog post. The last chapters are full of formulas and take time, as some of the formulas are not something I’ve encountered before. I learned stats with pen and paper, making countless calculations of averages, standard deviations, p-values, and such by hand. When I encounter a new stats formula, my brain wants to see the tables behind it, the charts behind it, the history behind it, and it refuses to move on by just accepting it.