A metric for an experimentation system’s maturity

~ Veselin ~ Leave a comment

Experiments in the context of software involve showing different groups of people different UIs or features and measuring the impact. This is one way to make informed decisions based on data rather than opinions. If experiments aren’t used, decision-making is usually subjective and prone to errors.

Humans are not good at predicting the future. We may be able to guess where the bus will stop next, but try guessing the price of Bitcoin next year. Being able to properly evaluate the outcome of decisions has real value, while relying on opinions can lead to irreversible losses.

Experiments are one way to get that data, and they’re something I’ve practiced a lot over the last two decades. I can say I have some experience, good or bad. But how far did I get with my own understanding of what a mature experimentation system should be?

The book I’m reading introduces a very interesting metric for experimentation maturity, based on the number of experiments, run per year. Metrics like this can certainly be criticized as superficial, it’s comparable to lines of code written or numbers of PRs. But having a metric is better than not having one, so here’s the maturity model from Trustworthy Online Controlled Experiments, Chapter 4. Numbers apply for large online services, like Google.

  1. Crawl phase: <10 experiments/year, few metrics, no internal system, manual post-experiment analysis
  2. Walk phase: up to 50/year, establishing standard metrics and some setup and data validation procedures, some internal systems and scorecards are available
  3. Run phase: up to 250/year, adoption at a level where most decisions are verified; the focus is on scaling, ease of use, increased number of standard metrics (100s to 1000s)
  4. Fly phase: 1,000s/year, a memory system is developed, the process is built around experimentation, shipping is easy, proper learnings and knowledge are preserved, metric drill downs (mentioned later in the book), thousands of metrics

The book isn’t exactly clear about when you’re out of Run and into Fly, given that one is around 250 experiments and the other is 1,000+, but it at least gives you a way to estimate roughly where you are if you’re running experiments.

It also doesn’t talk much about metric classification. As experimentation matures, metrics should be grouped into categories, with drill-downs for important characteristics. Some of the examples of what should be available felt very strong, like things I wish I knew sooner.

This book is great. I’m about 20 pages away from the end, and I have enough notes to write a chapter rather than a blog post. The last chapters are full of formulas and take time, as some of the formulas are not something I’ve encountered before. I learned stats with pen and paper, making countless calculations of averages, standard deviations, p-values, and such by hand. When I encounter a new stats formula, my brain wants to see the tables behind it, the charts behind it, the history behind it, and it refuses to move on by just accepting it.

Bush Family, Part Three

~ Veselin ~ 2 Comments

My recent theory that I found an orange mom + an orange dad cat family is no longer entirely viable. That tiny colony has at least 8 cats and while at least 5 are orange, some are grey tabbies. Here’s a photo of two nearly identical 3-4-5 month old kitten at the same spot. Their real size is tiny.

Update on the Bush Family

~ Veselin ~ 1 Comment

A few weeks ago, I posted a photo of two orange cats hiding in the bushes. I walked past the same area today and spotted three orange kittens playing together.

I think this is an uncommon case of a male and female orange cat producing a fully orange litter. I couldn’t get closer because they were playing in an area with restricted access, so I took the photo from outside. The three kittens appeared to be around 3-4 months old.

My Sister’s Grave by Robert Dugoni, Book Review

~ Veselin ~ Leave a comment

Tracy Crosswhite is the older sister in a family of sharp shooters. One day, the younger sister Sarah disappears after a shooting competition. A man is arrested and convicted of her murder, but there is no body. Tracy suspects that the wrong man has been framed. Twenty years later, the body is discovered.

For much of the book, this is a legal thriller in which Tracy tries to prove that the convicted murderer was framed, while somehow also looking for the real killer. Robert Dugoni drops clues about what actually happened along the way. I figured out a few details, but not the central mystery.

I enjoyed the book for the most part and ordered the next three. Maybe I over-reacted, I maybe next one would’ve been a better choice. It gets 4*/5 on my shelves.

Why four stars instead of five? I don’t know. I don’t like it when characters start doing horror-movie stupid things. Turning what was a pretty solid legal/crime thriller into something resembling a horror didn’t work for me. I also want to see Tracy using her sharp shooter skills. Hopefully, book two will deliver on that. It should be home tomorrow.