Tag: experiments

Reading in August

~ Veselin ~ Leave a comment

August was a good month for my shelves. I was on vacation and read a lot. Had a week with 3 completed Dugoni books, and a day in which I finished the entire Dungeon Crawler Carl in one go.

Best books

  1. Trustworthy Online Controlled Experiments – I have the feeling this will help me be better at my job but it just as well may not. It’s a guide for running better experiments, written by engineers from Google, Microsoft, and LinkedIn.
  2. The Trapped Girl by Robert Dugoni – out of the many Dugoni books I completed this month, this one felt the best. A crime/thriller with a good flow and difficult to guess culprit.
  3. Carl’s Doomsday Scenario by Matt Dinniman – Carl continues advancing through the Diablo 2 style maze. The book was a bit worse than Carl 1 but still 5/5. Looking forward to the 3rd part.
  4. Her Final Breath by Robert Dugoni – I finished 5 Dugoni books during the month and it’s difficult to remember which one is which 🙂
  5. My Sister’s Grave by Robert Dugoni
  6. In the Clearing by Robert Dugoni
  7. Close to Home by Robert Dugoni – probably the worst Dugoni book for the month, I think I gave that one 4/5. Still very good.

Worst books

  1. Therapy by Sebastian Fitzek, this was just bad. I have another book by the same author but I now question my choices and may just donate it so I don’t need to read it. Goodreads rating 3/5 but really 1/5.
  2. Glass Houses by Rachel Caine – I recommended two of her books about the Stillhouse Lake. However, this one is a YA fairytale and I exhausted my faith in the author to get to the end. 2/5
  3. Experimentation for Engineers: From A/B testing to Bayesian optimization by David Sweet – for the most part, it felt like the pages were filled with the purpose to provide a complete guide, rather than to teach me anything I have a remote chance of trying. I wasn’t able to separate the practical, useful bits from the filler, so I can’t tell for sure how much is filler, and how much is real content the author tried in their career. All the examples are hypothetical, there are no citations. The other book on the subject quoted a source for every claim. This one? Nope. Rating – on Goodreads was 3/5 but it could be 5 for the right person and 1 if I wrote it while in bad mood.

Experimentation for Engineers: From A/B testing to Bayesian optimization by David Sweet

~ Veselin ~ 9 Comments

I managed to complete this book, which is quite an achievement. Unlike the last two I read on the subject, this was not an easy read. Lots of abbreviations, terminology, python functions, charts that represent hypothetical scenarios I couldn’t relate to, and more questions than answers. Why does that work? How does it work?

I’m not sure if this is necessarily a bad book. Probably isn’t. The author clearly has experience in his line of work, which is very different than mine. So what this book showed me was that other methods exist and online controlled experiments is a broader topic than I assumed. Will I try using any of the python functions or the methods inside? Not likely. However, knowing what can be done is still valuable and may come in handy one day.

I think I wasn’t the right audience for it, which lead to some disappointment. However, I want to learn more about multi-armed bandits and contextual bandits and may check what’s available on Amazon in the area.

Trustworthy Online Controlled Experiments: Book Review

~ Veselin ~ 11 Comments

This book is a rare jewel that I would recommend to anyone, running A/B tests. Pretty happy with the purchase and my time spent with it.

After I figured out that something with my understanding of how experiments should run was off, my instinct was that it’s math and stats skills that were lacking. So I looked into improvements in the area of probability and statistics first and my last two self-improvement books where in that area. Trustworthy Online Controlled Experiments isn’t about math. It contains condensed experience from people, wrangling experiments at Google, Microsoft, and LinkedIn, and then has some advanced chapters, intended to make the book complete.

The area that stuck with me the most was how to pick metrics for evaluating experiments and what to do with them.

For example, one of my most beloved revenue metrics (average revenue per user) is considered a health metric by the authors and their reasoning very solid. They argue any primary metric should be around user journey, usability, and satisfaction, around the purpose for the specific service. Revenue is an indicator how well the overall system performs but better indicators exist that are also easier to move with an experiment.

Many other metrics I currently frequently check, the book categorizes under drill-downs of existing metrics. The book also highlights the importance of having classical log data, like server errors, in the experiment dashboard. The authors list 20-ish industry standard metric ideas but also state that their internal systems have 1000s of metrics. What could these be? This statement also makes me wonder, how do they even make any calls, if they have so much data? Sounds like a good problem to have.

Regarding the number of metrics, the next book on the subject I picked, called Experimentation for Engineers, has a lengthy chapter about optimizing for just one metric. Then another lengthy chapter about optimizing for two metrics. Is that just a matter of taste? Being spoiled by the ideas of Trustworthy Online Controlled Experiments, I didn’t like it.

Imagine a page with two buttons, and we make one of the buttons orange in the treatment variation. Two clear metrics here can be clicks on the modified button and clicks on the non-modified button, together with the general metrics, available for all experiments. If the orange button gets more clicks, the other one will get fewer clicks. Such is life and decisions should be informed based on both changes. It’s also possible that the total number of clicks goes down because people perceive the page as spammy due to the orange color, despite orange clicks going up.

Optimizing for one thing without knowing the others can work but only in scenarios where there is really just one thing. For example, algorithmic trading of a specific stock. Optimizing for one metric while staying oblivious about the others and can produce errors and eventually erode the trust in experimentation.

So, thanks for reading this brain dump. It’s not about a cat, it’s not a quite a book review either because it covers just one area the book touches, but what can I do. The things that fascinate me are sometimes unusual.

A metric for an experimentation system’s maturity

~ Veselin ~ Leave a comment

Experiments in the context of software involve showing different groups of people different UIs or features and measuring the impact. This is one way to make informed decisions based on data rather than opinions. If experiments aren’t used, decision-making is usually subjective and prone to errors.

Humans are not good at predicting the future. We may be able to guess where the bus will stop next, but try guessing the price of Bitcoin next year. Being able to properly evaluate the outcome of decisions has real value, while relying on opinions can lead to irreversible losses.

Experiments are one way to get that data, and they’re something I’ve practiced a lot over the last two decades. I can say I have some experience, good or bad. But how far did I get with my own understanding of what a mature experimentation system should be?

The book I’m reading introduces a very interesting metric for experimentation maturity, based on the number of experiments, run per year. Metrics like this can certainly be criticized as superficial, it’s comparable to lines of code written or numbers of PRs. But having a metric is better than not having one, so here’s the maturity model from Trustworthy Online Controlled Experiments, Chapter 4. Numbers apply for large online services, like Google.

  1. Crawl phase: <10 experiments/year, few metrics, no internal system, manual post-experiment analysis
  2. Walk phase: up to 50/year, establishing standard metrics and some setup and data validation procedures, some internal systems and scorecards are available
  3. Run phase: up to 250/year, adoption at a level where most decisions are verified; the focus is on scaling, ease of use, increased number of standard metrics (100s to 1000s)
  4. Fly phase: 1,000s/year, a memory system is developed, the process is built around experimentation, shipping is easy, proper learnings and knowledge are preserved, metric drill downs (mentioned later in the book), thousands of metrics

The book isn’t exactly clear about when you’re out of Run and into Fly, given that one is around 250 experiments and the other is 1,000+, but it at least gives you a way to estimate roughly where you are if you’re running experiments.

It also doesn’t talk much about metric classification. As experimentation matures, metrics should be grouped into categories, with drill-downs for important characteristics. Some of the examples of what should be available felt very strong, like things I wish I knew sooner.

This book is great. I’m about 20 pages away from the end, and I have enough notes to write a chapter rather than a blog post. The last chapters are full of formulas and take time, as some of the formulas are not something I’ve encountered before. I learned stats with pen and paper, making countless calculations of averages, standard deviations, p-values, and such by hand. When I encounter a new stats formula, my brain wants to see the tables behind it, the charts behind it, the history behind it, and it refuses to move on by just accepting it.

Trustworthy Online Controlled Experiments: … you got my attention

~ Veselin ~ Leave a comment

Just yesterday, I shared about a popular science book about Bayes. Today I started a new one, called Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. I picked it because I want to learn more about the industry standard way of doing controlled experiments, an area where I have some experience. This particular book started well.

Even the first chapters show that the authors do have experience I can relate to and can, hopefully, learn from.

This quote, for example, is included in their preface:

Any figure that looks interesting or different is usually wrong

— Twyman’s Law

It matches my experience with overly positive results or ambitious claims in analysis. Can’t tell you how many times I’ve fallen into the trap of believing something wins big time while there was really just a mistake, or a bias of some kind, twisting the results. But I had no clue someone postulated a law on it and got a Wikipedia page about it. I can now cite it and gain mana points 😀

So, I have a good feeling about this book and I hope to be able to complete it much faster than the last one. It just starts well.

Unfortunately, my copy arrived with significant damage. It was improperly packaged and delivered poorly. However, it is a book I read with a pen, underlining what I think is important, and leaving notes inside, so it will not get any better anyway.