I managed to complete this book, which is quite an achievement. Unlike the last two I read on the subject, this was not an easy read. Lots of abbreviations, terminology, python functions, charts that represent hypothetical scenarios I couldn’t relate to, and more questions than answers. Why does that work? How does it work?
I’m not sure if this is necessarily a bad book. Probably isn’t. The author clearly has experience in his line of work, which is very different than mine. So what this book showed me was that other methods exist and online controlled experiments is a broader topic than I assumed. Will I try using any of the python functions or the methods inside? Not likely. However, knowing what can be done is still valuable and may come in handy one day.
I think I wasn’t the right audience for it, which lead to some disappointment. However, I want to learn more about multi-armed bandits and contextual bandits and may check what’s available on Amazon in the area.
I seem to have a thing for catching a crime series and going as far as I can with it. This book was part 4 of a series of 6, makes you wish there were at least 5 more. Robert Dugoni needs to learn from Janet Evanovich and Alexandra Marinina. Six books is not enough for a successful Sherlock to fully develop. We need more.
Back to the book.
A crab poacher finds a trap with a dead woman inside, 20-25 meters underwater. Detective Tracy Crosswhite starts digging into the story, one clue at a time. It’s not very clear who the victim was but there was no shortage of people who had the means and motivation to harm her. The case is in good hands. Detective Crosswhite is tall, persistent, and a sharp shooter. Out of these qualities, she’ll most frequently need the first one, because clues are sometimes placed on high shelves.
The story is simple, the characters are believable, and vaguely remind you of someone. Multiple detectives are following the leads. No genius conclusions or superheroism. I wasn’t even close to guessing what really happened and why. Robert Dugoni is building up the protagonists slowly but surely. I won’t be surprised if this series turns into a romance within 3-4 books.
The second book from the Dungeon Crawler Carl Series didn’t disappoint. Action from start to finish, with a short break for promises and explanations about the future books. I’m sure I won’t remember any of that but it was nice to know that something outside of the dungeon exists.
The plot is simple. Carl, his cat Donut, and a few friends will grind through the maze, collecting experience points, and solving quests. Nothing sophisticated. I enjoyed it so much that completed the book in one go.
This book is a rare jewel that I would recommend to anyone, running A/B tests. Pretty happy with the purchase and my time spent with it.
After I figured out that something with my understanding of how experiments should run was off, my instinct was that it’s math and stats skills that were lacking. So I looked into improvements in the area of probability and statistics first and my last two self-improvement books where in that area. Trustworthy Online Controlled Experiments isn’t about math. It contains condensed experience from people, wrangling experiments at Google, Microsoft, and LinkedIn, and then has some advanced chapters, intended to make the book complete.
The area that stuck with me the most was how to pick metrics for evaluating experiments and what to do with them.
For example, one of my most beloved revenue metrics (average revenue per user) is considered a health metric by the authors and their reasoning very solid. They argue any primary metric should be around user journey, usability, and satisfaction, around the purpose for the specific service. Revenue is an indicator how well the overall system performs but better indicators exist that are also easier to move with an experiment.
Many other metrics I currently frequently check, the book categorizes under drill-downs of existing metrics. The book also highlights the importance of having classical log data, like server errors, in the experiment dashboard. The authors list 20-ish industry standard metric ideas but also state that their internal systems have 1000s of metrics. What could these be? This statement also makes me wonder, how do they even make any calls, if they have so much data? Sounds like a good problem to have.
Regarding the number of metrics, the next book on the subject I picked, called Experimentation for Engineers, has a lengthy chapter about optimizing for just one metric. Then another lengthy chapter about optimizing for two metrics. Is that just a matter of taste? Being spoiled by the ideas of Trustworthy Online Controlled Experiments, I didn’t like it.
Imagine a page with two buttons, and we make one of the buttons orange in the treatment variation. Two clear metrics here can be clicks on the modified button and clicks on the non-modified button, together with the general metrics, available for all experiments. If the orange button gets more clicks, the other one will get fewer clicks. Such is life and decisions should be informed based on both changes. It’s also possible that the total number of clicks goes down because people perceive the page as spammy due to the orange color, despite orange clicks going up.
Optimizing for one thing without knowing the others can work but only in scenarios where there is really just one thing. For example, algorithmic trading of a specific stock. Optimizing for one metric while staying oblivious about the others and can produce errors and eventually erode the trust in experimentation.
So, thanks for reading this brain dump. It’s not about a cat, it’s not a quite a book review either because it covers just one area the book touches, but what can I do. The things that fascinate me are sometimes unusual.
I’m glad I bought the second book in the Tracy Crosswhite series. The first one was a good thriller, but it also had some disappointing moments. The second one is great. I couldn’t figure out who the killer was. It fooled me well.
To some extent, the first two books are connected, as the murder is introduced in the first book, although it is only mentioned. In the second book, we will see Tracy Crosswhite work on it.
Here are some good parts about the book:
No superheroes or super-heroic traits. Any success is achieved by regular police work
More than one case at the same time, reminding me of the best books by Michael Connelly
Characters behave logically. We had some cases of comical stupidity in part one and while real life is full of that, I don’t like reading it in thrillers
More than one meaningful character. It’s not all about Tracy. Her colleagues and partner help significantly
Overall, a clear 5*/5. I’m already half-way through the third one, and the fourth is rolling on the floor next to my bed. It’s a solid series, won’t be surprised if it gets mentioned in the annual reading summary.
Veselin.blog 