Tag: Books

Trustworthy Online Controlled Experiments: Book Review

~ Veselin ~ 11 Comments

This book is a rare jewel that I would recommend to anyone, running A/B tests. Pretty happy with the purchase and my time spent with it.

After I figured out that something with my understanding of how experiments should run was off, my instinct was that it’s math and stats skills that were lacking. So I looked into improvements in the area of probability and statistics first and my last two self-improvement books where in that area. Trustworthy Online Controlled Experiments isn’t about math. It contains condensed experience from people, wrangling experiments at Google, Microsoft, and LinkedIn, and then has some advanced chapters, intended to make the book complete.

The area that stuck with me the most was how to pick metrics for evaluating experiments and what to do with them.

For example, one of my most beloved revenue metrics (average revenue per user) is considered a health metric by the authors and their reasoning very solid. They argue any primary metric should be around user journey, usability, and satisfaction, around the purpose for the specific service. Revenue is an indicator how well the overall system performs but better indicators exist that are also easier to move with an experiment.

Many other metrics I currently frequently check, the book categorizes under drill-downs of existing metrics. The book also highlights the importance of having classical log data, like server errors, in the experiment dashboard. The authors list 20-ish industry standard metric ideas but also state that their internal systems have 1000s of metrics. What could these be? This statement also makes me wonder, how do they even make any calls, if they have so much data? Sounds like a good problem to have.

Regarding the number of metrics, the next book on the subject I picked, called Experimentation for Engineers, has a lengthy chapter about optimizing for just one metric. Then another lengthy chapter about optimizing for two metrics. Is that just a matter of taste? Being spoiled by the ideas of Trustworthy Online Controlled Experiments, I didn’t like it.

Imagine a page with two buttons, and we make one of the buttons orange in the treatment variation. Two clear metrics here can be clicks on the modified button and clicks on the non-modified button, together with the general metrics, available for all experiments. If the orange button gets more clicks, the other one will get fewer clicks. Such is life and decisions should be informed based on both changes. It’s also possible that the total number of clicks goes down because people perceive the page as spammy due to the orange color, despite orange clicks going up.

Optimizing for one thing without knowing the others can work but only in scenarios where there is really just one thing. For example, algorithmic trading of a specific stock. Optimizing for one metric while staying oblivious about the others and can produce errors and eventually erode the trust in experimentation.

So, thanks for reading this brain dump. It’s not about a cat, it’s not a quite a book review either because it covers just one area the book touches, but what can I do. The things that fascinate me are sometimes unusual.

Her Final Breath by Robert Dugoni

~ Veselin ~ Leave a comment

I’m glad I bought the second book in the Tracy Crosswhite series. The first one was a good thriller, but it also had some disappointing moments. The second one is great. I couldn’t figure out who the killer was. It fooled me well.

To some extent, the first two books are connected, as the murder is introduced in the first book, although it is only mentioned. In the second book, we will see Tracy Crosswhite work on it.

Here are some good parts about the book:

  • No superheroes or super-heroic traits. Any success is achieved by regular police work
  • More than one case at the same time, reminding me of the best books by Michael Connelly
  • Characters behave logically. We had some cases of comical stupidity in part one and while real life is full of that, I don’t like reading it in thrillers
  • More than one meaningful character. It’s not all about Tracy. Her colleagues and partner help significantly

Overall, a clear 5*/5. I’m already half-way through the third one, and the fourth is rolling on the floor next to my bed. It’s a solid series, won’t be surprised if it gets mentioned in the annual reading summary.

My Sister’s Grave by Robert Dugoni, Book Review

~ Veselin ~ Leave a comment

Tracy Crosswhite is the older sister in a family of sharp shooters. One day, the younger sister Sarah disappears after a shooting competition. A man is arrested and convicted of her murder, but there is no body. Tracy suspects that the wrong man has been framed. Twenty years later, the body is discovered.

For much of the book, this is a legal thriller in which Tracy tries to prove that the convicted murderer was framed, while somehow also looking for the real killer. Robert Dugoni drops clues about what actually happened along the way. I figured out a few details, but not the central mystery.

I enjoyed the book for the most part and ordered the next three. Maybe I over-reacted, I maybe next one would’ve been a better choice. It gets 4*/5 on my shelves.

Why four stars instead of five? I don’t know. I don’t like it when characters start doing horror-movie stupid things. Turning what was a pretty solid legal/crime thriller into something resembling a horror didn’t work for me. I also want to see Tracy using her sharp shooter skills. Hopefully, book two will deliver on that. It should be home tomorrow.

Therapy by Sebastian Fitzek, Book Review

~ Veselin ~ 5 Comments

Viktor Larenz is a psychiatrist, whose twelve-year-old daughter, Josy, got sick and either disappeared or died in a very obscure scene.

One thing is very clear from the start, Viktor Lanetz sees things nobody else sees. The main mystery of the book is what is hallucination and what is real. My guess, after 20-30 pages read, was that 100% of all non-Viktor characters are hallucinations. This is what the author implies. However, this book has 4.1 on Goodreads, and I’m sure that 90% of the readers made the same guess and wouldn’t have rated it 4.1 if it was that simple.

On top of that, the tagline, what happened with Josy, turns out to also be quite mysterious, and it will be solved.

This is not a realistic book. It’s dark noir, made-up, annoying, with plot twists designed to play with your mind. It has the vibe of Sucker Punch, if you remember that movie, and The Poison Artist by Jonathan Moore. However, it’s not quite up to that bar, just leaning that way from a close distance.

I rated it 3/5 on Goodreads because for the most of the book, it felt too obvious. The stuff that wasn’t obvious felt too made up. However, the Goodreads audience disagrees and rated this over 4, which is pretty high. In general, my interpretation for Goodreads ratings is as follows:

  • <3.9 – I’ll be disappointed
  • 3.9-4.2 – readable, usually good books
  • 4.3+ – rating unclear, skewed by fans

This book is well within the readable range and it is not bad but I didn’t like it. I will, however, try another book by the same author, this time with adjusted expectations. Hopefully, I’ll like the next one more.

July in Books

~ Veselin ~ Leave a comment

I had a good month, completed some great books that are easy to recommend.

Best books

  1. The Devils by Joe Abercrombie. I forgot to share a review here, shame on me. But the book is a clear 5*, and a slightly lighter (not in weight though) than the regular Abercrombie story. Some tropes here and there, features lots of random fantasy species. 760 pages and very memorable, well-built characters. You have a priest and a vampire joining forces, not the most common occurrence in literature.
  2. Empire of the Vampire by Jay Kristoff – 1008 pages and a solid 5*/5. The sun stopped shining and the main deterrent to vampires spreading was gone. Also no review, pretty embarrassing, but at least I mentioned it. A must read.
  3. The Blacktongue Thief by Christopher Buehlman – I think that one was around 4.5*/5 but pretty good 4.5, one that would love to read once a second part shows up in the bookstores. The blacktongue thief is thief from a race that has black tongues. Epic fantasy with some drama.
  4. Head Cases by John McMahon – a satisfying thriller about nerds solving crimes. It got 4.5* on Goodreads by me but looking back, I think it’s closer to 4. Despite being a good story, the other three above were much better.
  5. Everything Is Predictable: How Bayes’ Remarkable Theorem Explains the World – pop-sci book about probabilities. I gave it 4*/5 but from the distance of time, I think it’s not up to the 4* bar.

Worst books

  1. Then She Was Gone by Lisa Jewell – Lisa Jewell is a gem, comparable to Freida McFadden. However, just like Freida McFadden, some of her books are pretty awful. Not that the book is poorly written, it’s just not a pleasant thing to read. I gave it 4/5, appreciating the good storytelling, translation and such. I hated this book so much that I’d rather not buy anything else by Lisa Jewell any time soon. Boo.