People make business decisions based on data all the time and think think they know what it means when they see the result ("up is up"). However, most people are not analyzing data in a scientifically valid way and are setting themselves up to be duped by data. By learning as much as I can about designing and conducting controlled experiments and analyzing the results, I'll be more effective at communicating why the details matter so much.
I will continue to follow the tried and true methods of the poker experiment:
(1) Gather and organize data.
(2) Datamine and identify an opportunity.
(3) Design and conduct an experiment to validate it.
(4) Exploit the opportunity and raise the stakes.
By repeatedly demonstrating the value of this approach in several different domains, the results will speak for themselves and more people will understand the value of data science.
Through the use of and advocating for a scientific mindset, I hope to expand the reliance on evidence-based thinking in some small way. When the world listens more to the data scientists, the world benefits.
My most memorable instance of data-driven decision-making was my development of a simple and profitable strategy for online poker. It also provided content for my guest lectures for Harvey Mudd College’s popular Mathematics of Games course on how the marriage of data and mathematics can result in successful strategies that defy conventional wisdom.
It all started when I saw a “poker corner” segment on TV stating that a player who is short-stacked (has few chips remaining) has only one move: all-in. This was presented as a bad situation, but in my mind it was a great opportunity to make the game tractable. Some poker sites allowed you to start with a short-stack, so if my hypothesis was correct, I could actually profit. Being somewhat risk-averse, I only ever desposited $50 into my online poker account.
After utilizing an initial all-in or fold strategy that allowed me to gather hand history files on my opponents, I engineered an exploitive strategy by calculating the expected call equity (value when my bet is called), fold equity (value when everyone folds), and the cost of patience (the blinds). Conventional wisdom states that repetitive strategies can’t work, and that your specific opponents and position at the table are the most important things to consider. However, my data was telling me that all of this was incorrect and that a handy profit could be made.
While the strategy was simple, the analysis was not. In addition to creating a predictive model to evaluate potential strategies, I also had to estimate my precise edge in the game, in order to use the Kelly Criterion to minimize exposure to bad luck while maximizing hourly winnings.
In the end, my $50 became $30,000, and after sharing the strategy with friends, we all collected crazy stories to tell disbelieving family members.