Why is the average age of death for male rappers under 30? Why are the best scoring schools the smallest ones? Are large earthquakes on the rise? Do dead salmon have brain activity when shown photographs? Are the Sophomore Slump and the Sport Illustrated Jinx real or imaginary? Do drugs for relaxation help students score higher on the SAT? Why does punishment seem to work better than reward? Why are movie sequels rarely as good as the originals? These are the types of questions that data scientists should be well positioned to answer, but knowing the specific answers isn’t as important as being familiar with the underlying principles and pitfalls which lead less careful thinkers astray.
Businesses call themselves "data driven" and think they know what data is telling them ("up is up"). However, many are not analyzing things in a scientifically valid way and are setting themselves up to be duped by data. My goal is to help train the next generation of data scientists and managers to avoid the pitfalls, whether it's through my book or by directly speaking to them about what I've learned. Most books contain success stories, but mine is mostly filled with "failure stories", which should be more instructive. Data science works, but only if you do it right.
I'm Jay and I am passionate about data science. I love reading about it, blogging about it, and I have a history of using data-driven techniques to tackle and solve problems that seemed unsolvable.
I earned my B.A. in Mathematics at Pomona College, but at first didn't really appreciate how many problems could be solved with a combination of mathematics and computer programming. It wasn’t until my final year at Pomona when the seeds for my lifelong love of data science were planted. My enjoyment of classes such as Investigational Statistics, Mathematical Modeling, Fundamental Concepts in Math (surprisingly similar to programming), and Probability and Its Applications provided the first clue that something like data science could be in my future.
Even after becoming a professional software developer, I hadn’t realized the magic of combining programming with math until my former professor Art Benjamin challenged me with a proof he was working on for his upcoming book “Proofs that Really Count: The Art of Combinatorial Proof.” I didn’t think I could possibly solve such a difficult problem, but somehow he had confidence that I could bring a unique approach to the problem. It turns out that he was right, as I created a custom computer program that used a kind of Monte Carlo approach to search possible solutions while also allowing the user to nudge the program in the right direction. Before long, I had surprised myself by conquering four problems and was excited to get mentioned in his book.
After this, I was a strategic advisor for the winning entry in the international 2007 AAAI Computer Poker Competition. In an article in the San Bernardino County Sun, Michael Bowling, from University of Alberta’s Computing Science Department, stated “they are going up against top-notch universities that are doing cutting-edge research in this area, so it was very impressive that they were not only competitive, but they won.”
Following 11 years as a software developer, I better aligned my career with my true interests by joining the Analytics team at Oversee.net. Early on, I overheard a couple co-workers discussing the statistics they used while running randomized A/B experiments and asked them to explain it to me. Something didn’t sound right and after digging into it, I found that their use of statistics was leading them to false and premature conclusions. Before long I was the manager of the testing pipeline armed with a new methodology for conducting experiments.
In the years after that, I had a front row seat to grand successes and epic failures related to the analysis and interpretation of data. Many of the stories were instructional enough to be included in the book The 9 Pitfalls of Data Science.
The field of data science is exploding right now and will undoubtedly lead to many incredible discoveries. I'm just excited to be going along for the ride and will enjoy learning as much as I can.