The Astros and leveraging data

Ben Reiter’s book follows the Astros’ journey to winning the World Series. If you’re a baseball fan the book provides great insights into the people and processes involved in building a winning team over the course of multiple seasons – and the key decisions that lead to winning the World Series in 2017.

If you’re in mobile games – or more generally working with data to make a human-facing, cultural product – then the book offers much more. The book provides great insights into the way the Astros’ organization harnessed the power of data, while remaining acutely aware of some of its shortcomings. The challenges faced by the Astros to integrate data in their decision-making process are the same challenges faced by anyone trying to make a mobile game. There are of course some key differences between a baseball team and a mobile game – or any entertainment product. The most obvious one concerns the different measures of success for each. In baseball it’s painfully clear. Success means beating the opposing team. There is no ambiguity. In mobile games – most of the time – success is never that clearly defined. However, despite all the differences between running a baseball team and designing and managing a mobile game, the concerns faced by the Astros will probably resonate with anybody who wants to leverage data to make the best decisions.

The main concern at the hearts of the Astros’ data driven process is the same faced by anyone trying to make a mobile game: how do you use data when there is no correct answer? Ultimately, for the Astros – just like it should be in mobile games – it’s not about the data and the result. It’s all about decisions. In math (or chemistry, or physics, or any other “hard” science) there is only one correct result. Finding the correct result will lead to taking the correct decision. When you are dealing with humans, that’s not the case. There are no correct results or decisions – some decisions are better than others, but there is no one correct decision to be made. And that’s because any outcome involving people is fundamentally uncertain. First there are many intangibles that come into play. And although those intangibles cannot be measured or incorporated into models and algorithms, they have a very real impact. Second, past behavior is not a foolproof predictor of future outcome. When it comes to individuals, past behavior will be a good indication of what is the most likely to happen – but it in no way determines or predicts what will happen. Finally, high levels of success imply beating the odds – and having a small part of luck. Winning the World Series involves winning the best of 7 games. There are so many contingencies involved that no model can tell you what decision to make. You need to make a judgment call, take a chance and hope things go your way. If you only rely on the data and algorithms at hand, then you will miss out on the longshots that can make the crazy play and win the game. Reaching the highest levels of success means sometimes you need to make some unreasonable bets. You can’t win the World Series by playing it safe. Just like you can’t make a top grossing game by simply playing it safe.

Data and intangibles

The case of Carlos Beltrán is the best example of the intangible contribution of a player. When signed in 2016 Beltrán’s tangible metrics were still decent – hits, batting angle, sprint speed, etc. But his biggest contribution ended up being something that no tracking system could measure. What the book identifies as Beltrán biggest impact lied in his ability to create a positive and inclusive team culture – to cut across the various fault-lines in the clubhouse (between Hispanics and Americans, between old and young players, all-stars and average players, etc.). That impact was impossible to measure but it was certainly real – and crucial in winning the title.

The same applies to features and mechanic in mobile games. You can quantify the improvement to D1 retention of having a log-in system, the impact in D0 conversion of an early IAP offer, or the impact on missions played of running a double XP weekend. But in mobile games – perhaps even more than in baseball – intangibles are key. If you’re in an RPG game, how do you quantify the impact of having a shard system to acquire characters (like Heroes Charge) or a character system (like Brave Frontier)? If you’re in a building simulation/game, what is the first building you should start with? Although it might only be a matter of skins and cosmetics – with no functional differences – first impressions are based on intangibles. And first impressions are key in the mobile space. No data will be able to inform those crucial decisions.

Predicting the exceptional

Models in baseball (and elsewhere) rely on past events to predict future ones. So, this means the predictions of a model work only inasmuch as nothing changes – and one the characteristics of key players is their ability to change the way they play, to learn and achieve new levels of greatness. Models don’t work for individuals who beat the odds, or individuals who transform themselves and the way the play in a fundamental manner. That’s the “growth mindset” that was so crucial for the Astros. The persistence and adaptability of a player that allows him to improve beyond what past occurrences suggested was possible.

Now say what’s a player in baseball becomes a feature or mechanic – or even a game. Up until Clash Royale, real-time PvP games had a poor track record (at best). The same can be said of AR-based games before Pokémon GO or Battle Royale games before Fortnite – the list of examples goes on. The mobile industry is so young past success and failures cannot be a foolproof guiding principle for new games or features. And there is one thing specific to mobile gaming that doesn’t occur in baseball (in baseball, the rules and the game remain the same). In mobile gaming, releasing something new, that has never been done before can be a route to huge success. This doesn’t mean radical innovation is a necessary condition for success – you just need to see what the top grossing landscape looks like. And there is such a thing as taking too many risks and bets. But in the entertainment industry there are many examples of innovators being very successful in the long run.

In Conclusion: using data and achieving greatness

Of course, all this doesn’t mean you can freestyle your way into the highest levels of success. Getting lucky is not a (sustainable) business strategy. Data must be at the core of the day-to-day operations – the fundamentals are in essence data-driven. But in order to reach the highest levels of success, you need that elusive x-factor that resists categorization and modelling. What the story of the Astros illustrates is that you need this x-factor to win it all, and fundamentally data cannot guide those decisions. Had the Astros only relied on data, they never would have drafted Correa, retained the five-foot-five Altuve, signed Beltrán as a 40-year-old free agent or traded for Verlander at $20 million a year. But those players were instrumental in creating a team that would ultimately win the World Series.

One of the key reasons for the Astros’ success was their ability to identify the opportunities for greatness – despite those opportunities not being confirmed by data (and sometimes outright contrary to what data suggested). This is why I like this story and I feel it’s relevant for people in the mobile space. The story of the Astros is the story of a mature use of data in the decision process. To achieve greatness, you need to defeat the odds. And at times that requires trusting one’s judgment, one’s gut – despite no supporting data (or in spite of contradictory data). Data is meant to identify regularities, not the exceptional occurrences, intangibles and outliers that make a champion (or exceptional features and game mechanics – if we’re talking about games). The Astros were able to see when to follow data, and when it couldn’t inform the type of decisions winning the World Series required. In a sense, being a data-driven organization means using the right data at the right moment. Sometimes that means knowing how to ask the right question to get the most relevant answer. And other times it means knowing no data will be able to inform the decision you need to make – and trusting your gut feeling.