To be relevant analytics should always be associated to decision-making – even if it’s only loosely. At times it can be easy to lose track of that. You turn to analytics to answer a specific question or to help make a specific decision, then one question leads to another, and then you are left trying to answer something that although interesting, is very abstract or descriptive and won’t have any operational impact. Finding an answer becomes an end in itself. In those cases, it’s no longer about informing a decision.
Analytics really only has value inasmuch as it allows us to make better or faster decisions (ideally both). Differently stated, analytics has a huge value because it is intrinsically practical. The one thing that sometimes gets lost in the picture is that analytics is not just practical because there are multiple data points that inform decisions. Ultimately the value of analytics is the greatest when it becomes a part of the team and the organization’s culture. Having a data-driven state of mind permeates the decision-making process, the evaluation criteria, and ultimately the value and nature of the product we are working on. It’s about having the numbers, but more importantly it’s about the role we give numbers in the development process, about knowing what type of questions we can ask, and when – and when not – to turn to data for answers.
Having an analytics culture means the standards for decision-making are both external and of the most rigorous nature. By external, I mean that analytics helps everyone emphasize the fact that the product is for an audience. The ultimate criterion of success is not what we like or what we think is best. The ultimate criterion for success is the end-user. We are not making games for ourselves. That means that making sense to us, or being consistent and coherent on paper cannot be a valid criteria. We are never measuring the performance of the game considered in a vacuum. The performance we measure is the user’s reaction and reception of the game. Because of that, user actual behavior is always the final criterion. Every feature or product is defined in terms of its output, in terms of how it impacts the end-user. A data-driven culture goes hand in hand with an emphasis on Objectives and Key Results (OKRs).
Standards of the highest nature means analytics can help us get a high level of precision and certainty. Analytics is the way to clearly, unambiguously and indubitably determine what the end-user is doing. Anytime you hear a statement along the lines of “users are playing PvP” or “users are buying PvP energy”, that’s something that can only be acceptable as long as data is backing it up. A strong analytics culture means there is no room for speculation when discussing tangible actions that can be objectively verified and quantitatively measured. A data-driven culture means that when factual matters are discussed, data is brought to the table. “What” questions are questions that are descriptive and can be clearly and unambiguously answered. For example, if you want to know how many players play PvP, the answer is straightforward. Only data can be legitimately considered for those types of questions – and if you don’t have data, then you should simply go get it.
On the other hand, there are some questions that cannot be answered and verified by data with the same criteria. Statements such as “why are players not playing PvP“ or “why is retention so low (insert angry face here)”. And of course, the fact that these questions cannot be measured or verified by data doesn’t mean that they are not important. On the contrary. Anything that pertains to user perception, preference or motivation is of the utmost importance. The most important aspect of any entertainment product is the audience’s preferences and perceptions. But subjective feelings, motivations and perceptions are not tangible. It’s one thing to say 75% of your DAU plays PvP. It’s another to say why your active users play PvP. Anything that has to do with motivations and perceptions cannot be measured or analyzed by the same criteria you would objective actions. “Why” questions are questions that have to do with user motivations and preferences. Those questions involve understanding the reasons behind the actions. And these questions can never be answered with the same criteria as what questions. There isn’t a single answer for why users like PvP. And that’s not because the tools, methodologies or models we use are not advanced enough. It’s because user motivations and preferences are intrinsically murky. And user motivations are not directly observable. User actions are the manifestation of their motivations. You must induce from the observed tangible actions users’ subjective motivations.
Making a game (or any entertainment product) means you are trying to make something the end-users will like and find appealing. It’s all about building a product that matches user expectations and preferences. Ultimately having a good understanding of the “why” questions is what’s most important. If you are considering adding a PvP mode, then no “what” question will provide you with the answer. Only understanding user motivations and preferences will help you decide to implement a PvP mode or make the game more social – or cancel a feature. Of course, data should inform your decision – but data won’t provide the answer. Analytics will help you color and assess the situation. But to a certain extent you’re on your own – and you need to embrace it and believe in yourself and your vision (alternatively, you can say with Pascal: “The eternal silence of these infinite spaces terrifies me”).
A data-driven culture of means knowing there are some cases where data cannot determine the best course of action. If there is a what question, then only data should answer. But when there are why questions, you need to know data can only help you so much.
Knowing which questions analytics can and cannot answer is not a theoretical consideration. It has very practical consequences. Turning to data for a question that cannot be answered by data is not just inefficient. It’s actively counter-productive: you are waiting for an answer that never comes. More crucially, you are not looking in the right spot to make your decisions. Data allows to have a clear and precise understanding of “what” questions. But anytime there is a “why” question you need to be able to embrace the uncertainty that goes with them. So, knowing when to turn to data – and when not to turn to data – is the first step in leveraging analytics the best way possible.
Of course, taking decisions in a context of uncertainty doesn’t mean anything goes. It’s about taking a calculated bet – not playing Russian roulette. Analytics should help evaluate the risks and identify where potential opportunities are the greatest. But the point remain that you are on your own and operating without a net. You need to make a judgment call based on your vision and your beliefs. But it also means that once you made a judgment call, you can turn to data to evaluate it. Data can help you measure and appreciate the course of action you took. OKRs are only valuable inasmuch as they can be measured, and goals can be unambiguously determined as being achieved or not.
In that respect, a data-driven culture is very close what Gary P. Pisano wrote in his HBR piece on innovative cultures. A data-driven culture means you know when data cannot determine the course of action and you must trust yourself. But it also means once its implemented, you confront your choices to the reality of their impact. You need to be willing to take risks, but then you need to be able to evaluate with no concessions the impact of what you implemented – with actual data.
Alexandre Macmillan 
3 comments