(Building data-driven teams in mobile games 1) How to think about your game

The reason you want to be data-driven is to have impact. End of the day, the thing you are trying to achieve is probably user in-app purchases or ad viewing (either at a cohort level to recoup your marketing investment or at the level of your active users to increase revenue). But there are no restrictions to the type of impact you could be after. And for a number of different reasons you are probably looking for intermediate-level impact to increase your lifetime value or arpdac (“increasing LTV” is hardly an actionable strategy): sessions played per day, average number of losses before a win, number of friend requests within 3 days of install, consecutive Sundays played, etc. Really, impact could be anything that is relevant for you.

There are many different types of impact you could be after. But one common thing throughout all this is that you want the impact to be real. And in order for that to happen – and to lead an organization where many different individuals are working to achieve a same impact – you need to be thinking of impact as something tangible, observable and measurable. That is why, regardless of the specific type of impact you’re looking for, in mobile games impact means player behavior. To be data-driven, you need to be thinking about what you are doing in terms of impact. And defining impact in a way it can be clearly observed and objectively measured by everybody involved in the process. That’s what will help you produce insights that can be generalized and applied in different circumstances.

The above statement might sound obvious. However, you are probably dealing with many different tendencies that might drive you to look at things differently (in a less productive way). If you’re in the mobile gaming business, you’re in the entertainment business (a very specific type of entertainment – different both from linear entertainment and I’d argue even console games in some key way). You’re creating a cultural product, you job requires creativity, and you are probably surrounded by creative people. When that happens, one natural tendency is to look at what your creation is. And when you look at things like that, you quickly step into what your creation should be. But that’s not necessarily the best way to look at your game, to work as a team, and to have significant impact. And if you look at things that way it might be harder to implement a data-driven culture.

For example: the impact you can be looking for can be “make the feature fun”. Alternatives could be: make the feature “consistent/fair/original” – the list is endless. This is defining impact in terms of what the game is – and this is what you want to avoid. When you are looking at what the game is – instead of looking at what the game does to player behavior – you are running into a series of different problems that will hurt your ability to increase LTV or revenue.

1. First, when you focus on what the game is or what it should be, you are thinking about your game in isolation from an audience. Your game in itself is nothing. The performance of your game exists at the intersection of a series of features you are building and an audience interacting with those features. If you focus on what your game is, you are leaving out a very important part of the equation.
2. Second, from a logistics point of view, it’s going to be very difficult to get everybody on the team to agree to what “fun” is. Ultimately what the game is is something highly subjective. For example, your definition of fun is probably not the same as mine. But both definitions of fun are equally valid – so it will be very difficult to move forward in a satisfactory way if we both want to be making the game fun. There is no common or objective standard you can appeal to to ensure people on the team are aligned.
3. Third, even if you were to somehow achieve this alignment (again, doubtful), “fun” refers to something subjective and intangible. You couldn’t clearly evaluate the impact of your actions. So even if you were to reach an agreement on what you wanted to achieve, you probably couldn’t come to an agreement when comes time to evaluate the impact of what you’ve achieved. And as a corollary you can’t really learn and improve because you’re not able to evaluate the impact your actions had.
4. Fourth (there are probably many more points that could be made), if you focus on what your game is rather than what it does, you might be basing your work on faulty assumptions. For example, you might feel like building a fun game is the way to make it successful financially (because a fun game naturally monetizes by virtue of it begin engaging – my belief is it doesn’t exactly work like that). I’d argue there are many examples of that not being the case, but that would involve me relying on my subjective definition of what fun is. And that’s precisely the problem here. The problem here is that you are defining the input subjectively (“fun”) and the output in a more objective way (“increase in spending”). So, it becomes nearly impossible to evaluate in a consistent and reproducible way the impact of what you’re doing – and applying that learning in a different context in the future.

That’s why you need to be thinking about your game in terms of what it does to player behavior. Behavior is something objective (the motivations behind the behavior are not). It can be defined in a way where everybody observes things the same way. The criteria of failure or success (are we seeing that behavior or not?) are objective and beyond debate. For example: “I want the new feature to increase customer average play time” is something everybody is likely to understand the same way.

In order to lead a data-driven organization and build successful products, you need to be keeping the entire loop at the level of player behavior: what is the behavioral impact I want to have with a given feature in order to generate another behavioral impact? For example

I want to change the points received in the event so 15% of players will be within 100 pts of achieving the event milestone in order to increase the percent of event participants spending (because you’ve observed the majority of users spending are within 100 pts of the event milestone)

The first step you need to do if you want to build a data-driven organization is stop looking at what the game is (or should be) and focus on what your game does. You need to consider your game as a machine that impacts player behavior.

This focus on impact is probably why a data-driven culture fits within an OKR framework (I cannot recommend enough the books written by Andy Grove and John Doerr). OKRs is the structured way to be working toward impact. Grove called it an “output-oriented approach to management”. And in order to achieve that, you need to be data-driven.

Focusing on what the game does means you need to be defining what you do in a way that is objective and not open to interpretation. You need to define things in an exhaustive, mutually exclusive way to everybody can understand things the same way and come to the same conclusion when evaluating impact.

Furthermore, looking at what the game does forces you to always to be intentional. You must be able to clearly assess and evaluate the impact of what you are doing: what specifically are you trying to achieve? What is the rationale behind the actions you are taking? When you are thinking about impact – and when you are focusing as an organization on impact – you need to have something that is measurable and that is objective. When you look at your game from the perspective of what it does, you are also putting yourself in a position to continuously improve. Defining something clearly in an unambiguous way is also a requirement to be making claims that can be falsified – and that is one key requirement to produce knowledge that can be reproducible.

That’s why to be data-driven, you need to think about your game from the perspective of what it does in terms of user behavior. Being data-driven is not so much about a number, a result. It’s about a specific outlook. It’s about looking at things in a way where your actions can have greater impact. And in order to produce learnings you can use to improve your game’s performance, you need to

1. Be able to reach clear conclusions when evaluating the impact of what you are doing
2. be able to generalize things at a level high enough to do it again in a different context
3. learn and course-correct in order to be continuously improving

You don’t get any of that when you focus on what your game is. All that you get by being data-driven. And the first step in being data-driven is focusing on what your game does.