In my previous post, I argued that being data-driven requires you to adopt a player centric approach. More specifically, a player-behavior centric approach. To be data-driven, you need to focus on a realm of things that are tangible and objective (as in, everybody can agree they are observing a same thing the same way) and measurable (so you can assess magnitude and impact). That’s why you need to focus on what your game makes your players do – not on what your game is in some ideal and subjective way.
However, there is an important thing to keep in mind here. You need to be considering your game from the perspective of what your game does to player behavior. And at the same time, your job involves releasing new features and balancing existing content. However, you don’t control the impact your features have on player behavior. The paradox here is that the thing you control (building and balancing a feature) does not directly determine the thing you are trying to achieve (player behavior). You control the features you build (of course because you’re working as part of a team the notion of “control” is meant in a collective way here). But you don’t control what those features make your players do. Because of that, being data-driven is not about outputs. Being data-driven is not about the features you build. Being data-driven is about processes. It’s about the way you go about building features, iterate and continuously improve to get closer to the desired outcome. Being data-driven is ultimately about the culture you need to take (calculated) risks to achieve the greatest impact, and continuously be iterating and learning. Being data-driven means always be thinking about the output in a temporary way and focus on being experimental and continuously be improving through iteration and hypothesis testing. That’s how being data-driven will help you gradually build features that will help you approximate the desired outcome – always defined in terms of player behavior.
Let me illustrate a bit the point that you don’t control the impact your features have on player behavior. If you are working on a game and implementing new features or changing the balancing of your economy and live ops, then hopefully you are seeing changes in your metrics. That’s usually how we naturally tend to think of KPIs. KPIs reflect the performance of your game. And changes in your KPIs reflect the impact of your features.
But that’s not always the case. Let’s take a specific example most of us can relate to. Say for example you are monitoring the day 3 LTV of your game, and all of a sudden notice it drops. It could be the result of a new feature you implemented and that’s not successful. Alternatively, you might have been featured in a low-monetizing country that on aggregate drags your overall d3 LTV down. That’s the reason why it’s always good practice to try to control for changes in your userbase – especially when looking at cohorted metrics (and always look at the performance of your game from the point of view of one segment – I usually like to look at US/iOS/organic).
This example shows 2 things. First, you can see that KPIs don’t simply reflect the performance of your features. The game’s performance doesn’t exist in a vacuum. There are 2 things impact your metrics: your game’s features themselves, and the audience playing your game and engaging with those features. There are the features and the audience engaging with those features. Your game’s performance lies at the intersection of the 2. Second, you don’t have control over the part that involves the audience’s behavior. You can sometimes influence it – but you don’t determine it. For example, you don’t control the fact that on average the monetization potential for Spanish installs will be lower than the average monetization potential of US installs. In the same vein, you don’t control whether or not players will find your game compelling enough to come back. If you make changes to your PvP ranking system, you need to hope those changes will fulfil a preference or need that is shared by a large enough portion of your userbase. You start with an assumption (and sometimes assumptions are pretty hard to validate), you start with a vision for what your game should deliver, and you can only hope that your game provides something a large enough audience will find meaningful and want to engage with. But the way players will engage with your game is out of your hands.
Once you release your game out in the wild, it’s no longer your game. You don’t determine what it does. You are releasing a cultural product, an entertainment product. And the way it will be received, used (and mis-used) is up to the audience interacting with it. Once you release your game on the market, its success depends on the audience. The best you can do is release something you believe in, something you have a vision for – and hope that it will be well received. In a sense, your game takes on a life of its own once it’s in contact with the public.
This is important because if you want to have impact and improve the impact you’re having, you need to focus your efforts on the aspects you can control. You can’t control player reception to your game. But you can control the features you build. More importantly, you can control the way you adapt to the player behavior and changes in player behavior you are observing. That’s one of the key aspects of being data-driven.
Begin data driven is not about building a finished feature that has the impact you are hoping for. You will never get the best outcome on your first attempt – although being data-driven and doing your homework before releasing the first iteration should help you be in the right ballpark. Being data-driven is not about the feature you build. It’s about the process of building those features – being experimental, course-correcting when you will inevitably mess things up and get it wrong, and produce learnings that will be general enough to be reapplied in another context.
That’s why it’s a good idea to start with the way players are playing, and try to reinforce those natural tendencies. You can’t make players do things (play a game mode, come back to your game, spend, etc.). But you can influence what they do – and in order to do that the most effective way I know is to be attentive to what they are already doing.
Adopting a data-driven process will allow you to take chances, focus on achieving the greatest impact, produce learnings, minimize the cost of failure and most importantly learn from those failures (and those learnings in themselves are tremendously valuable). So, building a data-driven culture requires you to build a solid culture of psychological safety where everybody is encouraged to try, to take chances and risk failure, and where failure is defined differently. The criteria of success are not going to be on whether or not your feature was successful (especially on the first try). Building successful features is of course your end goal. But that will derive from looking at what players are doing, formulating and validating assumptions, making mistakes, learning from those mistakes and improving. That’s what being data-driven is all about. Building successful features will be the natural side effect of being data-driven. The criteria of success will be on the way you approach building a feature, not directly on the impact of your feature. It will be on the attempt to achieve greater impact, and on the learning that will help you get there.
Begin data-driven is about adopting a process that allows you to continuously adapt and course correct to (hopefully) orient player behavior in the direction you want. Said in a more fancy way, being data driven is not a destination. It’s not about the features you build. Being data-driven is a journey. It’s about the way you approach game development and build features. Being data-driven is about the constant iteration, experimentation, and course-correcting that informs the way you build your feature.
Alexandre Macmillan 
One comment