Objectives and Key Results

OKRs are a planning tool. The purpose of OKRs is to improve production outputs and elicit peak performance. Because it’s a planning tool, it’s forward looking. The OKR framework is meant to help you set ambitious goals and achieve them (or at least get as close as possible to achieving them). OKRs are not an evaluation framework – you don’t look back on your key results to determine if you have succeeded or not. Objectives are what you want to achieve – they are the “what matters” of the process. Everything else  is a means to an end (including key results).

The OKR framework helps answer 2 different questions. The objective answers the question “where do I want to go?”. The key results answer the question “how will I pace myself to see if I am getting there?”. Andrew Grove – the author of “High Output Management” and the father of OKRs – gives a very clear example of this. Let’s say you want to drive to the airport to catch a plane that leaves in 2 hours (that’s the objective). Your key results might be: reach city A in 10 minutes, city B in 20 minutes and city C in 30 minutes. Your key results are what will help you get a sense of progression and help you pace yourself while you’re in route to the airport. If you’ve been driving for 30 minutes and haven’t reached city A yet, you know you’re behind on your plan (and need to do something about it – for example drive faster). But on the other hand, you might reach city C in 30 minutes and still not make your flight. Maybe you needed to have reached city C much earlier. Maybe you’ve been driving in the wrong way and those cities were not the ones you should have driven through in the first place. Maybe there was an important factor you didn’t take into account (the check in process and security check). The main point here is that key results are the indicators you will use along the way to see if you need to adapt and change your course of action. Hitting your key results is not a reason to celebrate. Hitting all your key results means nothing if in the end you don’t reach your objective. In a similar vein, not hitting any of your key results doesn’t matter if you reach your objectives (but in that case, you’re bad at planning and doing OKRs wrong).

Of course, no indicator is foolproof. The best structured plan will be useless if you’re not tracking the indicators that will help you fulfill your objective. Key results are milestones; they are indicators that will tell if you are on track while you are working towards your goal. Key results are indicators of progress relative to a plan (while you are working on completing it). Key results are not criteria of success. I insist on this because it can be very easy to make reaching the key result your main criterion of success. That’s because achieving your key results (or not) is usually something unambiguous and straightforward. It’s clear if you’ve hit your key results. And objectively evaluating progress relative to a target provides immediate gratification (even if it’s not the most relevant or meaningful target).

Grove insists on the way in which indicators have the power to draw your attention towards them. Any indicator you chose to monitor will have an impact – even if it’s unconscious – on your priorities, roadmap and efforts. Say you decide to track a minute per session KPI. Once you start tracking that KPI and monitoring its evolution, you are more likely to pay attention to variations of that metric. And you’re also more likely to try to impact it positively when you are designing features or tuning your game. The important question is: is increasing or decreasing that metric going to significantly help you achieve your ultimate objective (which more likely than not is increasing your LTV)? If the answer is no, then you are not utilizing your resources (both development and cognitive resources) in an optimal way.

In summary, you must be very selective with the key results you chose and the way you use them. First, key results are tools, indicators that are supposed to help you achieve your objective. You should be careful not to mistake your key results for your objective. Second, key results must be operational. So, you shouldn’t have so many indicators you lose track of what you’re trying to achieve. Tracking metrics is not an end in itself. And it’s very easy to have KPIs creep up to the point you have so many of them you no longer know how to move forward. You need to have a strong degree of confidence that those indicators are relevant. Here relevant means they will help you determine how you are progressing towards your goal, and you will be able to make the appropriate course adjustment depending on the way those key results vary. If that’s not the case, your key results are not helping you – and because you are dedicating resources and attention towards those key results, they are actually counter-productive and hindering your efforts towards your goal.

 

Define measurable objectives and key results

A common theme in the OKR literature is the cascading nature of objective and key results in an organization. OKRs trickle down from one layer of management to the lower one. That means the key results of your boss become your objectives. And your key results become the objectives of people below you on the organizational chart.

This matters when thinking about game development for a number of reasons – even if you are working in a flat and participative organization. In the OKR literature, there is a strong emphasis on the fact that objectives are the direction we want to go. They are aspirational – and they can be a matter of debate. One example of objective Doerr provides in his book is “dominate the mid-range microcomputer component business”. That objective is not unequivocal and requires some form of judgment and subjective appreciation (“dominate” and “mid-range” can be understood in a variety of ways and mean different things for different people).

On the other hand, key results are indicators. In order to have some functional value, key results must be unambiguous. You must clearly see if you’ve achieved your key results. But because someone’s key result is someone else’s objective, that means that both objectives and key results can be defined in a measurable way. You should be able to clearly answer “yes” or “no” to the question “did I achieve my key result?”. But the cascading principle means you should also be able to answer “yes” or “no” in the same unequivocal way for many of your organization’s objectives (especially the lower you go in the organizational chart). It’s beneficial to apply the same principles of clarity and objectivity to the objective part of OKRs. Whenever possible, you should define your objectives in a measurable, clear, unambiguous, and mutually exclusive way.

Ultimately, using OKRs this way for game development is a way to structure and formalize a data-driven approach to feature design and product management. When looking at the feature design process in free-to-play games (and probably many other areas), the objective of a feature needs to be measurable. And having clear unambiguous goals can in turn help guide the design and development process. OKRs can help you answer the question: “how should I build this feature in order for it to generate the desired outcome?”.

In general, the highest-level objective of a game will be financial – for example generate a given amount of booking or EBITDA in FY20. Defining an ambitious goal is crucial here. But having that information won’t help you design a game or a feature. Say you want to achieve that business goal by developing a new RPG. At that level the key results can involve a specific LTV, a threshold of UA performance, and operational optimization. It would look something like this.

 

Side note: a lot of my examples use a very specific user base – US/iOS/organic. That’s for 2 important reasons. First, US/iOS is probably one of the most important segments of users for any game that hopes to be successful (although that might change in the future, it’s still the reality for most western publishers today). Second, you need to always define your metrics relative to a specific segment of users. If not, changes in metrics might reflect changes in userbase rather than changes in the game itself. When you look at organics, you are controlling for the impact (and fluctuations) associated to UA campaigns. The LTVs of your paid installs could greatly change depending on your creatives or specific partners – not on the internal functioning of the game. Of course, changes in ASO will change the install rate (and therefore quality) of your organic installs. The point here is not that you will ever find an ideal situation, where your players are in a vacuum and unaffected by any external factor. But you can control for some important variations and minimize the impact of those external factors by rigorously defining what you’re measuring. The point is to define a key result in the best way possible to make sure you are tracking changes in your game – rather than changes associated to external factors.

Use OKRs to design your features

The challenge OKRs are meant to help you address is: what are the specific features/actions I need to implement in order to achieve my objective? The challenge is never really to know you need to increase your LTV. The real challenge consists in determining the in-game levers you need to play with to achieve that.

Now this isn’t chemistry. There isn’t a predefined recipe that will guarantee you will achieve your objective. But experience, a well thought out plan, and tremendous focus will increase your odds of succeeding. Again, OKRs are not a recipe for success. It’s a method to define the steps the most likely to get you to your desired destination. Postmortems are crucial here. As mentioned above you shouldn’t be looking retrospectively at your key results to see if you succeeded. The only measure of success is achieving your objective. But you definitely should be looking back on your previous OKRs to identify the key results that helped you achieve your objectives and those that didn’t. That way, you get better at identifying the key results (and the levers) that help achieve your goal.

Let’s continue with the previous example and assume we’re now at the level of the product owner (his/her main objective here is to reach the LTV target). For example, after evaluating the game you’re responsible for, you might see that you need to do more to get your d90 LTV up to $2.00. With experience, you might see that there are 3 big opportunities you should focus on to help reach your LTV goal: increasing your day 0 conversion for US/iOS/organics to 2%, day 7 redeposit rate for US/iOS/organics of 2.5%, and increase your customer next-day return rate to 90%. The last one doesn’t need to be specific US/iOS/organic. Focusing on customers is already a great way to control for a certain level of engagement and enthusiasm.

The plan for the rest of this example will be designing a feature to increase your customer next-day return rate. This is where OKRs benefit from having some experience – and being able to rely on some high level and abstract principles of player behavior. OKRs set the goals of the feature – so you should be able to determine some specific aspect of that feature’s design. If you decide to implement a new game mode with the objective of increasing the next-day return rate of your customers, what are the aspects of a feature that can make next-day return rate go up?

The measurable/quantifiable approach to OKRs probably stops when you start actually developing a feature in-game. There are no equations you can rely on. The feature in this example works like a black box. Grove discusses the concept in his book. A black box is a way to represent in a simple way the production process. As Grove writes: a “black box sorts out what the inputs, the outputs, and the labor are in the production process”. In this example, there is an input (in this case active customers) and a desired output (90% next day return rate). The feature is the “labor” that will produce the desired output.

Another way to refer to this is “data-driven design”. You know what your goals are. So, if you need to implement a feature that accomplishes that KPI goal, what should that feature look like. What are the levers the feature must have to accomplish that objective? Here it becomes kind of a task in applied sociology. What are the in-game mechanics and features that can have a given impact of your userbase?

What that means is that once you start designing the feature, there is a measurable objective – the output of the feature. But you are no longer thinking in terms of key results. Rather, the ORK framework will provide you with design guidelines you need to follow in order for your feature to have the desired impact.

 

For example:

Objective: Increase your customer next day-return rate

What you need:

1) You need to build a feature with a high-level of engagement (a high % of DAC interacting with the feature on a daily basis). If few customers engage with the feature, it won’t have a big impact on your customers’ overall engagement levels. In order for that to happen, you probably want to make that game mode fun. So, you should start by looking at the currently existing game modes with the highest level of daily engagement. What are the main characteristics of that game mode? Give your players more of what they already want. Also, you need to make it easy for customers to engage with that game mode. That means you should create a game mode that can be played and completed in a short amount of time. If you already have player data, you can get a sense of how long it a “short amount of time” is.

2) You need to create a high anticipation to play the feature. In order for that to happen, you can create some kind of scarcity and time limit. In other words, an appointment mechanism (imagine players can only play that feature once a day at a given moment). Creating that sense of urgency and scarcity on a daily basis can create a low commitment (effort-wise and cognitive-wise) for players to play. So having a contextual push notification with a clear call to action can reinforce customers’ anticipation of the feature and keep them playing day after day.

That would look something like this

 

Now this is an ideal example – in practice you can (and should) design things in such a way that one given feature should help increase multiple different metrics. In particular, I’d argue that any feature you invest time and resource into developing should directly monetize (have something about the feature you can buy). Every feature doesn’t have to be mainly about monetization. But at the very least each feature should have a feature to directly monetize players engaging with it. There is rarely anything to gain by expecting an engagement feature to indirectly impact monetization metrics (“if players stay longer in the game they’ll naturally spend more on their own” is a dangerous assumption). In this example, you can probably allow players to play the game mode outside the planned time slot for a cost, and also have some consumable and/or permanent items when you play the feature. The main idea here is that you shouldn’t build a new feature to get players to buy something that’s already for sale in your game. Fix your economy (or the thing you’re selling) instead. If you implement something new in your game, make sure it comes with something new that can be purchased.