In our previous blog post, we made the case for doing exploratory research in a game development context. In this post, we get into the details of actually doing it. Drawing from an example can help unpack the process. Imagine an executive producer tells a researcher the following:
“We want to add more features to the game so that players who have been playing the game for quite some time (e.g. since game launch) have something further to engage with. Let’s do a large sample survey to learn players’ preferences for a list of possible new features.”
Before diving into a proposal for a lengthy survey, the researcher attempts to dig a little deeper into what might be the executive producer’s intent on Slack:
After further discussion with the executive producer, you both determine that there are so many unknowns and before running a survey, it would be important to do some exploratory research to discover why players are motivated to play the game. The high-level plan of attack would then be to use those findings to later test preferences for features quantitatively. You are both excited to learn more about your players!
Framing Your Objectives
Work with your stakeholders to define what you intend the objectives of the research to be. A great way to do this is to discuss the assumptions stakeholders have going into the research and expectations they have on what might be uncovered in the process. As you have this discussion, call out all the possible outcomes by using words like learn, discover, uncover, reveal, dig deeper, understand, etc. Some example objectives might be:
- Learn why players who have played the game for an extended period of time continue to play and thus inform possible new features that tap into their current motivations
- Discover how they engage with the game in ways we do not know of (e.g. Facebook groups, etc)
- Uncover if players are engaging with various types of meta-features[1] in other games to see how they value mechanics such as collection, customization, and identity creation
- Reveal how players think about short and long term goals in games to inform quest or achievement system design
- Dig deeper into pain points or frustrations with meta systems[2] that our game can improve upon
- Understand what will be most important to include in later evaluative research plans
Research is not impactful unless it is acted upon. As you work with your teammates and stakeholders on what the unknowns are, pair each unknown with what action could be taken when answers are found. Doing so will help everyone see the value of the research before it is even conducted. Furthermore, getting the actions in writing will hold everyone accountable.
Designing Your Study
Researchers typically need to get buy-in from their stakeholders on aspects like budget and timeline. Once you have approvals on those items, it is time to get down to designing the details of your study. These details include:
- Creating the stimuli[3] for an unmoderated study
- Figuring out who to recruit to participate
- Determining how the resulting data will be analyzed
Creating Your Unmoderated Study
If working in an asynchronous tool [4] such as PlaytestCloud, a visual aid will be needed to guide the participant through the unmoderated study. One approach is a slide deck that guides the participant through a set of questions to answer or topics that you would like them to discuss freely– similar to the PlaytestCloud Concept test method. As you are using a slide deck, you can incorporate visual stimuli where appropriate. If you do include visual stimuli, don’t worry, these are not concept sketches that you need to have the development team create in advance. You – the researcher – have full control over every aspect of the study and will not need to wait on delivery of builds or art. The visual aids should only exist to clarify what you are looking for or further incite a reaction that you are hoping to understand. A great way to tackle your slide deck is to start broad and then narrow down. Since you will not be able to probe in real-time, you will have to think about how to extract deep insights in a different way. Here is an example set of slides that could do just that:
- Features outside of the core game loop like side quests or avatars. ↩︎
- E.g. rare items, currency systems, collectibles ↩︎
- Imagery, videos or other types of media used in a research study to bring about further thinking or reactions from a research participant. ↩︎
- Asynchronous tools allow you to do research without the researcher being present with the participant; often with the benefit of speed and other physical inhibitors not present. ↩︎
Slide NumberDescriptionVisual Aid1Intro slide thanking the participants and reminding them that we appreciate any and all detail regarding the following questions and that there is no such thing as a wrong answer.Welcome or Thank You themed graphics or emoji2Describe your journey from when you started playing [game title] all the way to where you are now. What has gone through your mind along the way?Images of the game you want them to focus on (so it is absolutely clear you get the data you need)3Describe what went through your mind the last time you created an in-game character that you can play [sometimes called an avatar]?Images of avatar creation tools in other games4What does collection or collecting things in games mean to you?Images of collection mechanics in other games5What comes to mind when you think about achievement in games?Images of achievement systems icons6Describe a goal you recently accomplished in [game title]. What was that like for you?Stock photos of people accomplishing goals7Anything else you want to tell us about [game title]?Blank8THANK YOU so much for your time. We are looking forward to listening to your story.THANK YOU emojis or stock photo imagery
Eight slides, that’s it! A slide deck like this, that allows for participant vocalizations, will extract insights that are much deeper than any free text survey response and doing so will be much less expensive and much quicker than planning for and traveling to do an in-person focus group. There is always the option to do some synchronous work later on, but imagine how quick it will be to simply start and begin climbing the mountain of unknowns and thus figure out where you need to go!
Considerations for Recruitment
Your recruitment goals should center around who you want to learn from, how much time you will have for analysis and when you believe you have collected enough data to uncover your unknowns.
Figuring out who to learn from depends on who you want to later build a product for. While our example points to those veteran players who had played for over a year, there may be cases where you are unsure of who to target. If you are starting completely from scratch, a recommended approach is to get together with your team and brainstorm the groups of people that might be coming to your game. Often called proto-personas or marketing personas, these are best guesses and are derivative of behaviors rather than demographics. Once you have your behavioral segment best-guesses, plan to recruit for at least two people in each segment. More will always be better. A good practice is to start with two or three and see if you are answering your unknowns. If not, run another iteration of your study.
Another factor to think about is how much time you might have to devote to watching the unmoderated study. Analysis per participant is at least double the length of the study. So if five people complete a 30-minute asynchronous study you should assume there will be at least 5 hours of analysis. With this in mind, you can decide if you want to tackle a larger group of participants or start smaller and see first what is uncovered.
Watch and Learn
Exploratory research will always take time, even if you do so in an unmoderated fashion. While a single researcher could take all the responsibility to watch every video and jot down notes, a more inclusive (and let’s be honest, much more fun) approach would be to involve the development team in the process. Not only does this speed up building a team-wide understanding of the player base, but it also prevents bias. In order to fast-track your development team to becoming expert analyzers, a worksheet or canvas to capture information can help (the UX canvas was originally designed as a facilitation tool to help cross-functional teams get alignment). This canvas can be created in whatever shared format works best for you and might look as follows (the example below was made in Google slides):
Analyze Data Collaboratively
Once you have all the unmoderated videos ingested, you should now have summaries and key findings for each participant. The next steps would be to rearrange the green, purple and blue blocks into categories.
In order to do so, have each observer present their summary and findings. You the researcher can listen closely and begin to uncover patterns. This step can be done in real-time, or, the researcher could take the worksheets and create clusters of the findings that go together. For each grouping, create a label to describe how you interpret the players’ underlying purposes.
You may be wondering, how do we know we have learned enough? Go back to your initial high-level objectives and see if you can answer the questions with the themes you have found. Discuss how confident you are that your unknowns are no longer such. Consider what the risks could be if you are not confident enough.
Make Recommendations for Next Steps
To make recommendations, answer the questions you drafted during the earlier phase. What did you need to know and what did you find out? This final step can also be done collaboratively. Go back to your original list of objectives and go through each one and assign the themes you have extracted to the objectives they align to. Use this new visual map you have created to unpack possible opportunities.
For any objective that does not have theme clusters, decide how to proceed. It may be that you need to run further studies. It also may be that you need to conduct a new study with more or different questions. Even if all of your objectives were not completely fulfilled, many of your unknowns should now be knowns. Likely the development team can put together their new list for the larger sample survey and you can begin to dive deeper into the insights to create game designs that you may not have considered.
Accessible Transcript of the chat with an Executive Producer (see above)
Researcher: Hey there! Before beginning my exploratory research I need a bit more information. Can I ask you some questions?
Producer: Yes
Researcher: Thanks
Researcher: Why do we want to add more features to the game?
Producer: The game has been around for quite some time and players who have stuck with the game since launch have seemed to max out on the content currently available.
Researcher: What do we know already about the players who have stuck with the game since launch?
Producer: We don’t know much. Previous research conducted was evaluative in nature, plus, long-time veterans of the game are essential to KPIs like lifetime value and it critically important for us to learn more about them.
Researcher: How do you hope the added features will change the player experience?
Producer: We hope to drive KPIs like retention and revenue similar to how a recent competitor title added new features and thus impacted the KPIs in focus.
Researcher: What is your thinking on testing preferences for a predetermined set of possible features?
Producer: The development team has been asked to put together a list of features that they are interested in implementing but they want to know what might be most preferred before diving into engineering.
Researcher: What groups of players might we need to focus on?
Producer: We should focus on long-term players who had an account with us since version 1.5 or earlier of the app.
Researcher: What will you do with the information obtained?
Producer: We expect to have an informed set of features that they could test at scale as the development team prefers some hard numbers before they dive into engineering.
Researcher: What outcomes are you expecting from the research and how will you take action on them?
Producer: Management expects us to learn why the long-term players have stuck around for so long, how to best design for them so they continue to play, and how to turn newer players into veterans that, in turn, also stick around.
