User Research Throughout the Game Development Process

As the prominence of user research in the tech industry continues to rise, more and more game development companies are looking to user researchers to improve the player experience within their title libraries. We recently spoke with two research strategists at Outfit7, Anja Arhar and Nives Đorđević. They confirmed for us that user research’s place within the game development life cycle has become much more involved throughout the whole process. What’s more, they said user researchers are better able to enact change throughout the course of a game’s development. This is in stark contrast to how such planning was previously established, whereby user research was generally considered quality assurance (QA), completed at the end of the development cycle, and only as a final pass.

Here, we will discuss how user research plays a critical role in the development of digital games, how that role has changed, and what types of research can be done at each step of the process to benefit your game’s first time user experience (FTUE) and beyond!

In the beginning: User research at the concept stage of development

According to the Nielsen Norman Group^[1], when an organization — or in this case, a game development studio — has reached a certain level of UX maturity, they have successfully integrated a user-centered design process within their UX or user research departments, and throughout their culture and at all leadership levels. This means they have placed adequate trust in user researcher involvement and initiative early on in a game’s development life cycle, have supported their integration into the development team, and have helped them enact positive change at all steps of the process.

As a user researcher, being involved early and often is ideal, and offers up a chance for researchers to assist with decision-making processes, and to steer your game towards better UX through their user research expertise.

Nives and Anja, for instance, pointed out that early on, methods like focus groups^[2] allowed their user research team to better understand how the game concept was being perceived, and whether that perception aligned with expectations from the game development team. This was also done with a convenience sample^[3] of players who had played previous titles of the game in the past. This allowed the research team to gather qualitative information about the general sentiment of a potential player’s reception to the concept being proposed, allowing the user research team to incorporate feedback from participants such as “needs” and “wants” into categories for further study.

Testing before the build: The importance of pre-production user research

Generally at this stage of the process, ideas are still amorphous and flexible, so being open-minded is key, as is testing all sorts of different concepts, as well as keeping lines of communication between the user research and development teams open.

Additionally, as a game is still in the pre-production state, this is an ideal time to complete a competitor analysis^[4] of games that may have similar mechanics or audiences. This exploration allows the design team to catch any potential issues of incongruence between audience and product, as well as to gather feedback on what players want or expect to see, even before they have a working build!

Finally, testing ideas to solidify them into assets in-game is an ideal target for this step, as there is still plenty of flexibility and opportunity for change. As Nives and Anja relayed in their talk, even testing icons with paper prototypes was found to be valuable (not to mention cost effective, thereby improving feedback accessibility) and will continue to aid designers further on in the development process as they learn what works best and what needs improvement.

Research while creating the game: User research during the production phase

While games are being produced, there are usually several benchmarks or milestones that production and development teams utilize to ensure the game is on track to release, and that each step of the development process is up to a satisfactory standard before moving on. Obviously, this method offers user researchers an opportunity to intervene and help the team better understand the overall game experience, knowledge, and usability from a player’s perspective. Nives and Anja described the ways in which detailed, iterative playtesting offers fantastic results during the production phase. Indeed, much of the game will have been fully formulated at that point, with the builds of the game offering players a real glimpse into what the final game will look and feel like.

This phase is important for user researchers on the team, since they can test several different elements of gameplay, including game mechanics, visual design, usability and accessibility issues, iconography, and interface design. As these elements become solidified within different builds, testing early and often becomes vital, and impacts the extent to which valuable feedback makes its way from players to the development team.

One important aspect Outfit7 chose to highlight is the tutorial process: a necessary but sometimes overlooked element of digital games that scaffolds necessary knowledge from the game to the player. In addition to gathering useful qualitative data through a think-out-loud protocol^[5], Nives and Anja mentioned that quantitative data, such as time-on-task to complete tutorials, can be helpful in discern player behavior such as whether players are rushing through the tutorial and will inevitably churn^[6]. Additionally, the use of A/B testing^[7] when implementing changes to specific parts of the build will allow players to compare and contrast those alterations, and ultimately express their preferences.

Out in the players’ hands: What user research can do during release and post-release

As with many mobile games, Nives and Anja - and the entire team at Outfit7 - focused on the first-time user experience (FTUE) during the production phase of My Talking Tom Friends, to ensure that players understood the game and would continue to play. While FTUE is vital to the success of many digital games, it is also important to understand how players are playing your game in the real-world, and how their play behaviors can change throughout the gameplay process. While many games use data analytics and telemetry data^[8] to understand what players are doing, an improved understanding of the qualitative side of how players are feeling is essential.

Anja and Nives discussed their usage of diary studies^[9], which, according to their practice, was a new methodology for them. This also spoke to their commitment to iteratively learning more about the method while also learning through the method. As is usually the case, their diary study was meant to look at potential reasons for player churn. Nives and Anja thus created a test plan which involved five days of gameplay at some point over a ten-day timeframe. Participants then recorded their thoughts and feelings during play sessions.

Anja and Nives’ diary study was used in combination with short surveys and wrap-up interviews at the end of the full study protocol. This exploration enabled in-house researchers to better understand player stories, their different playstyles, their feelings surrounding the game, and their motivations behind displayed behaviors, specifically from a player’s perspective.

As a result, developers were then better at referring to specific player needs, not just in terms of more skilled/less skilled players, but in terms of what motivates players, what keeps them engaged, what sorts of time commitments players are willing to use in the game, in addition to other important aspects which can ultimately guide development. These insights can inform future game updates, especially in games as a service (GaaS)^[10] models, where player retention is especially important.

Concluding thoughts: integrating user research from the start within the game development process

Game development can be a lengthy, arduous experience, yet it offers a plethora of opportunities for user researchers to improve the experience players have in your game. As Nives & Anja have discussed, user researchers can have a dramatic impact on a game’s development cycle, and they can play an influential role in elevating a game to be the best it can be.

As user experience research becomes more common in the tech industry, it is vital that every part of the team is working cohesively, and that they are communicating every step of the way - ideally starting this process as early as possible. This more than anything will have the greatest impact on both the product and on overall player experience.

https://www.nngroup.com/articles/ux-maturity-model/ ↩︎
a type of group interview with participants who have similar experiences or behaviors, to gain an understanding of their reactions and responses to predetermined research questions. ↩︎
where a sample is drawn from part of a population because they are easily accessible, though not always representative of the total population. ↩︎
an assessment of strengths and weaknesses, especially in comparison to your own product, of current or potential competitors in the same market space. ↩︎
a methodology that asks the user to engage with the system while actively verbalizing their thoughts and ideas as they are using the system. ↩︎
churn rate or churning is the act of users playing your game and then stopping, ultimately never returning. This is especially notable in the mobile and free-to-play markets. It is the opposite of a retention rate. ↩︎
a type of experiment where two different variants of a similar object/screen are shown to users at random, where the variant is a simple change that might affect user behavior. ↩︎
data that is remotely sent back to the developer from folks playing the game, such as time spent in specific areas, latency between players, and so on. ↩︎
collecting qualitative data from individuals over the course of an extended time period -- called longitudinally -- logging specific information about the activity that is being studied. ↩︎
where developers and publishers provide game content on a continual revenue model, monetizing video games after the initial purchase, or as a way to make money within a free-to-play model. ↩︎