Playtesting covers a range of very powerful methods for integrating player insight into your development process.
PlaytestCloud has recently added Live Playtests to its toolkit, in addition to traditional unmoderated research. However developers I speak to are often unsure which is appropriate for them, or how best to use them to test their games.
In this article I’ll introduce the differences between unmoderated and moderated research, and give a steer on when to use each.
Moderated vs Unmoderated Research
Unmoderated research has long been part of the PlaytestCloud suite of tools, and can allow teams to collect player data at scale.
In an unmoderated study, the tasks and questions given to players are defined in advance, and delivered to players without a moderator being there - meaning that the players are on their own for the playtest. They record a video of themselves playing, talking out loud about their thoughts and answer a survey at the end of the session. The game team can watch those videos, and analyze the data, at their leisure.
In contrast a Live study has someone watch the player live as they play. The moderator - usually a member of the team making the game, or a dedicated researcher, will be on hand to set tasks, ask questions, and interact with the player live. This unlocks a tremendous amount of potential for asking probing questions and understanding what players are thinking, to interpret their behavior.
Both methods have a place in the playtest toolkit - here are some factors to consider when deciding what’s the right one for your playtest.
What do you want to learn from your playtest?
Spending time one-on-one with the player will allow you to understand their exact experience with the game, and ask questions to see what’s happening inside their head - giving a better explanation of their behavior.
This is incredibly helpful for understanding both usability issues, and opinions. For usability issues, it allows you to ask prompting questions to understand why players get confused or stuck - giving you enough detail to successfully address the root of the problem. For opinions, it allows you to ask follow-up questions, and probe deep into players’ thoughts - much more thoroughly than in a survey.
Although unmoderated research will give some insight into players’ opinions, because there's no moderator to ask follow-up questions, the detail can be lacking. Instead unmoderated research’s strength is in scale - allowing you to use measurement questions to balance game difficulty, progression, or look out for retention or churn issues.
The most essential question to ask yourself before running a study is “what do I want to learn from this study”. If it’s a measurement question, consider unmoderated research and thorough surveys. If it’s the what and why of player behavior - choose Live research.
How much time do you have?
Live research requires the moderator to pay close attention to one player at a time. This is very valuable - and exposes a lot of really rich information about what is happening as they play the game. But it takes time.
Because unmoderated tests don’t require a moderator to watch, many unmoderated sessions can happen simultaneously. When combined with quantitative methods of collecting player’s opinions, such as a survey, this can allow teams to gather statistically significant answers to quantitative questions - like asking players to rate sections of gameplay, or looking out for difficulty spikes.
Relying on surveys can speed up data collection, and surveys can be analyzed efficiently - unlocking faster insights.
How reliable is the build?
It’s often very challenging to get an appropriate build ready for a playtest. For unmoderated research, traditionally the build has needed to be self-explanatory, and stable enough that bugs won’t disrupt players’ experience of the game. This can require dedicated development time, and custom builds - which are challenging to fit into demanding development schedules.
Live moderated research, and having a researcher live on a call with the player, unlocks testing on less ‘complete’ builds. The moderator can step in and use debug controls or help players avoid or overcome bugs, meaning less thorough QA is required for moderated builds. The moderator can also act as the tutorial - and explain aspects of the game where the tutorial isn’t implemented yet, allowing playtesting from earlier in the development process.
With the support of an experienced moderator, less polished builds can be playtested which will still generate valuable player insight. Running these playtests earlier on earlier builds gives development teams more time and scope to react to player experience issues, and ultimately makes better games.
Are you confident with your study design?
Live studies can also allow more exploration of unanticipated topics. Because the moderator is live with the player, they can adapt to emerging behavior or situations that occur in the game - generating new questions or tasks on the fly to probe into interesting areas.
This allows unanticipated questions to be explored, and live iteration of the study will help explain unexpected or emerging behaviors.
In contrast, unmoderated studies require the questions and tasks to be confirmed up-front. This standardization is very valuable for benchmarking and rigor, and essential for reliable quantitative analysis - but will require more upfront planning to get right. De-risk unmoderated studies by running small pilot tests (with a few participants) before opening up the study to a lot of players.
Is moderated or unmoderated research best?
There is no ‘perfect’ research method, and every method has strengths or weaknesses. For making a decision between unmoderated and moderated research, the most important question is “what do I want to learn”. Once you’ve nailed your objectives, the choice becomes a lot easier - and by working through the questions above, you should be able to come to a confident decision about which is most useful for your study.
