Recruiting the right participants for your user research studies is an essential step to guarantee you insightful findings.
Screener questionnaires, or “screeners”, are surveys people take before participating in a research study. They’re made up of a few questions, designed to exclude the respondents who aren’t your intended audience and identify the ones who are.
While they may seem straightforward, they’re actually very easy to get wrong. This article has one objective, giving you the tips and resources to design your next screener that will get results you can rely on.
Designing a screener is never the starting point of your research study. You should start by asking yourself “What is the goal of my research study?”
Consider what insight would be most useful, then work backward to figure out who could best provide that insight. Once you have the right participants in mind, you can start designing screening questions to recruit these players.
To go deeper here, we have published a Mobile Playtesting Playbook. There, you will find several chapters dedicated to:
We highly recommend you have a read of this book. The talented Gareth Lloyd made after years of experience in the game user research field, to deliver a true goldmine of information.
There are two types of information you want to collect from players before inviting them to a research study. These two targeting categories are the following:
Are you aiming your game at certain demographics, such as target age ranges and gender, or what devices players use or have experience using?
This may be important when you are testing appreciation of your game and understanding among your target audience, but there may also be certain playtests where you can test more widely.
Note that on PlaytestCloud, we by default collect and share this demographic information with you.
There is a second set of issues to consider here. They may broadly align with your demographic and audience definitions or could be different. These are around the current and previous gaming experience you want your playtesters to have.
Here, you are typically trying to discover the following audience traits:
How much experience are players required to have with a:
How much affinity do we want players to have with these?
Once you know the characteristics of your target audience (demographics) and you have laid out the specific criteria (gaming experience and preferences) for how you’ll identify the people who qualify, it’s time to write the questions that will help you recruit your participants.
First, you want to write questions that will help you assess if a respondent qualifies to be invited to your research study or not. The second part consists in structuring your screener questions, to organize them like a funnel to narrow down participants, starting with broad questions and moving to more specific questions.
Writing questions and getting started can be hard at times. This section of the article will try to compile the best practices you want to adopt, as well as the pitfalls you want to avoid while writing your questions.
Some of the best practices we follow at PlaytestCloud are:
At PlaytestCloud, we start our questionnaire with a free text question, to help respondents “get in the mood” of the screener.
This first question invites players to take a moment, to think about the games they play and articulate their answer in a written form. Knowing if a player is articulated before a 1:1 interview for example, is always important.
Q: When was the last time you played a game on mobile and which one was it?
We encourage you to use the plain language method, to ensure the respondents understand your survey as quickly, easily, and completely as possible.
We recommend that you explain all of the jargon terms you might use in your questions. It is also good practice to illustrate terms with examples.
❌ Which of these genres do you enjoy playing the most?
✅ Which of these game genres do you enjoy playing the most?
An open-ended question is a question that cannot be answered with a "yes" or a "no" response. They are pretty much the gold standard of question types, we use to screen our players.
Open-ended questions also give you the opportunity to collect additional information about affinity and frequency, while telling you about the respondent’s behavior.
❌ Do you play games on your mobile device, if at all?
✅ How much do you play games on your mobile device, if at all?
Yes or No questions should be kept for specific cases, where you want to share requirements or get your respondent’s approval before a study.
✅ If selected, you might be invited to a one hour interview with a game user researcher to discuss your gaming experience. Would you be interested in participating in a one hour interview after your playtest?
There is no need to be upfront about your research goals in your screener. During this step, you want to discover how the respondents behave.
Let’s say you are interested in recruiting players experienced with Candy Crush Saga to your next research study.
❌ Have you ever installed and played the game Candy Crush Saga, on your mobile device?
✅ Which of the following mobile games have you installed and played on your mobile device?
In this example we do not mention the target game in the question, letting the respondent select it or not, without leading their choice.
Note that it’s always best practice to include a catchall alternative option. In this case “None of the above”.
We have also added here, a dummy option with the game Caramel Crush Saga. This fake game is here to catch careless respondents, who might want to select all options available in order to maximize their chances to be invited to your study.
While we always think that our players are honest with us, we do sometimes get incorrect or untruthful responses. That is why we are using knowledge-based questions, to make sure players have the level of experience they say they have.
✅ If you have played LifeAfter, which one of these is the main currency used in the game?
New Dollar in LifeAfter by NetEase Games
Here we’ll share a couple of questions we like to use to see how frequently people play a game, and how much experience they have with it.
✅ Some people play Gardenscapes a lot and others do not play it at all. Which category do you fall into?
It helps us to combine terms like “sometimes” with a quantified metric (at least 2 days a week).
At PlaytestCloud, we have decided to measure the frequency of playing, using the number of days people play a game, over the course of a week. We find it a more reliable metric than let say, the number of hours spent in-game. It is easy to quickly assess how many times per week you are dedicated to playing a game, as opposed to keeping track of the number of hours. Great video games are very good at having us lose track of time :)
That said, if the number of hours played on a certain game is a crucial factor to define your audience, here is how we would go about writing the question.
✅ How would you describe how much you play Mobile Shooter games, if at all? (ex. PUBG Mobile, Call of Duty: Mobile, etc.)
Another way to measure affinity or experience, is to use questions with ratings with variable values. Here is a good example:
✅ When it comes to Mobile Shooter games, some people play them a lot, and others don't play them at all. Which category do you fall into?
When using a multiple choice question type, the choices presented to the respondents should not overlap and they should cover all possible answers to the question.
Trust us, you'll always want to make sure you're not adding overlapping values to your answer choices. This correct way of bordering values is very helpful when you are sorting out your data.
❌ In total, how much have you spent on in-app purchases in this game?
✅ In total, how much have you spent on in-app purchases in this game?
A well rounded screener doesn’t need to be long, in fact, it should be short. We like to keep our screeners short, sweet, and to the point, with a max length of 10 questions.
Screeners can be seen as a service respondents are doing to you. This voluntary step is only leading to a potential invite to a research study, so we like to keep that in mind.
Short questionnaires also avoid the “respondent fatigue”, making it easier to reduce careless responses.
If you can, the best would be to arrange your screener so your questions appear one after the other on separate pages. Keeping the whole questionnaire from being seen at once will help conceal what you are looking for.
On the same note, you might want to work on your questionnaire logic, so only a specific answer to a question will determine if another question is asked to your respondent. A structured logic will also help keep your screener short!
The order of the questions you ask should follow a simple structure. The goal is to group the questions in sections, and go from broad to specific questions.
Here is an order you could follow:
1 | Demographics[1]
2 | Usage and experience
3 | Affinity and validation
4 | Availability
To conclude this article, we would like to share a few important takeaways with you. These best practices will help you design a screener, to recruit the players you need for your next research project.