In this blog post
What is a Longitudinal Playtest
Running a Longitudinal Playtest
3 Tips to Help Plan Your Longitudinal Playtest
The Benefits of Longitudinal Playtesting
The Different Use Cases for Longitudinal Playtests
What is a Longitudinal Playtest | Mobile Game Playtesting
When PlaytestCloud first started out we called a ‘diary study’–or, a study that occurs over many days–a longitudinal study in our product line up. Now, almost a decade later, we’re still using this terminology as an homage to where we began… even though diary study is the common vernacular in the Games Research and Use Experience) GRUX industry. Consider it our little way of accepting that everyone is a novice with playtesting at some point or another. So, while we have perfected our games user research abilities by consulting with hundreds of games user researchers since then, we’re keeping this special part of our humble beginnings around for a little while longer. Just know that if you ever come across playtesting literature that mentions a diary study, a PlaytestCloud Longitudinal study (or playtest) is exactly the same thing!
But just what is Longitudinal study, exactly?
At PlaytestCloud, we like to say that Longitudinal playtests are the ‘make it or break it’ playtest for any mobile game trying to make it in the app stores. A Longitudinal Playtest is also one of the most popular long-form playtests we offer at PlaytestCloud, with the other being our Multi-session playtest. A Longitudinal playtest is seriously the granddaddy of all playtests, and likely the most important one you’ll run. Why?
Because with Longitudinal playtests, players don’t just play for one session and then provide feedback… they play across multiple days. That means players will get the chance to really become familiar with your game’s mechanics, providing nuanced, expert player opinions about what they like and don’t like about your game.
This is important since it is often said in research that, for players to be fully captured by (and committed to) your game, they need to interact with it once per day over the span of at least 7 days. By Day 7, that player will be much more likely to continue playing your game than a player with more intermittent plays in that critical first week.
Therefore, a Longitudinal playtest provides players with a day-by-day play environment, allowing you to test and target player milestones that come from multiple sessions of progressing gameplay.
What makes a longitudinal playtest different from a multi-session playtest?
Because longitudinal studies encompass a multi-session like component, studios can often get confused when trying to determine the difference between these two studies. Let’s lay that out clearly here, once and for all:
A multi-session playtest allows players to play the game in sequential intervals, where the focus is on how the breaks in between affect player opinions of certain game concepts, art, or mechanics. Basically, players engage for a short period of time, pause, then play again until the playtest is complete. Although these sessions can occur over multiple days like a longitudinal study, the main difference is in what each study focuses on–in this case the pauses. A longitudinal study or playtest, on the other hand, focuses on getting players in front of a game for at least 5 to 60 minutes over a prescribed number of days. In these cases, players will play only once per day for several days.
If you want to learn more about how multi-session playtesting works, read our latest use case article for more great content.
Running a Longitudinal Playtest
Although we like to make it very easy for game studios to run their playtests, we do understand that committing to a longitudinal playtest can be quite the undertaking for game studios at any stage of games user research. That’s why we’ve worked to make these test set-ups flexible and affordable. Still, for added help, we recommend you check out some of these resources on your Longitudinal playtesting journey:
- Longitudinal playtesting guide
- How should I plan for my longitudinal or multisession order?
- Why have players dropped out of my longitudinal studies?
- What is the best way to analyze my long form study results?
Then, once you’ve got a good plan in place for your user research aims, you can order your longitudinal study. Thankfully, a longitudinal playtest is uncomplicated to order, and can be completed in 3 steps using PlaytestCloud. All you have to do is:
- Set up your study online, or together with us. Whether you want to build your study through PlaytestCloud, or with the help of one of our games user research experts, you can quickly set out the parameters of the study and hit submit.
- Upload your iOS or Android build. Remember, you need to have several levels of gameplay completed so that players can get a handle on the true essence of your game. This is especially important if you’re testing late-game content.
- Review results as you go. Depending on how long you set the study for, you will get daily updates for each leg of the journey detailing how players are faring with your game, including think aloud audio feedback, screen capture gameplay, and daily surveys from players.
Just remember: the length of your playtest will depend on how many days you want players to play for–so if you run a 5-day playtest, we’ll likely include an extra 3 rest days in that period so we can realistically guarantee player engagement within the chosen time-frame. After all, real players may not be able to get to their games one per day for 7 consecutive days, for example. We want to give them a little wiggle room so the added pressure of playing your game within a certain time doesn’t impact the natural gameplay environment.
3 Tips to Help Plan Your Longitudinal Playtest
Before you jump in and order your Longitudinal playtest, we recommend you check out these 3 helpful planning tips:
- Make sure you have each game level you want to test ready to the point that it can be playtested.
- Find out what factors you want to test, and work independently (or with PlaytestCloud) to design a Longitudinal playtest to focus on those aims before you order.
- Contact PlaytestCloud in advance of your playtest order to let us know about your upcoming test and what you hope to get out of it. This can help us prepare for a successful playtest, and expedite the ordering process, leaving you with a faster and more thorough playtest experience.
Need more help throughout the entire playtest study process? Have a look at our HOW TO GUIDE for Longitudinal Playtests.
Ordering a Longitudinal Playtest
If you are thinking about ordering a Longitudinal playtest, know that they can be used to test just about anything, like for example long-term play and player motivation… but we’ll get to that in a moment.
Just know for now that if you’re thinking your playtest idea is too “out of the box” for testing, or if you think what you'd like feedback on is too specific. Don't worry, if you can dream it, we can test it. We can also do this with any player type, targeting your specific target audience; our online platform gives you direct access to 1.000.000+ players of all genres and demographics!
So, if you want to start a Longitudinal playtest right away, just sign-up now, or head here if you already have a PlaytestCloud account. If instead you want to know more about the benefits of Longitudinal playtests, including specific use cases of how it’s helped our clients, then just keep reading below.
The Benefits of Longitudinal Playtesting for Mobile Games
As you begin your game’s playtesting journey, you’re bound to come across a ton of product options to test different elements of your game.
So what playtest product is best for your game’s user research aims?
The main benefit of a Longitudinal playtests is that it helps you optimize your game’s engagement over the long run. That’s because they effectively simulate the experience that new players have from the moment they download your game. They also dramatically support the reduction of player churn rates, while improving the long-term fun and engagement of your game, which is the true goal of mobile gaming after all!
Along with this, Longitudinal playtests help you surpass the ‘first-stage’ thinking that many game studios get caught up in, so that you can really help players engage with later levels of your game. The added joy is that you get deep insight into how your game structure operates from start to finish–allowing you to take a big-picture view of your game as it heads into production.
Plus, since Longitudinal Playtests are completely tailor-made for your game, you are in full control of how long (and how often) players play the game during the study. As a result, you’ll find it super simple to:
- Watch players daily progress. You can follow your players’ journey from the first-time player experience to Day 7 and beyond.
- Track key metrics. Whether you want to measure ‘fun’ or just playability, you can gather data on specific information with daily surveys and recordings.
- See how players progress through your game. You can see whether your players find the game too difficult, or if they quickly understand the game’s mechanics as they move along.
Best of all, because we also take care of all the technical player logistics like NDAs and player payment, you will be free to focus wholly on your in-house user research goals and tasks, such as:
- Learning the root causes behind unexpected or unfavorable feedback during soft-launch or production.
- Understanding how to keep players excited for longer with your game.
- Discovering whether some levels are too hard, and what core game mechanics may need adjusting.
- Exploring why your game is experiencing low player retention, or low interaction with certain features as expected.
In other words, Longitudinal playtests can eliminate any uncertainty you may have about your game before launch day. That means you can rest easy knowing that players in your target audience genuinely like your game, and enjoy play-through beyond the first few levels of gameplay. Add to that PlaytestCloud’s advantage of fast setup and quick turnaround, and you’ll be receiving your first round of player recordings within 24 hours of beginning the playtest; that means you can start iterating while the Longitudinal study is still running.
Lastly (as with our other playtest products), Longitudinal playtests capture the whole gameplay experience of your game. Then, when you have your results, you can annotate videos, create reels, hashtags, and transcripts to relay playtest findings directly to members of your game studio team, ensuring you can implement the learnings from your Longitudinal playtest as quickly as possible.
The Different Use Cases for Longitudinal Playtests
We’ve outlined the specs for you, and now it’s time to learn how Longitudinal playtests can help you build incredible mobile games regardless of where you are in your development cycle. What problems can they solve? What questions can they answer? And what different setups can you achieve with them? Indeed, when creating your Longitudinal playtest you might want to know what testing prompts to include.
To answer these questions and more, below we give you 4 examples of how our clients have used Longitudinal playtests, showcasing the problems this service can solve, and the different setups each test can be run in – as well as their cost.
When you’re finished reading, we recommend you also check out an in-depth Longitudinal case study from the incredible studio in charge of the ‘Snake.io’ game and more from Kooapps.
Before we dig into the case examples below, however, let’s quickly learn a bit more about something essential for any PlaytestCloud user: Video Tokens.
What is a Video Token?
Video Tokens are like PlaytestCloud currency. They are tokens you can use to place orders – and come as a part of our subscription plans. They’re also how you pay for your one-off projects, for example.
Video Tokens are automatically added to your account every billing cycle so you can keep ordering more playtests!
How many Video Tokens would I spend on a Longitudinal playtest?
Longitudinal playtests come with with a lot of variables, all depending on your needs, and all of these affect cost. But to keep it simple, a longitudinal on the platform is priced with a set up fee of 8.7 Video Tokens. Then, you add to that price the number of days chosen as well as the respective play length per day. The number of players and their type of targeting will also determine the price of a longitudinal study, as depicted below:
Part 1: Price per player Playtime per session Price per session 15 min 1.01 VT 30 min 1.52 VT 60 min 2.03 VT Part 2: Target audience factor Standard targeting Included Advanced targeting Price per player x 1.5 Part 3: Setup fee 8.7 Video Tokens Players: 3 5 10 Playtime per day: 15 30 60 Days to play: 2 5 7 10 Calculate
Cost:
Now that that’s out of the way, let’s have a look at how Longitudinal playtests have helped our clients – and how they can help you.
4 Longitudinal Playtest Case Studies to Motivate Your Next Playtest
Longitudinal Testing with Experienced Players
Even when a studio gains a ton of success with their game build, their position in the industry is never guaranteed.
That was the thinking of a popular, Tier-A mobile game studio behind one of the most well-known mobile games on the market today. Their game has been in App Stores for many years, and yet they still commit to testing different features and game modes to make sure they’re fully cooked before being introduced live into the game.
Let’s take a look at their longitudinal playtest approach below, and what they gained from it:
Goal: To test multiple features and game modes for a top mobile game on the market before they are published live.
Cost: Given the price per player, the target audience cost, the number of sessions, and the setup fee, the total cost of this playtest was 41 Video Tokens broken down as follows:
4 Sessions of 15 minutes per day with 8 Players
Price per player: 8 VT (1.01 VT x 8 players x 15 min sessions)
Number of Sessions: 32 VT (4 sessions x 4 days)
Target Audience Cost: No additional cost for standard targeting
Setup Fee: 8.7 VT
Total Cost: 41 VT
Setup: Because this studio is quite experienced at playtesting, they have a tried and true method for the longitudinal studies they run. Often they measure gameplay over 4 or 5 days at about 15 to 20 minutes per day, with about 8 to 10 players per session. In this case, they placed an order of 4 sessions at 15 minutes per day, with 8 players who are active and experienced in the game being tested, and who are at level 400 or higher.
Owing to the flexibility of most playtesting and games user research, this game studio also formulated playtest tasks as a kind of survey on their own. They opted out of the written survey feature, and instead asked players ‘survey’ questions during the assigned play-task. From there, players gave verbal feedback on gameplay while playing the game. This type of ‘interactive’ approach ensured players shared their overall feelings and mindset as they encountered the new feature being tested.
Results: This studio has run playtests like this in the past, and they usually discover that their feature is ready to be released in the live version of their game. That’s what happened with this test–yet it’s worth mentioning there are times a new feature doesn't work as intended with players. Although that can be a hard hurdle to overcome, when a studio has to fix a ton of errors before launch, these types of discoveries make Video Tokens spent all the more worthwhile. They also ensure developers don’t have to spend a ton of extra time dealing with feature fixes in-game.
Longitudinal Testing to Maintain Game Popularity
When game studios create a break-out game that quickly becomes popular with mobile gamers, it’s important they keep the ball rolling to continue ramping up their success.
That’s just what happened with a small game studio who created a popular pair-matching puzzle game. With early success, they became a trail-blazer in the games market and wanted to innovate and improve to keep themselves relevant. However, when a game is popular, something as simple as a poorly executed feature launch can turn players away from the game. With a testing philosophy that says you need to spend valuable resources to get new features ready and create great returns, it’s no wonder this team chose a longitudinal playtest pre-feature launch, hoping to avoid a sudden bout of player churn.
To be specific, this studio’s development team had been working to launch their “Win Streaks” feature, which is designed to improve in-app purchase conversion rates. While the gameplay is already strategically designed to help users learn how to play (and enjoy the game) in the first 3 days, the idea of launching the “Win Streaks” feature would induce a significant change to core gameplay after that initial period. This would be a risk without testing, so let’s take a look at their longitudinal playtest approach below, and see what they gained from it:
Goal: To test a new in-game feature (Win Streaks) to understand whether a significant change to core gameplay and advanced functioning would help or harm the player onboarding experience.
Cost: Given the price per player, the target audience cost, the number of sessions, and the setup fee, the total cost of this playtest was 54 Video Tokens broken down as follows:
3 Sessions of 30 minutes per day with 10 Players
Price per player: 15 VT (1.52 VT x 10 players x 30 min sessions)
Number of Sessions: 45 VT (3 sessions x 3 days)
Target Audience Cost: No additional cost for standard targeting
Setup Fee: 8.7 VT
Total Cost: 54 VT
Setup: The game studio placed an order for a longitudinal playtest that would separate players into two distinct groups. While the first group would include 5 players with no experience playing the game title being tested, and who didn’t reject gameplay for pair-matching or puzzle games in their playtester profile, the second group was built of 5 players who have experience playing the game being tested.
Here, the players from the first group would start from Level 1, whereby researchers would watch players interact with traditional first-time user experience (FTUE) to learn their player type. Then, after players passed a certain level, they would gain access to the Win Streaks feature. At the same time, players in the second group would play from the level they currently hold within the game being tested. They would ascend through three levels, and then the Win Streak feature would be introduced.
From there, user researchers watched these experienced players for 3 days to see how they would interact with the feature. Because longitudinal playtests are the only way to watch how players interact over many days, it was the best choice for the team to observe player behavior with this feature over multiple sessions so they could gauge their long-term interactions with the game. In total, the team received about 15 hours of gameplay material, which was broken down to better understand how the Win Streaks feature affected gameplay. They then organized their files using hashtags, clips, and reels to help share highlight moments from the large content log with the rest of their team.
Results: Originally, the studio presumed their Win Streaks feature would be a bit advanced for new players, and that it would impact players negatively. However, after testing, they discovered the Win Streaks feature was much more powerful (and much more positively accepted) than they originally thought. In fact, for players who were at a greater risk of churn– out of either boredom or frustration–the Win Streak feature introduction would suddenly have them changing course and enjoying the game again.
These findings were decisively positive for the addition of Win Streaks to the game, and as a result the studio included the feature earlier in the new-player experience loop–i.e. levels that had not been adjusted for quite some time. It was the newfound confidence that led to an assurance that, with calculated improvements, the already perfected first levels of the game could better increase player retention if Win Streaks was introduced. After the feature was launched months later to great success, the game experienced the highest in-app purchase rates it has ever had.
Longitudinal Testing for In-Game Diagnostics
When game studios want to diagnose a certain issue they’re seeing with players, a longitudinal playtest can provide deep qualitative insights as to why.
That’s what happened when a large Tier A mobile game studio reached out to us to address an issue with churn rates, wanting to focus on the issue and design a solid solution to the problem from playtest results.
Let’s take a look at their playtest approach below, and what they gained from it:
Goal: To address a churn trend on a turn-based RPG game by diagnosing the issue and creating a solution to address the problem.
Cost: Given the price per player, the target audience cost, the number of sessions, and the setup fee, the total cost of this playtest was 145 Video Tokens broken down as follows:
10 Players at 30 minutes per day for 7 days + 10 Player interviews
Price per player: 15 VT (1.52 VT x 10 players x 15 min sessions)
Number of Sessions: 106.4 VT (7 sessions x 7 days)
Target Audience Cost: No additional cost for standard targeting
Setup Fee: 8.7 VT
Player Interview Pricing: 30 VT (10 players x 3)
Total Cost: 145 VT
Setup: The game studio placed an order for a Longitudinal playtest to include 8 men and 2 women who are fans of RPG games, but who have never played the particular title being tested. What was unique to this playtest was the exclusion of rest days (at the consent of players), whereas we would usually include these rest days to make the playtest more achievable. Nevertheless, the playtest went ahead with 30 minute sessions over 7 consecutive days of gameplay, each followed by a survey.
Every survey contained questions about the play experience, what the player enjoyed and didn’t enjoy, as well as whether they would continue playing the game if they weren’t being incentivized to playtest it. If the player chose ‘no’ for this final query, we would immediately mark that player for a follow-up PlaytestCloud Player Interview, where players could relay their reasons for the ‘simulated churn’. The team could then work towards updating and perfecting their game build to avoid high player turnover in the future.
Results: Even with the unusual set-up, the game studio found the results to be helpful in diagnosing their churn issue. In fact, right after the playtest phase, the team went straight into development mode to fix issues that had been discovered during testing. Once updates were ready, the same players we’d interviewed were invited back to play the new build, while new players were also invited to confirm whether these fixes had truly fixed the churn issue. So far, the results from these repeat players and playtests have been overwhelmingly positive.
Longitudinal Testing for Late-stage Gameplay
When game studios get into later stages of development, it's important to move on from those early tutorial stages and test whether new, late-game elements are just as successful as initial gameplay.
That’s just what a small mobile company developing a cooking game thought when they upgraded their usual Single-Session playtest order into a longitudinal study for late-game content.
Let’s take a look at their playtest approach below, and what they gained from it:
Goal: To ensure the compatibility, functionality, and enjoyability of late-stage game content for a new cooking-style mobile game.
Cost: Given the price per player, the target audience cost, the number of sessions, and the setup fee, the total cost of this playtest was 134 Video Tokens broken down as follows:
3 Sessions of 30 minutes per day with 25 Players
Price per player - Standard Targeting: 30.4 VT (1.52 VT x 20 players x 30 min sessions)
Price per player - Advanced Targeting: 7.6 VT (1.52 VT x 5 players x 30 min sessions)
Target Audience Cost: 11.4 VT
Number of Sessions: 125.4 VT (3 sessions x 3 days)
Target Audience Cost: No additional cost for standard targeting
Setup Fee: 8.7 VT
Total Cost: 134 VT
Setup: The game studio placed an order for a simple Longitudinal playtest with advanced targeting. In total, the team requested 4 target audiences, and each one was instructed to play the tested game normally for 3 days, 30 minutes each day, while providing their thoughts and feelings aloud during gameplay:
- Group 1: 5 players (3 men and 2 women) who are active casual RPG players
- Group 2: 5 players (3 men and 2 women) who are active hardcore RPG players
- Group 3: 5 players (2 men and 3 women) who are active casual strategy/puzzle game players
- Group 4: 5 players (2 men and 3 women) who are active players of the cooking mamma game series game players
Results: Based on the advice laid out in our Longitudinal How To Guide, the game studio was able to set aside plenty of work time to go through all of the playtesting results. Together, members of the team organized their findings from 30 total hours of gameplay content with hashtags, clips, and reels. Aside from some last-minute adjustments made to the game based on playtesting results, the studio was able to stick to their original release schedule to confidently release their game in App Stores.
Test Your Mobile Game with PlaytestCloud
As you can see, Longitudinal Playtesting can be used in a wide range of setups, and solve an equally wide range of problems. No matter the type of game you’re working on, or the players you’re hoping to reach, this type of playtest is essential in giving you all the feedback and insights you might need.
If you already have an account with PlaytestCloud, then when you’re ready to order a Longitudinal Playtest, you can choose “Longitudinal Study” from the New Playtest page on your account dashboard.
P.S. If you need more insight on how to order a multi-session playtest, we have a help center article for that!
If on the other hand you are new to PlaytestCloud (and maybe even new to creating and testing mobile games), then you can reach out to us with the form below and request your first Longitudinal playtest!
We are so excited to take care of all your playtesting needs, and look forward to hearing all about your next longitudinal adventure.
Looking for more Use Cases? Check out our other articles:
Single-Session Playtests
User Research Services
Multi-Session Playtests
Player Interviews
Concept Testing
Surveys
