How do you improve in-app purchases with a new feature?
Playdots, Inc. is a free-to-play mobile game studio with the successful launch of three mega-hit mobile games. But the pressure to continuously innovate and improve to stay relevant in the app store means there’s little time for celebration.
Drew McWhorter, a product manager on the game Two Dots, and his team were almost ready to launch a "Win Streaks" feature designed to improve the in-app purchases (IAP) conversion rate.
The team at Playdots never launches a feature without playtesting it first for any of their games. Their company's testing philosophy is that if you're spending valuable resources to develop a feature, it is critical to make sure the feature is ready by testing with external users prior to going live.
For new installs, the first three days of gameplay are strategically designed to help users learn how to play and enjoy the game. Because of this, these levels rarely see major changes, if ever. Onboarding is critical for every game, but with all of the Dots games, onboarding holds a little more weight. This is due to the unique game mechanics of the Dots franchise, so they want to make sure they get it right.
The Playtest: Longitudinal
Playdot worked with our team to set up a Longitudinal AB Playtest, separating players into two distinct groups: one group of new players starting gameplay on level 1, and one group of experienced Two Dots players starting on level 35.
Group one would be observed during the FTUE, while Drew and his team observed the players and learned their player type. Once these players passed level 38, the new Win Streaks feature would be introduced.
Group two would start three levels before the feature is introduced, and Drew would watch them play over three days to see how they interact with the Win Streaks feature over time.
A longitudinal playtest was the logical choice for this type of feature since it allows observing the same players interact over many days. Drew understood that to truly understand a player's long-term interactions with a game-changing feature like Win Streaks, he needed to observe their behavior over multiple game sessions.
In total, Drew received 16 hours of video from 10 players. His team incorporated user testing into the tail end of their design process and needed to work quickly to analyze the results before the last development sprint began.
The most successful launch in years
Playdots assumed that the Win Streaks feature would be a fairly advanced mechanic and could potentially impact new users negatively. But by watching the videos, it became very clear that Win Streaks were a much more powerful feature than originally thought.
The power of this update was confirmed over and over again in the player videos. Based on their results, the team pushed the feature earlier in the FTUE on new installs.
With these long-term qualitative study results, the team now had the data to take a calculated risk.
Following the launch, Playdots saw that the feature was even more impactful than they had hypothesized. In fact, based on what they learned through the longitudinal playtest, they were able to adjust and refine this feature.
Ultimately, Win Streaks was their most impactful feature launch in two years— leading to the highest IAP conversion rates the game has ever had. Drew was happy to inform us that running a longitudinal playtest was one of the key reasons for the sustained success of the Win Streaks feature.
