The Alpha Test: A Crucial Milestone
The alpha testing phase marks a pivotal moment in game development. It's the stage where doors are opened to testers, allowing them to dive headfirst into the world. Their feedback becomes the guiding light illuminating the path toward a polished and immersive gaming experience. In this article, I want to provide insight into what I have learned so far from the alpha test of The Pale Piper.
A little disclosure
As of now, I'm working with limited reach for gathering testers for my project. I don't have a significant following on platforms like YouTube or Twitter.
I recently established a Discord server, but it's still in its infancy, and I haven't yet built a substantial audience there either. But I did use other discord servers to announce the alpha test (of course only if there was a dedicated channel for that).
I do have mailing lists on two websites - one dedicated to Godot tutorials with approximately 100 subscribers, and my dev website with around 1200 subscribers (but it is hard to tell how many of them are bots or of that sort). To them, I sent out an invitation as well.
Additionally, I made an announcement here on IndieDB, which garnered around 305 views so far.
I've had absolutely no success on Reddit, as the platform seems to reliably flag my account as a bot or so. Consequently, every account I attempt to create gets deleted, likely because the IP address is already on a blacklist. I'm aware of the strict rules regarding Karma and other guidelines, and I made efforts to comply with them. However, despite my attempts, I still ended up banned after just a few posts. So no posts on Reddit for me.
Consider this as a point of comparison. Your following might be similar, greater, or lesser. But this provides an idea of what to expect, with a following and number of announcements similar to mine.
Amount of testers so far
The alpha test is currently underway on Steam, and it's open for everyone to join. As of the time of writing, there are about 112 passionate gamers registered. I'm eager to welcome even more players into the fold as I try to refine and improve the gaming experience.
From those 112 participants, so far I've received three outstanding feedbacks from dedicated players who have journeyed through all the levels. All of them also filled out the short form at the end of each room to the best of their ability. Additionally, I received three incomplete feedbacks, each containing valuable notes that shed light on various aspects of the game and information on why they stopped. I also received one recording, which is also helpful, since I can clearly see, where the struggles are in each room.
Following the implementation of achievements for visiting each room, there were instances where one or two testers skipped through all rooms solely to attain the achievement. I guess it has something to do with boosting some Steam achievement completion stat. Typically, I wouldn't mind, but such feedback only adds to my workload, requiring me to sift through and filter out irrelevant submissions. Additionally, they contribute to the Firebase quota. On Firestore, you get around 50k reads and 20k writes for free per day. In my testing process, I've never gone over 10k reads and 600 writes per day. However, this was with the limited number of testers I had. If there's a sudden flood of useless feedback (in the case of people just skipping every room), it could exceed the daily limit and therefore useful feedback could not reach the database for that day. This is important to consider.
So to address this, I introduced an additional achievement that unlocks only upon completing all rooms with a maximum of three skips. That way you cannot get 100% if you just skip every room, and for now this seems to have stopped that.
What stats do I track and how do I get them
Every feedback I receive is anonymous, ensuring privacy for the players. To still be able to track player progression, I generate a random number between 1 and a million. While technically it's possible for two players to have the same number, the odds are incredibly low, and any potential overlap would be negligible. I automatically record data such as player deaths and the time taken to complete each room, providing quantitative metrics for analysis. After each room, I prompt testers with three questions: their overall impression of the room, its perceived difficulty, and an open field for additional comments.
Furthermore, I monitor whether players collect the coins within each room and gather metadata such as the specific game build used, the room being played, and whether the player chose to skip the room. At the conclusion of the feedback process, I include a longer form focusing specifically on the player mechanics, since the alpha test in general is organized by mechanics.
This method allows me to pinpoint rooms that may be either too challenging or overly simplistic for the current stage of the game. Additionally, it helps me uncover frustrating mechanics or rooms that detract from the overall player experience. Conversely, I can identify rooms that resonate positively with players, offering valuable insights into what aspects of the game they enjoy most.
To collect and manage the data, I utilize a Google Firestore Database. Each entry includes various metrics:
I can then query and extract the needed data, allowing for in-depth analysis and visualization. By exporting the data to a spreadsheet, I gain insights into room difficulty and enjoyment, ultimately guiding the iterative development process:
Additionally, there are already hundreds of notes pertaining to specific mechanics in specific rooms. This qualitative feedback will be sorted and categorized by me, after which I'll assess each one.
The Good and the Bad
As I mentioned, I organized the test around player mechanics. There are several advantages and disadvantages to this approach. The main advantage is that testing each mechanic back-to-back allows for the discovery of any clunkiness or bugs specific to that mechanic. It also provides testers with a clear understanding of what to expect from each mechanic and focuses their feedback on that particular aspect.
However, a downside is that this approach may lead to increased frustration and make the alpha test feel more monotonous compared to the final game experience. In the actual game, mechanics will be introduced gradually, and as you progress through the levels, the rooms will feature a blend of both old and new mechanics. This design is intended to create a more enjoyable experience for players, in contrast to relying solely on one main mechanic throughout 70 rooms.
Additionally, the difficulty curve may not be very good, as the order of rooms will change in the final game. This means that while playing through one mechanic, there could be a significant difficulty spike from one room to the next. However, in the actual game, the player will already have improved their skills by the time they reach that room.
What happened since then
Since then, I've been actively implementing fixes based on the feedback received during the alpha testing phase. I prioritized addressing feedback that directly impacted the alpha, ensuring that future testers would benefit from an improved version of the game. For example, I revamped the player controller to enhance responsiveness and gameplay fluidity. I also fixed some bugs like a player freeze when punching. Additionally, in response to observations of players struggling in the initial levels with a new mechanic, I integrated tutorial text to provide clearer guidance and support. Moreover, I made it clearer, that each mechanic has its own designated start button.
The Road Ahead: Analyzing and Adapting
A lot of the feedback regarding room and mechanic improvements will not be incorporated into the alpha version, though, since I'm currently focused on completing a playable demo of the game's initial segment. Nevertheless, rest assured that I value every feedback note and will give each one serious consideration. Additionally, I am committed to making changes based on the quantitative data I gather as more people hopefully play through the alpha. Each feedback received is invaluable, serving as a guiding light toward achieving my vision of making this game the best it can possibly be. The alpha test is set to continue for the foreseeable future, with the plan being to proceed until the first demo is prepared.
Join the Alpha Test
In closing, I extend my heartfelt gratitude to all testers and supporters. And I invite everyone to participate in the alpha test as well. With a small download on Steam, you can get started in no time. Your feedback is invaluable and helps improve the game. Plus, you may find the challenge enjoyable.
If alpha testing isn't your cup of tea, I would greatly appreciate it if you could add the game to your Steam wishlist, even if you're just slightly interested. Your support would mean a lot to me and would greatly help when the game releases at some point in the future (fingers crossed).
Until next time!
