At the start we only had a mildly inactive beta server which got updated a couple of days before the first live release. But still a lot of issues made it into the live release maybe also related to the fact that only features “ready for release” were activated on Beta. We realized that we not just need to change our beta approach but also need additional help.

THE BETA TEAM

We started with setting up the base by first ramping up our beta team For this we asked some very experienced players if they would like to play a bigger role and have more influence on the development of the product. Luckily those players did not just have great game knowledge but also prior knowledge in testing so they were able to determine if an issue was a possible bug and could also investigate on them further. As also the communication between the beta and the development team was not ideal we introduced a few direct communication channels (mainly via Skype chats) and included QA members in it to have faster reaction times on questions, issues and feedback.

MISSIONS

After we had the base prepared we were ready for the next step. The project we wanted to build was called “Missions” with the main goal to reactivate our beta server in terms of testing activities. Especially for features with random content or vast amount of possible input we wanted to increase the issue spotting rates. But we did not just plan to use the Beta for testing but also for gathering early feedback for features which are still in development by using a slightly more regulated feedback approach.

How do those missions work? First of all we have set some rules which need to be followed without exception.  So we have set up a bug reporting template which included all information which was needed to investigate the reported issues but was also not to complicated for the players to understand. To not lose the overview we did not allow any duplicates, comments, discussions or issues “by design” in the bug reporting missions. As compensation we always run a feedback thread in parallel where they can discuss the features which eventually leads to changing some functionality or is saved for future improvements.

Each time boxed mission starts with the tasks which just contain a rough description of the feature to not spoil the whole fun for the beta players to explore them on their own and also the template to bring it back to their mind. During the testing questions are answered and problems which might arise are solved once they occur to have a smooth testing experience.  At the end of a mission all reported issues are getting a final evaluation and a test report is being prepared to show what the testers have achieved and which bonuses they can expect.

RESULTS & EXPERIENCE

After all this we already achieved quite some results as not just the testing activities increased but in combination with testing features even earlier we also registered a 25% reduction of those defects which were reported after the first live release. Due to the template usage our beta players got better insight on which information are needed for a good bug report which in return increased the overall quality of defect reports.

After having set up the missions on different products which differ in terms of team size, complexity and release cycles we noticed that we had a higher planning certainty in bigger teams as it was more likely a feature got finished in the predicted time period due to less bottlenecks due to sickness and vacations.

If you have only small content injections, small features or improvements beta players might lose interest faster due to missing challenges.  Thus regular feature updates can help to keep the interest of  the testers and to keep them motivated. In this sense it helped us a lot when we started to test parts of features which were still in development so there was a more steady flow of new stuff to test.

 

But even though we had a lot more issues reported through the missions the fluctuation of the testers was quite high which we want to tackle next.