There have been many a debate between both purists as well as future thinkers about the proper components for scale and load testing for MMO’s and online games. Due to recent research and laboratory developments, we’ve finally come to a conclusion;
To get the best results of a scale load test, you only need both components. About fifty live people, and a bot program (yes you heard us… a bot program!).
We’ve basically done tests using both types of environments. And in each scenario, results were both varied because of the different types variables that come into play such as;
- Location bandwith capabilities
- Dependability of over 600 personnel
- Dependability of a stable bot or load simulating program
- Stability of the simulated load program per se in its own server structure and how you do a network pathing technique to make it simulate a round trip around a couple of ISP’s to simulate consumer traffic.
- Direct and indirect data routs your tester-server-bot server-isp pathing would take. It really needs to make a full loop to simulate a consumer environment.
Turns out, that in both scenarios, there are similar bugs and errors that do occur. So in that regard, our client’s major concerns boiled down to the cost effective practicality of thousands of people in a live environment vis-a-vis a simulated load prior to beta release. The decision to retain 50 heads was practical since they were already spread out over several ISP’s to simulate what the game would look like to a consumer. Now while that was going on, we basically kicked in a server that would “simulate” an estimated 5000 ccu’s into the main game server. Although we pretty much broke the pipeline (or rather we were able to determine where the data choke points would occur), we considered this a successful test since we were able to identify a key breaking point.
Be weary though when considering a host for your online game or MMO. A lot of your coding problems will not be identified within the test environment. They will only occur when the data reaches the public ISP routes, and there will be a lot of errors here that the development team wouldn’t have considered since most tests were done in-house within their local lan network.
Of course did i mention, this was the cheaper route rather than entertaining the thought of setting up a large 1000 ccu test group that you have to pay for? That’s what open beta is for!
If you have any comments or suggestions, feel free to contact us over Linked In!
