My people have a saying: “Cate bordeie, atatea obiceie”; the rough translation of that would be that each household has its habits. I have witnessed the truth of that in my career as a game tester – truly each and every company I have worked for or with had its own approach to testing and quality assurance.
The first game company that employed me as a tester was a large European developer with thousands of employees across the world. They’ve been around for a while, and as such had well-established procedures and standards. That was rather fortunate for me, for it set my expectations for solid documentation and clear processes. It was also one of the few companies I have seen to use negative testing, although that was rather the testers’ own initiative rather than any official procedure. Another cookie to them for using test coverage assessments in functional tests, something I have yet to see in the other game developer companies I’ve worked with.
Having such a massive testing department, the focus was less on the quality of testers and more on reporting: there were very strict rules on how to submit bug reports, down to capitalization and punctuation. We had to write daily reports detailing the progress of testing, in a very specific format. However, during training we were never taught *how* to test. We were drilled into the use of the bug reporting system, the Great Tome of Wise Policies and Ergonomic Rules, we were shown how to check the Java files against the technical documentation, but when it came to actual testing, “Just play the game and find bugs” could sum up our training.
I find that this is an oversight that quite a few game developers are guilty of. They assume that gamers know how games should work in general, so they don’t need to be instructed on what it means to be a tester.
My current company, an independent middle-sized MMO developer, has initially made the same mistake. but since I am now the one writing the training materials, I am trying to correct that, teaching new testers how to… well, test. Still, we are recruiting testers from the ranks of our players, in the (not entirely mistaken) assumption that they are better suited to test the product. However, I find it that often it is the testers who have no previous knowledge of games who find the most glaring design and usability problems. They are not grown used to ignoring them by their years of gaming experience, where the same mistakes are being made over and over until they become almost features.
Thus, I am ever appreciative of the fresh perspective of my Indian team, which are testing for us as outsourced partners. I have heard saying that working with Asian testers is usually a pain, they have no initiative and no imagination. True, there are huge cultural differences between us, but what has been branded as a lack of initiative I have found to be rather a … shyness. Almost one year of working together has eroded that shyness and they are more efficient than ever. I can rely on their thoroughness to spot every little typographic error, every glitch and code error. I can rely on their implacable logic to spot design inconsistencies that are otherwise overlooked. They take such an immense pride in doing their job well – their team motto is “Let no bug go free”. And from what I hear these are common traits in Indian testing companies – this aspiration to excellence, the rigorous organization and amazing spreadsheet-fu.
Alas, one thing I still find difficult to teach them is negative testing. In fact, I believe it is a hard thing to learn for anyone, as I found precious few game testers who knew how to thoroughly break games. In a recent conversation with a tester for a relatively well-known North American game developer, I asked if they ever did negative testing and his reaction was rather bemused: “Err, no, we only tested to see if it works”. On the other hand, their company’s test process seemed more organized than our own, with prodigious checklists and test scripts, so I suppose a more formal testing leaves little room for the exploratory “let’s see how we can break this” method.
That being said though, I am very curious if any game developer out there uses formal test design techniques, like, say, orthogonal arrays or decision tables or equivalence class partitioning, since all that I know of only use exploratory testing and sometimes use-cases. One of or other test teams uses test driven development methods for their unit tests, but they are mostly programmers and don’t use any formal design techniques that I know of. This has been a subject that’s been haunting me for a while now – how can game developers adapt these techniques, many of them quite useful outside our domain, to our rather unique flavour of testing? But I guess that is the subject of another rambling story.
No food for test team unless you make with the clicky-clicky:
