I’m not sure what to think about Sony’s forthcoming reality show “The Tester“.

On one hand, they’re mocking quality assurance by turning it into a competition.

On the other hand they’ll hopefully get people to talk about testing issues and this is a rare public look into what is usually a secretive business.

Instead of that, I have to use my powers as a bugzilla admin to look through the user database and find the e-mail address of the person I want, since otherwise I might use the wrong one.

A god-damned e-mail address! What is this, 1998? Am I on a fucking NeXT here?

Give me a god damned list with AJAX and offer me a drop-down autocompleted list of people when I start typing their name.

There is also something very interesting in how Epic’s Mike Capps treats his employees, versus what he has said in the past when running for the igda board.

My one hope is that some day the United States comes around with this Working Time Directive style of doing business and eliminate the ability for any non-essential-social-services people to be worked beyond 50 or so hours.

This has been given a much better treatment in the light of day elsewhere.

Full video of Mike Capps dishing out his bizarre treatise on why it is OK to overwork people after the break, the warning here is that it is long, but well worth watching. Please let me know what you think in the comments.

Why?

Because then I can edit/add to bugs when I’m offline and come back later and have it actually do the changes.

It’ll make it more likely that I work through my bug lists instead of ignoring them.

What do you want to see from Bugzilla?

On a project nowadays, you may have forty to sixty temporary testers, after which maybe two or three are brought on as full-time testers.  In a department of forty to sixty full-time testers, you may have maybe five leads that cycle between products or ten leads that vacillate between leads and individual contributors.  Finally, you’ve usually only got a single QA Manager, although some organizations are splitting it up so that there is a manager per fifty testers or so.

In short, you have a major funnel from temp to tester to lead to manager and if any level is filled, your opportunities for advancement are severely restricted.

The question is…why should the funnel exist?  There are many ways that organizations can allow testers to grow without forcing them through the funnel.  These are a few of the possible ways that your organization can allow growth within QA without “breaking” the funnel.

Specialization: As time goes on, testers often find that they excel at testing a certain type of feature.  It may be console certification testing, it may be localization/globalization testing, it may be multiplayer testing, it may be UI automation…the point is that this person has shown that they excel at it.  If the workload allows it, it may benefit all projects to have a person be a specialized tester.  Another advantage of specialization is that if your testers are allowed direct access to your development team, the tester and developer of certain features are going to be interacting more often and the communication channels between these individuals will be a lot smoother, resulting in more bugs found and fixed.

Tiers: In many companies, there are only four levels a person can be: temp, tester, lead and manager.  This makes it very difficult for someone to advance, but you can also have sublevels or tiers within these levels.  After all, not every tester can or should be a lead, but that’s no reason to minimize their chances for career advancement.  Some companies have recognized this and have a “V” type system at each level where a tester can go be a lead, or become a Senior Tester or a mentor or something else.  It means an increased amount of responsibility and a structured career path.  People can choose to grow into their current tier or try to advance to the next tier without having to worry about being “managed up or managed out.”

Split Positions: Not everyone goes into QA to stay in QA.  Often, people go into QA as a stepping stone into other positions.  QA Managers can work with other departments to try to allow their testers who want to migrate to other departments to essentially let them “intern” in other departments for 2-3 days a week, usually interning Monday and Tuesday and test the remainder of the week.  In exchange, the other departments absorb the costs for those days.  This way, the employee gets a chance to grow, the departments get cheap labor and the QA manager still gets a tester without having to fund the entire position.

What other ways do your companies allow testers to grow when they hit a tier ceiling?

The first game company that employed me as a tester was a large European developer with thousands of employees across the world. They’ve been around for a while, and as such had well-established procedures and standards. That was rather fortunate for me, for it set my expectations for solid documentation and clear processes. It was also one of the few companies I have seen to use negative testing, although that was rather the testers’ own initiative rather than any official procedure. Another cookie to them for using test coverage assessments in functional tests, something I have yet to see in the other game developer companies I’ve worked with.

Having such a massive testing department, the focus was less on the quality of testers and more on reporting: there were very strict rules on how to submit bug reports, down to capitalization and punctuation. We had to write daily reports detailing the progress of testing, in a very specific format. However, during training we were never taught *how* to test. We were drilled into the use of the bug reporting system, the Great Tome of Wise Policies and Ergonomic Rules, we were shown how to check the Java files against the technical documentation, but when it came to actual testing, “Just play the game and find bugs” could sum up our training.

I find that this is an oversight that quite a few game developers are guilty of. They assume that gamers know how games should work in general, so they don’t need to be instructed on what it means to be a tester.

My current company, an independent middle-sized MMO developer, has initially made the same mistake. but since I am now the one writing the training materials, I am trying to correct that, teaching  new testers how to… well, test. Still, we are recruiting testers from the ranks of our players, in the (not entirely mistaken) assumption that they are better suited to test the product. However, I find it that often it is the testers who have no previous knowledge of games who find the most glaring design and usability problems. They are not grown used to ignoring them by their years of gaming experience, where the same mistakes are being made over and over until they become almost features.

Thus, I am ever appreciative of the fresh perspective of my Indian team, which are testing for us as outsourced partners. I have heard saying that working with Asian testers is usually a pain, they have no initiative and no imagination. True, there are huge cultural differences between us, but what has been branded as a lack of initiative I have found to be rather a … shyness. Almost one year of working together has eroded that shyness and they are more efficient than ever. I can rely on their thoroughness to spot every little typographic error, every glitch and code error. I can rely on their implacable logic to spot design inconsistencies that are otherwise overlooked. They take such an immense pride in doing their job well – their team motto is “Let no bug go free”. And from what I hear these are common traits in Indian testing companies – this aspiration to excellence, the rigorous organization and amazing spreadsheet-fu.

Alas, one thing I still find difficult to teach them is negative testing. In fact, I believe it is a hard thing to learn for anyone, as I found precious few game testers who knew how to thoroughly break games. In a recent conversation with a tester for a relatively well-known North American game developer, I asked if they ever did negative testing and his reaction was rather bemused: “Err, no, we only tested to see if it works”. On the other hand, their company’s test process seemed more organized than our own, with prodigious checklists and test scripts, so I suppose a more formal testing leaves little room for the exploratory “let’s see how we can break this” method.

That being said though, I am very curious if any game developer out there uses formal test design techniques, like, say, orthogonal arrays or decision tables or equivalence class partitioning, since all that I know of only use exploratory testing and sometimes use-cases. One of or other test teams uses test driven development methods for their unit tests, but they are mostly programmers and don’t use any formal design techniques that I know of. This has been a subject that’s been haunting me for a while now – how can game developers adapt these techniques, many of them quite useful outside our domain, to our rather unique flavour of testing? But I guess that is the subject of another rambling story.

However, it can be difficult as a lead or manager to figure out whois the poison pill in a team because most of the time, employees will openly bitch about other employees only within their circle and they rarely include management in their bitchfests for fear of being seen as complainers.  Because of that, we see the morale problem, but we have problems isolating the cause.

So how do we detect the problem people and once we do, how to we correct the problem?

Hopefully, you’ve been able to maintain a decent level of communication and trust with your testers.  If you have, the easiest way is just to pull them aside one by one and talk to them.  Start by asking about who is doing a really good job in their department, then segue into asking them if there have been any personality clashes in their department.  Assure them that their feedback is confidential.  You’ll usually see a pattern start to emerge around the poison in your midst.

The problem then becomes diagnosis.  You know who it is, but what kind of poison is it?  There are generally four types: incompetent, unmotivated, emotional black holes, and asshole.  Once you know the type of poison, you can start working on a way to manage the poison out of the department.

Incompetent testers are the easiest to handle.  Create performance goals, mentor them, and if they either can’t meet their goals, can’t accept the guidance they are provided or are still morale killers, it’s time for them to move on.

Unmotivated testers are a bit more difficult because you have to find out why they aren’t motivated.  Is it general project ennui?  Is it lack of advancement opportunities?  Is it the fact that after being a tester for five years he/she is still making less than any entry level programmer?  The causes behind lack of motivation are as varied as the bugs that get filed.  There may be times when the best thing to do is to actively help your tester find a position with another division or company.  As much as it pains you to lose that expertise, the cost to your other testers by keeping them is far higher.

Emotional black holes are people who have problems in their personal lives that are disrupting the team.  It isn’t always their fault.  They could be having medical issues, debt issues, family problems, etc.  You can’t correct their external problems, but you can try to be flexible by giving them time to try to correct them.  Allow them flex time or additional make-up time within reason so that they can handle the issues or find ways to leave the issues at home.  The nice thing is that just working with them tends to raise department morale because people see that you aren’t an unfeeling monster.  Everyone is going to have problems, and it makes people feel better knowing that if they have a problem as well, you’ll work with them on it.  Admittedly, there are limits to what you can do and you should lay those limits out.  You should also set a time limit based on what the issue is for it to be handled.

Finally, you’ve got the assholes.  QA tends to get more than their fair share simply because since the role of QA is to deliver bad news, you will inevitably get some people who enjoy it a bit too much.  These people not only affect morale within your department, but affect the department’s standing with other departments.  There are no easy ways to handle dealing with an asshole, so you may as well be blunt.  Tell them that what they are doing is wrong, explain why it is wrong, and tell them that if the behavior continues, they will be immediately dismissed.  At this point, it doesn’t matter if they are your best performer or not…assholes bring down the productivity of everyone they touch and must be dealt with appropriately.  At this point, one of three things will happen.  Either they won’t change and will be terminated, will quit, or they will make an effort to change.  If they are making an effort to change, work with them…nobody changes overnight.  Also do checkups with people who brought up the issues in your early meetings to see how the changes are being received.

Managing testers is hard.  Any time that the QA dichotomy of logic and emotion collide, there is bound to be friction.  Working to minimize that friction may not seem like the most effective use of your time, but if keeping your testers happy and healthy isn’t a worthwhile goal for you, what are you doing managing in the first place?

1. Hire observant testers, and then see that they are well trained on not only the platforms that they are going to be working on, but also on other platforms.  Knowledge of platform strengths and weaknesses can help an observant testers hypothesize where errors will be located.  Also have them test your competitor’s games.  Find the errors that they made and let slip through so you won’t have the same problems in your games.

2. Be sure that your testers have all of the equipment and testing tools that they need, and that they know how to use those tools.  Every platform has a wide array of testing tools available.  Your PC testers need to know how to use Process Explorer and Process Monitor at a minimum.  Both Microsoft and Sony provide a large range of testing tools for their platforms, although the documentation leaves a lot to be desired.  Learn them yourself and then train your testers on their use.

3. Help them plan their work to make the most effective use of their time.  As a lead, you should know about the architecture of the program and should be able to help them ensure that not only are they testing the entire feature, but that they are minimizing their efforts at retesting portions that have already been tested either by themselves or others during that test pass.  As a manager, you should be able to pick up on patterns in how a tester works that could be inefficient or scattershot and help them improve their testing abilities.

4. Stay in close touch with every tester.  Know what each one is doing.  Work with them.  Because testing is such a high turnover profession, we have very little in the realm of legacy knowledge.  Work with them not only to pass on the knowledge that you have, but to harvest the knowledge that they have gleaned themselves so that if they leave you can pass it on to the next generation of tester.

5. Keep up their enthusiasm.  Encourage every tester every day.  Very few jobs are harder on the psyche than being the bearer of bad news.  Not only that, but testers work long hours for little pay and often have little chance of career advancement or permanence.  It’s hard to keep them motivated, but recognizing accomplishments can go a long way.  Calling out when someone tracks down a difficult severity 1 bug, noticing when someone passes a major bug milestone on a project, celebrating team bug milestones, all of these are simple, effective ways of maintaining morale in a department that isn’t usually known for it.

But please, if you want to keep their morale up, don’t show them “Manos: The Hands of Fate.”