Interview of James Bach

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-james-bach-part-1/2009/12/)

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

JB: Years ago, Rational hired me to advise them about test processes, and for that reason I took the Rational Unified Process class. I had been working with the RUP team, and I wanted to see what the Rational instructor was saying about RUP. At lunch on the first day, I asked him if it was okay that I was asking challenging questions. He said that was fine. But at lunch on the second day he kicked me out of the class!

Most instructors don’t like me in their classes. So, when you say I’m forced to choose, it’s a question of which class I would prefer to disrupt. (Unless I have the choice of sitting in one of Cem Kaner’s classes at Florida Tech. I would definitely prefer that. Last time I took a class from Cem, I had to play solitaire furiously in the back of the room. It was the only way to stop myself from jumping up and shouting “Yes!” every three minutes.)

I would prefer to disrupt an ISTQB class.

uTest: A few months back you presented “How to Fake a Test Project” at STPCon, where you said “testers may accidentally find bugs because they don’t follow the test scripts precisely.” Are testing managers starting to catch on to this irony of such “best practices”, or are most of them still oblivious to this?

I don’t know. I think most test managers don’t think much about it. As I said in my talk, testing is easy to fake– *even by accident*– because testing is so intangible.

JB: Be honest, you’ve faked a few test projects in your day, right? C’mon you can tell us.

Yes. I remember walking over to my manager’s office at Borland– by this time I’d been doing test management for five years– carrying an estimate of how much testing was required for my project. The estimate was couched partly in terms of “test cases required.” But just as I reached his office, I realized that the numbers meant nothing.

Until that moment, I thought they meant something. Well, they did mean something, just not anything I could defend or explain. Suddenly it dawned on me that I was peddling an empty *story* about testing couched in numbers to make it look rational.

As I write this, that incident happened 17 years, 5 months, and 1 day ago. Since advising my boss that he should ignore my test case estimates at that meeting, I have never used test case counts as a metric on a test project. But that doesn’t explain why, for the 5 years, 1 month, and 21 days before that, I *did* use test case counts.

Of course, the explanation is that I hadn’t much thought about it. I went along with the convention. So, I can forgive people who fake test projects by accident. When we’re young we make mistakes. But we also learn and grow.

uTest: In your new book, Secrets of a Buccaneer-Scholar, you advocate alternatives to traditional schooling and certification – not just for testers, but for people in all professions. It seems to have worked well for you, but why are so many people still reluctant to ignore the certificates and degrees and just start testing? What’s holding them back?

JB: Fear and indoctrination. I recently argued with a fellow about why he was going through a certificate program that bored him and insulted him, instead of quitting that school and being a tester. He claimed that no one would give him a job without the certification. Well, he emailed me a few days ago and told me he just got a job, and that I was right.

Well of course! I’ve been a hiring manager, and the idea that hiring managers are zombies who only look at paper credentials is wrong. SOME managers do that, but you don’t really want to work for someone like, do you?

Seems to me uTest is a mechanism for building your reputation as a rapid tester. I encourage new testers to try it for exactly that reason.

uTest: What’s the most promising trend/development in testing?  And what’s the most dangerous or discouraging trend you’ve seen?

JB: I think the most promising trend is public peer conferences and mentoring. Things like the Weekend Testers, for instance, or the Rat Pack. Some of these people are inspired by guys like me, perhaps, but they are not followers. They are taking it upon themselves to re-invent testing by experiencing and examining it intensively. I’m proud of anything I have done or can do to encourage this sort of creativity.

The most dangerous trend is certification, which is systematically dumbing down our craft and preventing good people from getting work.

A more dangerous thing, which is not a trend but unfortunately a constant, is the astonishing incompetence of technology middle and upper management, who frequently demand stupid metrics and meaningless reports from the people who work for them, or who decide to outsource without having the first clue what they are giving up by dumping knowledgeable testers and sending critical work ten time zones away.

For some reason, test managers complain to me about this, but often don’t complain to their own bosses.

Last year, my brother created a world-class test team at LexisNexis, then someone many levels up from him decided to fire everyone and outsource. They never talked to my brother about this. Apparently they think good testing happens by magic. But here’s the thing, my brother decided that he was going to train the outsourcing company (who had claimed to know a lot about testing, but, surprise, didn’t actually know much at all) to do skilled exploratory testing. It worked. He’s pleased. And now his management at LexisNexis (former management, because they then fired him) must think outsourcing is a great strategy, having no idea that it works only because people like my brother have too much personal pride to let even senior management airheads experience a richly deserved failure.

uTest: We’re all familiar with your work, as well as that of your colleagues like Michael Bolton, Cem Kaner, Jon Bach and others. Are there any “great unknowns” out there in the testing blogosphere to whom we should be paying attention?  Anyone that should be ignored altogether as a dangerous thinker?

JB: I sometimes complain about people I think are dangerous. See my blog.

Basically, I hate bullies. I want a free and open craft. The most dangerous people are those whose personal ambition will mean discouraging innovation (perhaps through establishing some sort of baloney “testing standard” through the auspices of ISO), and encouraging the sort of test management practices that make software both expensive and sucky, while discouraging any smart ambitious person from entering the field in the first place.

As far as great unknowns, there are some. I think Lanette Creamer is someone to watch. Adam White, Ajay Balamurugadas, and Ben Yaroch, as well. There are the Weekend Tester guys, and a shadowy Context-Driven sub-community calling itself “The Rat Pack.” I come across testers, regularly, who are talented, but haven’t taken a step out into public discourse.

I am constantly recruiting for colleagues.

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.