Interview of Matthew Heusser

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-matt-heusser-part-1/2009/11/)

In this month’s installment of “Testing The Limits”, we sit down with Matt Heusser (@mheusser) — prolific blogger for STPCollaborative, thought leader and testing extraordinaire.  We’ll discuss the state of software testing, SpeedGeeking, the role of chaos in testing software, and the lack of fistfights at STPCon 2009

uTest:  We loved the SpeedGeeking session you led at STPCon, so we’re going to flip it on you – If you had just five minutes to teach, motivate or inspire the uTest audience about software testing, what would you say?

MH: Well, I’d start by asking the audience what they are doing today – what’s the greatest point or opportunity they feel – and asking what options they see to improve. Most of the time, I hear that testing is “too slow” or “the bottleneck” or something like that.

So I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting – it is very important the team stop the time tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the numbers for improvement. For example, many of the people I talk to can spend 80% of their time or more in meetings, working on documentation, working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from 80/20 to 60/40 will double the amount of time the team spends actually doing testing.

Another thing to look at is the amount of time spent trying to reproduce defects, document defects, file bug reports, “verify” fixes, and so on. We think of these activities as testing, and they can take a substantial chunk of that 20% – but they are really accidental. That’s not a testing bottleneck – it is a development bottleneck. If test can work with development to improve the quality of the software prior to code complete, that will improve the speed of the whole system. Realizing this, and having a little bit of data to “prove” it, can help the entire system improve.

So if I had five minutes, I would say start with measuring how you track your time, and ask yourself if this is the best use of your time and what can change. Sometimes, the big boss will say “no, we absolutely need you to fill out all seven pages of documentation per test run”, and you can say “ok.”  Six months from now, when someone asks why the big project is late, you can point out that the business made an explicit decision to pay the full price of defined process. You presented options and those were not accepted.

That won’t save this project — but it might save the next.  It also turns out that actually testing tends to be much more fulfilling than documentation and compliance activities. Who could have guessed?

Lots of contrasting opinions at last month’s STP Conference. While there were no fist fights (that we heard about anyway), what did you see as the most contentious issue? And where do you fall on this issue?

MH: I’m a little sad that we keep talking about best practices vs. context, scripting vs. exploratory. Of course we need a balanced breakfast of approaches to testing, and of course, “best practices” are a marketing term that do not exist in the engineering literature. Personally, I’m most embarrassed that you could get the crib notes from any conference from 2004 – or possibly 1999 – and see the same arguments.

Some of the problems are due to personalities. There are a few people who just don’t give past credit for ideas, or make wild claims without having actually done much software testing. Where do I fall on the issues? Well, let me first say, if anyone claims to be an expert on software testing, one place to start is to go to LinkedIn, look at what they actually claim to have done, and send out a few emails and try to verify those claims. If you can’t verify those claims, or realize they are written in a very specific way as to be non-falsifiable, well, that tells you a lot.

I was impressed that we managed to not fight about certification at the conference. There was no certification course before, after, or during, and we didn’t have to spend our time debating it’s merits or lack thereof – we mostly talked about, you know, actually testing and stuff. In that, I was pleased.

Congrats on getting your blog “Testing at the Edge of Chaos” exclusively featured on the STP Collaborative. What does it mean to you to test at the edge of chaos? What ideas are you most interested in getting across to the tester community?

MH: Uh, I think my blog has something to do with testing. And, um, like, something to with Chaos, or something.  Seriously, started blogging in 2001 or so to express myself and my ideas. The last iteration of that was my “Creative Chaos” Blog, where I talked covered creativity and innovation in the software process. “Testing at the Edge” is a little more tester-focused, with the goal of covering skills and innovation in the software testing space.

It seems to be that every year’s a new batch of graduates come out of MIT, Carnegie-Mellon, and the University of California at Berkely that know how to automate defined business processes – and take a look at testing and say “gee, there is a defined process, we should automate it.”

Four or five years later, they’ve learned a bit and turn around and say things like “gee, some of testing can be automated, but part of testing an investigative, feedback-oriented process.”  Then every May a new batch graduates and we start all over again.

I think “Creative Chaos” did a good job in reaching out to that audience, and with the publication of “Beautiful Testing”, we may be finally out-growing it. With the new blog “Testing at the Edge”, I hope to move into specific examples of good testing, how to get better at it, and how to set some appropriate boundaries around the testing activity – and discuss what those should be.

What’s the deal with your Testing Challenges? Are they still on-going, or have you simply run out of apps to test?

MH: I generally run test challenges in private, sometimes electronically, as part of a mentoring or training program. I do this both commercially and non-commercially, as part of my zero-profit “Miagi-Do School of Software Testing.” Recently, I’ve been asking people for what they want, and the idea of running test challenges publicly – and sharing the answers – keeps coming up. I ran one in October (link) that was well-recieved. Sure, I can do more of them if there is interest.

As I work full-time for software product company, I don’t see us running out of apps to test anytime soon. (Laughs)

A quick hypothetical: You’re banished from the software testing industry for five years. What do you do during that time? And don’t say developer.

MH: I’d probably challenge the authority of who it is that is trying to banish me! But I will take your question in the spirit you intended it, and answer what my next career choice would if, for some reason, I chose not to test. That would likely be writing about technology or business.

When you think about it, the investigative journalist shares a lot with the tester. Journalists are paid to look around, find something that seems correct on it’s surface but has a problem, and uncover information and evidence. They take careful notes, they work on projects that are different every time, think critically, and are guided by rules of thumb.

A surprising number of the leaders in the testing industry have an education of background in journalism – both Karen Johnson and Jonathan Bach were trained as journalists before they became testers, and Dr. Cem Kaner has a law degree as well as his PhD in Psychology.

What would the Matt Heusser from ten years ago think about today’s software testing landscape?

MH: Well let’s see … ten years ago Extreme Programming was a crazy idea that Kent Beck was trying at Chrysler, “agile” was spelled lower case and was an adjective that meant ‘bendy.’ The software development landscape was full of RUP and patterns and generalizations and abstractions, and testers were counting test cases. James Bach and Cem Kaner were using the term exploratory testing, but it was far from popular. Today, I think testers have more voices to listen to – the standard school, the context-driven school, and the Agile School all understand each other a bit better and have more clear comparisons. Also, test managers have more options, like the crowdsourced approach or perpetual beta.

Overall, there are more options, the profession is taken more seriously, and we are finally starting to evaluate ideas based on consequences and outcomes instead of ideology. It’s a great time to be a tester. I wouldn’t want to go back.

What other blogs, sites or message boards do you read to stay on the leading edge of all things testing?

MH: I’ve got a bit of a bias there, as I write a monthly column for Software Test and Performance Magazine. You can get the PDF version for free every month. I also subscribe to Better Software Magazine. For communities, I like softwaretestingclub.com and the agile-testing discussion list. For blogs, well, there’s James Bach, Cem Kaner, and Michael Bolton. Adam Goucher has been doing a lot of writing lately, including editing the new book, Beautiful Testing, which I contributed a chapter to. And I spent a fair amount of time working on my own blog, “Testing At The Edge of Chaos.”

Recently I’ve been getting to know people by working on projects with them; my friend Chris McMahon started a Google Group on Writing About Testing which I found to be a blast. Through that group (and the Miagi-Do School) I’ve met quite a few new bloggers: Markus Gaertner, Lanette Creamer, Marlena Compton, and Ajay Balamurugadas come immediately to mind.

What OS are you running right now? What’s your browser of choice? Anti-virus? Inquiring minds would like to know.

MH: Max OS X, and I have to support all of them, so I run all browsers. Anti-Virus? Dude, I told you, I use a mac, and I try to avoid the questionable websites that host the viruses. What have you been browsing lately, Mr. Johnston?  (Ed. Note: I’ll ask the questions around here.)

As more companies develop mobile apps, mobile testing is becoming a greater concern. Are you seeing a convergence of mobile app testing with traditional software testing? Or is mobile testing an separate entity altogether?

MH: My initial thought is that testing mobile apps is a nice little niche inside of web-based software testing. But mobile devices keep getting more and more standards-compliant, and thanks to sites like www.iphonetester.com, you can test handhelds without physically owning a dozen different makes and models. So while I believe there is value in identifying yourself as a mobile device tester, I suspect that value is going to decrease over time.

What attributes and skills make for the best software testers?

MH: My experience lately is with web-based and database-backed commercial and transactional applications, and so I’m going to give an answer that assumes that world view. First off, I think good testers understand the business itself – what the basic transactions are, and what happens if they go wrong. The strong tester will use this knowledge and seek out complexity and undefined behavior; bugs love to hang out there. Strong testers are good systems thinkers; they can perform modeling of the system to figure out where the weak points are. They have a strong memory; they can track a large number of variables in their head at one time. They may play chess, soduku, or maybe just board games by Avalon Hill, or possibly Steve Jackson Games in it’s prime. James Bach pointed me to curiosity, and I think he’s right – if you never ask “what happens when I click this?” it will be hard for you to click this. They’ll want to know common failure modes across all software, and will want to study “quick attacks”, or the “How to break … Software” literature. They will be solid critical and systems thinkers – the kind of skills you can get from listening to politicians and asking what they are not saying, or reading articles in the press and looking for the bias. They will have solid middle school math skills, but be able to figure out a 15% tip quickly, combined with a little bit of statistics and probability. An ability to do some programming, to understand HTML and javascript and such really helps.

I would suggest influence skills combined with a healthy dose of skepticism. It is very hard to find someone with both of these, but to find the problem you need to be skeptical of glib answers and false confidence – and to get anything done about it, you’ll need influence skills. Influence is more than conversation, it is written communication and situational awareness.

Another hard to measure skill is creativity, but it is so crucial. The industry may like to believe that test cases can be generated automatically by some standardized process, but the reality is someone has to think of them, and a creative mind can find all kinds of possible failures that a dull one will miss.

Finally, there’s discipline, which is an interesting one. I have to admit, I have a hard time denying myself as much as a cup of coffee. I’m not very good at forcing myself to testing – I actually want to test. If you’ll permit me just one brag, I think it is the actually wanting to test part that makes me good at it, not the discipline.

Of course it’s going to be different by the job, government and regulated work is very different than mass-market commercial software. And some organizations care a lot about technical skills like dotNet or Java or whatever. You might have a very effective boss who covers for a tester who lacks influence skills. If I had to pick one tester attribute to spark the rest, it would be curiosity. If you just don’t care “what would happen if?” you wouldn’t imagine it, and you won’t press the button to find out.

You seem to be big on tester training lately. From what you’ve read and observed, what’s the best way for a new tester to get up and running?

MH: I’m afraid I can’t give a simple answer like “just go to a conference for a week” – it would have to customized by the environment. So you’d start with some sort of exercise to look at where a tester is according to a skills inventory, where they fit in with the team, and what to work on next.

Likely, that would be a combination of job shadowing, directed readings, brown bag lunches, mentoring, and, possibly, external training. I do think sending a new tester to a conference is a good idea, mostly because of the rich variety of training options and idea exposure they will have that week. A conference offers an opportunity to customize training just by selecting which sessions to attend. If I didn’t have the budget for training, and wanted something local and online, I would consider something like the black box software testing course run by the Association for Software Testing.

Whatever the new tester does, I suggest it be some combination of theory and practical exercise, with the students aware of the exercise and built-in time for reflection on what happened.

Now, what do I recommend if you have a certain budget and want to train an entire team? Something customized in a similar way. So, for example, I am getting more and more excited about the idea of bringing the trainer on-site for a day or two of consulting by walking around. Based on what he observes, the trainer develops a day or two of custom training using similar technologies to those used in the actual workspace; then he comes back for a two days of actual testing, followed by a retrospective. The trainer might then come back every month or two for several brief engagements.

The idea there is to actually tie the training to the work being done, to move it from “Drive-by training” to “helping to collaboratively explore how to improve our work.” Or, in other words, to do outcome-based training. I know. Crazy.

Are great testers born or bred – asked differently is being a world-class tester something that anyone can learn or are you born with it in your blood?

Can I answer that by telling a story? Malcolm Gladwell has a new book out, called Outliers: The Story of Success. In the book he discusses the nature of language – that some cultures, such as South Korea, are high-ceremony cultures. These are the places where the conversations between superiors and subordinates is very formal and scripted. He also discusses low ceremony cultures, like the United States, that have more of a cowboy culture. Gladwell points out that, while human beings make mistakes, the high ceremony culture offers no way for a subordinate to point the mistake out and attempt to correct it. While an American co-pilot might say “we’re flying into a storm you ninny”, in other cultures that behavior is not allowed. That’s no big deal when you’re talking about how many lumps of sugar you’ve counted out, but when it comes to flying airplanes – or mission and life critical software – it matters. So I would say that culture and values or ideology, plays a huge part in the effectiveness of the testing. So much of our work is shining the light of truth when it would be more politically palatable to not notice.

So on the one hand we’ve got cultural values, and on another critical thinking, modeling skills, and curiosity. Certainly critical thinking and modeling can be strengthened through exercise. Curiosity? That’s a good question. You might make an argument that curiosity is inherent, that it “comes with the genes” so to speak — but I’ve seen enough children have their curiosity sucked out by schooling to believe that it must be possible, somehow, to create an environment to encourage it to come back in.