Interview of Scott Barber

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-scott-barber-part-i/2010/04/#more-5457)

Our Testing the Limits guest this month is testing guru Scott Barber, the Chief Technologist of PerfTestPlus. A speaker, writer, teacher and entrepreneur, Scott has one of the most impressive resumes in the business, particularly in the realm of customized testing methodologies, embedded systems testing, personal security systems and other topics – all of which are discussed on his blog.

In Part I of our 3-part interview, Scott discusses the Manifesto for Agile Development (almost ten years after it was created); the expectations of today’s testing managers; the notion of testers as an “unfortunate necessity”, the 1983 War Games movie and more.

uTest: As a signatory on the Manifesto for Agile Development, can you comment on the progress being made by software companies in upholding these principles? Have they exceeded your expectations, or is there still a long way to go?

SB: Honestly, I think the buzz around the “Agile movement” has, in many cases, taken the industry in an unfortunate direction. I meet far too many people and companies who are completely unfamiliar with the Agile Manifesto and think of Agile as a collection of practices, processes, and tools. The reality is that Agile is a far more a mindset and a culture than it is a collection of practices, processes and tools. Agile isn’t the best fit for every situation, or for every person.

I believe that the trend to “go Agile” is misguided. If a company is developing good software, the people involved in developing that software are happy working there, the software development is sustainable, and the business is being adequately served by that software, there’s really no need for them to try to be more or less Agile. Agile has challenges like any other culture, but the single biggest challenge I find is companies trying to solve development, process, management, and/or schedule problems by “going Agile.” Teams who have grown up in a culture that is fundamentally different than Agile simply will not find it easy to “go Agile.”

Think of it this way. Telling a team to “go Agile” between projects is like telling a professional baseball player to switch from being a right-handed hitter to being a left-handed hitter between seasons – and expecting that to break their hitting slump. You simply don’t see that. What you do see is batters working with coaches to re-evaluate their swing, work on their timing, spend more time in the weight room, etc.

Personally, I am happiest when I’m working in an Agile culture, but that is a personal preference, nothing more. What I do think would serve individuals and teams well, would be to read the Agile Manifesto and really think about whether or not those principles are a good fit for themselves, their team, and/or their organization. If so, I recommend embracing them.  If not, I recommend finding or developing an equivalent set of principles to build a culture around.

uTest: In a recent article, you highlight the critical point that “the perceived value of testing varies widely from individual to individual, team to team and company to company.” Can you further elaborate on how testing managers can do a better job of setting clear expectations?

SB: Consider these scenarios:

• Company A executives are looking for the testing to be their defense in case of failure and lawsuits
• Company B project managers want testing to say “ship it” so that if there are problems in production, they don’t get in trouble
• Company C developers want testing to show them the issues before management notices them and mandates changes to the development process… yet again.
• Company D is developing software for a client who expects testing to ensure their needs are met
• Company E just spent millions on a marketing campaign and is counting on testing to ensure that the company doesn’t get egg on its face when the campaign is released.

In each of these cases, the test manager is certainly the person who needs to lead the test team to adding the most value to the project. The challenge is that the test manager is often in the worst position to get straight answers.  Test managers are typically the lowest-level role in a company to hold the title “manager.” They don’t generally get to sit in the board meetings where defense against lawsuits are discussed, they don’t frequently have direct contact with the client, developers don’t often treat test managers as confidants, and who would tell a test manager that the purpose of his/her team is to take the blame if something goes wrong, even if it is completely out of their control?

How test managers, and testers, can do a better job is to take a step back and really consider what is going on in the company, what instructions they are being given, what questions they are being asked, and who seems happy or grumpy with the test results being provided.  Paying attention things like those will frequently make the “real story” pretty clear, and once the story starts to become clear, you can start asking smart questions that actually lead to pretty straight answers.

What doesn’t work is assuming that because you are the test manager, or the tester, that you *know* how your testing adds value to the company, that you *know* what the most important testing is, and that you *know* the managers and executives are idiots because of what they are telling you to test.  The fact is that even if your managers and executives are idiots, that doesn’t mean that they don’t have a sound and reasonable business reason to want what they are asking for. It’s not up to us to judge them for either being idiots or for making clumsy requests that don’t actually help them resolve their concern. It is up to us, however, to help them figure out what they actually need, help them figure out how to meet that need, and then guide our testing to best serve that need.

uTest: You have an impressive knack for drawing testing analogies from your lessons learned as an officer in the U.S. Army. What is the true value of outlining the situation, mission and intent of a software development/testing project?

SB: Thanks.  It’s not just the Army; it’s my life experience in general.  I suspect that has more to do with the fact that my parents were both educators and that I didn’t even know there was such a thing as software testing until I got hired to be a “Performance Engineer” and was introduced to the software test team on my way to my first client. So I pretty much had to figure it all out for myself with no real training in testing.

In this case, I was trying to figure out what my role really was. I clearly was not the decision-maker about whether or not to ship the product. I didn’t have the authority to tell the developers what should be fixed. So I applied the model I knew and asked myself “What is the situation here?”, “What is my mission?”, and “What is the overall intent of the client for the project as a whole?”

Over time, what I found was that the situation question helped me to understand things like where testing fit in the scheme of the project; who is panicked over what; how long I have to complete my tasks with what resources; who my allies are; what things are likely to block my success – that sort of thing. Much later, I learned that many testers refer to this as “context”, which I think is a very good word for it.

The mission question helped me figure out what was expected of me.  As it turned out, the folks I was reporting to often didn’t really know how to respond to my question of “What is my mission?”  They’d say things like “Well, test it and give me the results, of course.” My follow-up question of “Test it for what?” and “What kind of results?” didn’t help to clarify things either. So I learned to actually draft a little mission statement, like “Provide as much information as I can to management about issues I believe will cause lost clients or calls to tech support.” Sometimes someone would get very upset and change my mission statement, but that was fine by me because then I knew what their expectations were.

The intent question is what helped me stay focused on what really mattered.  The fact is that if the business we are working for fails, it doesn’t really matter how well tested a piece of software is.  No business pays testers for the sake of paying testers.  The truth is that most organizations feel that testers are an unfortunate necessity.  That truth makes it all the more important that our work add at least equal value to the business, as it costs the business to have us there in the first place.

It is at least as important as it is to have the situation, mission, and intent information individually, to look at those three items collectively. Often when I look at these three items collectively, what I find is that those three items contradict or work against one another. That is typically a good indication that either I have gotten things very confused, or it’s time for some stakeholders to have a meeting to figure out what is *really* important.

uTest: To our knowledge, no one has produced a movie entirely about software testers. However, there have been some notable software bugs in motion pictures (HAL from 2001: A Space Odyssey, the “Doomsday Machine” from Dr. Strangelove, the compound interest bug from Office Space, et al). If you had to suggest a movie that would inspire testers, what would it be and why? There are no wrong answers, except for The Net starring Sandra Bullock and Rocky 5.

SB: WarGames, 1983. All I have to say, is if you’ve built a computer capable of running WWIII simulations, and that is the same computer that controls the launch commands for nuclear warheads, and that computer is accessible via brute force hack of dialing phone numbers over a modem, you either didn’t have any testers or you didn’t listen to them.

uTest: There’s a lot of consensus among the Context-Driven School of Software Testing, but are there any fundamental differences? You can’t agree with guys like Michael Bolton and James Bach on everything, can you? Speak freely, your secrets are safe with us!

SB: I don’t know that there *is* a lot of consensus. I know that I am very much Context-Driven. In fact, my license plate says “CONTEXT”. That said, the buzz and counter-buzz surrounding the Context-Driven School of Software Testing is still causing as much harm as good.

I think the underlying problem is that the creation of software is widely treated as a software development activity, which may or may not benefit from related activities like testing. I find this thought process just plain silly. Software development, in my experience, is done for one of four reasons; altruism (minimal), education (some), research and development (slightly more than some), and/or financial/business advantage reasons (most). I think the fact that software is created for different reasons and purposes makes software development at least context-dependent by definition. The difference between being context-dependent and being context-driven is fundamentally the difference between being reactive vs. being proactive.

I think there never would have been a need to call out or name the Context-Driven School of Software Testing if testing hadn’t been so undervalued, misunderstood, and too often ignored entirely for so long that some entrepreneurial folks decided to step up and start talking about “the right way” to test software. While I’ll be one of the first to admit that many of these folks published “right ways” that were far superior to the state of the practice of the time, none of these “right ways” were right for everyone or every situation.

So, in an attempt to help people and organizations be aware that there are many ways to approach testing, and that the way that is “best” depends on a wide variety of factors, none of which involve which entrepreneurial person has the best “sales pitch” for their favorite approach.

Regardless of what the buzz may say, context-driven isn’t about Exploratory Testing, isn’t against test scripts, and doesn’t dictate who do what testing when. The way I talk about being driven by context is to regularly stop and ask yourself, “What can I do, right now that will be of most value to my team/project/company?” If you make decisions that way, instead of simply doing what someone who knows nothing about your situation recommends, I’d say that is at least the first step to being driven by context.

uTest: We’re always interested in how people got involved in software testing. What made you choose this path – what lit the spark?

SB:  The short story is that I didn’t choose software testing, it chose me.

I was dabbling in various technology and software related jobs after my time in the Army, when the company I was working for got bought by a big company who changed our compensation plan to something I couldn’t live with. While complaining to my friend, he said “Forget them, come work for us, we need a Performance Test Engineer.” In response, I asked “What’s that?”  He said “Don’t worry, you’ll like it.”

I started the job a few weeks later, and he was right, I did like it. I liked it so much, in fact, that I’m still happy being a tester over 10 years later. And to think, when I started that job I wasn’t even aware there were people who tested software for a living.

uTest: Follow-up question:  If you weren’t in testing, or anything related to software, what would you be doing right now?

SB: That’s hard to say. I went to college to become a Civil Engineer, and while I did very well in school and love the field, I was already too far removed from it by the time I left the Army to have a viable path back in. I’ve considered teaching secondary school or at a university, but there are aspects to those careers that I don’t think I would end up being very good at or enjoying. I’ve considered becoming the editor for a technical magazine, and becoming a conference coordinator for a technical conference. I’ve even considered going to work on the administrative side of a non-profit whose work inspires me.

My friends who know me well are always telling me that I should “retire into” sports announcing and/or coaching. I don’t know if that’s even vaguely possible, but I do know that there is nothing I seem to enjoy doing more than coaching and impressing people sitting around me by constantly pre-empting sports casters while watching a game.

uTest: Say we choose a software testing project at random. You don’t anything about it, except for the fact that there’s a major bottleneck somewhere in the process that is preventing quality software from getting released on-schedule. If you had to guess, what would the source of that bottleneck most likely be in a typical 2010 company? Management? Testers? Technology? Metrics? Give us your best educated guess and why.

SB: I believe it was in his book Secrets of Consulting that Jerry Weinburg stated “Every problem is a people problem.” Throughout my career, I have found this to be a great place to start. People make decisions, people devise and implement process, people write and test code, and people manage projects. If there is something preventing software from being released on time, it’s because someone made a mistake, a poor decision, an inaccurate estimate, or simply isn’t doing their job. It may or may not be relevant to figure out who did what that lead to the problem, but if you trace a problem far enough, eventually you end up at a person. The way I see it, people problems fall under management.

Add to this that quality education for managers and executives overseeing software development is at least as sparse as quality education for testers, and it makes “Management” a pretty high percentage place to start.  Of course, it’s also the most difficult to do anything about, so frequently organizations try to address their software development challenges by attacking symptoms rather than the cause.

uTest: Often times, testers will report bugs they deem “critical” (or high priority, P1, or whichever vernacular you choose) that are subsequently rejected by the client, product manager or dev lead as being ‘out of scope’ or ‘works as designed.’ What advice do you have for testers trying to make the case that a particular bug MUST be addressed before launch? Has this ever happened to you?

SB: It used to happen to me all the time, until one day I realized that I’m a tester, not an executive. My job is to provide information that my team, my employer, and whatever other stakeholders there are for my project want or need to know to make informed decisions. My job is to help them understand the state of the software and the possible implications of what I’ve found.

If, as a tester, you feel that a bug you have found is “extra important” to fix prior to launch, I recommend getting an audience with a decision maker and explaining the bug and its implications to items that matter to them (such as profitability, public relations, etc.) until you are convinced they can accurately represent your side of the argument to their peers and supervisors. After that you have two choices, as I see it, accept the decision that is made as being best for the business based on information that you’ll probably never have, or decide that your values and your company’s values are not in line and start looking for a job in a company with values closer to your own.

uTest: Favorite book that taught you an important testing lesson (even if the book wasn’t about testing per se)?  TV show?  Haiku?  Parable?  Off-Broadway play?  Shoot, just share any source of testing wisdom that has inspired you.

SB: General Systems Thinking by Jerry Weinberg, as I mentioned earlier. To be honest, it validated the way I have always thought about testing more than actually teaching me something new. That said, I’ve been teaching testers and other members of software development teams for almost 10 years and I can say with high degree of confidence that if all of those folks were reasonably skilled in systems thinking, software systems and their development, it would be in a much better place than it is today.

On a more pop-culture note, the television series House, M.D. is absolutely the best example of testing I’ve seen on a T.V. or movie screen. This is a show about diagnostic medicine, not software, but these people know testing. They understand the difference between positive and negative testing. They use scripted and exploratory testing. They aren’t satisfied with minimizing symptoms, they have an inner need to determine and eliminate root causes of unacceptable symptoms or behaviors. In fact, I’ve been influenced enough by the show that in one of my training classes, I have a segment titled “If Dr. House were a Performance Tester”, I’ve considered starting a blog under the same title, and I’ve seriously considered trying to popularize “Software Testing and Diagnostics Department” as a replacement for “Software Quality Assurance Department”.

uTest: It looks like you’ve hired your fair share of testers – what skills, experience and characteristics do you look for when making these critical hiring decisions?

SB: The most important things I look for in an individual are aptitude to learn whatever technologies, business information, processes, etc. that they will be expected to test – that and investigative curiosity.  Beyond that, I’m always looking to build a well-rounded and diverse testing team. I find that a team full of testers “like me” leads to us all missing the same kinds of bugs, so I’m always looking for a breadth of skills and experiences across the team. I’ve hired aircraft mechanics, medical billing specialists, call-center representatives, technical sales representatives, subject matter experts, developers… oh yeah, and even a few testers.

I know how to teach people technologies, the basics of a new programming language, how to report bugs, how to document process, and how to generate test ideas from whatever information we have.  What I don’t know how to teach people is how to be curious, the desire to investigate their curiosity, or to be excited by the thought of investigating things that strike them as odd.

So I guess I’m always on the lookout for someone with a tester’s attitude who fills a gap in perspective of the current team whose own knowledge or skill gaps fall in areas I know how to teach them or get them trained in.

uTest: Hypothetical: All testers are barred from practicing their craft for an entire year. What does the software industry – nay, the modern world – look like without their services? Please be as dramatic and hyperbolic as possible.

SB: Sadly, I think most of the world wouldn’t even notice. I think for the period of a year, non-testers would pick up enough of the slack to keep things moving, and enough individuals would perform enough “heroics” to keep the software world running. Sure, more bugs would make it into production, more software would ship late, and there would be more evening news reports of software failures, but I’m not sure the general public would even notice.

In fact, if software producers were smart, they’d pass some of the savings they were realizing by not having to pay testers on to the general public, making them even less likely to mind a few extra bugs.

But eventually, some kind of tipping point would be reached and the public would demand less buggy software. That is when the disaster would actually happen.  By the time companies re-engaged with testers, re-integrated them into their new (clearly, more streamlined, process), started getting results, realizing just how much re-work needed to be done and how long it would take to get software back up to the “old” standard, the public would be so furious that the “old” standard would no longer suffice.  Of course, achieving the new, higher, quality standard would take more time, and make the software cost more, and thus make the public demand yet *more* for their money.

Interestingly, if the software industry managed to survive for long enough to find a balance point at the tail of this downward, then upward spiral in software quality, I think it would be the best thing that could happen for testers. The publicity would lead to higher quality training for testers, more education about testing for developers and managers in school and on the job, better hiring methods and practices, and more respect for testing as a career. As the public became aware that software doesn’t have to be bug-infested (sure, all software will always have some bugs, but a couple of lady bugs are far more acceptable than a cockroach farm) more buyers will become more willing to spend a little extra for better quality, leading to more jobs for skilled testers, leading to more people wanting to become skilled testers, which would increase enrollment into education programs, leading to more research and development, and on, and on.

Maybe this isn’t exactly how it would go, but I do believe that it would take far too long for the public to notice and far too long for companies to start losing more money than they save by not paying testers. I also believe that unless something like this does happen, the public may never realize that they have been conditioned to accept far lower quality from software than they do from non-software items of equivalent cost. Think about it, how many times in the last year has a software bug ruined your day?  If the building materials used to construct your home were of the same quality of the software you use, how would you feel about your home?  What would you think if you had to patch a wall every time you patched your software?

Testers can’t solve the quality problem on our own. Higher quality has to be driven by conscientious business owners or disgruntled buyers. The best testers can do is try to make those executives as aware as we can of those bugs we can find while operating with not enough training, not enough time, not enough people, not enough equipment, and not enough support.

uTest: Are great testers born or built?  And if they’re built, then how the heck does that happen?  Share a little wisdom with the newbie testers and college kids who will join the ranks in the next year.

SB: I think that the potential to be a great tester is a part of a person’s personality that more or less exists or doesn’t by the time that person reaches college or the job market, but I don’t think that potential alone is enough to make someone a truly great tester.  Becoming a great tester is like becoming a great anything-else. It takes some natural ability, some opportunity, some good coaching, some relevant experiences to draw from, a lot of self-motivation, and a lot of practice.

Every now and again, someone might have so much of some of these items that they can get away with less of the others, but that is the exception. If you want to be a great tester, just like if you want to be a great musician, writer, or athlete, practice all you can, find a good coach or mentor, learn everything you can about technologies, designing experiments, systems, psychology, human interfaces, anything you can imagine being related to what you might end up testing or might help you better understand what might matter to the people who use what you might end up testing.

When you think you’re getting really good at something, start blogging it, then teaching it to others.  Don’t be discouraged when you find that it doesn’t really work for them at first.  Keep refining and revising until you believe those you are teaching are at least as good as you thought you were when you first started trying to teach them. When your students reach that level, pick the thing you are next best at and do the same thing.  You’ll know you’ve become a “great tester” when your current students have been referred by your former students, and your former students are trying to hire you instead of just trying to pick your brain.

uTest: [In the voice of deep, gravel-voiced movie preview guy] The year is 2020.  And in a world gone mad… where only one man knows what really happened… only one man can face the danger, beat the odds and reveal the truth… (insert pyrotechnic explosion here).  Okay, I don’t know precisely where I’m going with that, but what do you think testing look like in 2020?  What needs to change ASAP in the world of software design, development or testing in order to ensure a future of safe, reliable apps?

SB: The first thing that needs to happen is that testers, developers, managers, and testers need to learn what a “test” is.  Sounds silly, right?  It’s not.  Almost every day I encounter someone who wants to know “What test can I conduct to prove that…?”  While the answer (“There is no single test to prove that…”) it demonstrates a fundamental misunderstanding of what a test is.

Think about all the tests you have taken during your days as a student.  Do you believe that any of them proved that you had learned what the instructor had intended?  How about your driver’s test?  Do you believe that passing your driver’s test proved that you were going to be a good driver?  What about the last time you or someone you knew had a blood test?  Did you believe that blood test was going to prove the person was healthy?  Be honest now.

Of course you didn’t.  But somehow, most folks seem to think that software tests can prove that something will work in production.  The best we’re ever going to be able to do is demonstrate ways in which the software already isn’t working, demonstrate that the software doesn’t currently meet some measure of “goodness”, demonstrate that it is possible that the software may work in production, and/or collect information of potential value to someone who matters.  Even that depends on our tests being reliable and accurate.

Before testing can truly advance, folks need to recognize that testing is an application of the scientific method, that testing involves experimental design, and that tests don’t pass or fail, they provide useful information or they don’t.

Most of what happens today under the label of testing is nothing more than simple validation of expectations.  Testing is a search for information.  Testing is most valuable when expectations are not met.

Don’t get me wrong.  Someone had better be validating that expectations are being met or no one is going to want the software we’re developing, no matter how much we test it, or how few bugs are in it.  But we can’t stop there.  Technologies evolve too fast, users do things the most creative of us would never imagine before our software is retired for us to think that validation in some non-production environment is enough.

10 years ago, I heard a lot of managers and executives saying “What do you need testers for?  If you’d just develop it right in the first place, they’d have nothing to find.”  I don’t hear that very often anymore.  Today, executives understand that is an unreasonable sentiment.  Now they seem to think that testing can prove and/or control quality.  Maybe, 10 years from now, they will have figured out how to get the most value out of testing by asking them to actually test for the purposes of helping developers build better software and helping executives make better decisions.  I hope that is the case, but if there isn’t a change in the way “a test” is perceived, I don’t think it’s very likely.

uTest: We’ve been fortunate enough to interview the likes of Whittaker, Heusser, Bach, Bolton and Bach in the past for our ‘Testing The Limits’ series… who should we interview next?  Give us a few names who would continue this great (albeit young) tradition and blow our minds?

SB:

Rob Sabourin
Mike Kelly
Cem Kaner
Elizabeth Hendrickson