Interview of Jon Bach

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-jon-bach-part-i/2010/03/)

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

uTest: When there are delays in development process, why does testing always seem to take the blame? Is their role just misunderstood? Or is it really the testing team’s fault?  How can companies avoid this seemingly never-ending spiral?

JB: Often, we’re saved until last.  It’s like the novel is written, the movie is produced, but the reviews aren’t in yet, leaving the creators in a heightened state of worry and concern. It’s a lot of stress to be in that position. As testers shine the light on the product, looking for risks and vulnerabilities, the light is really shining on *us*.  We’re being tested just as much as that software, so we tend to be sensitive and aware of being in a critical position as others wait for our findings.

Not being an Agile guy, I’m finding that the values and principles of Scrum, Agile, Lean, and XP may be compelling enough to try.  I’m getting more into that domain and I see merit in the values and principles it is trying to instill.  One of my favorite colleagues (Elisabeth Hendrickson) is coaching me in a gentle way and her stories are enlightening because she lives this stuff. Like me, she used to focus on exploratory testing, but catching up with her recently, she seems to be the whole package — developer, tester, manager, coach — and seems to be immune from that never-ending blame spiral you asked about.  That’s compelling to me.  Some may call her one of those ardent “Agilista” types, but to me, she hasn’t lost her testing spirit or soul.  Though she may agree there’s no silver bullet, she has some great experiences that convince me that with those approaches, the blame spiral is being made irrelevant because of the way testers are more involved, earlier.

uTest: In Half-Baked Ideas for Rapid Test Management (PDF) you used a term called “Agile-fall” (a combination of Agile and waterfall). This seems to be the methodology that most companies follow, yet they always call themselves agile. Is there any shame in being Agile-Fall? And did you coin this term? We couldn’t find it anywhere else.

JB: “Agile-fall” was something I heard at LexisNexis from an awesome PM named Lance Thomas, but in a Google talk in 2005, Elisabeth Hendrickson called it “Scrumfall”, so search on that term and you’ll see that it refers to having the principles associated with Agile (daily stand-ups, sprints, burndown charts, etc.), done in a waterfall-y series of development steps.  Example: Sprint 1: Gather requirements, Sprint 2: Design your tests, Sprint 3, Run those tests, Sprint 4, Fix bugs, Sprint 5 regress those bugs.  There’s no shame in that if that’s what works, and when you’re going through a transition from Waterfall to Agile, that may be the best thing as opposed to a sudden lever-pull one day where you show up and your desk is next to someone else with no walls and there’s a stack of sticky notes and markers on your chair with an email to report to your first standup in 30-minutes.

uTest: It looks like software bugs are partly to blame for the recent Toyota recall debacle. Is this the worst nightmare for testing managers? What else keeps them up at night?

JB: I almost got a Toyota last month, and wish I had, just so I could find a cool defect.  But my nightmares in testing are: “Why didn’t you catch that bug?” Though I know several answers to that, and my favorite colleague Michael Bolton has a rich list of answers to that question, they don’t often overcome the strong emotional attachment that stakeholders have.  And I can’t blame them, really.  The more we call ourselves “Quality Assurance” (like we can guarantee quality), the more they will lean on us to assure them.  I can’t.  But I can certainly *assist* with the notion of  whether something has value to its intended customer.

What keeps me awake is how to know if I’m bringing the right value to the project.  There are a million things I could do, but which ones have the most value right now?  I’m worried I would stumble into an idea too late, or not at all.  That’s why I like heuristics and mnemonics and checklists to kick my mind into gear when I feel stuck or worried.

uTest: What’s the biggest difference between working for a large, mature enterprise and a small young startup (from a testing point of view, that is)?

JB: I’ve worked at both.  At Quardev, which is neither startup or enterprise, we’ve got a mature-but-startup mentality.  Since the word “Quardev” comes from an amalgam of QUality Assistance, Research, and DEVelopment, the balance of those gives us enough diverse opportunities to make it feel like a big company.

From a testing point-of view, test ideas are king no matter what size your company. Some may say the toolset is king, or dev skills are king, but when I’ve worked for start-ups, you don’t need as many signatures on things, and that helps creativity. You might fail faster with those test ideas because you have to do it cheaply, and that’s good.  It’s like projects I’ve been on that use notions of Agile compared to those that don’t.  I see those kinds of projects revealing problems faster because of the way the culture organizes to deliver working software.

At a small startup, it’s often expected that you’ll be brilliant.  That pressure can make work not-so-fun.  Maybe that’s why some of my best ideas came from working in enterprise environments (Microsoft, HP, LexisNexis). In those places, there was a perception that every test idea, method, tactic, and strategy had been tried before.  That carried an unspoken dare (in my view) that begged to be challenged, which few people took on.  For me, brilliance and innovation come when someone counts me out or doesn’t expect it.  I like that.  I like exceeding expectations no matter how big or small the crew.

What sums it up is a TED talk by Ken Robinson, who said: “If you’re not prepared to be wrong, you’ll never come up with anything original.”

uTest: Who plays Jon Bach in the made-for-TV-movie about your career in testing? And what’s the title?

JB: John Schneider, only because it has to be called “The Duke of Hazards.” It also fits because I told my friends in school that I was related to Cousin Daisy (played by Catherine Bach). Not true, but it helped keep the bullies away.

uTest: You have some great tips on how to handle bloated testing numbers and statistics: “Any number, any statistic is like software. It can be tested.” What other tips can you give testers when it comes to having the courage, diplomacy and patience to slow things down and get to the truth?

JB:  For me, the magic words that often make me feel more courageous, diplomatic and patient are: “I have been fooled before.”

No one will argue with that because it’s true.  Scammers often confess that the hardest person to fool is somebody who says “I can be fooled.”  So many times I’ve been so sure I was right just to meet someone who convinced me differently, sometimes in a matter of seconds.  So now say “I could be wrong”, and use other safety language like: “it could be”, “it seems like”, “it looks as if” and “maybe…”  That way, I don’t feel stupid when I’m shown refuting evidence to my claim.  If you practice that, chances are good that you will appear to be a kung fu master who, after having floored your 50th assailant with your skills, slowly backs out of the room on guard for the 51st.

Remember that testing is a craft.  It involves thinking about how things might be different.  Remembering to say “I have been fooled before” is consistent with that spirit.

uTest:  Testing certs: worthwhile or window dressing?

JB:  The only thing worthwhile about them is the debate they provoke.  Window dressing is an apt metaphor because it’s only meant to enhance a window’s *appearance*.  When there’s a flood or a storm or some other strong test of the window, the dressing often gets destroyed. Outside of the flood, people may prefer the look of the dressing; I just want to be a stronger window.  Passing multiple choice tests about so-called “best practices” don’t do that for me.

uTest: What are some of the blogs/sites that you read to stay on top of the latest QA trends and news?

JB: I’m mostly on Twitter because it leads to links to testing videos, newsletters, news stories, trivia, thoughts, questions, blogs and tools that I haven’t seen before.  For example, yesterday I learned about Scott Berkun and his excellent technology blog, Ken Robinson’s fantastic TED talk on rethinking education, the interesting notion of a testing “playbook”, and the Ignite Conferences (a series of 5-minute talks).  Thank you, tweeps!

The top 10 I follow are: James, Jim Benson, Lanette Creamer, Michael Bolton, Scott Hanselman, Elisabeth Hendrickson, Adam Goucher, Paul Boal, Steve Smith, and yes, I have to say, uTest.   All of these sources turned me on to such a wide variety of sources to edify myself that it’s no longer just a handful of blogs to read now and then.  Blogwise, I have to say James (of course), Scott Berkun, Jim Benson, Joel Spolsky, and Lanette Creamer are most worth my time.

uTest: Testers come from a wide range of backgrounds (and have a wide range of skills) yet they are often talked about as if they were all the same. In your experience, what’s the biggest misconception or stereotype of testers?

JB: That we like to “break stuff.”

Nah, we’re detectives… we FIND stuff.  It’s a treasure hunt, not a smash factory.  In fact, our reports are “findings”, like a scientist or an investigator would call them.

One of the first lessons in my career was my favorite — that software comes to testers *already broken*.  I like that.  I didn’t write the code that broke it (unless I did, like a unit test or a keyword-driven script).

Humans are tested the same way.  When we go on a job interview, the interviewer doesn’t “break” us, they find weaknesses and vulnerabilities that are already there in our programming.  The good news is that we can test ourselves before going into the interview — to find and address issues beforehand so we can show them our best value.

uTest: What qualities, characteristics and experiences do you look for when hiring testers?

JB: At Quardev as a hiring test manager, it is these: cautious, critical, curious, friendly, diplomatic, honest, insightful, thoughtful.  I want candidates to tell me about a cool bug they found, or give me their best test idea.  I want them to make me think. I want them to inspire me and make *me* curious*, even though the thing I give them to test when they come in has been tested by over 500 different testers over the last 10 years.  I’m not naive enough to think I’ve seen it all, because one in 20 testers will find something new or will put ideas together in ways I hadn’t thought of.

uTest: What advice would you give to new testers who want to stand out from the crowd and become tomorrow’s testing leaders/gurus/pundits?

JB: Give a talk. Practice in front of a high school class. Take an idea or a strong opinion you have and actually learn more about it, then present it.  Do a lightning talk at the closest Ignite Conference to you.  Read a testing book and email the author your comments.  Comment on blogs and sign your real name.  Question the gurus and the pundits. Call them out if you think their work is crap. Talk about your failures.  Tell us your *experience*, no matter how amateur you think it may be.  Could be you just invented the next cool method to try. It was like that for me with Open-Book Testing when I was on Microsoft Flight Sim.  It was a half-baked idea I had and now it’s blossomed into something that I keep in my intellectual briefcase to show prospective clients one way to teach, guide, and evaluate test teams.

uTest: One of the hottest areas of growth that we’ve seen at uTest is in the area of mobile app testing.  What unique challenges does mobile present to a testing manager?

JB: Finding the right minutes/text plan?  Those really add up when you’re testing!

Actually, I think uTest is in a great position to address the biggest challenge I would face if I was managing a mobile project right now – configuration testing.  I would love to have 1,000 testers at my disposal doing a variety of risk-based exploratory charters on their devices with all kinds of third-party apps installed, making calls on different service plans, texting, taking pictures, playing video, Tweeting, browsing, and maybe doing all of those actions at once from places around the world.

By the way, I read Patrick Copeland’s answer when you asked him the same question and really think he summed up the challenges – diversity of devices, input-output simulations, carrier networks — things that you’d have to be very technical, very savvy, and very organized to test.  But on top of all that, there’s the challenge of if you actually found a bug.  With all of those variables I mentioned, where is the real bug?  You may be just reporting the failure, so you’d have to have some good debugging and diagnostic tools to help you be confident that your concern was represented to those who could (or would) take action on it.

uTest:  If Al Gore hadn’t invented the interwebs, what would you be doing today (answers related to software or technology not allowed!)?

JB: I’d likely be a best-selling author, have a full head of hair, and would have retired at 18 because I would have invented Twitter in 1986 instead of messing around on the Apple II for hours on end.  Before Twitter, before Windows, before the Apple II, our family had the “chain letter” – a packet of letters sent from one member of my family to another, adding new content with each mailing until all 6 of us had caught up with each other’s news in the circle.  When the oldest person got it, they would add new content and mail it to the youngest, and round it would go again, replacing old content with new.  It was a great system and I would have turned that into Twitter as a system of snail-mailed 3×5 cards to everyone on the planet.

uTest: Rapid-fire Jon Bach pop trivia – Last book read?  Last movie read?  Last concert attended?  Favorite sport and team?  Favorite band or album?  Browser of choice?  What kind of cell phone are you carrying?  Blu-ray or DVD?  Paper or plastic?

JB: Last book read? You mean cover-to-cover, not skimming?  Ugh.  Oh!  Last night to my three year old: “How do Dinosaurs Say Good Night?”, after which, I dove back into “Extreme Programming Explored.” Actually, I think I was over-tired last night and got it backwards, which is why I suspect she woke up this morning talking about refactoring, code smells, and collective code ownership and why I dreamt about being a Gallimimus.

Last movie read? The screenplay for “Butch Cassidy and the Sundance Kid” by Goldman.

Last movie seen: the unfortunate and annoying “Where the Wild Things Are”

Concert: Sting, 1989 in Portland, Maine;

Sport: Football (Seattle Seahawks);

Album: Rush, Subdivisions;

Browser: Firefox;

cell: iPhone 3G;

DVD;

Plastic, because at least they spice up the boring look of those landfills!