What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.

********

uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.

Having said that, I do also feel on any software project that there is and always should be some level of manual testing. The ultimate users of the program are not robots or automated test frameworks. They are live human beings with lots of different skill sets, backgrounds, geographic location, and expectations. You need to experience the product with your own fingers and eyes on the keyboard and screen. Most automated tests are pretty superficial and have rather myopic validation methods for determining a pass or fail. Because of that, good manual validation–especially with a fully configured and integrated product–is a must in my book. So, what’s the appropriate mix? I would target 70% automated and 30% manual.

uTest: You were recently one of the keynote speakers at the Google Test and Automation Conference (GTAC) in India, where you presented on crowdsourced testing. How much easier has it gotten to speak on this topic in the last few years? Is it starting to register with most people that crowdsourcing (particularly crowdsourced testing) is a viable method for helping to get important work done?

ME: I am relatively new to the concept of crowdsourcing and crowdsourced testing. It was a topic I started to get very interested in a couple of years ago. When I was at Agitar, we started a free service for JUnit test generation called JUnit Factory. It was a pretty successful open project that attracted more than ten thousand users. It was a win-win in that users received pretty good unit tests for their Java projects, and we got crowdsourced testing on our latest code base. Since then, I’ve kept an eye on crowdsourcing progress and I am pleased that it has made it to the software testing field. I do believe it is the right model for a lot of testing tasks, especially in the web and mobile space. So, I’m glad that uTest is making a go of it. I think the explosion of social networking over the past years has really made it apparent that the virtual crowd is a very powerful thing. It can solve lots of interesting problems we know about now, and I’m sure we will be discovering areas where crowdsourcing can be very effective in the future. So, I think crowdsourcing is a practice that is here to stay and will be with us in some form for as long as the web stays running.

uTest: The Mozilla community is unmatched in terms of sheer size, at nearly a million members. How can you possibly manage that much feedback on a regular basis? How do you separate the valuable data from the junk (aka: signal-to-noise ratio)? In short, how do you make something that sounds so chaotic seem so orderly?

ME: It certainly is a challenge and is a constant battle to stay on top of it. And the honest answer is that we do the best we can. As you have surmised there is a mountain of feedback that is received by Mozilla on a daily basis. Primarily the feedback we receive on Firefox is in the form of crash reports, Bugzilla bug reports, beta feedback and twitter. That list is in priority order of what we continuously monitor every day. We are constantly searching for any trends where a particular crash or reported bug starts to become and issue within the Firefox user population. Even a bug that affects a relatively small percentage of users say 0.5%, we are still talking about two million users. So there’s a constant vigilance to detect of there are any emerging issues within the code base.

In the latest beta releases of Firefox 4, there is the feedback feature at the top right of the browser. This allows users to quickly and with limited interaction give a thumbs up or down whether they are happy with the current behavior of the browser. This along with an optional comment is sent to a collection server for categorization and display: http://input.mozilla.com/en-US/. This has been a pretty successful method to get instantaneous feedback, although it does suffer from  a lot of noise contain within the data. But given the sorting and filtering tools we have certainly have received some key insights where we are weak and need more attention with our testing. We are exploring ways to allow users to participate more directly in the test effort through these types of feedback tool features.

uTest: People often (wrongly) view crowdsourcing as a passing fad, or something that only involves unskilled labor. In your experience, what’s been the biggest misconception when it comes to crowdsourcing or community-driven projects?

ME: I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard.  The reality is that crowdsourcing and community based projects are as diverse as the people that are involved.  The interaction and participation of the project members can vary drastically from project to project. The mechanisms for interaction can foster high communication and collaboration among the members or complete isolation and independent input.  There is such a big spectrum of crowd-sourced project implementations and it isn’t correct to have crowd-sourced and community-driven projects under the same umbrella. Strict crowd-sourced projects tend to be targeted towards tasks that are discrete and distributed in nature and is better suited for projects that require a number of paralleled tasks. In strict crowd-sourced projects members tend not to collaborate together to get a consensus results and may even be in a competitive situation to get the best result.

Community based approaches tend to be longer lived projects and are typically for public benefit. Membership is considered a privilege and betterment of the community as whole is usually at the top of the list of shared goals amongst the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Mozilla certainly is an example of of a community-based project. Many people would be surprised at our mission statement: “to promote openness, innovation and opportunity on the web”. It just so happens that building a world-class competitive browser helps us achieve the mission. Community citizenship and passion for the project has been the key success factors for Mozilla success in my opinion.

uTest: You were the QA Director for Palm when they launched the Palm Pre Smartphone, as well as the WebOS apps and services. What’s been the biggest difference (if any) between launching a mobile product and a web product?

ME: The biggest difference between a web product and mobile device is the amount of testing and certification that must precede the launch of a mobile product. A smartphone such as the Pre is an incredibly complex and highly integrated piece of technology–much more so than a typical web application. First off, a smartphone contains a fully-functional OS, usually based on some variant of Linux running on very constrained hardware. It must perform all of its concurrent services utilizing limited memory and limited CPU horsepower. The smartphone must also respond correctly to the multitude of many current events, from those generated from the environment–like switching from wifi to a WAN internet connection–to handling data input from the user, as well as handling events from the onboard applications.

Launching a mobile product requires exhaustive certification of individual hardware components such as the CPU, modems, codecs, and displays. Even then, the finished product is really launched by the carrier and must go through their exhaustive certification tests as well. Testing an onboard mobile application is also a much harder testing task. There are so many conditions and constraints that are involved in testing a mobile application. A typical mobile application is nearly functionally equivalent to any counterpart desktop client-side web application. Take, for example, a mobile email application. It must behave and interact with the server-side application in nearly the same way as a desktop web client. The established protocols were designed for a stable communications environment, but this is just not the case in a mobile environment. The internet connection may be lost and reconnected very rapidly. The connection may even be lost for long periods of time. The application may, at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of system conditions happen in a smartphone that would rarely or never happen in the context of a desktop web client application. However, a mobile application must perform its main functional operations of retrieving and sending messages flawlessly with no loss of data and full operational integrity. Testing mobile applications under these environmental scenarios is a huge challenge. In short, testing a web application is no easy task, but mobile applications and products represent a much tougher and larger testing challenge.

uTest: There’s been a lot of bold predictions about the future of native apps, the mobile web, tablets, and just about anything that relates to mobile. Having lived (and worked) through the dotcom boom and bust of the last decade, do you see any similarities with regard to mobile?

ME: Not really. In the dotcom boom and bust era, the problem was such rampant speculation about how we as consumers would utilize the web for just about every type of activity and commerce transaction. It seemed at that time that anyone with a slick PowerPoint presentation and an outrageous business plan could get startup funding. The main problem was that the technology of the day just couldn’t deliver the user experience that all the web startup companies promised. Here we are, nearly a decade after the bust, and I believe we are just beginning to fulfill some of the wild web usage predictions and capitalization by users and consumers. That’s quite a gap between the vision that was promised and when it was actually delivered. No wonder there was such startup company roadkill during the bust.

Mobile computing, like any other surge in adoption of technology, will have winners and losers. However, there seems to be no end in sight for all things mobile. There is such innovation happening in this space. Take, for instance, the iPad. It is a game-changer, in my opinion. It has defined a whole new computing platform between what we consider a mobile handset and a laptop, and surely is adding to Apple’s vast bank vault. Smartphones continue to wow us with their capabilities and the ability to combine consumer-based technologies into such a convenient package, and we are quickly seeing how these devices are changing how we interact and communicate with each other. Mozilla encourages people to think big thoughts about the future of mobile computing. Just check out this video about the concept mobile phone called seabird and you will see what I mean. I sure would like the phone for Christmas.

So here’s the big difference I see between the dotcom boom/bust period a decade ago: In this mobile era we are leading with innovation and seeing how the marketplace utilizes the technologies. Where as, in the dotcom/bust era it was a lot of empty promises about technology and investor riches that ultimately were never delivered. The current stage in mobile and web computing in general is set for an explosive boom possibly much bigger and broader than than the dotcom period.  But this time I think it will be sustainable. Mobile computing and platforms are here to stay and will be the magnet for innovation for the foreseeable future. The current biggest hurdles are battery life and communication bandwidth. The latter is being addressed as we speak. 4G networks will have broadband connectivity. Hopefully battery life will also be improved shortly. A bright future ahead is my prediction.

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

ME: It’s always been hard to find good software testers. Good or bad economy, a seasoned and talented software tester is rarely unemployed. Typically, many good testers wind up in development because it often appears to be the growth path for a software technology career. Software testing as a career path is still viewed as something lesser than development. However, there has been a lot of gained respect for the software testing career path in recent years. The industry and practice has definitely matured since I started as a QA engineer when breaking software meant dumping your card deck all over the floor. So, I think the pool of quality testing folks is better now than ever, but it is still very hard and time consuming to finally hire someone. How do you fix this? We are always in search mode. We never stop looking for good QA engineers. Also having a team of great recruiters that can poach well is very key as well.

uTest: Which blogs/sites/magazines/conferences do you frequent to stay on top of the latest trends and tools in testing?

ME: I haven’t been to many conferences in the last ten years, except of course the recent GTAC conference in Hyderabad. Mostly, this is because I’ve been involved in startups which didn’t have the budget for sending folks to conferences, and I couldn’t afford the time as well. I have followed James Bach’s blog over the years; he’s as much an entertainer as he is a software tester. I also read a lot from Michael Bulton, mostly because I think he is the E. F. Hutton of the testing world. He is definitely worth listening to. Last year, I took some time off between jobs. It was a great time to have the luxury to really do a lot of web-surfing and getting caught up on things. I became rather fascinated with Yvette Francino. She’s a wonderful writer/blogger, and she chronicled her story of getting laid off as software test manager. It was quite the feel-good story to see how her use of blogging and online networking landed her a job as editor of SearchQuality.com, a site I still frequent today. Lastly, it was great to finally meet James Whittaker in person at GTAC this year. He’s quite a speaker, and I’ve learned a lot from his posts and books. I can’t wait to see how he will avenge being called the “octomom of testing books” at the GTAC conference .

uTest: What could software companies (eg: Palm/HP) learn from the open source movement about launching great products?  And vice versa?

ME: Traditional software companies are skeptical of open source tools and projects. They generally think that they have lesser quality that commercial counterparts. That may have been true some years ago, but I don’t think it is true today. Open source projects have one great advantage over commercially developed ones. The rate of updates and fixes is typically is much higher than commercial projects. I’m mostly talking about the bigger projects such as the popular linux distributions, apache tools and even Mozilla. We are unparalleled in our transparency and response to reported security flaws for instance. In fact being more transparent across the development cycle is a competitive advantage to open source. It encourages user feedback and therefore course corrections can be done incrementally and continuously and ultimately deliver a product a user is expecting. I think that software companies should be more transparent and allow their customers to participate more in the development process. What can open source projects learn from commercial companies? If there is anything it would probably be discipline around scope of the project. Too often, I see that open source projects get feature bloat or accumulate features that are cool to build but are not really needed by the user. Since, most open source developers are working on the projects for the pure enjoyment of it, they are often attracted to working on new stuff and are less interested in improving or fixing regression bugs.

uTest: Put on your prognosticator’s hat – what’s the next big “wow” feature or innovation in the world of browsers?

ME: I think Aza Raskin of Mozilla Labs fame has it right. Identity and browser identity awareness are the next big wow browser features in my opinion. Let me try to explain, although I’m sure Aza does a much better job of it. Your browser knows or could know all about you: your browsing habits, where you browse, what you buy, where you login to, who you communicate with, etc. etc. Currently, much if not all of this data is being captured on the web in many, many places. I won’t go into any of the arguments about whether your captured browsing habits and history are truly anonymous data or whether this data is being used only for commercial benevolence. But one thing is clear: you don’t own this data, and you can’t access or alter it either. And for many, that is a real problem. However, this data can be very useful to you. It is mostly used for a better experience on the web–or at least a merchant’s version of what it thinks will be a better experience for you. Well, your browser could do a much better job of it. It could capture this data directly and it would only be about you. This data could be stored locally, or even in the cloud using services such as Firefox Sync. It could do many cool things like real targeted pre-fetching of your favorite blogs, and then scan them for keywords that are important to you and arrange them in priority order. It could sense what tabs are used more often and arrange for a better workflow experience. I think there is a lot of assistance a browser could do either in the context of the browser interaction, e.g., automate tasks or perform operations based on the web content. Well, there is one huge obstacle, and that is security. How do we ensure that this personal data that absolutely defines you is kept safe and secure? And how do we prove it to you so that you feel safe in allowing a browser to continue capturing this data and making operational decisions based on this data? That may take awhile, but little by little we are making inroads towards this goal such as the Firefox Sync feature that securely stores your profile data in the cloud across instances of Firefox running on desktops and mobile devices. We are getting there, but I suspect it will be a while before you let your browser do your online banking for you.

uTest: What’s the greatest challenge that tech execs will face around app quality in 2011?

ME: The biggest challenge I see is around compatibility testing and maintaining a consistent user experience across the landscape of platforms. To be competitive in today’s web-enabled software market, you need to support a variety of platform interfaces. It is now expected that any web-accessible service will include feature-rich clients for iPhones, iPads, tablets, smartphones, desktops, and kiosks. The compatibility matrix starts to explode when you add in device models, OS versions, and other configuration-dependent attributes. In my experience, you do need to test on real hardware and configurations. Emulation capabilities do help, but at the end of the day you can not be sure of software behavior unless you test it on live hardware. Its a real problem that doesn’t seem to get better with the plethora of new platforms available to users.

Myth One: “You only need to unit test. TDD testing is sufficient”

For the vast majority of commercial developments this simply isn’t true. Even strong proponents of agile development recognise the need for their armory to include a range of testing techniques. For example, Scott W. Ambler has a list of twenty-one different testing techniques as part of his FLOOT (Full Life Cycle Object-Oriented Testing) methodology, including white box, black box, regression testing, stress testing and UAT. (http://www.ambysoft.com/essays/floot.html)

TDD programmers rely on these tests to verify their code is correct. If the requirements (test cases) are specified incorrectly, then you’ll build robust code that fails to meet the objective. Therefore, most agile projects include investigative testing efforts (black-box), which support rather than compete with white-box testing. Good investigative  testing will reveal problems that the developer missed before they get too far downstream.

Myth Two: “You can reuse unit tests to build a regression test suite”

Some TDD proponents argue that conventional downstream testing is unnecessary because every line of application code has a corresponding test case; they suggest that by reassembling unit tests, everything from User Acceptance Tests to Regression Tests can be performed.

Although this sounds feasible, it isn’t realistic, and here’s why: The granularity and objectives of white-box unit tests developed in TDD serve a different purpose from downstream black-box testing. While the overall objective of a unit test is designed to prove that the code will do what is expected, the aim of regression testing is to ensure that no  unexpected effects result from the application code being changed. These two objectives are not synonymous – e.g. checking that an attribute has a valid date format, is not the same as checking that for a given input, the value of the field contains an expected date.

Myth Three: “We no longer need testers, or automation tools”

Professional testers fulfill a different and equally valid role from their development colleagues. It is widely recognised  that traditional test automation tools have failed to live up to the hype of the vendors. Original Software’s roots are as  providers of products that improve the productivity of the database testing environment. It was because there were  no adequate tools to provide User interface testing that we extended our solutions into this market sector; our heritage is aligned to effective testing rather than screen-scraping automation. When we developed TestDrive, we did it with the benefit of twenty years hindsight of missed opportunities from the traditional vendors. Often, TDD projects have at least as much test code as application code and, therefore, are themselves software applications. This test code needs to be maintained for the life of the target application.

Myth Four: “Unit tests remove the need for manual testing”

Manual testing is a repetitive task; it’s expensive, boring and error-prone. While TDD can reduce the amount of manual effort needed for functional testing; it does not, remove the need for further black-box testing (manual and automated). By automatically capturing the tester’s process and documenting their keystrokes and mouse-clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating. Though manual testing is a time-consuming (and therefore expensive) way to find errors, the costs of not finding them are often much higher.

Myth Five: “User Acceptance Testing is no longer necessary”

Within agile development, acceptance testing is often positioned as working with the customer to resolve “incorrect requirements”, rather than correcting functionality that does not map to what the user needs. When the user initially defines their requirements, they do it based on their initial expectations. When they see the system “in the flesh” they will invariably come up with different, or additional, requirements. While agile methods might reduce the frequency of this happening, it is unreasonable to expect the approach to resolve them entirely, so there should be no expectation that user acceptance testing will be obviated. This is especially true when it comes to the business user signing off approval of the User Interface, since they may have envisaged something different from the developer; which brings us nicely to myth six…

Myth Six: “Automation is impossible”

Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. To begin with, almost all testing from a user and QA perspective will be manual but this testing effort and design can be beneficial later if captured and re-used. Knowing  the right time to automate is tough, so using technology that proactively supports early manual testing but provides a path to evolve this into automation is key.

Myth Seven: “Developers have adequate testing skills”

If testing was easy everybody would do it and we’d deliver perfect code every time. An independent testing team serves as an objective third-party, able to see “the big picture”, to validate the functionality and quality of the deliverable. While developers tend towards proving the system functions as required, a good tester will be detached enough to ask “what happens if…?” When you include business user testing as well, you are more likely to have a system that is fit for purpose. Finally, and I’m sure this point will be disputed, most developers don’t actually want to spend much time first writing tests and then developing code to prove the tests work. Using the collaborative process described below, the developer gets ample assistance in describing the functional tests and can focus on writing lean, accurate and robust code.

Myth Eight: “The unit tests form 100% of your design specification”

Whichever development method you use, before you develop code you have to think about the requirements. While TDD “done well” may identify a fair percentage of the design specification, there are still concerns about gaps between the unit tests. The argument, that you need TDD to prove the requirements are captured accurately, isn’t substantiated by history.

The V-model, for example, is a valid approach to understanding testing requirements and by implication, functional requirements. Like TDD, the V-model and most other models, fall down when the practitioner’s approach is rigid, while development processes are fluid. Whichever approach you choose, correct thinking challenges each user requirement by asking “how would I test that?” The important factor here is that test cases are challenged before they are committed to code, otherwise you’ll be doing a lot of recoding and calling it “refactoring”. When requirements are refined through collaboration, the developer receives a more robust specification that is less likely to change because it has been openly appraised from several people’s perspectives.

Myth Nine: “TDD is applicable on every project”

As the size of the project increases, the time required to run the tests also increases. This can be addressed by partitioning either/both the project and/or the tests. Either route results in tests that run at different frequencies depending upon their relevance to the current coding effort.

This approach introduces the need for test planning and execution management. To achieve this efficiently, in addition to white-box testing, you need to be thinking about:

Integration Testing – “Which tests do I need to run to ensure the new code works seamlessly with the surrounding code?”

System Testing – “Does the functionality supported by the new code dovetail with functionality elsewhere in this system, or in other systems within the process flow?”

Regression testing – “How often do I need to run a regression test to ensure there are no unforeseen impacts of  the new code?” Automated regression testing provides a safety net that can affirm agile development techniques”

User Acceptance Testing – “While TDD (in collaboration with business users) should ensure that a specific function performs correctly, is the cumulative impact of changes still acceptable to the business users?”

However, in today’s environment it is unacceptable to think of these testing stages as discrete serial activities. Often they will need to be run in parallel as we get a new ‘code drop’. As the size of the project team increases (along with testing effort), it is no longer acceptable for the tests to be considered “self-documenting”. Whenever the number of participants increases, the project’s risks become exposed to its members’ different interpretations of the requirements – the definition of what constitutes “success” can be misconstrued.

As the size of the project increases, so would the amount of test code that needs to be developed. Any test code developed needs to be supported for the life of the application – effectively doubling the ongoing maintenance effort.

As the size of the testing workload increases the project needs to add test automation to its armory, to minimize the human intervention and elapsed times for each of these testing cycles.

Myth Ten: “Developers and Testers are like oil and water”

Since the dawn of time there has often been a “them and us” tension between developers and testers. This is usually a healthy symbiotic relationship which, when working correctly, provides a mutually beneficial relationship between the two groups resulting in a higher quality deliverable for the customer.

The discussion should be focused on:

• Software delivery that meets business objectives (fit for purpose, on time and on budget), not who owns which part of the process.

• What can testers contribute to the requirements gathering phase to ensure they are involved in the TDD process?

• How can testers maximize reuse of the assets created during the development phase?

• Is there a role for the “traditional tester” in TDD, or should they (like the developers) be acquiring new skills to enable them to adapt to the new paradigm?

• How can developers and testers find mutual benefit in exploiting new advances in software development and testing tools?

Just as improvements in developer’s software tools and methods have enabled a shift in development approaches, next generation technology for test automation is similarly reframing the opportunities for testers to automate earlier in the delivery cycle without incurring the heavy burden of script maintenance so often associated with traditional automation tools.

Conclusion
Agile projects are in fact an excellent opportunity for QA to take leadership of the agile processes; who else is better
placed to bridge the gap between users and developers, understand both what is required, how it can be achieved and
how it can be assured prior to deployment? QA should have a vested interest in both “the how” and “the result”, as well as continuing to ensure that the whole evolving system meets the business objectives and is fit for purpose. But it requires QA professionals to be fluid and agile themselves, discarding previous paradigms and focusing on techniques  to optimise a new strategy to testing.

The Reality of Software Testing in an Agile Environment
http://www.origsoft.com/whitepapers/agile-environment/myths.php

Our Testing the Limits guest this month is Ben Simo. Known as the “Quality Frog” on Twitter, Ben is one of the most insightful and entertaining testers in the business. A proponent of the context-driven school, Ben has more than 19 years of experience testing software and developing testing tools. He currently lives in Colorado with his wife, two children, two dogs, five cats and fourteen – count ‘em – fourteen goldfish. For the full Ben Simo experience, go to his blog.

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

I’d like to be able to say that I carefully selected the name and that it signifies that I care about quality and I’m amphibious like a frog. I’d like to say something along the lines that I started life as a tadpole in the waters of programming and later grew legs to live on land and be a tester.  I could even say that as a Quality Frog, I now eat bugs for breakfast and help keep the waters clear. While such thoughts may have come to mind, the truth is that I came up with Quality Frog while pairing a variety of words with Quality in search of an available domain name. Frog came to mind as something that ate bugs and the domain name was available. Since then, I’ve continued to use it as a handle.

uTest: You’re a self-described “defensive pessimist”, which seem like good qualities for a tester to have. What other attributes come in handy in this line of work?

Simo: The term “defensive pessimist” comes from Dr. Julie Norem, a psychology professor at Wellesley College.  In her book, “The Positive Power of Negative Thinking”, Dr Norem describes defensive pessimists as people who typically perform worse when pressured to look on the bright side and be optimistic about things that concern them.  Rather than trying to think happy thoughts and only look at the positive, defensive pessimists imagine the worst case scenario; not to get depressed and become immobilized, but to develop solutions for what might go wrong in order to be better prepared. Defensive pessimists can make great testers, and are likely to annoy many optimists.

The third guiding principle of the Association for Software Testing states “AST views software testing as a cognitively complex activity that requires critical thinking, effective communication, and rapid self-directed learning.”  I fully agree with this.  Therefore, I find it essential that testers be critical thinkers, effective communicators, and self-educators.  Any one of these three things without the others will make us less effective as testers.

This doesn’t mean that every tester must master all three of these on their own.  My ideal test team would be comprised of people with a diversity of aptitudes, skills, and experience. I don’t want a team of clones.

uTest: Much like our previous Testing the Limits’ guests, you’re a critic of testing certifications. Yet some still see certifications as the only way to stand out from the “unskilled labor” crowd. Tell us a bit about why you’re a skeptic/critic of certs – and how they could be improved and made more relevant/useful/predictive.

Simo: I am not against certifications per se. I am against bad certifications.  I am against certifications that are presented to be something other than what they are. I am against certification bodies and trainers that prey upon people’s desire to stand out and tell people they can improve and certify their competence as testers with few days of training and a multiple-false test. In his keynote at the Conference of the Association for Software Testing (CAST) this year, Cem Kaner stated that If one can become certified in their profession in three days, they are a commodity, and don’t deserve much more pay than unskilled labor. I agree with Dr. Kaner. Rather than educate and help people stand out from the unskilled labor crowd, such certifications trivialize testing and encourage wrong thinking that testing is unskilled labor.  I want testers to be more than a commodity.

Many IT certifications, including testing certifications, are more about marketing than education.  These certifications are not good measures of skill, competency, professionalism, quality, or any of the many things those on the receiving end (of the marketing) care about. In my experience interviewing job candidates, tester certifications have not been an indicator of applicants’ testing abilities.

Software testing is a rich and diverse field. It is also a young field. Rather than feign maturity and simplicity where there is none, let’s embrace the diversity and youth. Let’s continue to learn. Let’s not lock in a set of context-free definitions and practices and make them a standard. Such standards will hurt the quality of software, not improve it.

Rather than pursue a certification, I encourage testers to get involved in a professional community. Find colleagues that challenge you and help you learn.  Seek real education that comes through interaction and doing over memorizing information useful in passing a multiple-false test. The Association for Software Testing offers a series of online software testing courses that facilitate deeper learning that you are likely to find in training focused on helping you pass a certification exam.

uTest: Jon Bach mentioned that changing the meaning of “QA” to Quality Assistance would help outsiders (engineers, executives, et al) better understand the role of this discipline.  Agree or disagree?

Simo: I believe I first heard  “Quality Assistance”  from Cem Kaner.  I agree with Jon. When testers bear the title Quality Assurance, it often implies that they actually assure the quality of other people’s work. Testers are in a position to help assist quality; not assure it. Let’s not assist the setting of unrealistic expectations with inappropriate titles.

uTest: While we’re on the subject, are you anyway related to James and Jon Bach? The resemblance is uncanny.
Simo: I don’t think so. I’m available for adoption if the Bach family is interested.

uTest: You’ve said that you frequently use automated tools, but that you don’t trust them entirely (back to that whole defensive pessimist thing again). What advice do you have for testers and managers wanting to strike a healthy balance? And what’s currently in your arsenal of automated tools?

Simo: My mistrust in tools is based on the fact that tools can’t think for me. Automated checking can only process whatever decision rules someone thought to program when the checks were created. Automation will consistently do what it is programmed to do and consistently not do what it is not explicitly programmed to do. I find test automation to be useful. In fact, there are some things I’d not want to even try to do manually. I do, however, distrust the green bar. When automated checking passes, I ask myself what the automation does not tell me. I also try to keep aware that people who don’t understand what the automation does are likely to assume that it does more than it does.

Tools are much more than test automation. Tools are essential for testing. I don’t want to test without tools. I have some old programming books that promote testing in which a programmer manually executes code, step-by-step, with pencil and paper in order verify that the code works as expected. This is manual testing. This is a testing practice that came from a time when computer time was rare and cost more than people. We’d now laugh at someone proposing testing in this manner.

Today, whether we call it “manual” or “automated”, tools and automation are part of coding and testing software.
Programmers use fancy integrated development environments that do a great deal of the tedious work of managing and checking code. Programmers have unit testing tools that allow them to write tests as they write the code. We use tools to track and version source code. We have continuous integration tools that automatically run pre-defined tests and report results when programmers save code into shared code repositories. Today, programmers rely on interactive tools to do work that used to be done manually. These tools support people coding software rather than trying to replace them with batch processes.

Many of the tools built for testers seem to still be stuck in the world of batch processing. Rather than assist testers by helping with tedious tasks, they try to replace test execution. Some of this comes from the design-all-tests-up-front-in-procedurally-scripted-detail thinking that made sense when computer time cost more than people.

I worked as a tester for nine years before I encountered a record and playback type test automation tool. In those nine years, tools played an essential role in my testing. We built all our tools in-house. We wrote macros and scripts to do tedious data processing. We wrote code to verify data formats. We created test interfaces for databases. We created test management systems. We depended on tools but did not have a single test case that was fully automated. This automation made testers both more efficient and effective.

Rather than look at testing as something to be either manual or automated, I encourage people to look at individual tasks that are part of testing and try to identify ways that automation can help testers evaluate software. Rather than limit tools to automation of test execution, I look for ways that automation can assist testers – to turn them into powerful cyborgs.

These tools include automated test execution tools, data setup and teardown scripts, communication tools, test management tools, test logging and reporting tools, test data manipulation tools, and more. Along with test execution automation tools (take your pick), my most-used testing tools are Excel, Picasa (for screenshots), a screen recorder, and the GNU Unix utilities (also available for Windows).

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

Simo: I’ve not noticed great distinctions in testing practices based on industry.

What I have noticed is that testing practices seem to evolve within individual companies and test groups. Rather than learn from what others have done, it often appears that each tester, each test organization, and each company starts in some stone age and evolves on its own – making the same mistakes that others did long before.  Lessons learned don’t seem to be shared much across companies.

I’ve also noticed that “quality assurance” groups in large commercial businesses, regardless of industry, tend to be dominated by factory school thinking. Instead of viewing testing as cognitively complex work that requires critical thinking, good communication, and continual learning: testing is thought to be predictable repeatable validation work that can be done by automation or cheap labor.  These companies tend to view quality and process as things to be scripted, controlled, and quantitatively measured. In doing so, I believe these companies dehumanize their people and their customers.  These companies tend to have what Jerry Weinberg called “overstructured management” in a paper written nearly 30 years ago; management which attempts to manage people like code: with sequential work models, binary either-or choices, and mechanical repetition. These things may be good for computers, but aren’t great for people.

While many companies are still recursively calling on the mistakes decades past, others are recognizing that people aren’t to be controlled like machines.  While they may not be familiar with either document, these companies exhibit the values expressed in the Agile Manifesto and practice the principles of the Context-Driven School. These companies value people over systems.

uTest: Who should testers be following on Twitter to stay current on the latest industry trends (I mean, besides you and us, of course)?

Simo: I’m not as interested in trends as much as I am in experience reports, and new ideas and challenges. Some of my favorites who are active on Twitter are Pradeep Soundararajan (@TesterTested), Shrini Kulkarni (@shrinik), Matt Heusser (@mheusser), Lanette Creamer (@lanettecream), Elisabeth Hendrickson (@TestObsessed), Michael Bolton (@MichaelBolton), Jon Bach (@jbtestpilot), and James Bach (@JamesMarcusBach). Beyond that, I suggest people see who these people, and I, follow and interact with on Twitter. Use Twitter’s search feature to find people tweeting about things that interest you. Don’t limit yourself to only testers or those people you like. We can learn a great deal from those outside our field or with whom we disagree.

uTest: Let’s go back in time for a second: How did you get into the craft? What was the first application you tested? What was testing like back in the early 90s?

Simo: Providence. It was providence that got me into testing.

I was young, in love, and planning to get married.  I had been doing some part time database development work, but needed a full time job before the wedding.  I submitted letters and resumes to dozens of companies. I was willing to do almost anything that would pay the rent.  I lived in a city where the local job market was dominated by defense contractors. I quickly learned that many of them called nearly everyone who applied for anything in for an interview; so they could learn about people and add them to databases of potential hires for matching to work they did not yet have.  These companies would then present these people to the government as their available workforce when bidding on contracts. This made it frustrating for those of us looking for work. It often wasn’t clear, when going in for an interview, if it was for a real job or for a potential position that might come at some time in the future if that company were to be awarded a government contract.

I interviewed with the company for which my fiancé (now my wife of 19 years) Sophie worked. It appeared to be one of those information gathering interviews without an actual position to fill. I was asked a lot of questions but none seemed related to a specific opening.  At the end of the interview, the interviewer said he’d be calling me.  Time went by without any more contact. Nearly a month later, I got a call asking if I could start the next morning.

I was one of a handful of people hired to work as “data collectors”; to execute tests and collect results for an interoperability demonstration of digital imagery systems.  These systems, from a variety of different vendors, were used by the military for manipulating and exchanging digital photographs.  This test was the second phase of an effort to validate a new communications protocol standard. Our mission was to assess to what extent systems made by different companies could exchange information over a variety of communications systems. We were given a huge matrix of imagery systems and communications equipment (from encrypted telephones to push-to-talk radios) combinations to test. We quickly learned that this required we learn the communications systems, the imagery systems, and the protocol used to exchange data.  I enjoyed this challenge. By the end of the project, I found myself in the role of being the writer of the test report and numerous interface documents. I also learned that I enjoyed testing.

If I hadn’t received that call which led to becoming a tester, I would have likely taken a job as a pay telephone technician. We all know how cell phones have since killed the pay telephone business.

So, what was testing like in the early 90s?  For me, working for an independent fee-for-service testing organization, it was a great learning experience.  I learned that my role as a tester was to help our clients produce better products; not to just pass or fail their products.  I learned that every company followed different development and testing practices – regardless of what labels they gave them.  I learned to seek guidance from others who could mentor me. I learned that continual learning is essential in testing. I learned that testing is a better assessment tool than an assurance tool. I learned to plan and script testing only to the level of detail that made sense; understanding that some detail cannot be known up front.  I learned that testing is exploratory in nature.  I learned to communicate with stakeholders and developers. I learned that the real requirements aren’t necessarily the things documented in a technical requirements document.  I learned to create tools to help me accomplish tedious tasks and enable doing things I could not do manually. While I didn’t realize it at the time, I was becoming a context-driven tester.

After nearly a decade, I moved to working for commercial software development firms — with fear and trepidation.  I wasn’t sure that my skills would transfer to being an in-house tester. I soon discovered that they transferred well.  I spent a short time with some misguided “quality cop” thinking after I was given a “Quality Assurance” title before better understanding and returning to my context-driven roots.

Rapid Fire Q&A:

• Favorite movie of all time: My favorite movie of all time would have to be “Star Wars, Episode IV: A New Hope”. It is both one of my earliest and favorite childhood movie memories, and I still love it

• Favorite movie… that’s about testing or taught you something about testing: I don’t know about it being a favorite, but “Man of the Year” comes to mind. The movie was disappointing in that it wasn’t the comedy it was advertised to be; and the software “bug” on which the plot was based made little technical sense; however, it illustrates the potential risks of software not working as it should – either due to oversight or ill intent.

• Mac or PC: Commodore. I still do everything on my Commodore 64. Seriously, I’m a PC. I’ve not used a Mac in over 15 years.

• Browser of choice: I’ve recently come to like Chrome for most of my personal browsing. However, I prefer Firefox for testing. There are many cool add-in tools for Firefox.

• Brand of smartphone: Motorola Droid X. After seven years of using Windows Mobile devices, I just switched to Android.

• Favorite US president (no fair picking Millard Fillmore… he’s my fav): Teddy Roosevelt. I can’t help but love an adventurer.

• Top tech innovator of all-time (Edison, Gates, Jobs, Andreesen, et al): Apart from the modern computer, Thomas Edison has likely had a great impact on humanity.  So, if I had to pick only one, I’d say Edison. When it comes to computers, Charles Babbage and Steve Wozniak come to mind. The ideas of Charles Babbage led to the programmable computers we have today. And Steve Wozniak was essential in making computers personal and bringing them into our homes

• Best concert you’ve ever seen in person: I haven’t been to a real concert in over 20 years. However, I have fond memories of Barry McGuire sitting on the edge of a stage singing songs to the last couple hundred stragglers who remained.after someone else’s concert.

• Hobby that would surprise our readers: Since I recently quit my job and am looking for new work, I could say that software testing is now a hobby for me. It will hopefully be paying work again, real soon. I could call my obsession with old programming books a hobby. I collect and read old programming and testing books to better understand the history of software development.

Photography and exploring Colorado are two things I could also call hobbies; but those won’t surprise anyone who follows me on Twitter. Also, we’ve got a bit of a zoo at our house with cats, dogs, and fish. That too should not be a surprise.

What may be a surprise is that I’m a recovering collector of NASCAR die cast cars. Yes, I was once an addict with new cars regularly arriving in the mail.  I’ve now been clean, and not added any new cars to my collection, for well nearly two years.

uTest: If Al Gore had never invented the inter-web and software testing didn’t exist… what would you be doing for a living?

Simo: I’d likely be a starving photographer. When I was eighteen, I was considering pursuing a career in computer programming or photography.  My father advised me that he’d encountered more starving photographers than computer programmers. So, I pursued software over photography.

It was one year ago (June ’09) that James Whittaker helped us christen our ‘Testing The Limits’ interview series by being our first guest. And for much of the year, he held the distinction of generating the most page views… and then some guy named Patrick Copeland came along and took the lead a few months back.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

What we are working toward is a browser that is capable of testing applications that run inside it. The idea is to build the test framework into the browser, in our case Chrome, so that test artifacts don’t have to be moved around and installed in test labs and on test machines. Also since the browser and automation are one, problems such as Ajax and self-updating web pages won’t confuse automation. This is true cloud-based test automation. What I want is the ability to launch one of these WTF-equipped browsers and point it at the web site/app I want to test and have all the back-end test case storage, bug reporting, progress reporting etc managed for me, automatically, in the cloud. Tests that I wrote or others wrote can be consumed by the browser and applied en mass. These assets all exist in the cloud at Google, why not make them universally available?

I’m betting you want the same thing.

uTest: You’ve argued that the past two decades of software testing innovation have been “the disposable decades” – too tied to the app, the domain and the technology. What does the next decade look like to you? What will surprise us most?

JW: At GTAC 2008 and EuroStar the same year I gave a talk on the Future of Testing and I am returning to GTAC in October 2010 to give an update on my vision for the next decade. I don’t want to preempt that talk, but I will say that my vision is far more aggressive now than it was then. What I thought was 10-20 years out two years ago will be here in less than 5. Things are changing fast and I am anxious to push them faster. We are working unnecessarily hard to test our applications and it is limiting innovation and creating a market full of sub par software. This has got to stop.

The central theme of my vision of the future is getting rid of this disposable nature of our test artifacts. We are reinventing the wheel every time we test wrt planning, test development, automation, reporting and metrics. Let’s get together and stop this. One of the reasons I am so nice to you guys at uTest is that you are working with the idea of leveraging the cloud and the crowd to perform testing. There is much more collective intelligence involved in what you guys are doing and I like that. I am anxious to get our testing tools in the hands of your crowd.

uTest: In your most recent blog post, you describe a (very near) future where test data will be more like a cellophane wrapper around a web app UI, an overlay that would replace the tiresome querying of multiple databases. Just how close are we to achieving this innovative bug and test case overlay and what would be the significance of such a feat?

JW: You keep trying to pull my GTAC talk out of me prematurely! But let me give a bit of a preview. Test managers need data to make decisions about how to optimize their testing and their testing resources. Most technical test managers I know end up creating a set of SQL queries that they use to get information about features, bugs, test cases, test runs and so forth. All the data is there, it’s just hard to access. Video gamers have a similar problem, there is a lot of information in the game and it is in a lot of different places. Our solution in testing should be the solution gamers use: a heads up display – a single location to surface all the information a tester and test manager needs to make the right decisions during a testing campaign. We are building a heads up display that can consume all this information and use it to guide the hand of the tester and test manager. This is part of what I meant earlier when I said that manual testing needs automated assistance. People doing exploratory testing without a HUD are like gamers who play without one. In the latter case, the gamer’s character gets killed, in the former case the tester is less effective.

uTest: You are a busy man, James, having less and less time to write witty blog posts (which we miss)! What has been keeping you most busy and what have your greatest accomplishments been in the past 12 months?

JW: Busy or just having so much fun I can’t take the time to write a lot? My greatest accomplishment happens every day: people who work for me questioning every aspect of our business and pushing innovative solutions to other testers to improve the way they execute. I had a direct tell me two weeks ago that he couldn’t wait for Sunday to pass so he could come to work on Monday. Nothing else I do compares to that.

uTest: One of our top testers, Madhukar Jain, would like to know: Do you see cloud computing as the next evolving challenge in testing? We would take this one step further and also ask if the cloud represents or enables the fourth gen of testing, or testsourcing, you’ve often alluded to? What is the cloud’s potential to empower the crowd (third gen)?

JW: I see cloud computing as the way to solve the testing problem. And yet again you are forcing me to reveal my GTAC talk prematurely. So be it. Imagine for a moment every test case database, bug repository, build script, static analysis tool, metrics engine, and so forth in the cloud. Every tool and scrap of information a tester will ever need stored in the cloud and surfaced in his HUD when it is most relevant. It’s already stored on some server somewhere, putting it the cloud is a small step. Over time (hopefully a short time) we’ll be able to use this information seamlessly. Instead of the Internet, you have the Testernet capable of answering any question you have about testing and being there to guide you through whatever testing tasks you are performing. Other industries are already doing this. We’re behind the curve in testing.

Do you remember Star Trek The Next Generation? My friend and fellow Googler Dan Massey and I like to hold this series up as the type of technology we want to build. Geordie and Data were constantly “reconfiguring the subroutines” to solve some important task. They could reprogram the phasers, the photon torpedos, the holodeck, anything on the ship! Plug and play “subroutines!” They did this from any console on the ship. They did this without installing any back end repositories or test assets because it was all available in the cloud. They did this without running any all-night build scripts or writing new test cases. The build happened automatically, test assets were applied automatically. The reason Geordie and Data could do this programming so quickly is because they only had to solve the problem they were working on, they would ignore all the back-end build and test nonsense better suited for a computer.

I bet Geordie had one hell of a testing HUD and never had to wade through the complexity of partially automated build systems and SQL queries to find the tests and test data he needs, in fact I bet he didn’t even know anything about these backend systems. They were hidden in the cloud, exactly where they belong, allowing him to concentrate on his programming.

uTest: The Microsoft vs. Google battle continues to play out very publicly in the media. Just last week, Computerworld wrote this story: “Microsoft: No Matter What Google Says, Windows Is Secure.” Having been at both companies, we think you have a unique perspective on this one. Any thoughts?

JW: Let me say right away that I enjoyed my time at Microsoft and admire the company and the smart people who work there. As a resident of Seattle, it is in my best interest for Microsoft to prosper! But the two companies are vastly different regarding the way their talent is managed and their products are built. Google is an engineering-centric company where innovation comes from individuals who are empowered to do whatever they need to get ideas into production. Much has been made of Google’s game-theory approach to managing people where rewards are given quickly for impactful behavior. It works. Morale is high and people work very hard and take quality very seriously.

Does this mean we produce more secure or more reliable products? We try hard to do so; Microsoft tries equally hard. I think we have the advantage of less legacy and a more modern and reliable platform (the Web as opposed to client operating systems) to work from. But the secret sauce at both companies is the same: hard work and due diligence.

uTest: You shared with us (as the pioneer of Testing the Limits posts) that your first assignment at Google was “To raise the level of testing precision and diligence.” So, how did it go?

JW: It didn’t take long. Google was mostly already there so I can’t really take credit for it. Now I am busy raising the bar further.

uTest: Top tester Glory Leung is curious: What are your views on SCRUM testing in general? Are people doing it properly? What is the ideal situation?

JW: Scrum is just a name. I don’t like names, they feel too confining and people have their own ideas of what they mean. I took a lot of flak for using the name ‘exploratory testing’ for my book. People love to confine you to how they view a specific named idea or technique. Flexibility is required.

The ideal situation is that the best testers for the job be involved. For unit testing and TDD this means the individual devs who wrote the code. No one is more qualified to test a function in isolation than the person who wrote it. However the dev is often the worse person to test how his function interacts within a larger feature or with its operating environment. At Google we expect devs to do the former, all of it. Then and only then have they earned the right to have testers do the latter. Having broken or missing unit tests is a great way to ensure that testers get taken off your product.

The question I am struggling with now is product expertise. What is the right mixture of testers who are close enough to the product to become real experts with it and specialist testers who are expert breakers? Call it whatever you like but product expertise and testing expertise are the two ingredients that I find make for good testing, regardless of your ship cycles or how your team develops their code.

uTest: There are many varieties of “testing frameworks” out there that guide testers in choosing test cases and defines the way they approach test design. Out of the ones you’ve recently seen (e.g. Input Domain, Divide & Conquer, Fishbowl, Storybook, Pessimists), which of these works best? Is it a combination of two or three? Does it depend on the job?

JW: More names, less meaning and more chances of some consultant claiming ownership over them! Like I said in the previous answer to me the thing that I find most valuable are people who know the product and people who know how to test. Stir those together and you have a winning formula.

uTest: Doc JW rapid-fire (in no particular order):

1. Last book read? The Hobbit with my 12 year old son. It was his first time (how I envied him for that!)
2. Last movie watched? Avatar, 4 times.
3. Favorite band or album? Neil Young, Led Zeppelin, and Nirvana … in that order.
4. Favorite Kevin Bacon movie? Kevin who?
5. Browser of choice? Chrome!!!
6. Cats or dogs? Dogs, not even close.
7. Favorite video game? I played Monkey Ball 14 years ago, once.
8. Which mobile device are you packin’? Android Nexus One
9. Favorite mobile app? Google Sky Map.

uTest: What would you be doing with your career if you were not a Test Director (e.g. bullfighter, gourmet chef, cobbler, explorer)?

JW: Stand up comedian. I still intend to do it some day.

uTest: Can you give us a sneak peek into any of your upcoming guest lectures, books, articles or presentations on the horizon?

JW: Alan Page inspired me with his book on how Microsoft does testing, perhaps I will write one on how Google does it. There is almost no overlap. I have also doodled a memoir of sorts trying to figure out how a guy like me managed to succeed. I’ve only shared it with a single reader who thinks I should wait until I retire to publish it.

uTest: It’s graduation time…  what would you tell college students who want to enter the world of professional software testing?  What should they do (or avoid) to get their career started on the right track?

JW: Simple, learn computer science. Then, get a job at a company that takes testing seriously and be prepared to learn. A degree in CS is, for me, the first and most important background.

uTest: Since you literally wrote the book on the subject, what separates a bad exploratory tester from a good exploratory tester?  And what’s the difference between good and great in this area?

JW: Bad exploratory testers spend little time preparing and are only seeking bugs as their primary outcome. Good exploratory testers apply structure to their work and use tools to collect data so that their testing sessions are more meaningful than just bug finding exercises. Exploratory testing should be connected to the rest of the testing process and not just a late cycle bug finding activity.

In this month’s Testing the Limits interview, we’ll put Patrick Copeland on the hot seat. Patrick is the Senior Engineering Director for a promising young upstart named Google (we’re not familiar with them ourselves, but we’ve heard good things) where he oversees a global team of about 800 engineers. But this isn’t his first rodeo –  prior to Google, Patrick spent a decade at Microsoft, where he specialized in all things related to software engineering.

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

• Strategic: Maybe a new feature, but in a market with existing competition (like Android).
• Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
• Customer usage: For example, popular high-traffic applications like GMail.
• Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
• Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

By the way, what I meant is that the surface area of the challenges we face in software has grown exponentially over time. Just in the past few years we’ve witnessed a paradigm shift to cloud computing that stretches our imagination and challenges the limits of software. Our process for developing that software is going through an equally dramatic revolution. We are reconsidering the appropriateness of the lessons we’ve taken from traditional software development companies. Testing teams need to continue to evolve quickly and innovate just keep up with what’s happening in the industry.

uTest: Not too long ago, you added testing guru and author, James Whittaker to the Google team, who is well-known to the uTest community. Why was he such an important addition, and has there been anything that’s surprised you since he came on board?

PC: James has been a great addition to the team. His passion for testing is incredible. The biggest surprise is that he’s having fun being a manager. James accepted our offer without knowing exactly what he was going to do. This is a pretty common situation for new senior people at Google. I remember giving him our offer and he said repeatedly, “I’m not interested in being a manager.” I kept saying, “ok, ok, no manager position for you.” But he’s decided to run the Kirkland and Seattle Test Teams, in addition to several groups in Mountain View. He loves it. I think that might even be a surprise even to him.

uTest: Google seems to break into another market every week, with the new Nexus One phone being the most recent example. How much of your time is spent looking forward to new apps and categories, as opposed to concentrating on existing products?

PC: At Google we have a 70-20-10 rule for investing in technology. While the numbers are not carved in stone, the basic concept is that we invest ~70% on our existing core products, ~20% in related but new categories and ~10% on riskier bets or areas outside our existing core competencies.

Innovation is a high order bit of course. We’re having to do a lot of innovation on supportive infrastructure to keep up with the pace. For instance, we’ve created an infrastructure that allows us to innovate very rapidly while maintaining high quality despite a huge and complex code base. On any given day, there are thousands of projects under development with over 5000 CPU cores producing 65K compilations of build targets, and over 100M executed test suites. The build and test system accommodates the load, and an annual growth rate of 75%, using distributed execution, caching, and work avoidance techniques. The average build and test cycle take just 4 minutes and we are quickly moving toward a model where developers can receive results before they formally check-in code changes (‘pre-submit checking’).

In our model, product teams maintain a build that’s always green (compiling and testing without error). We built the supporting system based on three priorities:

• Speed: All test and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data.
• Feedback: The focus of test systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not fixing code that is broken.
• Simplicity: Engineers should not have to understand how the underlying systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workflow that allows them to take appropriate action.

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

PC: We have a pretty open culture where engineers have a good degree of freedom to invest in areas that interest them. We have an idea called “20% time” where engineers are encourage to invest a portion of their time in projects outside of their primary area.

Does culture help to make better software? I think so, but it’s a tough thing to quantify. On occasion the idea of collecting and measuring “productivity metrics” has come up (ex. lines of code, # of check-ins). We uniformly resist this because similar metrics can result in undesirable outcomes as people try to “succeed” in the system. Our biggest measure of performance, besides the velocity of innovation being delivered to customers, is quarterly peer reviews. This type of peer system reinforces ideas like teamwork, making sure you are having an impact, and building respect from peers. It’s more subjective, but harder to “game” and, ultimately, much more valuable in reflecting the real value and contribution of an individual.

BTW, “Food-pellets” wouldn’t work because our engineers already have free access to gourmet food.

uTest: What’s the difference between a good tester and a great tester?

PC: Good testers can be trained. Among many others, the basics you need to be good are: depth in computer fundamentals, an understanding of the application domain, an strong understanding of the use case, empathy of the user, mastery of metrics, and a focus on the development process.

Great testers, meaning the 90th percentile, are rare and special. Not everyone can be truly great. In my experience great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. From the 100s of interviews I’ve done, “great” boils down to: 1) a special predisposition to finding problems and 2) a passion for testing to go along with that predisposition. In other words, they love testing and they are good at it. They also appreciate that the challenges of testing are, more often than not, equal or greater than the challenges of programming. A great “career” tester with the testing gene and the right attitude will always be able to find a job. They are gold.

uTest: What’s the greatest barrier to designing, developing and launching high-quality products on a consistent basis (deadlines, prioritizing, competing visions, etc)? Said different, why can’t every company launch world-class apps every time?

PC: Every product I’ve worked on required a unique approach. There are so many variables in play and most of them are situational. Some of them you can control almost completely (e.g. technology choices). Many others you can control somewhat. Others still are completely outside of your control (ex. what the competition decides to do).

Some companies try to normalize their development process. One thread common to formal development models are that they focus on a few of the many variables: improving efficiency, predictable process, estimation of quality, or others. Process-heavy development models may work well for manufacturing airplanes and have been successfully applied by some companies, they have been viewed by many developers as burdensome and contrary to the creative nature of writing innovative software. Conversely, “process-less process”, can lead to a heroic culture that’s unable to repeatedly deliver. There needs to be balance.

Consider the physics of flight as an analogy to software process. In addition to reasonable flying conditions and an experienced pilot, the key to getting airborne is having an appropriate balance of factors that match the situation: too much weight or too little thrust can be disastrous depending on the situation. Similarly, teams, products and process all have virtual physics. For instance, adding engineers late in a product cycle doesn’t necessarily provide more lift. Adopting a new process may give a team more thrust momentarily, but may also incur a longer term drag that makes them incapable of innovation.

The popularity of Agile, while not a wholesale rejection of more rigid processes, indicates that developers desire more balance and creativity. Whatever we do to make software higher quality and more predictable to build, we must maintain a balance that encourages the innovative aspects of the art form. We need to motivate smart minds to solve hard problems and deliver rich features to customers. In other words, we need to be adaptable to stay airborne for the long term.

uTest: What advice would you give to young software turks who aspire to become a senior tech exec at a global powerhouse?

PC: Here’s my advice:

1. Be technical: At Google all of our engineering execs have very strong technical backgrounds. They were programmers and many of them can – and do – still program today. Advice: remember your computer science and stay sharp.
2. Be a connector and innovator: One exec commented that his job could be described as being “a very expensive email router.” Get connecting within your company and establish a network of strong peers.
3. Drive your career: Don’t wait for someone to recognize you. Inform people about your plans and successes. You are your own best billboard. People will create a myth about you, like it or not. If you are smart you’ll help them create the myth you want them to have.
4. Play to your strengths: Pick projects that you are passionate about and where you can rock worlds.
5. Bite the hand that feeds you (a little): Don’t be a jerk, but make sure you don’t go passively along with bad ideas. A weak willed tester is a bad thing. Make your concerns known…and know when to fold.
6. Go where there’s opportunity for growth: Not all projects or companies can offer you an ability to grow. Move when you think you are topping out. The biggest mistake I’ve seen is staying too long in a place that hinders growth.

uTest: Google is booming in the mobile apps space; what QA testing challenges do mobile apps present that web or desktop apps do not?

PC: Where do I begin?

A good starting place is the huge diversity of devices. For instance, significant variations in HW and SW: OS, memory, processors, screen size, and input methods. Just having access to a fraction of the devices on the market for testing purposes is a challenge.

On top of that, if you want any automation, you have the problem of simulating input and capturing the actual output on the device. We needed to develop tools that allow up to test and build against multiple platforms. Normalizing the testing is tough. We debated record and playback vs writing abstraction layers vs other methods. All can be brittle in the extreme. We also needed to think about the long term cost of maintaining tests suites on fast moving SUTs.

A couple other issues are the carrier networks and the numerous languages. As I said, I could go on for a long time, but I believe I hit some of the major pain points. Even with all of the automation in the world, a community based test approach can be complimentary.

uTest: Is there a particular programming language to which you are loyal (PHP, Java, Ruby, etc)?

PC: Nope. Not really. At Google the most common ones are Java, JavaScript, Python, Perl, and C++. Perhaps a personal failing, but I still think in C and my code ends up looking kind of C-ish even if it’s in another language. My preference also depends on the thing I’m trying to do, obviously each has strengths and weaknesses.

On a similar topic, I recently met famed computer language innovator, Niklaus Wirth at the last GTAC conference in Zurich. He had in interesting comment on testing and quality, “The experience, judgment, and intuition of programmers who have survived the rigors of testing are what make programs of the present day useful, efficient, and correct.” In other words, you can’t test quality in and quality comes from experience.

uTest: Any new year’s resolutions for Google’s testing & QA efforts?

PC: Yes. Every year I write up a “one pager” describing what I think we need to do. I try to stay focused on general productivity and not specifically testing. Testing is one tool among many we use to make software high quality and delivery faster. We are doing well on many levels, but we aspire to do even better. Here are a few of the high level bullets:

• Greater release velocity. We apply appropriate use of automation, tools, process, and information that solve Google challenges. We introduce and use metrics that fit naturally into the product cycle. We are introducing key metrics that are actionable and representative of the health of the project that allow team members to understand issues and take action quickly.
• Cohesive tools. We’ve built an impressive set of tools that have made significant impact. In our next stage, we need to better integrate some of the data and tools and focus on accelerating the end to end developer workflow. This includes identifying bottle necks that slow people down and keeping an eye on innovative ideas coming from product teams. And of course, continual improvement of build and test speed.
• Driving improvement upstream. We will take advantage of our ability to make good ideas useful across the company. We use our broad view of projects to push adoption of ideas and methodologies that work. We look for ways to compliment existing product strategy by identifying changes where the adoption is in a project’s self interest. We constantly evaluate our own practices by looking at their effectiveness in relation to customer and business needs.

uTest: You went to school at University of Arizona, and USC; you’ve worked at Microsoft and Google; are you a lifetime west coaster or has it just worked out that way? Also, Dodgers or Giants (or D-backs)?

PC: Are there any good East Coast teams? I heard it was so foggy the other day that the Washington Nationals couldn’t even see who was beating them.

They may not be sports teams, but I’m a fan of the testing teams we have on the east coast: NYC “Bubble Sorts”, Boston “Red SOX”, and Pittsburgh “Open Sourcers”. BTW, we just hired a Test Director on the East Coast named John Turek who’s looking for a few good players.

BTW, I actually have lived in other places. My family lived in Maryland and Tallahassee for a bit. I also lived in Madagascar, which is definably on the East Coast (of Africa).

uTest: What sites, blogs or magazines do you read to stay ahead of the curve in the areas of business, technology, etc?

PC: Track and Field, Cosmo and, occasionally, Monster Truck Weekly . Seriously, I have a bunch of custom news feeds on certain topics, like Google, our competitors, and testing. I also read wsj on line and at least one interesting industry technical paper per quarter.

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

JB: Years ago, Rational hired me to advise them about test processes, and for that reason I took the Rational Unified Process class. I had been working with the RUP team, and I wanted to see what the Rational instructor was saying about RUP. At lunch on the first day, I asked him if it was okay that I was asking challenging questions. He said that was fine. But at lunch on the second day he kicked me out of the class!

Most instructors don’t like me in their classes. So, when you say I’m forced to choose, it’s a question of which class I would prefer to disrupt. (Unless I have the choice of sitting in one of Cem Kaner’s classes at Florida Tech. I would definitely prefer that. Last time I took a class from Cem, I had to play solitaire furiously in the back of the room. It was the only way to stop myself from jumping up and shouting “Yes!” every three minutes.)

I would prefer to disrupt an ISTQB class.

uTest: A few months back you presented “How to Fake a Test Project” at STPCon, where you said “testers may accidentally find bugs because they don’t follow the test scripts precisely.” Are testing managers starting to catch on to this irony of such “best practices”, or are most of them still oblivious to this?

I don’t know. I think most test managers don’t think much about it. As I said in my talk, testing is easy to fake– *even by accident*– because testing is so intangible.

JB: Be honest, you’ve faked a few test projects in your day, right? C’mon you can tell us.

Yes. I remember walking over to my manager’s office at Borland– by this time I’d been doing test management for five years– carrying an estimate of how much testing was required for my project. The estimate was couched partly in terms of “test cases required.” But just as I reached his office, I realized that the numbers meant nothing.

Until that moment, I thought they meant something. Well, they did mean something, just not anything I could defend or explain. Suddenly it dawned on me that I was peddling an empty *story* about testing couched in numbers to make it look rational.

As I write this, that incident happened 17 years, 5 months, and 1 day ago. Since advising my boss that he should ignore my test case estimates at that meeting, I have never used test case counts as a metric on a test project. But that doesn’t explain why, for the 5 years, 1 month, and 21 days before that, I *did* use test case counts.

Of course, the explanation is that I hadn’t much thought about it. I went along with the convention. So, I can forgive people who fake test projects by accident. When we’re young we make mistakes. But we also learn and grow.

uTest: In your new book, Secrets of a Buccaneer-Scholar, you advocate alternatives to traditional schooling and certification – not just for testers, but for people in all professions. It seems to have worked well for you, but why are so many people still reluctant to ignore the certificates and degrees and just start testing? What’s holding them back?

JB: Fear and indoctrination. I recently argued with a fellow about why he was going through a certificate program that bored him and insulted him, instead of quitting that school and being a tester. He claimed that no one would give him a job without the certification. Well, he emailed me a few days ago and told me he just got a job, and that I was right.

Well of course! I’ve been a hiring manager, and the idea that hiring managers are zombies who only look at paper credentials is wrong. SOME managers do that, but you don’t really want to work for someone like, do you?

Seems to me uTest is a mechanism for building your reputation as a rapid tester. I encourage new testers to try it for exactly that reason.

uTest: What’s the most promising trend/development in testing?  And what’s the most dangerous or discouraging trend you’ve seen?

JB: I think the most promising trend is public peer conferences and mentoring. Things like the Weekend Testers, for instance, or the Rat Pack. Some of these people are inspired by guys like me, perhaps, but they are not followers. They are taking it upon themselves to re-invent testing by experiencing and examining it intensively. I’m proud of anything I have done or can do to encourage this sort of creativity.

The most dangerous trend is certification, which is systematically dumbing down our craft and preventing good people from getting work.

A more dangerous thing, which is not a trend but unfortunately a constant, is the astonishing incompetence of technology middle and upper management, who frequently demand stupid metrics and meaningless reports from the people who work for them, or who decide to outsource without having the first clue what they are giving up by dumping knowledgeable testers and sending critical work ten time zones away.

For some reason, test managers complain to me about this, but often don’t complain to their own bosses.

Last year, my brother created a world-class test team at LexisNexis, then someone many levels up from him decided to fire everyone and outsource. They never talked to my brother about this. Apparently they think good testing happens by magic. But here’s the thing, my brother decided that he was going to train the outsourcing company (who had claimed to know a lot about testing, but, surprise, didn’t actually know much at all) to do skilled exploratory testing. It worked. He’s pleased. And now his management at LexisNexis (former management, because they then fired him) must think outsourcing is a great strategy, having no idea that it works only because people like my brother have too much personal pride to let even senior management airheads experience a richly deserved failure.

uTest: We’re all familiar with your work, as well as that of your colleagues like Michael Bolton, Cem Kaner, Jon Bach and others. Are there any “great unknowns” out there in the testing blogosphere to whom we should be paying attention?  Anyone that should be ignored altogether as a dangerous thinker?

JB: I sometimes complain about people I think are dangerous. See my blog.

Basically, I hate bullies. I want a free and open craft. The most dangerous people are those whose personal ambition will mean discouraging innovation (perhaps through establishing some sort of baloney “testing standard” through the auspices of ISO), and encouraging the sort of test management practices that make software both expensive and sucky, while discouraging any smart ambitious person from entering the field in the first place.

As far as great unknowns, there are some. I think Lanette Creamer is someone to watch. Adam White, Ajay Balamurugadas, and Ben Yaroch, as well. There are the Weekend Tester guys, and a shadowy Context-Driven sub-community calling itself “The Rat Pack.” I come across testers, regularly, who are talented, but haven’t taken a step out into public discourse.

I am constantly recruiting for colleagues.

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

uTest: When there are delays in development process, why does testing always seem to take the blame? Is their role just misunderstood? Or is it really the testing team’s fault?  How can companies avoid this seemingly never-ending spiral?

JB: Often, we’re saved until last.  It’s like the novel is written, the movie is produced, but the reviews aren’t in yet, leaving the creators in a heightened state of worry and concern. It’s a lot of stress to be in that position. As testers shine the light on the product, looking for risks and vulnerabilities, the light is really shining on *us*.  We’re being tested just as much as that software, so we tend to be sensitive and aware of being in a critical position as others wait for our findings.

Not being an Agile guy, I’m finding that the values and principles of Scrum, Agile, Lean, and XP may be compelling enough to try.  I’m getting more into that domain and I see merit in the values and principles it is trying to instill.  One of my favorite colleagues (Elisabeth Hendrickson) is coaching me in a gentle way and her stories are enlightening because she lives this stuff. Like me, she used to focus on exploratory testing, but catching up with her recently, she seems to be the whole package — developer, tester, manager, coach — and seems to be immune from that never-ending blame spiral you asked about.  That’s compelling to me.  Some may call her one of those ardent “Agilista” types, but to me, she hasn’t lost her testing spirit or soul.  Though she may agree there’s no silver bullet, she has some great experiences that convince me that with those approaches, the blame spiral is being made irrelevant because of the way testers are more involved, earlier.

uTest: In Half-Baked Ideas for Rapid Test Management (PDF) you used a term called “Agile-fall” (a combination of Agile and waterfall). This seems to be the methodology that most companies follow, yet they always call themselves agile. Is there any shame in being Agile-Fall? And did you coin this term? We couldn’t find it anywhere else.

JB: “Agile-fall” was something I heard at LexisNexis from an awesome PM named Lance Thomas, but in a Google talk in 2005, Elisabeth Hendrickson called it “Scrumfall”, so search on that term and you’ll see that it refers to having the principles associated with Agile (daily stand-ups, sprints, burndown charts, etc.), done in a waterfall-y series of development steps.  Example: Sprint 1: Gather requirements, Sprint 2: Design your tests, Sprint 3, Run those tests, Sprint 4, Fix bugs, Sprint 5 regress those bugs.  There’s no shame in that if that’s what works, and when you’re going through a transition from Waterfall to Agile, that may be the best thing as opposed to a sudden lever-pull one day where you show up and your desk is next to someone else with no walls and there’s a stack of sticky notes and markers on your chair with an email to report to your first standup in 30-minutes.

uTest: It looks like software bugs are partly to blame for the recent Toyota recall debacle. Is this the worst nightmare for testing managers? What else keeps them up at night?

JB: I almost got a Toyota last month, and wish I had, just so I could find a cool defect.  But my nightmares in testing are: “Why didn’t you catch that bug?” Though I know several answers to that, and my favorite colleague Michael Bolton has a rich list of answers to that question, they don’t often overcome the strong emotional attachment that stakeholders have.  And I can’t blame them, really.  The more we call ourselves “Quality Assurance” (like we can guarantee quality), the more they will lean on us to assure them.  I can’t.  But I can certainly *assist* with the notion of  whether something has value to its intended customer.

What keeps me awake is how to know if I’m bringing the right value to the project.  There are a million things I could do, but which ones have the most value right now?  I’m worried I would stumble into an idea too late, or not at all.  That’s why I like heuristics and mnemonics and checklists to kick my mind into gear when I feel stuck or worried.

uTest: What’s the biggest difference between working for a large, mature enterprise and a small young startup (from a testing point of view, that is)?

JB: I’ve worked at both.  At Quardev, which is neither startup or enterprise, we’ve got a mature-but-startup mentality.  Since the word “Quardev” comes from an amalgam of QUality Assistance, Research, and DEVelopment, the balance of those gives us enough diverse opportunities to make it feel like a big company.

From a testing point-of view, test ideas are king no matter what size your company. Some may say the toolset is king, or dev skills are king, but when I’ve worked for start-ups, you don’t need as many signatures on things, and that helps creativity. You might fail faster with those test ideas because you have to do it cheaply, and that’s good.  It’s like projects I’ve been on that use notions of Agile compared to those that don’t.  I see those kinds of projects revealing problems faster because of the way the culture organizes to deliver working software.

At a small startup, it’s often expected that you’ll be brilliant.  That pressure can make work not-so-fun.  Maybe that’s why some of my best ideas came from working in enterprise environments (Microsoft, HP, LexisNexis). In those places, there was a perception that every test idea, method, tactic, and strategy had been tried before.  That carried an unspoken dare (in my view) that begged to be challenged, which few people took on.  For me, brilliance and innovation come when someone counts me out or doesn’t expect it.  I like that.  I like exceeding expectations no matter how big or small the crew.

What sums it up is a TED talk by Ken Robinson, who said: “If you’re not prepared to be wrong, you’ll never come up with anything original.”

uTest: Who plays Jon Bach in the made-for-TV-movie about your career in testing? And what’s the title?

JB: John Schneider, only because it has to be called “The Duke of Hazards.” It also fits because I told my friends in school that I was related to Cousin Daisy (played by Catherine Bach). Not true, but it helped keep the bullies away.

uTest: You have some great tips on how to handle bloated testing numbers and statistics: “Any number, any statistic is like software. It can be tested.” What other tips can you give testers when it comes to having the courage, diplomacy and patience to slow things down and get to the truth?

JB:  For me, the magic words that often make me feel more courageous, diplomatic and patient are: “I have been fooled before.”

No one will argue with that because it’s true.  Scammers often confess that the hardest person to fool is somebody who says “I can be fooled.”  So many times I’ve been so sure I was right just to meet someone who convinced me differently, sometimes in a matter of seconds.  So now say “I could be wrong”, and use other safety language like: “it could be”, “it seems like”, “it looks as if” and “maybe…”  That way, I don’t feel stupid when I’m shown refuting evidence to my claim.  If you practice that, chances are good that you will appear to be a kung fu master who, after having floored your 50th assailant with your skills, slowly backs out of the room on guard for the 51st.

Remember that testing is a craft.  It involves thinking about how things might be different.  Remembering to say “I have been fooled before” is consistent with that spirit.

uTest:  Testing certs: worthwhile or window dressing?

JB:  The only thing worthwhile about them is the debate they provoke.  Window dressing is an apt metaphor because it’s only meant to enhance a window’s *appearance*.  When there’s a flood or a storm or some other strong test of the window, the dressing often gets destroyed. Outside of the flood, people may prefer the look of the dressing; I just want to be a stronger window.  Passing multiple choice tests about so-called “best practices” don’t do that for me.

uTest: What are some of the blogs/sites that you read to stay on top of the latest QA trends and news?

JB: I’m mostly on Twitter because it leads to links to testing videos, newsletters, news stories, trivia, thoughts, questions, blogs and tools that I haven’t seen before.  For example, yesterday I learned about Scott Berkun and his excellent technology blog, Ken Robinson’s fantastic TED talk on rethinking education, the interesting notion of a testing “playbook”, and the Ignite Conferences (a series of 5-minute talks).  Thank you, tweeps!

The top 10 I follow are: James, Jim Benson, Lanette Creamer, Michael Bolton, Scott Hanselman, Elisabeth Hendrickson, Adam Goucher, Paul Boal, Steve Smith, and yes, I have to say, uTest.   All of these sources turned me on to such a wide variety of sources to edify myself that it’s no longer just a handful of blogs to read now and then.  Blogwise, I have to say James (of course), Scott Berkun, Jim Benson, Joel Spolsky, and Lanette Creamer are most worth my time.

uTest: Testers come from a wide range of backgrounds (and have a wide range of skills) yet they are often talked about as if they were all the same. In your experience, what’s the biggest misconception or stereotype of testers?

JB: That we like to “break stuff.”

Nah, we’re detectives… we FIND stuff.  It’s a treasure hunt, not a smash factory.  In fact, our reports are “findings”, like a scientist or an investigator would call them.

One of the first lessons in my career was my favorite — that software comes to testers *already broken*.  I like that.  I didn’t write the code that broke it (unless I did, like a unit test or a keyword-driven script).

Humans are tested the same way.  When we go on a job interview, the interviewer doesn’t “break” us, they find weaknesses and vulnerabilities that are already there in our programming.  The good news is that we can test ourselves before going into the interview — to find and address issues beforehand so we can show them our best value.

uTest: What qualities, characteristics and experiences do you look for when hiring testers?

JB: At Quardev as a hiring test manager, it is these: cautious, critical, curious, friendly, diplomatic, honest, insightful, thoughtful.  I want candidates to tell me about a cool bug they found, or give me their best test idea.  I want them to make me think. I want them to inspire me and make *me* curious*, even though the thing I give them to test when they come in has been tested by over 500 different testers over the last 10 years.  I’m not naive enough to think I’ve seen it all, because one in 20 testers will find something new or will put ideas together in ways I hadn’t thought of.

uTest: What advice would you give to new testers who want to stand out from the crowd and become tomorrow’s testing leaders/gurus/pundits?

JB: Give a talk. Practice in front of a high school class. Take an idea or a strong opinion you have and actually learn more about it, then present it.  Do a lightning talk at the closest Ignite Conference to you.  Read a testing book and email the author your comments.  Comment on blogs and sign your real name.  Question the gurus and the pundits. Call them out if you think their work is crap. Talk about your failures.  Tell us your *experience*, no matter how amateur you think it may be.  Could be you just invented the next cool method to try. It was like that for me with Open-Book Testing when I was on Microsoft Flight Sim.  It was a half-baked idea I had and now it’s blossomed into something that I keep in my intellectual briefcase to show prospective clients one way to teach, guide, and evaluate test teams.

uTest: One of the hottest areas of growth that we’ve seen at uTest is in the area of mobile app testing.  What unique challenges does mobile present to a testing manager?

JB: Finding the right minutes/text plan?  Those really add up when you’re testing!

Actually, I think uTest is in a great position to address the biggest challenge I would face if I was managing a mobile project right now – configuration testing.  I would love to have 1,000 testers at my disposal doing a variety of risk-based exploratory charters on their devices with all kinds of third-party apps installed, making calls on different service plans, texting, taking pictures, playing video, Tweeting, browsing, and maybe doing all of those actions at once from places around the world.

By the way, I read Patrick Copeland’s answer when you asked him the same question and really think he summed up the challenges – diversity of devices, input-output simulations, carrier networks — things that you’d have to be very technical, very savvy, and very organized to test.  But on top of all that, there’s the challenge of if you actually found a bug.  With all of those variables I mentioned, where is the real bug?  You may be just reporting the failure, so you’d have to have some good debugging and diagnostic tools to help you be confident that your concern was represented to those who could (or would) take action on it.

uTest:  If Al Gore hadn’t invented the interwebs, what would you be doing today (answers related to software or technology not allowed!)?

JB: I’d likely be a best-selling author, have a full head of hair, and would have retired at 18 because I would have invented Twitter in 1986 instead of messing around on the Apple II for hours on end.  Before Twitter, before Windows, before the Apple II, our family had the “chain letter” – a packet of letters sent from one member of my family to another, adding new content with each mailing until all 6 of us had caught up with each other’s news in the circle.  When the oldest person got it, they would add new content and mail it to the youngest, and round it would go again, replacing old content with new.  It was a great system and I would have turned that into Twitter as a system of snail-mailed 3×5 cards to everyone on the planet.

uTest: Rapid-fire Jon Bach pop trivia – Last book read?  Last movie read?  Last concert attended?  Favorite sport and team?  Favorite band or album?  Browser of choice?  What kind of cell phone are you carrying?  Blu-ray or DVD?  Paper or plastic?

JB: Last book read? You mean cover-to-cover, not skimming?  Ugh.  Oh!  Last night to my three year old: “How do Dinosaurs Say Good Night?”, after which, I dove back into “Extreme Programming Explored.” Actually, I think I was over-tired last night and got it backwards, which is why I suspect she woke up this morning talking about refactoring, code smells, and collective code ownership and why I dreamt about being a Gallimimus.

Last movie read? The screenplay for “Butch Cassidy and the Sundance Kid” by Goldman.

Last movie seen: the unfortunate and annoying “Where the Wild Things Are”

Concert: Sting, 1989 in Portland, Maine;

Sport: Football (Seattle Seahawks);

Album: Rush, Subdivisions;

Browser: Firefox;

cell: iPhone 3G;

DVD;

Plastic, because at least they spice up the boring look of those landfills!

Our Testing the Limits guest this month is testing guru Scott Barber, the Chief Technologist of PerfTestPlus. A speaker, writer, teacher and entrepreneur, Scott has one of the most impressive resumes in the business, particularly in the realm of customized testing methodologies, embedded systems testing, personal security systems and other topics – all of which are discussed on his blog.

In Part I of our 3-part interview, Scott discusses the Manifesto for Agile Development (almost ten years after it was created); the expectations of today’s testing managers; the notion of testers as an “unfortunate necessity”, the 1983 War Games movie and more.

uTest: As a signatory on the Manifesto for Agile Development, can you comment on the progress being made by software companies in upholding these principles? Have they exceeded your expectations, or is there still a long way to go?

SB: Honestly, I think the buzz around the “Agile movement” has, in many cases, taken the industry in an unfortunate direction. I meet far too many people and companies who are completely unfamiliar with the Agile Manifesto and think of Agile as a collection of practices, processes, and tools. The reality is that Agile is a far more a mindset and a culture than it is a collection of practices, processes and tools. Agile isn’t the best fit for every situation, or for every person.

I believe that the trend to “go Agile” is misguided. If a company is developing good software, the people involved in developing that software are happy working there, the software development is sustainable, and the business is being adequately served by that software, there’s really no need for them to try to be more or less Agile. Agile has challenges like any other culture, but the single biggest challenge I find is companies trying to solve development, process, management, and/or schedule problems by “going Agile.” Teams who have grown up in a culture that is fundamentally different than Agile simply will not find it easy to “go Agile.”

Think of it this way. Telling a team to “go Agile” between projects is like telling a professional baseball player to switch from being a right-handed hitter to being a left-handed hitter between seasons – and expecting that to break their hitting slump. You simply don’t see that. What you do see is batters working with coaches to re-evaluate their swing, work on their timing, spend more time in the weight room, etc.

Personally, I am happiest when I’m working in an Agile culture, but that is a personal preference, nothing more. What I do think would serve individuals and teams well, would be to read the Agile Manifesto and really think about whether or not those principles are a good fit for themselves, their team, and/or their organization. If so, I recommend embracing them.  If not, I recommend finding or developing an equivalent set of principles to build a culture around.

uTest: In a recent article, you highlight the critical point that “the perceived value of testing varies widely from individual to individual, team to team and company to company.” Can you further elaborate on how testing managers can do a better job of setting clear expectations?

SB: Consider these scenarios:

• Company A executives are looking for the testing to be their defense in case of failure and lawsuits
• Company B project managers want testing to say “ship it” so that if there are problems in production, they don’t get in trouble
• Company C developers want testing to show them the issues before management notices them and mandates changes to the development process… yet again.
• Company D is developing software for a client who expects testing to ensure their needs are met
• Company E just spent millions on a marketing campaign and is counting on testing to ensure that the company doesn’t get egg on its face when the campaign is released.

In each of these cases, the test manager is certainly the person who needs to lead the test team to adding the most value to the project. The challenge is that the test manager is often in the worst position to get straight answers.  Test managers are typically the lowest-level role in a company to hold the title “manager.” They don’t generally get to sit in the board meetings where defense against lawsuits are discussed, they don’t frequently have direct contact with the client, developers don’t often treat test managers as confidants, and who would tell a test manager that the purpose of his/her team is to take the blame if something goes wrong, even if it is completely out of their control?

How test managers, and testers, can do a better job is to take a step back and really consider what is going on in the company, what instructions they are being given, what questions they are being asked, and who seems happy or grumpy with the test results being provided.  Paying attention things like those will frequently make the “real story” pretty clear, and once the story starts to become clear, you can start asking smart questions that actually lead to pretty straight answers.

What doesn’t work is assuming that because you are the test manager, or the tester, that you *know* how your testing adds value to the company, that you *know* what the most important testing is, and that you *know* the managers and executives are idiots because of what they are telling you to test.  The fact is that even if your managers and executives are idiots, that doesn’t mean that they don’t have a sound and reasonable business reason to want what they are asking for. It’s not up to us to judge them for either being idiots or for making clumsy requests that don’t actually help them resolve their concern. It is up to us, however, to help them figure out what they actually need, help them figure out how to meet that need, and then guide our testing to best serve that need.

uTest: You have an impressive knack for drawing testing analogies from your lessons learned as an officer in the U.S. Army. What is the true value of outlining the situation, mission and intent of a software development/testing project?

SB: Thanks.  It’s not just the Army; it’s my life experience in general.  I suspect that has more to do with the fact that my parents were both educators and that I didn’t even know there was such a thing as software testing until I got hired to be a “Performance Engineer” and was introduced to the software test team on my way to my first client. So I pretty much had to figure it all out for myself with no real training in testing.

In this case, I was trying to figure out what my role really was. I clearly was not the decision-maker about whether or not to ship the product. I didn’t have the authority to tell the developers what should be fixed. So I applied the model I knew and asked myself “What is the situation here?”, “What is my mission?”, and “What is the overall intent of the client for the project as a whole?”

Over time, what I found was that the situation question helped me to understand things like where testing fit in the scheme of the project; who is panicked over what; how long I have to complete my tasks with what resources; who my allies are; what things are likely to block my success – that sort of thing. Much later, I learned that many testers refer to this as “context”, which I think is a very good word for it.

The mission question helped me figure out what was expected of me.  As it turned out, the folks I was reporting to often didn’t really know how to respond to my question of “What is my mission?”  They’d say things like “Well, test it and give me the results, of course.” My follow-up question of “Test it for what?” and “What kind of results?” didn’t help to clarify things either. So I learned to actually draft a little mission statement, like “Provide as much information as I can to management about issues I believe will cause lost clients or calls to tech support.” Sometimes someone would get very upset and change my mission statement, but that was fine by me because then I knew what their expectations were.

The intent question is what helped me stay focused on what really mattered.  The fact is that if the business we are working for fails, it doesn’t really matter how well tested a piece of software is.  No business pays testers for the sake of paying testers.  The truth is that most organizations feel that testers are an unfortunate necessity.  That truth makes it all the more important that our work add at least equal value to the business, as it costs the business to have us there in the first place.

It is at least as important as it is to have the situation, mission, and intent information individually, to look at those three items collectively. Often when I look at these three items collectively, what I find is that those three items contradict or work against one another. That is typically a good indication that either I have gotten things very confused, or it’s time for some stakeholders to have a meeting to figure out what is *really* important.

uTest: To our knowledge, no one has produced a movie entirely about software testers. However, there have been some notable software bugs in motion pictures (HAL from 2001: A Space Odyssey, the “Doomsday Machine” from Dr. Strangelove, the compound interest bug from Office Space, et al). If you had to suggest a movie that would inspire testers, what would it be and why? There are no wrong answers, except for The Net starring Sandra Bullock and Rocky 5.

SB: WarGames, 1983. All I have to say, is if you’ve built a computer capable of running WWIII simulations, and that is the same computer that controls the launch commands for nuclear warheads, and that computer is accessible via brute force hack of dialing phone numbers over a modem, you either didn’t have any testers or you didn’t listen to them.

uTest: There’s a lot of consensus among the Context-Driven School of Software Testing, but are there any fundamental differences? You can’t agree with guys like Michael Bolton and James Bach on everything, can you? Speak freely, your secrets are safe with us!

SB: I don’t know that there *is* a lot of consensus. I know that I am very much Context-Driven. In fact, my license plate says “CONTEXT”. That said, the buzz and counter-buzz surrounding the Context-Driven School of Software Testing is still causing as much harm as good.

I think the underlying problem is that the creation of software is widely treated as a software development activity, which may or may not benefit from related activities like testing. I find this thought process just plain silly. Software development, in my experience, is done for one of four reasons; altruism (minimal), education (some), research and development (slightly more than some), and/or financial/business advantage reasons (most). I think the fact that software is created for different reasons and purposes makes software development at least context-dependent by definition. The difference between being context-dependent and being context-driven is fundamentally the difference between being reactive vs. being proactive.

I think there never would have been a need to call out or name the Context-Driven School of Software Testing if testing hadn’t been so undervalued, misunderstood, and too often ignored entirely for so long that some entrepreneurial folks decided to step up and start talking about “the right way” to test software. While I’ll be one of the first to admit that many of these folks published “right ways” that were far superior to the state of the practice of the time, none of these “right ways” were right for everyone or every situation.

So, in an attempt to help people and organizations be aware that there are many ways to approach testing, and that the way that is “best” depends on a wide variety of factors, none of which involve which entrepreneurial person has the best “sales pitch” for their favorite approach.

Regardless of what the buzz may say, context-driven isn’t about Exploratory Testing, isn’t against test scripts, and doesn’t dictate who do what testing when. The way I talk about being driven by context is to regularly stop and ask yourself, “What can I do, right now that will be of most value to my team/project/company?” If you make decisions that way, instead of simply doing what someone who knows nothing about your situation recommends, I’d say that is at least the first step to being driven by context.

uTest: We’re always interested in how people got involved in software testing. What made you choose this path – what lit the spark?

SB:  The short story is that I didn’t choose software testing, it chose me.

I was dabbling in various technology and software related jobs after my time in the Army, when the company I was working for got bought by a big company who changed our compensation plan to something I couldn’t live with. While complaining to my friend, he said “Forget them, come work for us, we need a Performance Test Engineer.” In response, I asked “What’s that?”  He said “Don’t worry, you’ll like it.”

I started the job a few weeks later, and he was right, I did like it. I liked it so much, in fact, that I’m still happy being a tester over 10 years later. And to think, when I started that job I wasn’t even aware there were people who tested software for a living.

uTest: Follow-up question:  If you weren’t in testing, or anything related to software, what would you be doing right now?

SB: That’s hard to say. I went to college to become a Civil Engineer, and while I did very well in school and love the field, I was already too far removed from it by the time I left the Army to have a viable path back in. I’ve considered teaching secondary school or at a university, but there are aspects to those careers that I don’t think I would end up being very good at or enjoying. I’ve considered becoming the editor for a technical magazine, and becoming a conference coordinator for a technical conference. I’ve even considered going to work on the administrative side of a non-profit whose work inspires me.

My friends who know me well are always telling me that I should “retire into” sports announcing and/or coaching. I don’t know if that’s even vaguely possible, but I do know that there is nothing I seem to enjoy doing more than coaching and impressing people sitting around me by constantly pre-empting sports casters while watching a game.

uTest: Say we choose a software testing project at random. You don’t anything about it, except for the fact that there’s a major bottleneck somewhere in the process that is preventing quality software from getting released on-schedule. If you had to guess, what would the source of that bottleneck most likely be in a typical 2010 company? Management? Testers? Technology? Metrics? Give us your best educated guess and why.

SB: I believe it was in his book Secrets of Consulting that Jerry Weinburg stated “Every problem is a people problem.” Throughout my career, I have found this to be a great place to start. People make decisions, people devise and implement process, people write and test code, and people manage projects. If there is something preventing software from being released on time, it’s because someone made a mistake, a poor decision, an inaccurate estimate, or simply isn’t doing their job. It may or may not be relevant to figure out who did what that lead to the problem, but if you trace a problem far enough, eventually you end up at a person. The way I see it, people problems fall under management.

Add to this that quality education for managers and executives overseeing software development is at least as sparse as quality education for testers, and it makes “Management” a pretty high percentage place to start.  Of course, it’s also the most difficult to do anything about, so frequently organizations try to address their software development challenges by attacking symptoms rather than the cause.

uTest: Often times, testers will report bugs they deem “critical” (or high priority, P1, or whichever vernacular you choose) that are subsequently rejected by the client, product manager or dev lead as being ‘out of scope’ or ‘works as designed.’ What advice do you have for testers trying to make the case that a particular bug MUST be addressed before launch? Has this ever happened to you?

SB: It used to happen to me all the time, until one day I realized that I’m a tester, not an executive. My job is to provide information that my team, my employer, and whatever other stakeholders there are for my project want or need to know to make informed decisions. My job is to help them understand the state of the software and the possible implications of what I’ve found.

If, as a tester, you feel that a bug you have found is “extra important” to fix prior to launch, I recommend getting an audience with a decision maker and explaining the bug and its implications to items that matter to them (such as profitability, public relations, etc.) until you are convinced they can accurately represent your side of the argument to their peers and supervisors. After that you have two choices, as I see it, accept the decision that is made as being best for the business based on information that you’ll probably never have, or decide that your values and your company’s values are not in line and start looking for a job in a company with values closer to your own.

uTest: Favorite book that taught you an important testing lesson (even if the book wasn’t about testing per se)?  TV show?  Haiku?  Parable?  Off-Broadway play?  Shoot, just share any source of testing wisdom that has inspired you.

SB: General Systems Thinking by Jerry Weinberg, as I mentioned earlier. To be honest, it validated the way I have always thought about testing more than actually teaching me something new. That said, I’ve been teaching testers and other members of software development teams for almost 10 years and I can say with high degree of confidence that if all of those folks were reasonably skilled in systems thinking, software systems and their development, it would be in a much better place than it is today.

On a more pop-culture note, the television series House, M.D. is absolutely the best example of testing I’ve seen on a T.V. or movie screen. This is a show about diagnostic medicine, not software, but these people know testing. They understand the difference between positive and negative testing. They use scripted and exploratory testing. They aren’t satisfied with minimizing symptoms, they have an inner need to determine and eliminate root causes of unacceptable symptoms or behaviors. In fact, I’ve been influenced enough by the show that in one of my training classes, I have a segment titled “If Dr. House were a Performance Tester”, I’ve considered starting a blog under the same title, and I’ve seriously considered trying to popularize “Software Testing and Diagnostics Department” as a replacement for “Software Quality Assurance Department”.

uTest: It looks like you’ve hired your fair share of testers – what skills, experience and characteristics do you look for when making these critical hiring decisions?

SB: The most important things I look for in an individual are aptitude to learn whatever technologies, business information, processes, etc. that they will be expected to test – that and investigative curiosity.  Beyond that, I’m always looking to build a well-rounded and diverse testing team. I find that a team full of testers “like me” leads to us all missing the same kinds of bugs, so I’m always looking for a breadth of skills and experiences across the team. I’ve hired aircraft mechanics, medical billing specialists, call-center representatives, technical sales representatives, subject matter experts, developers… oh yeah, and even a few testers.

I know how to teach people technologies, the basics of a new programming language, how to report bugs, how to document process, and how to generate test ideas from whatever information we have.  What I don’t know how to teach people is how to be curious, the desire to investigate their curiosity, or to be excited by the thought of investigating things that strike them as odd.

So I guess I’m always on the lookout for someone with a tester’s attitude who fills a gap in perspective of the current team whose own knowledge or skill gaps fall in areas I know how to teach them or get them trained in.

uTest: Hypothetical: All testers are barred from practicing their craft for an entire year. What does the software industry – nay, the modern world – look like without their services? Please be as dramatic and hyperbolic as possible.

SB: Sadly, I think most of the world wouldn’t even notice. I think for the period of a year, non-testers would pick up enough of the slack to keep things moving, and enough individuals would perform enough “heroics” to keep the software world running. Sure, more bugs would make it into production, more software would ship late, and there would be more evening news reports of software failures, but I’m not sure the general public would even notice.

In fact, if software producers were smart, they’d pass some of the savings they were realizing by not having to pay testers on to the general public, making them even less likely to mind a few extra bugs.

But eventually, some kind of tipping point would be reached and the public would demand less buggy software. That is when the disaster would actually happen.  By the time companies re-engaged with testers, re-integrated them into their new (clearly, more streamlined, process), started getting results, realizing just how much re-work needed to be done and how long it would take to get software back up to the “old” standard, the public would be so furious that the “old” standard would no longer suffice.  Of course, achieving the new, higher, quality standard would take more time, and make the software cost more, and thus make the public demand yet *more* for their money.

Interestingly, if the software industry managed to survive for long enough to find a balance point at the tail of this downward, then upward spiral in software quality, I think it would be the best thing that could happen for testers. The publicity would lead to higher quality training for testers, more education about testing for developers and managers in school and on the job, better hiring methods and practices, and more respect for testing as a career. As the public became aware that software doesn’t have to be bug-infested (sure, all software will always have some bugs, but a couple of lady bugs are far more acceptable than a cockroach farm) more buyers will become more willing to spend a little extra for better quality, leading to more jobs for skilled testers, leading to more people wanting to become skilled testers, which would increase enrollment into education programs, leading to more research and development, and on, and on.

Maybe this isn’t exactly how it would go, but I do believe that it would take far too long for the public to notice and far too long for companies to start losing more money than they save by not paying testers. I also believe that unless something like this does happen, the public may never realize that they have been conditioned to accept far lower quality from software than they do from non-software items of equivalent cost. Think about it, how many times in the last year has a software bug ruined your day?  If the building materials used to construct your home were of the same quality of the software you use, how would you feel about your home?  What would you think if you had to patch a wall every time you patched your software?

Testers can’t solve the quality problem on our own. Higher quality has to be driven by conscientious business owners or disgruntled buyers. The best testers can do is try to make those executives as aware as we can of those bugs we can find while operating with not enough training, not enough time, not enough people, not enough equipment, and not enough support.

uTest: Are great testers born or built?  And if they’re built, then how the heck does that happen?  Share a little wisdom with the newbie testers and college kids who will join the ranks in the next year.

SB: I think that the potential to be a great tester is a part of a person’s personality that more or less exists or doesn’t by the time that person reaches college or the job market, but I don’t think that potential alone is enough to make someone a truly great tester.  Becoming a great tester is like becoming a great anything-else. It takes some natural ability, some opportunity, some good coaching, some relevant experiences to draw from, a lot of self-motivation, and a lot of practice.

Every now and again, someone might have so much of some of these items that they can get away with less of the others, but that is the exception. If you want to be a great tester, just like if you want to be a great musician, writer, or athlete, practice all you can, find a good coach or mentor, learn everything you can about technologies, designing experiments, systems, psychology, human interfaces, anything you can imagine being related to what you might end up testing or might help you better understand what might matter to the people who use what you might end up testing.

When you think you’re getting really good at something, start blogging it, then teaching it to others.  Don’t be discouraged when you find that it doesn’t really work for them at first.  Keep refining and revising until you believe those you are teaching are at least as good as you thought you were when you first started trying to teach them. When your students reach that level, pick the thing you are next best at and do the same thing.  You’ll know you’ve become a “great tester” when your current students have been referred by your former students, and your former students are trying to hire you instead of just trying to pick your brain.

uTest: [In the voice of deep, gravel-voiced movie preview guy] The year is 2020.  And in a world gone mad… where only one man knows what really happened… only one man can face the danger, beat the odds and reveal the truth… (insert pyrotechnic explosion here).  Okay, I don’t know precisely where I’m going with that, but what do you think testing look like in 2020?  What needs to change ASAP in the world of software design, development or testing in order to ensure a future of safe, reliable apps?

SB: The first thing that needs to happen is that testers, developers, managers, and testers need to learn what a “test” is.  Sounds silly, right?  It’s not.  Almost every day I encounter someone who wants to know “What test can I conduct to prove that…?”  While the answer (“There is no single test to prove that…”) it demonstrates a fundamental misunderstanding of what a test is.

Think about all the tests you have taken during your days as a student.  Do you believe that any of them proved that you had learned what the instructor had intended?  How about your driver’s test?  Do you believe that passing your driver’s test proved that you were going to be a good driver?  What about the last time you or someone you knew had a blood test?  Did you believe that blood test was going to prove the person was healthy?  Be honest now.

Of course you didn’t.  But somehow, most folks seem to think that software tests can prove that something will work in production.  The best we’re ever going to be able to do is demonstrate ways in which the software already isn’t working, demonstrate that the software doesn’t currently meet some measure of “goodness”, demonstrate that it is possible that the software may work in production, and/or collect information of potential value to someone who matters.  Even that depends on our tests being reliable and accurate.

Before testing can truly advance, folks need to recognize that testing is an application of the scientific method, that testing involves experimental design, and that tests don’t pass or fail, they provide useful information or they don’t.

Most of what happens today under the label of testing is nothing more than simple validation of expectations.  Testing is a search for information.  Testing is most valuable when expectations are not met.

Don’t get me wrong.  Someone had better be validating that expectations are being met or no one is going to want the software we’re developing, no matter how much we test it, or how few bugs are in it.  But we can’t stop there.  Technologies evolve too fast, users do things the most creative of us would never imagine before our software is retired for us to think that validation in some non-production environment is enough.

10 years ago, I heard a lot of managers and executives saying “What do you need testers for?  If you’d just develop it right in the first place, they’d have nothing to find.”  I don’t hear that very often anymore.  Today, executives understand that is an unreasonable sentiment.  Now they seem to think that testing can prove and/or control quality.  Maybe, 10 years from now, they will have figured out how to get the most value out of testing by asking them to actually test for the purposes of helping developers build better software and helping executives make better decisions.  I hope that is the case, but if there isn’t a change in the way “a test” is perceived, I don’t think it’s very likely.

uTest: We’ve been fortunate enough to interview the likes of Whittaker, Heusser, Bach, Bolton and Bach in the past for our ‘Testing The Limits’ series… who should we interview next?  Give us a few names who would continue this great (albeit young) tradition and blow our minds?

SB:

Rob Sabourin
Mike Kelly
Cem Kaner
Elizabeth Hendrickson

We’re thrilled to have Michael Bolton as the latest victim of our Testing the Limits series. As the founder of DevelopSense, Michael has traveled the world teaching the craft of software testing to businesses and individuals alike. Since 2005, he has specialized in courses on Rapid Software Testing – which he co-authored with James Bach. Michael is also a prolific writer, and his publications include hundreds of articles, essays and columns. Aside from his blog, you can keep tabs on his latest work through Twitter.

In Part I of the “trilogy” we discuss the Weekend Testers, testing abroad, how numbers can enslave managers, and of course, his pop-star namesake.

uTest: You’ve been a thought leader in the testing space for a while now, but people still seem to get you confused with Michael Bolton (the singer) on Twitter. Ever thought about creating a tester alias? Or have you considered asking him to change his name since “he’s the one that sucks.” Assuming you (and our readers) have seen Office Space, I bet this joke never gets old.

MB: Yeah, it never gets old.  Try renting a car with this name.

A couple of things on that. First, Office Space captures very well what it’s like to have my name. Second, it’s not his real name; he changed it already. Way back when, before Office Space, I was working in tech support at Quarterdeck Canada.  American callers would occasionally turn north when there were long phone queues in Santa Monica. On one call, when I introduced myself to the customer, he laughed. “Really? That’s your real name?” “Yes, really,” I said, expecting one of the usual jokes. He said, “You know, it isn’t his real name. I used to be his bass player.”  The singer’s real name is Bolotin, but according to the bass player, there was no hope that radio DJs would ever pronounce “Bolotin” right, so he changed it.

uTest: We recently interviewed your friend and colleague James Bach, who had high praise for a group called the Weekend Testers. Can you give our readers a quick recap of what this group does, and whether or not you’re on board with their testing philosophy?

MB: James hadn’t heard about Weekend Testers (or maybe he had heard, but it hadn’t registered) until I raved about them to him. When I was in India in November 2009, I attended a talk by Ajay Balamurugadas, who told an amazing story. They’re a bunch of fairly young testers from Bangalore. The core organizers (Ajay plus Sharath Byregowda, Manoj Nair, and Parimala Shankaraiah) are students of our colleague Pradeep Soundararajan. They gather online every week to spend an hour or so testing some freeware or some Web service, and then they spend an hour or so debriefing each other.

There’s a weekly leader who sets a mission for the session. Sometimes the focus is on finding bugs quickly; sometimes it’s on choosing a particular approach, sometimes it’s on note taking. In the talk, Ajay described testers taking responsibility for their own skills development; testers building a self-critical community; testers overcoming obstacles like power cuts and buggy messaging tools; testers ignoring meaningless certifications; testers providing service to the open source development community; testers discovering testing for themselves, owning it.

At the end of the presentation, I whooped, stood up and applauded like I was at a rock concert. At Q & A time, I raised my hand to say that I didn’t have a question, but that this was the most exciting talk on testing I had seen in years, maybe ever.  At the time of the presentation, there were already chapters forming in other cities in India.  At the beginning of this year, Markus Gartner and Anna Baik announced a European chapter.  It’s spreading!

Am I on board with their philosophy?  Hell yeah!  These people, and people like them, are the future of skilled testing.

uTest: It looks like you’ve been “testing abroad” for quite some time now. What’s the biggest thing you’ve learned about testing in locales other than the US and Canada (your two primary residences)?

MB: Scandinavia—Sweden in particular—and New Zealand seem to be percolating more excellent testing than other places.  Meanwhile, I observe that many Western firms—mostly the Americans—are making life difficult for testers in India and other developing countries. These firms didn’t know much about testing to begin with. They viewed it as a rote, clerical activity, piecework, checking work, commodity work that delivered little value. They knew how to do checking, sort of, very slowly and very expensively. But they didn’t know how to do testing, or how to increase its value, so they focused on cost and outsourced it.

The developing countries have millions of intelligent people who want to develop skills, but the West is still requiring these overprescribed, expensive, low-value, confirmatory approaches in which smart human testers are being asked to behave like slow, dumb machines. Confirmatory tests do find problems, but to a great degree, programmers should be pairing and low-level automated tests to squash those problems before testers ever see them. Then free up the testers to look for higher-level problems and previously unanticipated risks.

uTest: Your testing philosophy seems to draw a lot of heat from some circles. For instance, a lot of people seem to think you’re an anti-numbers guy (as you mentioned recently on our blog). What is it these people are so opposed to? Or are they simply misinterpreting your approach to testing?

MB: Excellent testing, the way we teach it, involves thinking critically about the product and developing a story about it. In order to do that expertly, we have to think critically about how we’re getting that story and how we’re telling it in ways that are important to our clients. Testing is about trying to make sure that our clients aren’t being fooled. If we’re fooling  ourselves, we’re likely miss important problems.

When I object to people using numbers badly or using numbers excessively, some people perceive that I object to using numbers at all – not so. It’s just that I love numbers so much that I hate to see the poor things abused.

Some people enslave numbers.  They make numbers work too hard, and too often. I’ve seen organizations collect piles of data about defect escape ratios and defect detection percentages. They hire market research firms and calculate the ratio of happy customers to unhappy customers. But the aggregated data doesn’t tell you anything specific on how to make things better for the unhappy customers.

When you enslave numbers, they eventually rise up, revolt, and enslave you. These organizations spend so much time collecting the data and talking about it and justifying it and trying to duck blame that they don’t seem to have time to do anything about the actual problems, which generally fall into two categories. One:  the organizations are trying to do more work than they can handle with the approaches they’re using. Two:  they’re not listening to people that matter—neither to their customers, nor to their own front-line staff, many of whom are closest to the customers. VPs could learn a ton of useful business information from their own customer service and technical support reps, and they could learn plenty about the project by listening to their programmers and their testers. Product and project knowledge gets mediated by middle managers and numbers; it turns from information into data. When your car is about to go off a cliff, it’s a weird time to be thinking about gas mileage and drag coefficients; better to take the right control action—look out the window and steer or use the brake until you’re back on course. Once you’re back to being productive, then you can start thinking about optimizing, if you think you need to. I wrote about that here and here .

Besides, people don’t decide things based on numbers anyway. They decide based on how they feel about the numbers.  On their own, numbers don’t tell you what is or isn’t okay. Your judgment does that. Your judgment is governed by the synthesis of observations and inferences and facts and feelings, in your head and in your gut.  Effective decisions require both head and gut. Each can mislead the other, which ends up with someone being fooled, or worse. Neuroscientists and psychologists and economists (people like Antonio Damasio, Dan Ariely, Daniel Kahneman and the late Amos Tversky, Daniel Gilbert, Gerd Gigerenzer, Daniel Goleman) have been looking into each others’ fields to discover fascinating stuff about the links between emotion and intelligence. (If you want to do research in that area, apparently it helps to be named Dan.)

Some people seem genuinely scared by the human issues and the uncertainty that’s inherent in software development.  They want testing to be this simple technical problem, a confirmatory thing. Questions related to exploration and discovery and investigation and learning emphasize the fact that we don’t know everything from the outset when we’re dealing with complicated systems. We can’t. We prepare what we think are the requirements, but we often don’t understand what we want until we can compare it to something we’ve got. We have to live in this messy, uncertain, continuously changing world that we can’t control very well.

In that kind of world, repeatability isn’t that big a deal; computers are good at that. The big deal is adaptability; can our software help people to solve their problems, not only reliably but also flexibly?  Designing software to do that, putting all the design work up front, is difficult—I’d say impossible. But we can do a bit of focused work, gather information (that is, test), and then tune things where they need tuning, add things where they need to be added, drop them where they’re not needed any more. Then we repeat the cycle often, with lots of little variations.  That’s what Agile development is about. That’s what evolution is all about, really. In that kind of world, testing is an important service, and the kind of testing that James and I teach is designed to make that service sophisticated and powerful and fast. I think the need for testing annoys people who think everything should be known in advance. Alas, I believe that there are lots of people, even many testers, who think like that.

uTest: There’s a lot of passion amongst testing thought leaders about the best way to test, or the best way to manage or train testers.  Often that passion overflows into heated debates.  How can this passion best be channeled to improve the state of testing?

MB: First of all, we should welcome debate, and we should welcome skilled argumentation as part of the art of construction and practice of persuasion. I’ve found, though, that it helps to remember that we’re exploring and challenging ideas. That means it’s good not to get too personally invested in certain ideas, because we’re always learning more, and because changes in context can mean big changes in what needs to be done.

That said, there are some ideas that seem robust for me. I believe that it’s unethical to dumb down people or the work that they do. I believe that we should focus our craft on learning, and learning how to learn rapidly. How can we improve the state of testing? By recognizing that software development is a web of people who are related in service to each other. That means putting people and social issues first. Get that right, and everything else will follow.

Suggestions are cool.  Standards are something else.  No group should be dictating to other people how they must test unless there are compelling human health and safety reasons for it. Do you really believe that the standards people know anything useful about your business? That the force of government-supported regulation, created by busybodies, should weigh on how you do your daily work? And if your answer is No, what are you going to do to get it stopped?

I talked to one fellow at a conference who said that managers would have no way to filter candidates without certifications.  “No way?” I asked  “Really?”  “Well, what other way is there?” he asked. “What other ways can you think of?” I asked him. He was stumped.  He didn’t seem to know about personal references, well-crafted resumes, networking, interviewing, auditioning, hiring from within and training people.  He did know about those things, but he was so fixated on certification that he couldn’t come up with a single one of them on the spot. Now:  do you really want someone like that, with that kind of limited mindset, setting the standards for how you test your product? Big money and people’s livelihoods are riding on your answer.

uTest: We sat in on your STPCon presentation in October, where you said “Pay attention to your emotions while testing. If something strikes you as being off, then chances are it probably is.” How did you come up with this piece of advice? And can you give testers a brief real-world example?

MB: I fly a lot, and so I often have to book travel using buggy airline Web sites. A couple of years ago, I had to book a flight to Europe. In the course of five minutes, I went through surprise (the site kept revising the dates I had chosen), confusion (why was it doing that?), frustration (the airline’s own Web site didn’t offer flights directly from Toronto to Amsterdam, even I knew though travel agency sites and personal experience that there were two such flights a day), impatience (why is this so hard and taking so long, for crying out loud?) and annoyance (I don’t liked being stymied and having my time wasted). I wrote an article about the experience; you can find it here.

We all want to do testing better and faster, right? Tests of the business rules and the calculations are really important—no question about that—but emotional responses are important too, and they have the advantage of being immediate, both in the “right now” sense and the “no intermediary” sense; they’re direct experience.  Our customers will have emotional responses to their experiences too, and you never get a second chance to make a first impression.

In the Rapid Testing course that James Bach and I write and teach, we say that a bug is “something that bugs somebody who matters”.  People, including testers, are bugged by stuff that doesn’t work.  Being bugged is a feeling that starts with an emotional reaction.  The feeling is a signal.  It’s data, but not information; you still need to interpret the signal’s meaning and significance.  So instead of saying “If something strikes you as being off, then chances are it probably is,” I’d prefer not to be too certain about that.  Look into what you’re feeling, for sure, but consider the possibility that something else is the source of the feeling.  Maybe what you’re seeing is okay, but you need to learn something about the business rules.  Maybe it’s not consistent with an older version of the product, but the new way of doing things is really better and you’re just not used to it. Maybe it feels like a big deal for you, but it’s okay for your client or your end users.

We teach people to pay attention not only to the product, but also to their testing. If you’re bored while testing, use that feeling as a trigger heuristic to prompt a question:  why am I bored? Is it because my testing isn’t revealing anything new? Is it because I’m doing repetitive and un-engaging work that I could delegate to a machine? Is it because this is useless work that I shouldn’t bother with at all? Am I missing the point or paying attention to the wrong things?  If you’re uncomfortable for some reason—if you feel like you’re being pressured, for example—there’s a possibility that you’ll be distracted, or that you’ll rush, or that you’ll be biased towards noticing some kinds of problem at the expense of others. But maybe the pressure is telling you something about the significance of risk, or maybe it’ll remind you to work rapidly, or something else that’s good in your context.

In Jerry Weinberg’s Quality Software Management Vol. 1, Systems Thinking, he points out that decisions about quality are always political and emotional, but people want to appear to be rational. In fact, that in itself causes an emotional reaction in people who want to preserve their own illusions that they’re rational. At a conference recently, a tester approached me and said that she had been intrigued by the role of emotion in software development. She mentioned this to her boss. He replied, “There’s no room for emotion in software development.”  She said, “Really?  No room at all?” He got very agitated, and repeated, shouting this time, “THERE’S NO ROOM FOR EMOTION IN SOFTWARE DEVELOPMENT!” She told me he was quite serious. Naturally, we laughed about that. Our emotional reaction was a pointer to the incongruence between what he was claiming and how he was feeling.

uTest: What are the greatest threats right now to the software testing discipline?  What are the greatest hopes for a brighter testing future?

MB: The greatest threat at the moment, I think, is a systemic set of misconceptions about testing, especially at the middle management level. We don’t do quality assurance; people with the power to make decisions about the nature of the product and the project do that. We call those people managers. Software testing is not like manufacturing quality assurance, where we’re trying to make a zillion identical copies of something, and then checking to make sure that each one is just like the prototype. Every software development project is an attempt to build something new—otherwise we’d use a product that had been built already. So software testing has more in common with design quality assurance, where we’re trying to figure out the relationships between a product and its stakeholders, and trying to understand the product as we’re designing and building the first instance of it. But even then, we don’t assure quality; we question it on behalf of our clients. The people who are building and managing the product assure quality.

When we model software testing on manufacturing quality assurance, we emphasize checking at the expense of testing.  I’ve written a lot on my blog about the distinction between the two since August of 2009.  (You can see it at http://www.developsense.com/blog.shtml). Checking is important, but it’s mostly a confirmatory activity, verifying what we knew to be true before, or what we hope is still true, making sure that we’re getting the right answers. Testing is a bigger deal, focused on finding new information, identifying new risks, asking new questions.

Lots of people—managers, programmers, and even many testers—conflate testing and checking. Again, checking is really important, and it’s important to do it skillfully.  In particular, I think it’s important to automate it skillfully.  But when people confuse testing and checking, they don’t think clearly about cost, value and risk, and they tend to dumb testing down.  That’s a big problem.

uTest: Looking back over your testing career, you’ve seen a lot of changes to tools and techniques and trends.  What do you think software testing will look like in 2020?

MB: 2020 the year, or 20/20 the vision? I predict that, in 2020, there will have been a lot of changes to tools, techniques, and trends, and that someone will ask a tester to predict what testing will be like in 2030.

At this point in history, no sensible person should make specific predictions about technology beyond dinner time; read The Black Swan for more about that. But as for testing, the fundamentals will still be the same. People will want stuff, and programmers and engineers will design it. There will be miscommunication, misunderstanding, unexpected incompatibilities, delays, mistakes, emergent properties. So we’ll still need to investigate requirements and designs and products and problems skillfully.

It’s like writing. After a few thousand years of development on paper, we’ve got people writing for print and online media and television and radio. People’s work can be prepared and consumed anywhere in the world, 24/7, in any number of languages. But we still have people to manage the writers, and we still have people to help the writers—editors and research assistants and designers and so forth.

uTest: Hypothetical: You’ve been banished from testing – nay, ALL software-related activities – for the rest of your days. What will you to earn a living?  What hobbies would you pick up to fill the intellectual void?

MB: Who knows?  For fun, I’d keep playing mandolin, probably. Teach, maybe. Write. I’ve worked in theatre stage management, been a book-keeper, tended bar, worked in a comedy club. In high school I worked in mail rooms during the summer. Whatever I’ve picked up in life, it was because something needed to be done and I was there to do it.  If it didn’t seem like much at first, I started to learn about it quickly. When you invest a little bit of effort to figure out your job, you learn how to makes it faster and better and more interesting. It turns into this great feedback loop. Any job can be more fun when you set out to master it.

uTest: Tell our testing community something about you that your most avid readers don’t know.

MB: While walking through the woods on an island near Vancouver recently, I found myself being quiet and brief, which I like from time to time. Practically nobody knows that.

Lots of people probably don’t know how much I’m eager to help people out. All of my work—courses, articles, conference presentations, this interview—comes with lifetime free technical support. Have a question? Just ask. I might not answer right away—supporting the family with paying work takes precedence over supporting the community—but I’ve never knowingly turned anybody down, so if I don’t answer right away, be persistent. James Bach makes the same offer, by the way. We’ve found that it’s a great way not only to help people, but also to explore problems and come up with solutions and learn things that can help our clients.

uTest: If you were talking to a newbie tester, what advice would you give him or her to set their professional journey off on the right foot?  How about for a 10-year veteran tester?

MB: To both: learn, and don’t stop learning. Practice in your work, but also on open source projects, and in your life generally. Study testing, but note that the great ideas are likely to come from outside the field—at least the narrow vision of the field that the process enthusiasts and “certificationists” present. Testing is a marvelously interdisciplinary craft.  One implication is that whatever you bring to the table from your life and your experience and your education can inform new ideas about testing.  Join or start communities of skilled testers in your area. Join online groups; I like the Software Testing Mailing List and the Software Testing Club particularly. Learn about your craft and build your reputation by asking questions and seeking answers. As with testing itself, the really excellent work starts with asking questions.

uTest: In your opinion, what characteristics, skills or experience separate a good tester from a great tester?  How about in a testing manager?

MB: My colleagues and I have been talking about skills of great testers lately.  (In term of public postings, James and Jon Bach have podcast a nice pair of conversations which you can find here. James also did a CodingQA podcast here. One key skill is the ability to identify context quickly, which helps to avoid all kinds of pathologies—wasting time, failing understand the mission, testing the wrong stuff, investigating unimportant risks, and so forth.  Get the context right, and you’ve got a better shot at doing excellent testing. We teach people to consider and ask questions about the test lab, the test team, the development model and the programmers, the requirements, and the mission to help with that.

Another skill—which we also teach—is something that James has named test framing, which is the same kind of thing in the other direction. It’s the high-level skill—a skill set, really—associated with linking the context, your motivation, the actions you perform, the observations you make, the inferences you draw, and the story you tell about all that stuff.  It’s the ability to follow and describe the threads of logic forward and backward, from the mission to the outcome of the test and back again. It’s the ability to competently address and answer the questions “How, specifically, does this test relate to your mission?” as you prepare to test and “How does the outcome fulfill your mission?” after you’re done.

We’ve seen some good testers whom we can’t quite call great, because they have a hard time with some aspect of that skill set. They seem to lose the ability to link premises and conclusions at some point.

I think there’s a strong relationship to the skills of a great test manager there. A great test manager needs to be able to do the same kind of framing with respect to the test project and the test strategy. A manager that can frame the project can negotiate for people and resources more effectively, can train and deploy her people more efficiently, can explain the work of her team, can defend herself skillfully, can adapt to rapidly changing circumstances. That’s really important, because most projects are messy and confusing. If they weren’t we wouldn’t need testers.

Here’s an example: Test managers get asked questions like “When is testing going to be done?” That question is hard for some test managers because it’s not the test manager’s job to make the decision. They feel pressured to provide an answer, but they can’t frame the problem. A great test manager can deal with that pressure. First: Problems in a product get introduced almost entirely by people other than the testers. Second: Neither those people nor the testers know what problems are there, or where they are. Third: Testing is done in cycles that reveal as-yet unknown information about problems that may or may not need to be fixed. Fourth: The decision to fix the problems depends on the programmers and on the product owner. Fifth: Testing isn’t done until the problems in the product are fixed to the product owner’s satisfaction.

If those things are true (and they are), the amount of investigation and reporting and bug verification testers have to do is unpredictable until you’ve actually tested the product, made decisions about what’s to be fixed, and fixed the problems. Well… it’s predictable, but the prediction doesn’t have any real validity, because there are too many variables that could throw the whole prediction off. “Accurate” predictions in this business come mostly self-fulfilling prophecy—and from ignoring changes in scope or staffing or schedules or budgets or tools that just happened to make the prediction come true. If you set only a couple of data points for your targets, all the other stuff can change without people noticing much.

So testing cycles can be scheduled, but testing cannot be done until the last problem deemed important by the product owner is fixed, by the programmers, to his satisfaction. The great test manager can help the product owner to identify and decide upon what information he needs to know about the product. His goal is to be satisfied in his technical awareness of the product—in relation to the business needs—such that it makes business sense to ship or deploy the product. The test manager and the product owner collaborate in identifying the testing tasks that might help in providing that information.  That turns into the mission for that cycle of testing, test manager and her team can then set about fulfilling the mission—and adapting it when the product owner changes his mind, or when other things don’t go according to plan.  But in the end, the question of when testing will be done depends very little upon the test manager, and far more upon the programmers, the product owner, and a bunch of information that is presently unknown.  A great test manager can explain that in a way that satisfies the product owner, without being trapped into an unachievable commitment.

uTest: At STPCon, you also said that “Testing can be assisted by automation, but automation is not the arbiter of value. We are.” Does this mean that testing will never be replaced by machines? And does this include cyborgs?

MB: Automation is a tool; it’s a means to an end.  It’s a medium, in the McLuhan sense—it extends or enhances or enables or accelerates or intensifies some human capability, but it doesn’t replace that capability. Writing and email and wikis extend our ability to communicate, but they don’t replace talking.

Machines can’t make decisions about what’s best for us, because they’re only extensions of some human faculty. Value is about what humans want, what they like, what they’re willing to pay or do. People don’t even like other people making those decisions for them.

uTest: You’re an active blogger, teacher and speaker – do you enjoy these things more than testing itself?

MB: That’s a little like asking about someone’s favorite Beatle album. I love testing, I love teaching it, I love talking about it. It’s a feedback loop. I wouldn’t be able to do the other stuff if I didn’t test, and the more testing and research about testing that I do, the more I find that’s interesting to talk about. When we teach Rapid Testing, we offer a free day of hands-on testing and coaching. We offer that separately too, and it’s always a source of great learning and great fun.  I’d be delighted for clients to involve me more directly more often.

uTest: Who do you read in the world of testing (blogs, sites, journalists, et al)?

MB: I have a very expansive view of the world of testing. To me, the world of testing is the world of human knowledge and inquiry, of critical thinking, of science. That includes software-related stuff, but it includes a ton of other disciplines too:  medicine (Jerome Groopman, Atul Gawande), aviation (Patrick Smith on Salon.com), education and other social issues (David Cayley), advertising and marketing (Terry O’Reilly), economics (James Surowiecki, John Cassidy, Herbert Simon), security (Bruce Schneier), risk (Nassim Taleb).  All those folks and their specialties are Googleable. I really like Malcolm Gladwell’s writing, too; he’s harder to pigeonhole, but he’s good at finding intriguing sides to the basic story. I first encountered most of the writers via The New Yorker and most of the broadcasters via CBC Radio. I like starting with very accessible stuff, and then if something interests me, I follow the lines of references, bibliographies and notes.

A couple of testing books that came out recently have few notes and no bibliography. That’s like publishing malpractice, so far as I’m concerned. You’ll notice that I’ve mentioned a lot of names already. That’s specifically so that people who are reading this interview can check out the stuff that’s fascinated me and helped to spark new ideas. I hope it helps to inspire people to talk about their own discoveries. Thank you for this opportunity to chat about mine!

Editor’s note: We hope you’ve enjoyed our three-part chat with Michael Bolton. If there’s a question you wanted us to ask that we didn’t get to, you can ask Michael directly at: michael@developsense.com.

Once a month, we’re going to “test the limits”, interviewing a leading thinker in the world of testing and quality.  It could be a journalist, an industry analyst or an exec from a top software company.  To kick this program off, we could think of no better person than our good friend, Dr. James Whittaker.  So we recently interviewed James by bouncing emails back & forth over the course of a few days.

Several of these questions came directly from our community of testers.  The whole exchange is fairly lengthy, so we’re splitting it into two posts.  Come back and check out the 2^nd half later this week.

uTest:  So the news is out about your move to Google. What prompted you to make this move?

JW:  I didn’t so much leave Microsoft and I did join Google. I was attracted by all the Googlers I met at conferences and what I read on their blogs about the way they test. When they offered me the opportunity to be a part of it, one might even argue an important part of it, I found it impossible to decline.

uTest:  Is there something about Microsoft you’ll miss the most?

JW:  Yes, the breadth of both products and expertise. You literally have every type of software imaginable and a chance to collaborate with the people who make that software. From an intellectual standpoint, Microsoft is mind-blowing.

uTest:  What specific work at Microsoft did you enjoy the most?

JW:  Mentoring – making other people better. I love working with smart people and finding their passions. When you take a smart person and help them find and focus their passions, you have a powerful mix. Smart people who aren’t passionate creates a bad vibe. I think I was pretty good at helping people find that combination and enabling them to become the company’s top performers.

uTest:  That’s interesting. What is your approach to mentoring? Can you give us an example?

JW:  To attract passionate people, you have to be a passionate person. I don’t go out of my way to select mentees, we just discover each other. When I was at Florida Tech I supported a lot of students. Some came looking for the wages or to improve their chances of landing a job at Microsoft or Google, but the ones who were drawn by their intellectual curiosity where the ones that saw the most success. It’s been the same at Microsoft. I mentored folks who came and went and others that really got it and became stars.

uTest:  Can you tell us a little about the interview process at your former and current employers? They are both famously difficult and, now that you’ve experienced them both, do you have any tips for our readers?

JW:  During my Noogler orientation I saw stats on the accept rate of Google applicants — it’s astoundingly low. And although I never saw such stats about Microsoft, I have to imagine its low too. But the interviews themselves were fairly similar.

A lot of hard technical questions, both from a coding and testing perspective and also problem solving. Google kept me in a single room and brought the interviews to me. Microsoft shuffled me around from office to office. I think I prefer Microsoft’s approach from that standpoint, but the actual interviews were fairly similar. I think Google took a page from Microsoft’s playbook on that one.

Anyone out there who plans to apply at either place should be ready for problems that test your knowledge and your creativity. Good luck!

uTest:  With this new role, will you still be able to write, teach and evangelize testing?

JW:  I wouldn’t accept a job where I didn’t have to teach! Googlers will have every reason to get sick of me and I am still doing my STAR tutorial and occasional keynotes.

uTest:  So where can everyone find the soothing, savvy writings of Doc JW?

JW:  On the Google Testing Blog, of course! http://googletesting.blogspot.com/

uTest:  And when all is said and done what will be the professional accomplishment you’ll look back on with the most pride?

JW:  Creating an actual discipline around software quality. Note I said quality and not testing. I want software projects as a whole to run more smoothly and more predictably. I really think that’s what software testing is all about — reducing the uncertainty of software development and finding ways to muscle errors out of the process. A process in which mistakes are harder than doing the right thing is the ultimate goal. We can’t eliminate them, but we can make doing the right thing to be the easiest thing to do.

uTest:  What’s your first assignment at Google?

JW:  To raise the level of testing precision and diligence. Google has a lot of smart testers, my job is to help mold them into a serious fighting force and let our bugs beware. But this isn’t so much an individual commitment. Google has a culture of collaboration that I am fascinated by as a Noogler.

We share offices (which might explain their interview strategy), inhabit common areas, collaborate constantly and work as a community. If I am successful, there will be many people who can take credit and if I fail, I won’t go down alone! I think the whole free food thing is at the heart of this as food is often the centerpiece for bringing people together. Lots of work gets done while your mouth is full. I hope to succeed before I have to buy bigger clothes.

uTest:  Rumor has it that you have a new book coming out.  What’s it about and when will it hit Amazon’s shelves?

JW:  It’s in production now. I hope it’s available later this summer. The title is Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Manual Testing.

[uTest note:  Since James is too shy to promote it, we'll do it for him.  The new book is available for pre-order at Amazon... get your advance copy today!]

uTest:  Will you be implementing test tours at Google? If so, which ones in particular?

JW:  That’s going to be up to Google’s engineers. I will most certainly teach them everything I know and be there to work alongside them. Good ideas tend to stick, so this little Google adventure will help decide whether the tours work here or not!

uTest:  In your opinion, what does the future of software testing look like?

JW:  I actually just wrote it up. It’s chapter 8 of my new book. But I am working on a pre-publication version of this particular topic.  I’m not yet sure whether it will be out as an eBook form or a presentation that I do separately. Watch the Google test blog for updates on this!

uTest:  How do automated and manual testing coexist in the future – are the compliments or substitutes?

JW:  They will co-exist better than they do today, but this question requires a longer answer.  Suffice it to say that every good automated test I have ever seen started out as a manual test. How’s that for coexistence!

[uTest note:  this would probably make a good topic for a future webinar with James]