Interview of Matt Evans

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-matt-evans-part-i/2010/12/)

What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.

********

uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.

Having said that, I do also feel on any software project that there is and always should be some level of manual testing. The ultimate users of the program are not robots or automated test frameworks. They are live human beings with lots of different skill sets, backgrounds, geographic location, and expectations. You need to experience the product with your own fingers and eyes on the keyboard and screen. Most automated tests are pretty superficial and have rather myopic validation methods for determining a pass or fail. Because of that, good manual validation–especially with a fully configured and integrated product–is a must in my book. So, what’s the appropriate mix? I would target 70% automated and 30% manual.

uTest: You were recently one of the keynote speakers at the Google Test and Automation Conference (GTAC) in India, where you presented on crowdsourced testing. How much easier has it gotten to speak on this topic in the last few years? Is it starting to register with most people that crowdsourcing (particularly crowdsourced testing) is a viable method for helping to get important work done?

ME: I am relatively new to the concept of crowdsourcing and crowdsourced testing. It was a topic I started to get very interested in a couple of years ago. When I was at Agitar, we started a free service for JUnit test generation called JUnit Factory. It was a pretty successful open project that attracted more than ten thousand users. It was a win-win in that users received pretty good unit tests for their Java projects, and we got crowdsourced testing on our latest code base. Since then, I’ve kept an eye on crowdsourcing progress and I am pleased that it has made it to the software testing field. I do believe it is the right model for a lot of testing tasks, especially in the web and mobile space. So, I’m glad that uTest is making a go of it. I think the explosion of social networking over the past years has really made it apparent that the virtual crowd is a very powerful thing. It can solve lots of interesting problems we know about now, and I’m sure we will be discovering areas where crowdsourcing can be very effective in the future. So, I think crowdsourcing is a practice that is here to stay and will be with us in some form for as long as the web stays running.

uTest: The Mozilla community is unmatched in terms of sheer size, at nearly a million members. How can you possibly manage that much feedback on a regular basis? How do you separate the valuable data from the junk (aka: signal-to-noise ratio)? In short, how do you make something that sounds so chaotic seem so orderly?

ME: It certainly is a challenge and is a constant battle to stay on top of it. And the honest answer is that we do the best we can. As you have surmised there is a mountain of feedback that is received by Mozilla on a daily basis. Primarily the feedback we receive on Firefox is in the form of crash reports, Bugzilla bug reports, beta feedback and twitter. That list is in priority order of what we continuously monitor every day. We are constantly searching for any trends where a particular crash or reported bug starts to become and issue within the Firefox user population. Even a bug that affects a relatively small percentage of users say 0.5%, we are still talking about two million users. So there’s a constant vigilance to detect of there are any emerging issues within the code base.

In the latest beta releases of Firefox 4, there is the feedback feature at the top right of the browser. This allows users to quickly and with limited interaction give a thumbs up or down whether they are happy with the current behavior of the browser. This along with an optional comment is sent to a collection server for categorization and display: http://input.mozilla.com/en-US/. This has been a pretty successful method to get instantaneous feedback, although it does suffer from  a lot of noise contain within the data. But given the sorting and filtering tools we have certainly have received some key insights where we are weak and need more attention with our testing. We are exploring ways to allow users to participate more directly in the test effort through these types of feedback tool features.

uTest: People often (wrongly) view crowdsourcing as a passing fad, or something that only involves unskilled labor. In your experience, what’s been the biggest misconception when it comes to crowdsourcing or community-driven projects?

ME: I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard.  The reality is that crowdsourcing and community based projects are as diverse as the people that are involved.  The interaction and participation of the project members can vary drastically from project to project. The mechanisms for interaction can foster high communication and collaboration among the members or complete isolation and independent input.  There is such a big spectrum of crowd-sourced project implementations and it isn’t correct to have crowd-sourced and community-driven projects under the same umbrella. Strict crowd-sourced projects tend to be targeted towards tasks that are discrete and distributed in nature and is better suited for projects that require a number of paralleled tasks. In strict crowd-sourced projects members tend not to collaborate together to get a consensus results and may even be in a competitive situation to get the best result.

Community based approaches tend to be longer lived projects and are typically for public benefit. Membership is considered a privilege and betterment of the community as whole is usually at the top of the list of shared goals amongst the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Mozilla certainly is an example of of a community-based project. Many people would be surprised at our mission statement: “to promote openness, innovation and opportunity on the web”. It just so happens that building a world-class competitive browser helps us achieve the mission. Community citizenship and passion for the project has been the key success factors for Mozilla success in my opinion.

uTest: You were the QA Director for Palm when they launched the Palm Pre Smartphone, as well as the WebOS apps and services. What’s been the biggest difference (if any) between launching a mobile product and a web product?

ME: The biggest difference between a web product and mobile device is the amount of testing and certification that must precede the launch of a mobile product. A smartphone such as the Pre is an incredibly complex and highly integrated piece of technology–much more so than a typical web application. First off, a smartphone contains a fully-functional OS, usually based on some variant of Linux running on very constrained hardware. It must perform all of its concurrent services utilizing limited memory and limited CPU horsepower. The smartphone must also respond correctly to the multitude of many current events, from those generated from the environment–like switching from wifi to a WAN internet connection–to handling data input from the user, as well as handling events from the onboard applications.

Launching a mobile product requires exhaustive certification of individual hardware components such as the CPU, modems, codecs, and displays. Even then, the finished product is really launched by the carrier and must go through their exhaustive certification tests as well. Testing an onboard mobile application is also a much harder testing task. There are so many conditions and constraints that are involved in testing a mobile application. A typical mobile application is nearly functionally equivalent to any counterpart desktop client-side web application. Take, for example, a mobile email application. It must behave and interact with the server-side application in nearly the same way as a desktop web client. The established protocols were designed for a stable communications environment, but this is just not the case in a mobile environment. The internet connection may be lost and reconnected very rapidly. The connection may even be lost for long periods of time. The application may, at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of system conditions happen in a smartphone that would rarely or never happen in the context of a desktop web client application. However, a mobile application must perform its main functional operations of retrieving and sending messages flawlessly with no loss of data and full operational integrity. Testing mobile applications under these environmental scenarios is a huge challenge. In short, testing a web application is no easy task, but mobile applications and products represent a much tougher and larger testing challenge.

uTest: There’s been a lot of bold predictions about the future of native apps, the mobile web, tablets, and just about anything that relates to mobile. Having lived (and worked) through the dotcom boom and bust of the last decade, do you see any similarities with regard to mobile?

ME: Not really. In the dotcom boom and bust era, the problem was such rampant speculation about how we as consumers would utilize the web for just about every type of activity and commerce transaction. It seemed at that time that anyone with a slick PowerPoint presentation and an outrageous business plan could get startup funding. The main problem was that the technology of the day just couldn’t deliver the user experience that all the web startup companies promised. Here we are, nearly a decade after the bust, and I believe we are just beginning to fulfill some of the wild web usage predictions and capitalization by users and consumers. That’s quite a gap between the vision that was promised and when it was actually delivered. No wonder there was such startup company roadkill during the bust.

Mobile computing, like any other surge in adoption of technology, will have winners and losers. However, there seems to be no end in sight for all things mobile. There is such innovation happening in this space. Take, for instance, the iPad. It is a game-changer, in my opinion. It has defined a whole new computing platform between what we consider a mobile handset and a laptop, and surely is adding to Apple’s vast bank vault. Smartphones continue to wow us with their capabilities and the ability to combine consumer-based technologies into such a convenient package, and we are quickly seeing how these devices are changing how we interact and communicate with each other. Mozilla encourages people to think big thoughts about the future of mobile computing. Just check out this video about the concept mobile phone called seabird and you will see what I mean. I sure would like the phone for Christmas.

So here’s the big difference I see between the dotcom boom/bust period a decade ago: In this mobile era we are leading with innovation and seeing how the marketplace utilizes the technologies. Where as, in the dotcom/bust era it was a lot of empty promises about technology and investor riches that ultimately were never delivered. The current stage in mobile and web computing in general is set for an explosive boom possibly much bigger and broader than than the dotcom period.  But this time I think it will be sustainable. Mobile computing and platforms are here to stay and will be the magnet for innovation for the foreseeable future. The current biggest hurdles are battery life and communication bandwidth. The latter is being addressed as we speak. 4G networks will have broadband connectivity. Hopefully battery life will also be improved shortly. A bright future ahead is my prediction.

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

ME: It’s always been hard to find good software testers. Good or bad economy, a seasoned and talented software tester is rarely unemployed. Typically, many good testers wind up in development because it often appears to be the growth path for a software technology career. Software testing as a career path is still viewed as something lesser than development. However, there has been a lot of gained respect for the software testing career path in recent years. The industry and practice has definitely matured since I started as a QA engineer when breaking software meant dumping your card deck all over the floor. So, I think the pool of quality testing folks is better now than ever, but it is still very hard and time consuming to finally hire someone. How do you fix this? We are always in search mode. We never stop looking for good QA engineers. Also having a team of great recruiters that can poach well is very key as well.

uTest: Which blogs/sites/magazines/conferences do you frequent to stay on top of the latest trends and tools in testing?

ME: I haven’t been to many conferences in the last ten years, except of course the recent GTAC conference in Hyderabad. Mostly, this is because I’ve been involved in startups which didn’t have the budget for sending folks to conferences, and I couldn’t afford the time as well. I have followed James Bach’s blog over the years; he’s as much an entertainer as he is a software tester. I also read a lot from Michael Bulton, mostly because I think he is the E. F. Hutton of the testing world. He is definitely worth listening to. Last year, I took some time off between jobs. It was a great time to have the luxury to really do a lot of web-surfing and getting caught up on things. I became rather fascinated with Yvette Francino. She’s a wonderful writer/blogger, and she chronicled her story of getting laid off as software test manager. It was quite the feel-good story to see how her use of blogging and online networking landed her a job as editor of SearchQuality.com, a site I still frequent today. Lastly, it was great to finally meet James Whittaker in person at GTAC this year. He’s quite a speaker, and I’ve learned a lot from his posts and books. I can’t wait to see how he will avenge being called the “octomom of testing books” at the GTAC conference .

uTest: What could software companies (eg: Palm/HP) learn from the open source movement about launching great products?  And vice versa?

ME: Traditional software companies are skeptical of open source tools and projects. They generally think that they have lesser quality that commercial counterparts. That may have been true some years ago, but I don’t think it is true today. Open source projects have one great advantage over commercially developed ones. The rate of updates and fixes is typically is much higher than commercial projects. I’m mostly talking about the bigger projects such as the popular linux distributions, apache tools and even Mozilla. We are unparalleled in our transparency and response to reported security flaws for instance. In fact being more transparent across the development cycle is a competitive advantage to open source. It encourages user feedback and therefore course corrections can be done incrementally and continuously and ultimately deliver a product a user is expecting. I think that software companies should be more transparent and allow their customers to participate more in the development process. What can open source projects learn from commercial companies? If there is anything it would probably be discipline around scope of the project. Too often, I see that open source projects get feature bloat or accumulate features that are cool to build but are not really needed by the user. Since, most open source developers are working on the projects for the pure enjoyment of it, they are often attracted to working on new stuff and are less interested in improving or fixing regression bugs.

uTest: Put on your prognosticator’s hat – what’s the next big “wow” feature or innovation in the world of browsers?

ME: I think Aza Raskin of Mozilla Labs fame has it right. Identity and browser identity awareness are the next big wow browser features in my opinion. Let me try to explain, although I’m sure Aza does a much better job of it. Your browser knows or could know all about you: your browsing habits, where you browse, what you buy, where you login to, who you communicate with, etc. etc. Currently, much if not all of this data is being captured on the web in many, many places. I won’t go into any of the arguments about whether your captured browsing habits and history are truly anonymous data or whether this data is being used only for commercial benevolence. But one thing is clear: you don’t own this data, and you can’t access or alter it either. And for many, that is a real problem. However, this data can be very useful to you. It is mostly used for a better experience on the web–or at least a merchant’s version of what it thinks will be a better experience for you. Well, your browser could do a much better job of it. It could capture this data directly and it would only be about you. This data could be stored locally, or even in the cloud using services such as Firefox Sync. It could do many cool things like real targeted pre-fetching of your favorite blogs, and then scan them for keywords that are important to you and arrange them in priority order. It could sense what tabs are used more often and arrange for a better workflow experience. I think there is a lot of assistance a browser could do either in the context of the browser interaction, e.g., automate tasks or perform operations based on the web content. Well, there is one huge obstacle, and that is security. How do we ensure that this personal data that absolutely defines you is kept safe and secure? And how do we prove it to you so that you feel safe in allowing a browser to continue capturing this data and making operational decisions based on this data? That may take awhile, but little by little we are making inroads towards this goal such as the Firefox Sync feature that securely stores your profile data in the cloud across instances of Firefox running on desktops and mobile devices. We are getting there, but I suspect it will be a while before you let your browser do your online banking for you.

uTest: What’s the greatest challenge that tech execs will face around app quality in 2011?

ME: The biggest challenge I see is around compatibility testing and maintaining a consistent user experience across the landscape of platforms. To be competitive in today’s web-enabled software market, you need to support a variety of platform interfaces. It is now expected that any web-accessible service will include feature-rich clients for iPhones, iPads, tablets, smartphones, desktops, and kiosks. The compatibility matrix starts to explode when you add in device models, OS versions, and other configuration-dependent attributes. In my experience, you do need to test on real hardware and configurations. Emulation capabilities do help, but at the end of the day you can not be sure of software behavior unless you test it on live hardware. Its a real problem that doesn’t seem to get better with the plethora of new platforms available to users.

Ten QA Myths of Agile Testing

There are a number of comments and statements regarding TDD and the QA function beginning to appear in articles on the internet by so-called specialists, that are, at best, misguided. This article responds to some of these myths and highlights challenges that QA teams will need to face up to and address in order to be successful in an increasingly agile world.

Myth One: “You only need to unit test. TDD testing is sufficient”

For the vast majority of commercial developments this simply isn’t true. Even strong proponents of agile development recognise the need for their armory to include a range of testing techniques. For example, Scott W. Ambler has a list of twenty-one different testing techniques as part of his FLOOT (Full Life Cycle Object-Oriented Testing) methodology, including white box, black box, regression testing, stress testing and UAT. (http://www.ambysoft.com/essays/floot.html)

TDD programmers rely on these tests to verify their code is correct. If the requirements (test cases) are specified incorrectly, then you’ll build robust code that fails to meet the objective. Therefore, most agile projects include investigative testing efforts (black-box), which support rather than compete with white-box testing. Good investigative  testing will reveal problems that the developer missed before they get too far downstream.

Myth Two: “You can reuse unit tests to build a regression test suite”

Some TDD proponents argue that conventional downstream testing is unnecessary because every line of application code has a corresponding test case; they suggest that by reassembling unit tests, everything from User Acceptance Tests to Regression Tests can be performed.

Although this sounds feasible, it isn’t realistic, and here’s why: The granularity and objectives of white-box unit tests developed in TDD serve a different purpose from downstream black-box testing. While the overall objective of a unit test is designed to prove that the code will do what is expected, the aim of regression testing is to ensure that no  unexpected effects result from the application code being changed. These two objectives are not synonymous – e.g. checking that an attribute has a valid date format, is not the same as checking that for a given input, the value of the field contains an expected date.

Myth Three: “We no longer need testers, or automation tools”

Professional testers fulfill a different and equally valid role from their development colleagues. It is widely recognised  that traditional test automation tools have failed to live up to the hype of the vendors. Original Software’s roots are as  providers of products that improve the productivity of the database testing environment. It was because there were  no adequate tools to provide User interface testing that we extended our solutions into this market sector; our heritage is aligned to effective testing rather than screen-scraping automation. When we developed TestDrive, we did it with the benefit of twenty years hindsight of missed opportunities from the traditional vendors. Often, TDD projects have at least as much test code as application code and, therefore, are themselves software applications. This test code needs to be maintained for the life of the target application.

Myth Four: “Unit tests remove the need for manual testing”

Manual testing is a repetitive task; it’s expensive, boring and error-prone. While TDD can reduce the amount of manual effort needed for functional testing; it does not, remove the need for further black-box testing (manual and automated). By automatically capturing the tester’s process and documenting their keystrokes and mouse-clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating. Though manual testing is a time-consuming (and therefore expensive) way to find errors, the costs of not finding them are often much higher.

Myth Five: “User Acceptance Testing is no longer necessary”

Within agile development, acceptance testing is often positioned as working with the customer to resolve “incorrect requirements”, rather than correcting functionality that does not map to what the user needs. When the user initially defines their requirements, they do it based on their initial expectations. When they see the system “in the flesh” they will invariably come up with different, or additional, requirements. While agile methods might reduce the frequency of this happening, it is unreasonable to expect the approach to resolve them entirely, so there should be no expectation that user acceptance testing will be obviated. This is especially true when it comes to the business user signing off approval of the User Interface, since they may have envisaged something different from the developer; which brings us nicely to myth six…

Myth Six: “Automation is impossible”

Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. To begin with, almost all testing from a user and QA perspective will be manual but this testing effort and design can be beneficial later if captured and re-used. Knowing  the right time to automate is tough, so using technology that proactively supports early manual testing but provides a path to evolve this into automation is key.

Myth Seven: “Developers have adequate testing skills”

If testing was easy everybody would do it and we’d deliver perfect code every time. An independent testing team serves as an objective third-party, able to see “the big picture”, to validate the functionality and quality of the deliverable. While developers tend towards proving the system functions as required, a good tester will be detached enough to ask “what happens if…?” When you include business user testing as well, you are more likely to have a system that is fit for purpose. Finally, and I’m sure this point will be disputed, most developers don’t actually want to spend much time first writing tests and then developing code to prove the tests work. Using the collaborative process described below, the developer gets ample assistance in describing the functional tests and can focus on writing lean, accurate and robust code.

Myth Eight: “The unit tests form 100% of your design specification”

Whichever development method you use, before you develop code you have to think about the requirements. While TDD “done well” may identify a fair percentage of the design specification, there are still concerns about gaps between the unit tests. The argument, that you need TDD to prove the requirements are captured accurately, isn’t substantiated by history.

The V-model, for example, is a valid approach to understanding testing requirements and by implication, functional requirements. Like TDD, the V-model and most other models, fall down when the practitioner’s approach is rigid, while development processes are fluid. Whichever approach you choose, correct thinking challenges each user requirement by asking “how would I test that?” The important factor here is that test cases are challenged before they are committed to code, otherwise you’ll be doing a lot of recoding and calling it “refactoring”. When requirements are refined through collaboration, the developer receives a more robust specification that is less likely to change because it has been openly appraised from several people’s perspectives.

Myth Nine: “TDD is applicable on every project”

As the size of the project increases, the time required to run the tests also increases. This can be addressed by partitioning either/both the project and/or the tests. Either route results in tests that run at different frequencies depending upon their relevance to the current coding effort.

This approach introduces the need for test planning and execution management. To achieve this efficiently, in addition to white-box testing, you need to be thinking about:

Integration Testing – “Which tests do I need to run to ensure the new code works seamlessly with the surrounding code?”

System Testing – “Does the functionality supported by the new code dovetail with functionality elsewhere in this system, or in other systems within the process flow?”

Regression testing – “How often do I need to run a regression test to ensure there are no unforeseen impacts of  the new code?” Automated regression testing provides a safety net that can affirm agile development techniques”

User Acceptance Testing – “While TDD (in collaboration with business users) should ensure that a specific function performs correctly, is the cumulative impact of changes still acceptable to the business users?”

However, in today’s environment it is unacceptable to think of these testing stages as discrete serial activities. Often they will need to be run in parallel as we get a new ‘code drop’. As the size of the project team increases (along with testing effort), it is no longer acceptable for the tests to be considered “self-documenting”. Whenever the number of participants increases, the project’s risks become exposed to its members’ different interpretations of the requirements – the definition of what constitutes “success” can be misconstrued.

As the size of the project increases, so would the amount of test code that needs to be developed. Any test code developed needs to be supported for the life of the application – effectively doubling the ongoing maintenance effort.

As the size of the testing workload increases the project needs to add test automation to its armory, to minimize the human intervention and elapsed times for each of these testing cycles.

Myth Ten: “Developers and Testers are like oil and water”

Since the dawn of time there has often been a “them and us” tension between developers and testers. This is usually a healthy symbiotic relationship which, when working correctly, provides a mutually beneficial relationship between the two groups resulting in a higher quality deliverable for the customer.

The discussion should be focused on:

• Software delivery that meets business objectives (fit for purpose, on time and on budget), not who owns which part of the process.

• What can testers contribute to the requirements gathering phase to ensure they are involved in the TDD process?

• How can testers maximize reuse of the assets created during the development phase?

• Is there a role for the “traditional tester” in TDD, or should they (like the developers) be acquiring new skills to enable them to adapt to the new paradigm?

• How can developers and testers find mutual benefit in exploiting new advances in software development and testing tools?

Just as improvements in developer’s software tools and methods have enabled a shift in development approaches, next generation technology for test automation is similarly reframing the opportunities for testers to automate earlier in the delivery cycle without incurring the heavy burden of script maintenance so often associated with traditional automation tools.

Conclusion
Agile projects are in fact an excellent opportunity for QA to take leadership of the agile processes; who else is better
placed to bridge the gap between users and developers, understand both what is required, how it can be achieved and
how it can be assured prior to deployment? QA should have a vested interest in both “the how” and “the result”, as well as continuing to ensure that the whole evolving system meets the business objectives and is fit for purpose. But it requires QA professionals to be fluid and agile themselves, discarding previous paradigms and focusing on techniques  to optimise a new strategy to testing.

The Reality of Software Testing in an Agile Environment
http://www.origsoft.com/whitepapers/agile-environment/myths.php

Interview of Ben Simo

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-ben-simo-part-i/2010/08/)

Our Testing the Limits guest this month is Ben Simo. Known as the “Quality Frog” on Twitter, Ben is one of the most insightful and entertaining testers in the business. A proponent of the context-driven school, Ben has more than 19 years of experience testing software and developing testing tools. He currently lives in Colorado with his wife, two children, two dogs, five cats and fourteen – count ‘em – fourteen goldfish. For the full Ben Simo experience, go to his blog.

uTest: Your “Is There a Problem Here?” series has been a big hit in the testing community. What’s the absolute worst bug that’s ever been submitted? And what can testers and developers learn from these type of mistakes?

Simo: Many of the bugs on IsThereAProblemHere.com could be argued to not be bugs. The software works or catches and reports an error condition; but in a way that it unnecessarily frustrates users. My hope is that people involved in creating and testing software can learn from these examples. Rather than only look for the obvious technical bugs, we need to be asking ourselves “Is there a problem here?”

We build software for the benefit of people. Software fails when it does something other than solve human problems.  Although not the worst items submitted, two items come to mind.

The first occurred on Christmas Day last year.  Twitter was full of complaints by people who received Sony’s new electronic book Reader device as Christmas gifts. The device worked except that Sony was not prepared for the Christmas Day rush on their servers as people attempted to install software and purchase books.  By not sufficiently preparing for the Christmas rush on their servers, Sony turned joy into frustration for many new customers. As a performance tester, I take this as a warning to seriously consider what events may cause a surge of demand for the systems I test.

The second problem that comes to mind is one I’ve repeatedly encountered with Blogger’s auto-save feature. I like features that help prevent users from losing their data.  While auto-save features usually indicate that software designers value their customers’ data, Blogger provides a great example of how auto-save can make things worse.  The Ctrl-Z undo option in users’ web browsers goes away after an auto-save occurs.  If a user fat-fingers text in a way that deletes content just before an auto-save occurs, there is no going back. An accidental Ctrl-A instead of a Ctrl-Z or Ctrl-X followed by another keystroke can permanently delete a document in an instant.

uTest: Gotta ask about the “Quality Frog” handle on Twitter. What’s the origin of this moniker?

Simo: A few people have told me “Quality Frog” looks like two random words from a Facebook captcha.

I’d like to be able to say that I carefully selected the name and that it signifies that I care about quality and I’m amphibious like a frog. I’d like to say something along the lines that I started life as a tadpole in the waters of programming and later grew legs to live on land and be a tester.  I could even say that as a Quality Frog, I now eat bugs for breakfast and help keep the waters clear. While such thoughts may have come to mind, the truth is that I came up with Quality Frog while pairing a variety of words with Quality in search of an available domain name. Frog came to mind as something that ate bugs and the domain name was available. Since then, I’ve continued to use it as a handle.

uTest: You’re a self-described “defensive pessimist”, which seem like good qualities for a tester to have. What other attributes come in handy in this line of work?

Simo: The term “defensive pessimist” comes from Dr. Julie Norem, a psychology professor at Wellesley College.  In her book, “The Positive Power of Negative Thinking”, Dr Norem describes defensive pessimists as people who typically perform worse when pressured to look on the bright side and be optimistic about things that concern them.  Rather than trying to think happy thoughts and only look at the positive, defensive pessimists imagine the worst case scenario; not to get depressed and become immobilized, but to develop solutions for what might go wrong in order to be better prepared. Defensive pessimists can make great testers, and are likely to annoy many optimists.

The third guiding principle of the Association for Software Testing states “AST views software testing as a cognitively complex activity that requires critical thinking, effective communication, and rapid self-directed learning.”  I fully agree with this.  Therefore, I find it essential that testers be critical thinkers, effective communicators, and self-educators.  Any one of these three things without the others will make us less effective as testers.

This doesn’t mean that every tester must master all three of these on their own.  My ideal test team would be comprised of people with a diversity of aptitudes, skills, and experience. I don’t want a team of clones.

uTest: Much like our previous Testing the Limits’ guests, you’re a critic of testing certifications. Yet some still see certifications as the only way to stand out from the “unskilled labor” crowd. Tell us a bit about why you’re a skeptic/critic of certs – and how they could be improved and made more relevant/useful/predictive.

Simo: I am not against certifications per se. I am against bad certifications.  I am against certifications that are presented to be something other than what they are. I am against certification bodies and trainers that prey upon people’s desire to stand out and tell people they can improve and certify their competence as testers with few days of training and a multiple-false test. In his keynote at the Conference of the Association for Software Testing (CAST) this year, Cem Kaner stated that If one can become certified in their profession in three days, they are a commodity, and don’t deserve much more pay than unskilled labor. I agree with Dr. Kaner. Rather than educate and help people stand out from the unskilled labor crowd, such certifications trivialize testing and encourage wrong thinking that testing is unskilled labor.  I want testers to be more than a commodity.

Many IT certifications, including testing certifications, are more about marketing than education.  These certifications are not good measures of skill, competency, professionalism, quality, or any of the many things those on the receiving end (of the marketing) care about. In my experience interviewing job candidates, tester certifications have not been an indicator of applicants’ testing abilities.

Software testing is a rich and diverse field. It is also a young field. Rather than feign maturity and simplicity where there is none, let’s embrace the diversity and youth. Let’s continue to learn. Let’s not lock in a set of context-free definitions and practices and make them a standard. Such standards will hurt the quality of software, not improve it.

Rather than pursue a certification, I encourage testers to get involved in a professional community. Find colleagues that challenge you and help you learn.  Seek real education that comes through interaction and doing over memorizing information useful in passing a multiple-false test. The Association for Software Testing offers a series of online software testing courses that facilitate deeper learning that you are likely to find in training focused on helping you pass a certification exam.

uTest: Jon Bach mentioned that changing the meaning of “QA” to Quality Assistance would help outsiders (engineers, executives, et al) better understand the role of this discipline.  Agree or disagree?

Simo: I believe I first heard  “Quality Assistance”  from Cem Kaner.  I agree with Jon. When testers bear the title Quality Assurance, it often implies that they actually assure the quality of other people’s work. Testers are in a position to help assist quality; not assure it. Let’s not assist the setting of unrealistic expectations with inappropriate titles.

uTest: While we’re on the subject, are you anyway related to James and Jon Bach? The resemblance is uncanny.
Simo: I don’t think so. I’m available for adoption if the Bach family is interested.

uTest: You’ve said that you frequently use automated tools, but that you don’t trust them entirely (back to that whole defensive pessimist thing again). What advice do you have for testers and managers wanting to strike a healthy balance? And what’s currently in your arsenal of automated tools?

Simo: My mistrust in tools is based on the fact that tools can’t think for me. Automated checking can only process whatever decision rules someone thought to program when the checks were created. Automation will consistently do what it is programmed to do and consistently not do what it is not explicitly programmed to do. I find test automation to be useful. In fact, there are some things I’d not want to even try to do manually. I do, however, distrust the green bar. When automated checking passes, I ask myself what the automation does not tell me. I also try to keep aware that people who don’t understand what the automation does are likely to assume that it does more than it does.

Tools are much more than test automation. Tools are essential for testing. I don’t want to test without tools. I have some old programming books that promote testing in which a programmer manually executes code, step-by-step, with pencil and paper in order verify that the code works as expected. This is manual testing. This is a testing practice that came from a time when computer time was rare and cost more than people. We’d now laugh at someone proposing testing in this manner.

Today, whether we call it “manual” or “automated”, tools and automation are part of coding and testing software.
Programmers use fancy integrated development environments that do a great deal of the tedious work of managing and checking code. Programmers have unit testing tools that allow them to write tests as they write the code. We use tools to track and version source code. We have continuous integration tools that automatically run pre-defined tests and report results when programmers save code into shared code repositories. Today, programmers rely on interactive tools to do work that used to be done manually. These tools support people coding software rather than trying to replace them with batch processes.

Many of the tools built for testers seem to still be stuck in the world of batch processing. Rather than assist testers by helping with tedious tasks, they try to replace test execution. Some of this comes from the design-all-tests-up-front-in-procedurally-scripted-detail thinking that made sense when computer time cost more than people.

I worked as a tester for nine years before I encountered a record and playback type test automation tool. In those nine years, tools played an essential role in my testing. We built all our tools in-house. We wrote macros and scripts to do tedious data processing. We wrote code to verify data formats. We created test interfaces for databases. We created test management systems. We depended on tools but did not have a single test case that was fully automated. This automation made testers both more efficient and effective.

Rather than look at testing as something to be either manual or automated, I encourage people to look at individual tasks that are part of testing and try to identify ways that automation can help testers evaluate software. Rather than limit tools to automation of test execution, I look for ways that automation can assist testers – to turn them into powerful cyborgs.

These tools include automated test execution tools, data setup and teardown scripts, communication tools, test management tools, test logging and reporting tools, test data manipulation tools, and more. Along with test execution automation tools (take your pick), my most-used testing tools are Excel, Picasa (for screenshots), a screen recorder, and the GNU Unix utilities (also available for Windows).

uTest: Certain industries appear to be ahead of the curve when it comes to testing practices, while others remain in the proverbial stone age. Is this an accurate statement? Or have testing practices evolved at similar pace across all industries? As someone who has spent time in many sectors, we’re interested to hear your thoughts on this.

Simo: I’ve not noticed great distinctions in testing practices based on industry.

What I have noticed is that testing practices seem to evolve within individual companies and test groups. Rather than learn from what others have done, it often appears that each tester, each test organization, and each company starts in some stone age and evolves on its own – making the same mistakes that others did long before.  Lessons learned don’t seem to be shared much across companies.

I’ve also noticed that “quality assurance” groups in large commercial businesses, regardless of industry, tend to be dominated by factory school thinking. Instead of viewing testing as cognitively complex work that requires critical thinking, good communication, and continual learning: testing is thought to be predictable repeatable validation work that can be done by automation or cheap labor.  These companies tend to view quality and process as things to be scripted, controlled, and quantitatively measured. In doing so, I believe these companies dehumanize their people and their customers.  These companies tend to have what Jerry Weinberg called “overstructured management” in a paper written nearly 30 years ago; management which attempts to manage people like code: with sequential work models, binary either-or choices, and mechanical repetition. These things may be good for computers, but aren’t great for people.

While many companies are still recursively calling on the mistakes decades past, others are recognizing that people aren’t to be controlled like machines.  While they may not be familiar with either document, these companies exhibit the values expressed in the Agile Manifesto and practice the principles of the Context-Driven School. These companies value people over systems.

uTest: Who should testers be following on Twitter to stay current on the latest industry trends (I mean, besides you and us, of course)?

Simo: I’m not as interested in trends as much as I am in experience reports, and new ideas and challenges. Some of my favorites who are active on Twitter are Pradeep Soundararajan (@TesterTested), Shrini Kulkarni (@shrinik), Matt Heusser (@mheusser), Lanette Creamer (@lanettecream), Elisabeth Hendrickson (@TestObsessed), Michael Bolton (@MichaelBolton), Jon Bach (@jbtestpilot), and James Bach (@JamesMarcusBach). Beyond that, I suggest people see who these people, and I, follow and interact with on Twitter. Use Twitter’s search feature to find people tweeting about things that interest you. Don’t limit yourself to only testers or those people you like. We can learn a great deal from those outside our field or with whom we disagree.

uTest: Let’s go back in time for a second: How did you get into the craft? What was the first application you tested? What was testing like back in the early 90s?

Simo: Providence. It was providence that got me into testing.

I was young, in love, and planning to get married.  I had been doing some part time database development work, but needed a full time job before the wedding.  I submitted letters and resumes to dozens of companies. I was willing to do almost anything that would pay the rent.  I lived in a city where the local job market was dominated by defense contractors. I quickly learned that many of them called nearly everyone who applied for anything in for an interview; so they could learn about people and add them to databases of potential hires for matching to work they did not yet have.  These companies would then present these people to the government as their available workforce when bidding on contracts. This made it frustrating for those of us looking for work. It often wasn’t clear, when going in for an interview, if it was for a real job or for a potential position that might come at some time in the future if that company were to be awarded a government contract.

I interviewed with the company for which my fiancé (now my wife of 19 years) Sophie worked. It appeared to be one of those information gathering interviews without an actual position to fill. I was asked a lot of questions but none seemed related to a specific opening.  At the end of the interview, the interviewer said he’d be calling me.  Time went by without any more contact. Nearly a month later, I got a call asking if I could start the next morning.

I was one of a handful of people hired to work as “data collectors”; to execute tests and collect results for an interoperability demonstration of digital imagery systems.  These systems, from a variety of different vendors, were used by the military for manipulating and exchanging digital photographs.  This test was the second phase of an effort to validate a new communications protocol standard. Our mission was to assess to what extent systems made by different companies could exchange information over a variety of communications systems. We were given a huge matrix of imagery systems and communications equipment (from encrypted telephones to push-to-talk radios) combinations to test. We quickly learned that this required we learn the communications systems, the imagery systems, and the protocol used to exchange data.  I enjoyed this challenge. By the end of the project, I found myself in the role of being the writer of the test report and numerous interface documents. I also learned that I enjoyed testing.

If I hadn’t received that call which led to becoming a tester, I would have likely taken a job as a pay telephone technician. We all know how cell phones have since killed the pay telephone business.

So, what was testing like in the early 90s?  For me, working for an independent fee-for-service testing organization, it was a great learning experience.  I learned that my role as a tester was to help our clients produce better products; not to just pass or fail their products.  I learned that every company followed different development and testing practices – regardless of what labels they gave them.  I learned to seek guidance from others who could mentor me. I learned that continual learning is essential in testing. I learned that testing is a better assessment tool than an assurance tool. I learned to plan and script testing only to the level of detail that made sense; understanding that some detail cannot be known up front.  I learned that testing is exploratory in nature.  I learned to communicate with stakeholders and developers. I learned that the real requirements aren’t necessarily the things documented in a technical requirements document.  I learned to create tools to help me accomplish tedious tasks and enable doing things I could not do manually. While I didn’t realize it at the time, I was becoming a context-driven tester.

After nearly a decade, I moved to working for commercial software development firms — with fear and trepidation.  I wasn’t sure that my skills would transfer to being an in-house tester. I soon discovered that they transferred well.  I spent a short time with some misguided “quality cop” thinking after I was given a “Quality Assurance” title before better understanding and returning to my context-driven roots.

Rapid Fire Q&A:

• Favorite movie of all time: My favorite movie of all time would have to be “Star Wars, Episode IV: A New Hope”. It is both one of my earliest and favorite childhood movie memories, and I still love it

• Favorite movie… that’s about testing or taught you something about testing: I don’t know about it being a favorite, but “Man of the Year” comes to mind. The movie was disappointing in that it wasn’t the comedy it was advertised to be; and the software “bug” on which the plot was based made little technical sense; however, it illustrates the potential risks of software not working as it should – either due to oversight or ill intent.

• Mac or PC: Commodore. I still do everything on my Commodore 64. Seriously, I’m a PC. I’ve not used a Mac in over 15 years.

• Browser of choice: I’ve recently come to like Chrome for most of my personal browsing. However, I prefer Firefox for testing. There are many cool add-in tools for Firefox.

• Brand of smartphone: Motorola Droid X. After seven years of using Windows Mobile devices, I just switched to Android.

• Favorite US president (no fair picking Millard Fillmore… he’s my fav): Teddy Roosevelt. I can’t help but love an adventurer.

• Top tech innovator of all-time (Edison, Gates, Jobs, Andreesen, et al): Apart from the modern computer, Thomas Edison has likely had a great impact on humanity.  So, if I had to pick only one, I’d say Edison. When it comes to computers, Charles Babbage and Steve Wozniak come to mind. The ideas of Charles Babbage led to the programmable computers we have today. And Steve Wozniak was essential in making computers personal and bringing them into our homes

• Best concert you’ve ever seen in person: I haven’t been to a real concert in over 20 years. However, I have fond memories of Barry McGuire sitting on the edge of a stage singing songs to the last couple hundred stragglers who remained.after someone else’s concert.

• Hobby that would surprise our readers: Since I recently quit my job and am looking for new work, I could say that software testing is now a hobby for me. It will hopefully be paying work again, real soon. I could call my obsession with old programming books a hobby. I collect and read old programming and testing books to better understand the history of software development.

Photography and exploring Colorado are two things I could also call hobbies; but those won’t surprise anyone who follows me on Twitter. Also, we’ve got a bit of a zoo at our house with cats, dogs, and fish. That too should not be a surprise.

What may be a surprise is that I’m a recovering collector of NASCAR die cast cars. Yes, I was once an addict with new cars regularly arriving in the mail.  I’ve now been clean, and not added any new cars to my collection, for well nearly two years.

uTest: If Al Gore had never invented the inter-web and software testing didn’t exist… what would you be doing for a living?

Simo: I’d likely be a starving photographer. When I was eighteen, I was considering pursuing a career in computer programming or photography.  My father advised me that he’d encountered more starving photographers than computer programmers. So, I pursued software over photography.

Interview of James Whittaker

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-james-whittaker-part-i/2010/06/)

It was one year ago (June ’09) that James Whittaker helped us christen our ‘Testing The Limits’ interview series by being our first guest. And for much of the year, he held the distinction of generating the most page views… and then some guy named Patrick Copeland came along and took the lead a few months back.

Well, in honor of our one-year anniversary, James has accepted our invite to be our first-ever return guest – and this marks the start of a new trend. In our 2nd year of Testing the Limits, we’re going to be revisiting some of the past legends and leaders to see what’s changed since they last spoke with us. Of course, we’ll also be blending in some voices we haven’t heard from yet (we’re looking at you, Cem Kaner and Elizabeth Hendrickson) so stay tuned!

In this interview, James discusses his present role at Google; the emergence of Web Test Framework (aka WTF); the next decade of testing innovations; cloud computing and much more. When you’re done with this one, go read Part II.

uTest: A year ago, the big news was about your move from Microsoft to Google. Now that you’re no longer a Noogler, how has this year changed your perspective on testing and the testing industry? What has surprised you most?  Can you share any favorite stories?

JW: Four years ago I made the decision to leave the comfy confines of academe and consultancy and do something more real. It seems there is a steady supply of ex-industry folks going into consulting and not much of a flow the other way. I thought it would challenge me more than anything else I could do. Unfortunately, Microsoft just wasn’t the place to pull that off, ship schedules in the client-server domain simply didn’t allow a fast enough pace to suit me. I’ve been part of more software development in the past year at Google than I had my entire time at Microsoft and my consulting career combined. Things I didn’t think possible like shipping a product from concept to production in a matter of weeks, doing software development in a way that makes testing mostly invisible and creating completely new uses for test techniques that I had dismissed as amateur earlier in my career (e.g., record and playback) have not only surprised me but also now make my days a lot more interesting.

Another perspective that has changed is my appreciation of automated testing has grown. I’ve written extensively about manual testing and the importance of having a brain in-the-loop and I haven’t given it enough credit to automation in the past. Automation is really important and I think the detractors to it, simply don’t know how to do it right or simply don’t have enough experience with it. At the same time my appreciation for manual testing has grown as well, but I no longer advocate doing it without a lot of automated assistance. I’ll explain more about that later.

uTest: In the spirit of “WTF”, can you tell us more about the new, appropriately named, Web Test Framework and the unique control that Chrome and Chrome OS will offer web apps, browsers and the operating systems they are running on?

JW: I work with a developer who believes that WTF (the real meaning of the acronym) is the only appropriate response to a tester who creates yet another test framework. I have to admit, it is a response I often employ as well. Does the world really need another test framework? What the —-?

Well the world needs this one.

What we are working toward is a browser that is capable of testing applications that run inside it. The idea is to build the test framework into the browser, in our case Chrome, so that test artifacts don’t have to be moved around and installed in test labs and on test machines. Also since the browser and automation are one, problems such as Ajax and self-updating web pages won’t confuse automation. This is true cloud-based test automation. What I want is the ability to launch one of these WTF-equipped browsers and point it at the web site/app I want to test and have all the back-end test case storage, bug reporting, progress reporting etc managed for me, automatically, in the cloud. Tests that I wrote or others wrote can be consumed by the browser and applied en mass. These assets all exist in the cloud at Google, why not make them universally available?

I’m betting you want the same thing.

uTest: You’ve argued that the past two decades of software testing innovation have been “the disposable decades” – too tied to the app, the domain and the technology. What does the next decade look like to you? What will surprise us most?

JW: At GTAC 2008 and EuroStar the same year I gave a talk on the Future of Testing and I am returning to GTAC in October 2010 to give an update on my vision for the next decade. I don’t want to preempt that talk, but I will say that my vision is far more aggressive now than it was then. What I thought was 10-20 years out two years ago will be here in less than 5. Things are changing fast and I am anxious to push them faster. We are working unnecessarily hard to test our applications and it is limiting innovation and creating a market full of sub par software. This has got to stop.

The central theme of my vision of the future is getting rid of this disposable nature of our test artifacts. We are reinventing the wheel every time we test wrt planning, test development, automation, reporting and metrics. Let’s get together and stop this. One of the reasons I am so nice to you guys at uTest is that you are working with the idea of leveraging the cloud and the crowd to perform testing. There is much more collective intelligence involved in what you guys are doing and I like that. I am anxious to get our testing tools in the hands of your crowd.

uTest: In your most recent blog post, you describe a (very near) future where test data will be more like a cellophane wrapper around a web app UI, an overlay that would replace the tiresome querying of multiple databases. Just how close are we to achieving this innovative bug and test case overlay and what would be the significance of such a feat?

JW: You keep trying to pull my GTAC talk out of me prematurely! But let me give a bit of a preview. Test managers need data to make decisions about how to optimize their testing and their testing resources. Most technical test managers I know end up creating a set of SQL queries that they use to get information about features, bugs, test cases, test runs and so forth. All the data is there, it’s just hard to access. Video gamers have a similar problem, there is a lot of information in the game and it is in a lot of different places. Our solution in testing should be the solution gamers use: a heads up display – a single location to surface all the information a tester and test manager needs to make the right decisions during a testing campaign. We are building a heads up display that can consume all this information and use it to guide the hand of the tester and test manager. This is part of what I meant earlier when I said that manual testing needs automated assistance. People doing exploratory testing without a HUD are like gamers who play without one. In the latter case, the gamer’s character gets killed, in the former case the tester is less effective.

uTest: You are a busy man, James, having less and less time to write witty blog posts (which we miss)! What has been keeping you most busy and what have your greatest accomplishments been in the past 12 months?

JW: Busy or just having so much fun I can’t take the time to write a lot? My greatest accomplishment happens every day: people who work for me questioning every aspect of our business and pushing innovative solutions to other testers to improve the way they execute. I had a direct tell me two weeks ago that he couldn’t wait for Sunday to pass so he could come to work on Monday. Nothing else I do compares to that.

uTest: One of our top testers, Madhukar Jain, would like to know: Do you see cloud computing as the next evolving challenge in testing? We would take this one step further and also ask if the cloud represents or enables the fourth gen of testing, or testsourcing, you’ve often alluded to? What is the cloud’s potential to empower the crowd (third gen)?

JW: I see cloud computing as the way to solve the testing problem. And yet again you are forcing me to reveal my GTAC talk prematurely. So be it. Imagine for a moment every test case database, bug repository, build script, static analysis tool, metrics engine, and so forth in the cloud. Every tool and scrap of information a tester will ever need stored in the cloud and surfaced in his HUD when it is most relevant. It’s already stored on some server somewhere, putting it the cloud is a small step. Over time (hopefully a short time) we’ll be able to use this information seamlessly. Instead of the Internet, you have the Testernet capable of answering any question you have about testing and being there to guide you through whatever testing tasks you are performing. Other industries are already doing this. We’re behind the curve in testing.

Do you remember Star Trek The Next Generation? My friend and fellow Googler Dan Massey and I like to hold this series up as the type of technology we want to build. Geordie and Data were constantly “reconfiguring the subroutines” to solve some important task. They could reprogram the phasers, the photon torpedos, the holodeck, anything on the ship! Plug and play “subroutines!” They did this from any console on the ship. They did this without installing any back end repositories or test assets because it was all available in the cloud. They did this without running any all-night build scripts or writing new test cases. The build happened automatically, test assets were applied automatically. The reason Geordie and Data could do this programming so quickly is because they only had to solve the problem they were working on, they would ignore all the back-end build and test nonsense better suited for a computer.

I bet Geordie had one hell of a testing HUD and never had to wade through the complexity of partially automated build systems and SQL queries to find the tests and test data he needs, in fact I bet he didn’t even know anything about these backend systems. They were hidden in the cloud, exactly where they belong, allowing him to concentrate on his programming.

uTest: The Microsoft vs. Google battle continues to play out very publicly in the media. Just last week, Computerworld wrote this story: “Microsoft: No Matter What Google Says, Windows Is Secure.” Having been at both companies, we think you have a unique perspective on this one. Any thoughts?

JW: Let me say right away that I enjoyed my time at Microsoft and admire the company and the smart people who work there. As a resident of Seattle, it is in my best interest for Microsoft to prosper! But the two companies are vastly different regarding the way their talent is managed and their products are built. Google is an engineering-centric company where innovation comes from individuals who are empowered to do whatever they need to get ideas into production. Much has been made of Google’s game-theory approach to managing people where rewards are given quickly for impactful behavior. It works. Morale is high and people work very hard and take quality very seriously.

Does this mean we produce more secure or more reliable products? We try hard to do so; Microsoft tries equally hard. I think we have the advantage of less legacy and a more modern and reliable platform (the Web as opposed to client operating systems) to work from. But the secret sauce at both companies is the same: hard work and due diligence.

uTest: You shared with us (as the pioneer of Testing the Limits posts) that your first assignment at Google was “To raise the level of testing precision and diligence.” So, how did it go?

JW: It didn’t take long. Google was mostly already there so I can’t really take credit for it. Now I am busy raising the bar further.

uTest: Top tester Glory Leung is curious: What are your views on SCRUM testing in general? Are people doing it properly? What is the ideal situation?

JW: Scrum is just a name. I don’t like names, they feel too confining and people have their own ideas of what they mean. I took a lot of flak for using the name ‘exploratory testing’ for my book. People love to confine you to how they view a specific named idea or technique. Flexibility is required.

The ideal situation is that the best testers for the job be involved. For unit testing and TDD this means the individual devs who wrote the code. No one is more qualified to test a function in isolation than the person who wrote it. However the dev is often the worse person to test how his function interacts within a larger feature or with its operating environment. At Google we expect devs to do the former, all of it. Then and only then have they earned the right to have testers do the latter. Having broken or missing unit tests is a great way to ensure that testers get taken off your product.

The question I am struggling with now is product expertise. What is the right mixture of testers who are close enough to the product to become real experts with it and specialist testers who are expert breakers? Call it whatever you like but product expertise and testing expertise are the two ingredients that I find make for good testing, regardless of your ship cycles or how your team develops their code.

uTest: There are many varieties of “testing frameworks” out there that guide testers in choosing test cases and defines the way they approach test design. Out of the ones you’ve recently seen (e.g. Input Domain, Divide & Conquer, Fishbowl, Storybook, Pessimists), which of these works best? Is it a combination of two or three? Does it depend on the job?

JW: More names, less meaning and more chances of some consultant claiming ownership over them! Like I said in the previous answer to me the thing that I find most valuable are people who know the product and people who know how to test. Stir those together and you have a winning formula.

uTest: Doc JW rapid-fire (in no particular order):

1. Last book read? The Hobbit with my 12 year old son. It was his first time (how I envied him for that!)
2. Last movie watched? Avatar, 4 times.
3. Favorite band or album? Neil Young, Led Zeppelin, and Nirvana … in that order.
4. Favorite Kevin Bacon movie? Kevin who?
5. Browser of choice? Chrome!!!
6. Cats or dogs? Dogs, not even close.
7. Favorite video game? I played Monkey Ball 14 years ago, once.
8. Which mobile device are you packin’? Android Nexus One
9. Favorite mobile app? Google Sky Map.

uTest: What would you be doing with your career if you were not a Test Director (e.g. bullfighter, gourmet chef, cobbler, explorer)?

JW: Stand up comedian. I still intend to do it some day.

uTest: Can you give us a sneak peek into any of your upcoming guest lectures, books, articles or presentations on the horizon?

JW: Alan Page inspired me with his book on how Microsoft does testing, perhaps I will write one on how Google does it. There is almost no overlap. I have also doodled a memoir of sorts trying to figure out how a guy like me managed to succeed. I’ve only shared it with a single reader who thinks I should wait until I retire to publish it.

uTest: It’s graduation time…  what would you tell college students who want to enter the world of professional software testing?  What should they do (or avoid) to get their career started on the right track?

JW: Simple, learn computer science. Then, get a job at a company that takes testing seriously and be prepared to learn. A degree in CS is, for me, the first and most important background.

uTest: Since you literally wrote the book on the subject, what separates a bad exploratory tester from a good exploratory tester?  And what’s the difference between good and great in this area?

JW: Bad exploratory testers spend little time preparing and are only seeking bugs as their primary outcome. Good exploratory testers apply structure to their work and use tools to collect data so that their testing sessions are more meaningful than just bug finding exercises. Exploratory testing should be connected to the rest of the testing process and not just a late cycle bug finding activity.

Interview of Patrick Copeland

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-patrick-copeland-part-i/2010/02/)

In this month’s Testing the Limits interview, we’ll put Patrick Copeland on the hot seat. Patrick is the Senior Engineering Director for a promising young upstart named Google (we’re not familiar with them ourselves, but we’ve heard good things) where he oversees a global team of about 800 engineers. But this isn’t his first rodeo –  prior to Google, Patrick spent a decade at Microsoft, where he specialized in all things related to software engineering.

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

• Strategic: Maybe a new feature, but in a market with existing competition (like Android).
• Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
• Customer usage: For example, popular high-traffic applications like GMail.
• Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
• Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

By the way, what I meant is that the surface area of the challenges we face in software has grown exponentially over time. Just in the past few years we’ve witnessed a paradigm shift to cloud computing that stretches our imagination and challenges the limits of software. Our process for developing that software is going through an equally dramatic revolution. We are reconsidering the appropriateness of the lessons we’ve taken from traditional software development companies. Testing teams need to continue to evolve quickly and innovate just keep up with what’s happening in the industry.

uTest: Not too long ago, you added testing guru and author, James Whittaker to the Google team, who is well-known to the uTest community. Why was he such an important addition, and has there been anything that’s surprised you since he came on board?

PC: James has been a great addition to the team. His passion for testing is incredible. The biggest surprise is that he’s having fun being a manager. James accepted our offer without knowing exactly what he was going to do. This is a pretty common situation for new senior people at Google. I remember giving him our offer and he said repeatedly, “I’m not interested in being a manager.” I kept saying, “ok, ok, no manager position for you.” But he’s decided to run the Kirkland and Seattle Test Teams, in addition to several groups in Mountain View. He loves it. I think that might even be a surprise even to him.

uTest: Google seems to break into another market every week, with the new Nexus One phone being the most recent example. How much of your time is spent looking forward to new apps and categories, as opposed to concentrating on existing products?

PC: At Google we have a 70-20-10 rule for investing in technology. While the numbers are not carved in stone, the basic concept is that we invest ~70% on our existing core products, ~20% in related but new categories and ~10% on riskier bets or areas outside our existing core competencies.

Innovation is a high order bit of course. We’re having to do a lot of innovation on supportive infrastructure to keep up with the pace. For instance, we’ve created an infrastructure that allows us to innovate very rapidly while maintaining high quality despite a huge and complex code base. On any given day, there are thousands of projects under development with over 5000 CPU cores producing 65K compilations of build targets, and over 100M executed test suites. The build and test system accommodates the load, and an annual growth rate of 75%, using distributed execution, caching, and work avoidance techniques. The average build and test cycle take just 4 minutes and we are quickly moving toward a model where developers can receive results before they formally check-in code changes (‘pre-submit checking’).

In our model, product teams maintain a build that’s always green (compiling and testing without error). We built the supporting system based on three priorities:

• Speed: All test and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data.
• Feedback: The focus of test systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not fixing code that is broken.
• Simplicity: Engineers should not have to understand how the underlying systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workflow that allows them to take appropriate action.

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

PC: We have a pretty open culture where engineers have a good degree of freedom to invest in areas that interest them. We have an idea called “20% time” where engineers are encourage to invest a portion of their time in projects outside of their primary area.

Does culture help to make better software? I think so, but it’s a tough thing to quantify. On occasion the idea of collecting and measuring “productivity metrics” has come up (ex. lines of code, # of check-ins). We uniformly resist this because similar metrics can result in undesirable outcomes as people try to “succeed” in the system. Our biggest measure of performance, besides the velocity of innovation being delivered to customers, is quarterly peer reviews. This type of peer system reinforces ideas like teamwork, making sure you are having an impact, and building respect from peers. It’s more subjective, but harder to “game” and, ultimately, much more valuable in reflecting the real value and contribution of an individual.

BTW, “Food-pellets” wouldn’t work because our engineers already have free access to gourmet food.

uTest: What’s the difference between a good tester and a great tester?

PC: Good testers can be trained. Among many others, the basics you need to be good are: depth in computer fundamentals, an understanding of the application domain, an strong understanding of the use case, empathy of the user, mastery of metrics, and a focus on the development process.

Great testers, meaning the 90th percentile, are rare and special. Not everyone can be truly great. In my experience great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. From the 100s of interviews I’ve done, “great” boils down to: 1) a special predisposition to finding problems and 2) a passion for testing to go along with that predisposition. In other words, they love testing and they are good at it. They also appreciate that the challenges of testing are, more often than not, equal or greater than the challenges of programming. A great “career” tester with the testing gene and the right attitude will always be able to find a job. They are gold.

uTest: What’s the greatest barrier to designing, developing and launching high-quality products on a consistent basis (deadlines, prioritizing, competing visions, etc)? Said different, why can’t every company launch world-class apps every time?

PC: Every product I’ve worked on required a unique approach. There are so many variables in play and most of them are situational. Some of them you can control almost completely (e.g. technology choices). Many others you can control somewhat. Others still are completely outside of your control (ex. what the competition decides to do).

Some companies try to normalize their development process. One thread common to formal development models are that they focus on a few of the many variables: improving efficiency, predictable process, estimation of quality, or others. Process-heavy development models may work well for manufacturing airplanes and have been successfully applied by some companies, they have been viewed by many developers as burdensome and contrary to the creative nature of writing innovative software. Conversely, “process-less process”, can lead to a heroic culture that’s unable to repeatedly deliver. There needs to be balance.

Consider the physics of flight as an analogy to software process. In addition to reasonable flying conditions and an experienced pilot, the key to getting airborne is having an appropriate balance of factors that match the situation: too much weight or too little thrust can be disastrous depending on the situation. Similarly, teams, products and process all have virtual physics. For instance, adding engineers late in a product cycle doesn’t necessarily provide more lift. Adopting a new process may give a team more thrust momentarily, but may also incur a longer term drag that makes them incapable of innovation.

The popularity of Agile, while not a wholesale rejection of more rigid processes, indicates that developers desire more balance and creativity. Whatever we do to make software higher quality and more predictable to build, we must maintain a balance that encourages the innovative aspects of the art form. We need to motivate smart minds to solve hard problems and deliver rich features to customers. In other words, we need to be adaptable to stay airborne for the long term.

uTest: What advice would you give to young software turks who aspire to become a senior tech exec at a global powerhouse?

PC: Here’s my advice:

1. Be technical: At Google all of our engineering execs have very strong technical backgrounds. They were programmers and many of them can – and do – still program today. Advice: remember your computer science and stay sharp.
2. Be a connector and innovator: One exec commented that his job could be described as being “a very expensive email router.” Get connecting within your company and establish a network of strong peers.
3. Drive your career: Don’t wait for someone to recognize you. Inform people about your plans and successes. You are your own best billboard. People will create a myth about you, like it or not. If you are smart you’ll help them create the myth you want them to have.
4. Play to your strengths: Pick projects that you are passionate about and where you can rock worlds.
5. Bite the hand that feeds you (a little): Don’t be a jerk, but make sure you don’t go passively along with bad ideas. A weak willed tester is a bad thing. Make your concerns known…and know when to fold.
6. Go where there’s opportunity for growth: Not all projects or companies can offer you an ability to grow. Move when you think you are topping out. The biggest mistake I’ve seen is staying too long in a place that hinders growth.

uTest: Google is booming in the mobile apps space; what QA testing challenges do mobile apps present that web or desktop apps do not?

PC: Where do I begin?

A good starting place is the huge diversity of devices. For instance, significant variations in HW and SW: OS, memory, processors, screen size, and input methods. Just having access to a fraction of the devices on the market for testing purposes is a challenge.

On top of that, if you want any automation, you have the problem of simulating input and capturing the actual output on the device. We needed to develop tools that allow up to test and build against multiple platforms. Normalizing the testing is tough. We debated record and playback vs writing abstraction layers vs other methods. All can be brittle in the extreme. We also needed to think about the long term cost of maintaining tests suites on fast moving SUTs.

A couple other issues are the carrier networks and the numerous languages. As I said, I could go on for a long time, but I believe I hit some of the major pain points. Even with all of the automation in the world, a community based test approach can be complimentary.

uTest: Is there a particular programming language to which you are loyal (PHP, Java, Ruby, etc)?

PC: Nope. Not really. At Google the most common ones are Java, JavaScript, Python, Perl, and C++. Perhaps a personal failing, but I still think in C and my code ends up looking kind of C-ish even if it’s in another language. My preference also depends on the thing I’m trying to do, obviously each has strengths and weaknesses.

On a similar topic, I recently met famed computer language innovator, Niklaus Wirth at the last GTAC conference in Zurich. He had in interesting comment on testing and quality, “The experience, judgment, and intuition of programmers who have survived the rigors of testing are what make programs of the present day useful, efficient, and correct.” In other words, you can’t test quality in and quality comes from experience.

uTest: Any new year’s resolutions for Google’s testing & QA efforts?

PC: Yes. Every year I write up a “one pager” describing what I think we need to do. I try to stay focused on general productivity and not specifically testing. Testing is one tool among many we use to make software high quality and delivery faster. We are doing well on many levels, but we aspire to do even better. Here are a few of the high level bullets:

• Greater release velocity. We apply appropriate use of automation, tools, process, and information that solve Google challenges. We introduce and use metrics that fit naturally into the product cycle. We are introducing key metrics that are actionable and representative of the health of the project that allow team members to understand issues and take action quickly.
• Cohesive tools. We’ve built an impressive set of tools that have made significant impact. In our next stage, we need to better integrate some of the data and tools and focus on accelerating the end to end developer workflow. This includes identifying bottle necks that slow people down and keeping an eye on innovative ideas coming from product teams. And of course, continual improvement of build and test speed.
• Driving improvement upstream. We will take advantage of our ability to make good ideas useful across the company. We use our broad view of projects to push adoption of ideas and methodologies that work. We look for ways to compliment existing product strategy by identifying changes where the adoption is in a project’s self interest. We constantly evaluate our own practices by looking at their effectiveness in relation to customer and business needs.

uTest: You went to school at University of Arizona, and USC; you’ve worked at Microsoft and Google; are you a lifetime west coaster or has it just worked out that way? Also, Dodgers or Giants (or D-backs)?

PC: Are there any good East Coast teams? I heard it was so foggy the other day that the Washington Nationals couldn’t even see who was beating them.

They may not be sports teams, but I’m a fan of the testing teams we have on the east coast: NYC “Bubble Sorts”, Boston “Red SOX”, and Pittsburgh “Open Sourcers”. BTW, we just hired a Test Director on the East Coast named John Turek who’s looking for a few good players.

BTW, I actually have lived in other places. My family lived in Maryland and Tallahassee for a bit. I also lived in Madagascar, which is definably on the East Coast (of Africa).

uTest: What sites, blogs or magazines do you read to stay ahead of the curve in the areas of business, technology, etc?

PC: Track and Field, Cosmo and, occasionally, Monster Truck Weekly . Seriously, I have a bunch of custom news feeds on certain topics, like Google, our competitors, and testing. I also read wsj on line and at least one interesting industry technical paper per quarter.

Interview of James Bach

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-james-bach-part-1/2009/12/)

In the December episode of our Testing the Limits series, we rapid fire some questions back and forth with James Bach (@jamesmarcusbach).  James is one of the most thought-provoking, outspoken, earnest thought leaders in the testing space.  Check out his blog if you don’t believe us.

Today we’ll be discussing James’ disdain for tester certifications, faking test projects, werewolf hunting in parallel universes and what he would do if he were king (or an angel) for a year. Don’t worry, it’ll all make sense soon. Update: Here’s Part 2 of the interview.

uTest: You’ve been an outspoken critic of traditional certs and classroom education. If you were king for a year, how would you fix testing certifications?  And how would you change a college’s curriculum?

JB: Kings are not powerful enough. I want to be an angel for a year.

You see, certification is promoted by frightened people who feel they need elaborate content-free ceremonies in order to feel competent. But in their hearts they know they are faking it. The fear of being exposed as imposters keeps them from doing much about it. So, in that year I would travel at relativistic speed around the industry. I would visit, by night, the hearts of testers everywhere, giving them inspiration to become excellent at their craft. The ones already certified would wake up and take a long cleansing shower, then write blog posts– by the thousands!– repudiating ISEB, ISTQB, CSQE, and all such blight. They would declare themselves reborn as students of the craft. (The ones not certified will just feel strangely cheerful, at least for testers.)

A spirit of exploration, experimentation, and debate would spread around the industry. It will seem to come from everywhere at once.

Weekend Testers would become Weekday Testers. TMap textbooks would be beaten into plowshares… and then recycled. Test plan templates and TPS reports would blow forgotten through streets lined with cheering crowds playing tester games designed to hone practical reasoning skill. By the thousands! FOR THE WIN!!

As far as university goes, I’ve already been doing my part. I helped found and run the Workshops on Training Software Testers, which brings university professors together to examine how to teach testing better.

I served on an advisory board for the Rochester Institute of Technology when they were trying to set up their degree program in software engineering, too.

But if I were king (not the modern Swedish kind but the old-school Caesar kind) I would make school a lot harder (much easier to expel a bad student) and instead of paying tuition, students would be paid.

Also, there would be no classes, as such, just constant projects and training. In other words, it would be almost exactly like Silicon Valley in the eighties, except with better corporate libraries.

uTest: If a parallel universe where you weren’t involved in testing or software at all – what would you be?

JB: If the parallel universe is before the industrial revolution, then any TWO of the following:

• A freelance sentry.
• A small-time warlord.
• An itinerant geometer.
• Werewolf.
• Werewolf hunter.
• A member of the 1735 French Geodetic expedition, but not the one who got killed by the mob at the bullfight (he had it coming).
• Zorro.
• A gentleman naturalist.
• A buccaneer.
• Gandalf.

uTest: A full day at an ISTQB seminar, or a full day in a college-level computing class – you’re forced to choose one. What’s it gonna be?

JB: Years ago, Rational hired me to advise them about test processes, and for that reason I took the Rational Unified Process class. I had been working with the RUP team, and I wanted to see what the Rational instructor was saying about RUP. At lunch on the first day, I asked him if it was okay that I was asking challenging questions. He said that was fine. But at lunch on the second day he kicked me out of the class!

Most instructors don’t like me in their classes. So, when you say I’m forced to choose, it’s a question of which class I would prefer to disrupt. (Unless I have the choice of sitting in one of Cem Kaner’s classes at Florida Tech. I would definitely prefer that. Last time I took a class from Cem, I had to play solitaire furiously in the back of the room. It was the only way to stop myself from jumping up and shouting “Yes!” every three minutes.)

I would prefer to disrupt an ISTQB class.

uTest: A few months back you presented “How to Fake a Test Project” at STPCon, where you said “testers may accidentally find bugs because they don’t follow the test scripts precisely.” Are testing managers starting to catch on to this irony of such “best practices”, or are most of them still oblivious to this?

I don’t know. I think most test managers don’t think much about it. As I said in my talk, testing is easy to fake– *even by accident*– because testing is so intangible.

JB: Be honest, you’ve faked a few test projects in your day, right? C’mon you can tell us.

Yes. I remember walking over to my manager’s office at Borland– by this time I’d been doing test management for five years– carrying an estimate of how much testing was required for my project. The estimate was couched partly in terms of “test cases required.” But just as I reached his office, I realized that the numbers meant nothing.

Until that moment, I thought they meant something. Well, they did mean something, just not anything I could defend or explain. Suddenly it dawned on me that I was peddling an empty *story* about testing couched in numbers to make it look rational.

As I write this, that incident happened 17 years, 5 months, and 1 day ago. Since advising my boss that he should ignore my test case estimates at that meeting, I have never used test case counts as a metric on a test project. But that doesn’t explain why, for the 5 years, 1 month, and 21 days before that, I *did* use test case counts.

Of course, the explanation is that I hadn’t much thought about it. I went along with the convention. So, I can forgive people who fake test projects by accident. When we’re young we make mistakes. But we also learn and grow.

uTest: In your new book, Secrets of a Buccaneer-Scholar, you advocate alternatives to traditional schooling and certification – not just for testers, but for people in all professions. It seems to have worked well for you, but why are so many people still reluctant to ignore the certificates and degrees and just start testing? What’s holding them back?

JB: Fear and indoctrination. I recently argued with a fellow about why he was going through a certificate program that bored him and insulted him, instead of quitting that school and being a tester. He claimed that no one would give him a job without the certification. Well, he emailed me a few days ago and told me he just got a job, and that I was right.

Well of course! I’ve been a hiring manager, and the idea that hiring managers are zombies who only look at paper credentials is wrong. SOME managers do that, but you don’t really want to work for someone like, do you?

Seems to me uTest is a mechanism for building your reputation as a rapid tester. I encourage new testers to try it for exactly that reason.

uTest: What’s the most promising trend/development in testing?  And what’s the most dangerous or discouraging trend you’ve seen?

JB: I think the most promising trend is public peer conferences and mentoring. Things like the Weekend Testers, for instance, or the Rat Pack. Some of these people are inspired by guys like me, perhaps, but they are not followers. They are taking it upon themselves to re-invent testing by experiencing and examining it intensively. I’m proud of anything I have done or can do to encourage this sort of creativity.

The most dangerous trend is certification, which is systematically dumbing down our craft and preventing good people from getting work.

A more dangerous thing, which is not a trend but unfortunately a constant, is the astonishing incompetence of technology middle and upper management, who frequently demand stupid metrics and meaningless reports from the people who work for them, or who decide to outsource without having the first clue what they are giving up by dumping knowledgeable testers and sending critical work ten time zones away.

For some reason, test managers complain to me about this, but often don’t complain to their own bosses.

Last year, my brother created a world-class test team at LexisNexis, then someone many levels up from him decided to fire everyone and outsource. They never talked to my brother about this. Apparently they think good testing happens by magic. But here’s the thing, my brother decided that he was going to train the outsourcing company (who had claimed to know a lot about testing, but, surprise, didn’t actually know much at all) to do skilled exploratory testing. It worked. He’s pleased. And now his management at LexisNexis (former management, because they then fired him) must think outsourcing is a great strategy, having no idea that it works only because people like my brother have too much personal pride to let even senior management airheads experience a richly deserved failure.

uTest: We’re all familiar with your work, as well as that of your colleagues like Michael Bolton, Cem Kaner, Jon Bach and others. Are there any “great unknowns” out there in the testing blogosphere to whom we should be paying attention?  Anyone that should be ignored altogether as a dangerous thinker?

JB: I sometimes complain about people I think are dangerous. See my blog.

Basically, I hate bullies. I want a free and open craft. The most dangerous people are those whose personal ambition will mean discouraging innovation (perhaps through establishing some sort of baloney “testing standard” through the auspices of ISO), and encouraging the sort of test management practices that make software both expensive and sucky, while discouraging any smart ambitious person from entering the field in the first place.

As far as great unknowns, there are some. I think Lanette Creamer is someone to watch. Adam White, Ajay Balamurugadas, and Ben Yaroch, as well. There are the Weekend Tester guys, and a shadowy Context-Driven sub-community calling itself “The Rat Pack.” I come across testers, regularly, who are talented, but haven’t taken a step out into public discourse.

I am constantly recruiting for colleagues.

uTest:  Do you see the quality of resources in the testing field increasing or decreasing (tools, training, certs, et al)?  What do you think are some of the drivers of that change?

JB: There are many good resources out there, and yes there are resources getting better. There’s testingeducation.org and the Weekend Testers project, to name two. At the same time there are terrible things out there (such as certification and all the stupidity that goes with that). You have to be a smart consumer, because it seems to me that the bad stuff has always outweighed the good stuff by an order of magnitude or so. Maybe by two orders of magnitude.

uTest: When it comes to automated checking, what are some of the key opportunities to employ it that generally generate a positive ROI? Are there any good rules of thumb that can be used, i.e. if you plan on executing the same test 7 times, then it is a candidate (understanding of course that some assumptions need to be made to answer this)?

JB: Here’s how I think of it:

- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.

- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.

- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?

- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.

- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.

- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.

- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.

- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.

- When I automate, I do it incrementally, in small bits.

I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check.

uTest:  What characteristics and practices make for a good tester?  How about a great tester?

JB: To be a good tester you must be curious about technology, and eager to learn it. You must be able to ask questions and make explanations. You must be skeptical, but you must have at least a little faith about one thing: the possibility of undiscovered trouble.

Anyone who wants to can rapidly develop themselves into a good tester, with or without any special training. The reason that so few people are good testers is they just don’t try. They don’t care to be good.

To be a great tester requires that you develop your mind into a disciplined instrument of analysis and observation. This is an ongoing life-long process of practice and refinement. Also, you need to learn how technology works on a rather intimate level (in order to gray-box risk analysis and to understand programmer chalk talks), you need to understand epistemology (in order to reason systematically when necessary) and cognitive psychology (in order to design tests with the limitations and capabilities of human perception in mind).

(Here’s something funny: There’s a blogger from Microsoft out there who actually wrote a blog post attacking the idea that testers need to learn epistemology. That may sound fine, except in order to prepare a rational argument against *anything* you need to know how logic works and how it relates to rhetoric. BUT THAT ITSELF IS EPISTEMOLOGY. Hence, this blogger from Microsoft merely demonstrated that he literally does not know what he is doing. His every word silently testified against his thesis. A much more powerful way to oppose my view that epistemology is important is to make a non-rational argument against it, such as by swinging at me with a bar stool. I could then at least respect him for being philosophically coherent.)

If you want to be a great tester, you need to set yourself testing problems and vigorously solve them. Even over-solve them. Critique yourself and encourage other to do the same. This is why I love doing coaching over Skype. Although I have too many students now, so I’ll have to start charging for that, soon, to make the volume manageable.

uTest:  You spoke recently at the Oredev conference.  What did you talk about?

JB: I got to speak about just what I wanted and I said exactly what I wanted to say… Except, I ran out of time on my testing efficiency talk and didn’t get to talk about simulated annealing and its relevance to exploratory software testing. Basically, simulated annealing demonstrates that the path to efficiency might well be to wander around randomly.

uTest:  When the most prominent testing minds get together, it seems there are often loud, heated disagreements – why is that?

JB: It’s not prominent minds causing this, it’s different cultures of testing. Also, you have a sampling bias: you notice heated disagreements more than the absence of them. Why don’t I get credit for all the times I *didn’t* argue with Boris Beizer under an escalator?

We have different cultures of testing. They are basically at war with each other. I wish the other guys would surrender and come into the light, but Rex, Stuart, Bernard, Dot, Lloyd, et al don’t take my advice.

The way not to have over-heated debate is to have A) agreement, or B) apathy, or C) a culture of professional pluralism. Professional pluralism means that even if you disagree with someone, you listen to, track, and respond to their point of view. You try to understand their ideas from their own context.

The Context-Driven School of testing was founded on the idea of pluralism. We are one school of testing thought among many.

But most other schools of testing that oppose the Context-Driven school don’t admit that they are schools. Each testing culture tends to think– not that they are the best, that’s not a problem– but that they are the ONLY way to think about testing. This makes for some strange confusion, such as Stuart Reid still telling people he thinks that my opposition to certification is staged for effect, rather than representing a serious and considered point of view. I’ve spent about 10 hours in public debate (including the three hours we spent at the pub) with the guy. He has a PhD. And he still seems to have no understanding or even apparently any memory of the arguments and evidence I put before him.

uTest:  How are the flying lessons going?  What appeals to you about flying?

JB: Yes, I learned to fly many years ago, and then forgot most of what I know. So my father is teaching me to fly again. It’s wonderful.

What appeals to me about it, other than the prospect of impressing Dad, is that I love the complexity and danger. I love that combination. When you fly power planes, lots of little skills must come together all at once, stick and rudder, altitude control, judging the weather, engine control, sharing the sky with other planes, radio, ground avoidance, navigation, and how to make the GPS unit to give you the correct radio frequency for the airspace you’re flying through while simultaneously not flying into a mountain.

Flying safely involves asking “what if…?” almost continuously, which appeals to my tester brain.

uTest: Which blogs and sites do you read for insights and learning?

JB: I read Sciencedaily.com, Wired.com, and Cracked.com. I also follow someone on Twitter (@ashalynd and @ashalynd_feed) who posts great links. I love Ted talks. I’ve been learning Javascript and CSS recently and love w3schools.com.

Finally, I’m addicted to Sporcle.com and Chess.com. Mostly, my colleagues alert me to what I should look at.

uTest: The testing of mobile apps is clearly at a different stage of maturity than testing web or desktop apps (in terms of tools, methods, the apps themselves) – how is mobile app testing the same and how is it different than the testing that’s been done for the last 10 years?

JB: That’s more a question for someone with direct personal experience.  But one thing I’ll say: it seems that automating checks is hard to do with those fancy multi-touch screens.

Interview of Jon Bach

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-jon-bach-part-i/2010/03/)

After Twitter-stalking him, making some harassing phone calls and sending threatening letters, Jon Bach (@jbtestpilot) cheerily agreed to take part in our Testing the Limits series. Much like his brother, Jon has a remarkable understanding of software testing – both in theory and in practice. Having worked for companies like Quardev, LexisNexis, HP and Microsoft, Jon is also a blogger, author and software testing consultant. An expert, in the truest sense of the term.

In the first installment of our two-part interview, we get Jon’s thoughts on sibling rivalry; the blame spiral of software development; the emergence of “agile-fall”;  testing at a startup vs. testing in the enterprise; John Schneider as Jon Bach and more.

uTest: A few months back, we asked your buddy Andy Muns who’d win a fight between you and your brother (this was a big debate in the uTest office). He said you would win hands down. Would he be right? And since you and your brother seem to share the same testing philosophy, what would do you think the fight would be about?

JB: It’s hard to fight with someone who stayed in their room for most of our childhood.  He was either reading or doing science experiments with a microscope or the chemistry set.  It got worse when we got the TRS-80 in 1980.  In fact, that’s probably the last time we fought — over who got computer time next.  My memory may be fuzzy, but just when it came to blows, he programmed a user name and password dialog? Something clever like that. Now it’s better just to learn from him and do my best to keep up — but that’s true for all younger brothers, I think.

As for modern-day fighting, sponsor me for a testing certification and let’s see what he’d do.

uTest: Say you’re named grand poobah of the QA universe… what’s your first decree?

JB: Effective today, “Quality Assurance” is now “Quality Assistance”.

(Try it.  Watch what happens when you start using it.)

uTest: When there are delays in development process, why does testing always seem to take the blame? Is their role just misunderstood? Or is it really the testing team’s fault?  How can companies avoid this seemingly never-ending spiral?

JB: Often, we’re saved until last.  It’s like the novel is written, the movie is produced, but the reviews aren’t in yet, leaving the creators in a heightened state of worry and concern. It’s a lot of stress to be in that position. As testers shine the light on the product, looking for risks and vulnerabilities, the light is really shining on *us*.  We’re being tested just as much as that software, so we tend to be sensitive and aware of being in a critical position as others wait for our findings.

Not being an Agile guy, I’m finding that the values and principles of Scrum, Agile, Lean, and XP may be compelling enough to try.  I’m getting more into that domain and I see merit in the values and principles it is trying to instill.  One of my favorite colleagues (Elisabeth Hendrickson) is coaching me in a gentle way and her stories are enlightening because she lives this stuff. Like me, she used to focus on exploratory testing, but catching up with her recently, she seems to be the whole package — developer, tester, manager, coach — and seems to be immune from that never-ending blame spiral you asked about.  That’s compelling to me.  Some may call her one of those ardent “Agilista” types, but to me, she hasn’t lost her testing spirit or soul.  Though she may agree there’s no silver bullet, she has some great experiences that convince me that with those approaches, the blame spiral is being made irrelevant because of the way testers are more involved, earlier.

uTest: In Half-Baked Ideas for Rapid Test Management (PDF) you used a term called “Agile-fall” (a combination of Agile and waterfall). This seems to be the methodology that most companies follow, yet they always call themselves agile. Is there any shame in being Agile-Fall? And did you coin this term? We couldn’t find it anywhere else.

JB: “Agile-fall” was something I heard at LexisNexis from an awesome PM named Lance Thomas, but in a Google talk in 2005, Elisabeth Hendrickson called it “Scrumfall”, so search on that term and you’ll see that it refers to having the principles associated with Agile (daily stand-ups, sprints, burndown charts, etc.), done in a waterfall-y series of development steps.  Example: Sprint 1: Gather requirements, Sprint 2: Design your tests, Sprint 3, Run those tests, Sprint 4, Fix bugs, Sprint 5 regress those bugs.  There’s no shame in that if that’s what works, and when you’re going through a transition from Waterfall to Agile, that may be the best thing as opposed to a sudden lever-pull one day where you show up and your desk is next to someone else with no walls and there’s a stack of sticky notes and markers on your chair with an email to report to your first standup in 30-minutes.

uTest: It looks like software bugs are partly to blame for the recent Toyota recall debacle. Is this the worst nightmare for testing managers? What else keeps them up at night?

JB: I almost got a Toyota last month, and wish I had, just so I could find a cool defect.  But my nightmares in testing are: “Why didn’t you catch that bug?” Though I know several answers to that, and my favorite colleague Michael Bolton has a rich list of answers to that question, they don’t often overcome the strong emotional attachment that stakeholders have.  And I can’t blame them, really.  The more we call ourselves “Quality Assurance” (like we can guarantee quality), the more they will lean on us to assure them.  I can’t.  But I can certainly *assist* with the notion of  whether something has value to its intended customer.

What keeps me awake is how to know if I’m bringing the right value to the project.  There are a million things I could do, but which ones have the most value right now?  I’m worried I would stumble into an idea too late, or not at all.  That’s why I like heuristics and mnemonics and checklists to kick my mind into gear when I feel stuck or worried.

uTest: What’s the biggest difference between working for a large, mature enterprise and a small young startup (from a testing point of view, that is)?

JB: I’ve worked at both.  At Quardev, which is neither startup or enterprise, we’ve got a mature-but-startup mentality.  Since the word “Quardev” comes from an amalgam of QUality Assistance, Research, and DEVelopment, the balance of those gives us enough diverse opportunities to make it feel like a big company.

From a testing point-of view, test ideas are king no matter what size your company. Some may say the toolset is king, or dev skills are king, but when I’ve worked for start-ups, you don’t need as many signatures on things, and that helps creativity. You might fail faster with those test ideas because you have to do it cheaply, and that’s good.  It’s like projects I’ve been on that use notions of Agile compared to those that don’t.  I see those kinds of projects revealing problems faster because of the way the culture organizes to deliver working software.

At a small startup, it’s often expected that you’ll be brilliant.  That pressure can make work not-so-fun.  Maybe that’s why some of my best ideas came from working in enterprise environments (Microsoft, HP, LexisNexis). In those places, there was a perception that every test idea, method, tactic, and strategy had been tried before.  That carried an unspoken dare (in my view) that begged to be challenged, which few people took on.  For me, brilliance and innovation come when someone counts me out or doesn’t expect it.  I like that.  I like exceeding expectations no matter how big or small the crew.

What sums it up is a TED talk by Ken Robinson, who said: “If you’re not prepared to be wrong, you’ll never come up with anything original.”

uTest: Who plays Jon Bach in the made-for-TV-movie about your career in testing? And what’s the title?

JB: John Schneider, only because it has to be called “The Duke of Hazards.” It also fits because I told my friends in school that I was related to Cousin Daisy (played by Catherine Bach). Not true, but it helped keep the bullies away.

uTest: You have some great tips on how to handle bloated testing numbers and statistics: “Any number, any statistic is like software. It can be tested.” What other tips can you give testers when it comes to having the courage, diplomacy and patience to slow things down and get to the truth?

JB:  For me, the magic words that often make me feel more courageous, diplomatic and patient are: “I have been fooled before.”

No one will argue with that because it’s true.  Scammers often confess that the hardest person to fool is somebody who says “I can be fooled.”  So many times I’ve been so sure I was right just to meet someone who convinced me differently, sometimes in a matter of seconds.  So now say “I could be wrong”, and use other safety language like: “it could be”, “it seems like”, “it looks as if” and “maybe…”  That way, I don’t feel stupid when I’m shown refuting evidence to my claim.  If you practice that, chances are good that you will appear to be a kung fu master who, after having floored your 50th assailant with your skills, slowly backs out of the room on guard for the 51st.

Remember that testing is a craft.  It involves thinking about how things might be different.  Remembering to say “I have been fooled before” is consistent with that spirit.

uTest:  Testing certs: worthwhile or window dressing?

JB:  The only thing worthwhile about them is the debate they provoke.  Window dressing is an apt metaphor because it’s only meant to enhance a window’s *appearance*.  When there’s a flood or a storm or some other strong test of the window, the dressing often gets destroyed. Outside of the flood, people may prefer the look of the dressing; I just want to be a stronger window.  Passing multiple choice tests about so-called “best practices” don’t do that for me.

uTest: What are some of the blogs/sites that you read to stay on top of the latest QA trends and news?

JB: I’m mostly on Twitter because it leads to links to testing videos, newsletters, news stories, trivia, thoughts, questions, blogs and tools that I haven’t seen before.  For example, yesterday I learned about Scott Berkun and his excellent technology blog, Ken Robinson’s fantastic TED talk on rethinking education, the interesting notion of a testing “playbook”, and the Ignite Conferences (a series of 5-minute talks).  Thank you, tweeps!

The top 10 I follow are: James, Jim Benson, Lanette Creamer, Michael Bolton, Scott Hanselman, Elisabeth Hendrickson, Adam Goucher, Paul Boal, Steve Smith, and yes, I have to say, uTest.   All of these sources turned me on to such a wide variety of sources to edify myself that it’s no longer just a handful of blogs to read now and then.  Blogwise, I have to say James (of course), Scott Berkun, Jim Benson, Joel Spolsky, and Lanette Creamer are most worth my time.

uTest: Testers come from a wide range of backgrounds (and have a wide range of skills) yet they are often talked about as if they were all the same. In your experience, what’s the biggest misconception or stereotype of testers?

JB: That we like to “break stuff.”

Nah, we’re detectives… we FIND stuff.  It’s a treasure hunt, not a smash factory.  In fact, our reports are “findings”, like a scientist or an investigator would call them.

One of the first lessons in my career was my favorite — that software comes to testers *already broken*.  I like that.  I didn’t write the code that broke it (unless I did, like a unit test or a keyword-driven script).

Humans are tested the same way.  When we go on a job interview, the interviewer doesn’t “break” us, they find weaknesses and vulnerabilities that are already there in our programming.  The good news is that we can test ourselves before going into the interview — to find and address issues beforehand so we can show them our best value.

uTest: What qualities, characteristics and experiences do you look for when hiring testers?

JB: At Quardev as a hiring test manager, it is these: cautious, critical, curious, friendly, diplomatic, honest, insightful, thoughtful.  I want candidates to tell me about a cool bug they found, or give me their best test idea.  I want them to make me think. I want them to inspire me and make *me* curious*, even though the thing I give them to test when they come in has been tested by over 500 different testers over the last 10 years.  I’m not naive enough to think I’ve seen it all, because one in 20 testers will find something new or will put ideas together in ways I hadn’t thought of.

uTest: What advice would you give to new testers who want to stand out from the crowd and become tomorrow’s testing leaders/gurus/pundits?

JB: Give a talk. Practice in front of a high school class. Take an idea or a strong opinion you have and actually learn more about it, then present it.  Do a lightning talk at the closest Ignite Conference to you.  Read a testing book and email the author your comments.  Comment on blogs and sign your real name.  Question the gurus and the pundits. Call them out if you think their work is crap. Talk about your failures.  Tell us your *experience*, no matter how amateur you think it may be.  Could be you just invented the next cool method to try. It was like that for me with Open-Book Testing when I was on Microsoft Flight Sim.  It was a half-baked idea I had and now it’s blossomed into something that I keep in my intellectual briefcase to show prospective clients one way to teach, guide, and evaluate test teams.

uTest: One of the hottest areas of growth that we’ve seen at uTest is in the area of mobile app testing.  What unique challenges does mobile present to a testing manager?

JB: Finding the right minutes/text plan?  Those really add up when you’re testing!

Actually, I think uTest is in a great position to address the biggest challenge I would face if I was managing a mobile project right now – configuration testing.  I would love to have 1,000 testers at my disposal doing a variety of risk-based exploratory charters on their devices with all kinds of third-party apps installed, making calls on different service plans, texting, taking pictures, playing video, Tweeting, browsing, and maybe doing all of those actions at once from places around the world.

By the way, I read Patrick Copeland’s answer when you asked him the same question and really think he summed up the challenges – diversity of devices, input-output simulations, carrier networks — things that you’d have to be very technical, very savvy, and very organized to test.  But on top of all that, there’s the challenge of if you actually found a bug.  With all of those variables I mentioned, where is the real bug?  You may be just reporting the failure, so you’d have to have some good debugging and diagnostic tools to help you be confident that your concern was represented to those who could (or would) take action on it.

uTest:  If Al Gore hadn’t invented the interwebs, what would you be doing today (answers related to software or technology not allowed!)?

JB: I’d likely be a best-selling author, have a full head of hair, and would have retired at 18 because I would have invented Twitter in 1986 instead of messing around on the Apple II for hours on end.  Before Twitter, before Windows, before the Apple II, our family had the “chain letter” – a packet of letters sent from one member of my family to another, adding new content with each mailing until all 6 of us had caught up with each other’s news in the circle.  When the oldest person got it, they would add new content and mail it to the youngest, and round it would go again, replacing old content with new.  It was a great system and I would have turned that into Twitter as a system of snail-mailed 3×5 cards to everyone on the planet.

uTest: Rapid-fire Jon Bach pop trivia – Last book read?  Last movie read?  Last concert attended?  Favorite sport and team?  Favorite band or album?  Browser of choice?  What kind of cell phone are you carrying?  Blu-ray or DVD?  Paper or plastic?

JB: Last book read? You mean cover-to-cover, not skimming?  Ugh.  Oh!  Last night to my three year old: “How do Dinosaurs Say Good Night?”, after which, I dove back into “Extreme Programming Explored.” Actually, I think I was over-tired last night and got it backwards, which is why I suspect she woke up this morning talking about refactoring, code smells, and collective code ownership and why I dreamt about being a Gallimimus.

Last movie read? The screenplay for “Butch Cassidy and the Sundance Kid” by Goldman.

Last movie seen: the unfortunate and annoying “Where the Wild Things Are”

Concert: Sting, 1989 in Portland, Maine;

Sport: Football (Seattle Seahawks);

Album: Rush, Subdivisions;

Browser: Firefox;

cell: iPhone 3G;

DVD;

Plastic, because at least they spice up the boring look of those landfills!

Interview of Scott Barber

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-scott-barber-part-i/2010/04/#more-5457)

Our Testing the Limits guest this month is testing guru Scott Barber, the Chief Technologist of PerfTestPlus. A speaker, writer, teacher and entrepreneur, Scott has one of the most impressive resumes in the business, particularly in the realm of customized testing methodologies, embedded systems testing, personal security systems and other topics – all of which are discussed on his blog.

In Part I of our 3-part interview, Scott discusses the Manifesto for Agile Development (almost ten years after it was created); the expectations of today’s testing managers; the notion of testers as an “unfortunate necessity”, the 1983 War Games movie and more.

uTest: As a signatory on the Manifesto for Agile Development, can you comment on the progress being made by software companies in upholding these principles? Have they exceeded your expectations, or is there still a long way to go?

SB: Honestly, I think the buzz around the “Agile movement” has, in many cases, taken the industry in an unfortunate direction. I meet far too many people and companies who are completely unfamiliar with the Agile Manifesto and think of Agile as a collection of practices, processes, and tools. The reality is that Agile is a far more a mindset and a culture than it is a collection of practices, processes and tools. Agile isn’t the best fit for every situation, or for every person.

I believe that the trend to “go Agile” is misguided. If a company is developing good software, the people involved in developing that software are happy working there, the software development is sustainable, and the business is being adequately served by that software, there’s really no need for them to try to be more or less Agile. Agile has challenges like any other culture, but the single biggest challenge I find is companies trying to solve development, process, management, and/or schedule problems by “going Agile.” Teams who have grown up in a culture that is fundamentally different than Agile simply will not find it easy to “go Agile.”

Think of it this way. Telling a team to “go Agile” between projects is like telling a professional baseball player to switch from being a right-handed hitter to being a left-handed hitter between seasons – and expecting that to break their hitting slump. You simply don’t see that. What you do see is batters working with coaches to re-evaluate their swing, work on their timing, spend more time in the weight room, etc.

Personally, I am happiest when I’m working in an Agile culture, but that is a personal preference, nothing more. What I do think would serve individuals and teams well, would be to read the Agile Manifesto and really think about whether or not those principles are a good fit for themselves, their team, and/or their organization. If so, I recommend embracing them.  If not, I recommend finding or developing an equivalent set of principles to build a culture around.

uTest: In a recent article, you highlight the critical point that “the perceived value of testing varies widely from individual to individual, team to team and company to company.” Can you further elaborate on how testing managers can do a better job of setting clear expectations?

SB: Consider these scenarios:

• Company A executives are looking for the testing to be their defense in case of failure and lawsuits
• Company B project managers want testing to say “ship it” so that if there are problems in production, they don’t get in trouble
• Company C developers want testing to show them the issues before management notices them and mandates changes to the development process… yet again.
• Company D is developing software for a client who expects testing to ensure their needs are met
• Company E just spent millions on a marketing campaign and is counting on testing to ensure that the company doesn’t get egg on its face when the campaign is released.

In each of these cases, the test manager is certainly the person who needs to lead the test team to adding the most value to the project. The challenge is that the test manager is often in the worst position to get straight answers.  Test managers are typically the lowest-level role in a company to hold the title “manager.” They don’t generally get to sit in the board meetings where defense against lawsuits are discussed, they don’t frequently have direct contact with the client, developers don’t often treat test managers as confidants, and who would tell a test manager that the purpose of his/her team is to take the blame if something goes wrong, even if it is completely out of their control?

How test managers, and testers, can do a better job is to take a step back and really consider what is going on in the company, what instructions they are being given, what questions they are being asked, and who seems happy or grumpy with the test results being provided.  Paying attention things like those will frequently make the “real story” pretty clear, and once the story starts to become clear, you can start asking smart questions that actually lead to pretty straight answers.

What doesn’t work is assuming that because you are the test manager, or the tester, that you *know* how your testing adds value to the company, that you *know* what the most important testing is, and that you *know* the managers and executives are idiots because of what they are telling you to test.  The fact is that even if your managers and executives are idiots, that doesn’t mean that they don’t have a sound and reasonable business reason to want what they are asking for. It’s not up to us to judge them for either being idiots or for making clumsy requests that don’t actually help them resolve their concern. It is up to us, however, to help them figure out what they actually need, help them figure out how to meet that need, and then guide our testing to best serve that need.

uTest: You have an impressive knack for drawing testing analogies from your lessons learned as an officer in the U.S. Army. What is the true value of outlining the situation, mission and intent of a software development/testing project?

SB: Thanks.  It’s not just the Army; it’s my life experience in general.  I suspect that has more to do with the fact that my parents were both educators and that I didn’t even know there was such a thing as software testing until I got hired to be a “Performance Engineer” and was introduced to the software test team on my way to my first client. So I pretty much had to figure it all out for myself with no real training in testing.

In this case, I was trying to figure out what my role really was. I clearly was not the decision-maker about whether or not to ship the product. I didn’t have the authority to tell the developers what should be fixed. So I applied the model I knew and asked myself “What is the situation here?”, “What is my mission?”, and “What is the overall intent of the client for the project as a whole?”

Over time, what I found was that the situation question helped me to understand things like where testing fit in the scheme of the project; who is panicked over what; how long I have to complete my tasks with what resources; who my allies are; what things are likely to block my success – that sort of thing. Much later, I learned that many testers refer to this as “context”, which I think is a very good word for it.

The mission question helped me figure out what was expected of me.  As it turned out, the folks I was reporting to often didn’t really know how to respond to my question of “What is my mission?”  They’d say things like “Well, test it and give me the results, of course.” My follow-up question of “Test it for what?” and “What kind of results?” didn’t help to clarify things either. So I learned to actually draft a little mission statement, like “Provide as much information as I can to management about issues I believe will cause lost clients or calls to tech support.” Sometimes someone would get very upset and change my mission statement, but that was fine by me because then I knew what their expectations were.

The intent question is what helped me stay focused on what really mattered.  The fact is that if the business we are working for fails, it doesn’t really matter how well tested a piece of software is.  No business pays testers for the sake of paying testers.  The truth is that most organizations feel that testers are an unfortunate necessity.  That truth makes it all the more important that our work add at least equal value to the business, as it costs the business to have us there in the first place.

It is at least as important as it is to have the situation, mission, and intent information individually, to look at those three items collectively. Often when I look at these three items collectively, what I find is that those three items contradict or work against one another. That is typically a good indication that either I have gotten things very confused, or it’s time for some stakeholders to have a meeting to figure out what is *really* important.

uTest: To our knowledge, no one has produced a movie entirely about software testers. However, there have been some notable software bugs in motion pictures (HAL from 2001: A Space Odyssey, the “Doomsday Machine” from Dr. Strangelove, the compound interest bug from Office Space, et al). If you had to suggest a movie that would inspire testers, what would it be and why? There are no wrong answers, except for The Net starring Sandra Bullock and Rocky 5.

SB: WarGames, 1983. All I have to say, is if you’ve built a computer capable of running WWIII simulations, and that is the same computer that controls the launch commands for nuclear warheads, and that computer is accessible via brute force hack of dialing phone numbers over a modem, you either didn’t have any testers or you didn’t listen to them.

uTest: There’s a lot of consensus among the Context-Driven School of Software Testing, but are there any fundamental differences? You can’t agree with guys like Michael Bolton and James Bach on everything, can you? Speak freely, your secrets are safe with us!

SB: I don’t know that there *is* a lot of consensus. I know that I am very much Context-Driven. In fact, my license plate says “CONTEXT”. That said, the buzz and counter-buzz surrounding the Context-Driven School of Software Testing is still causing as much harm as good.

I think the underlying problem is that the creation of software is widely treated as a software development activity, which may or may not benefit from related activities like testing. I find this thought process just plain silly. Software development, in my experience, is done for one of four reasons; altruism (minimal), education (some), research and development (slightly more than some), and/or financial/business advantage reasons (most). I think the fact that software is created for different reasons and purposes makes software development at least context-dependent by definition. The difference between being context-dependent and being context-driven is fundamentally the difference between being reactive vs. being proactive.

I think there never would have been a need to call out or name the Context-Driven School of Software Testing if testing hadn’t been so undervalued, misunderstood, and too often ignored entirely for so long that some entrepreneurial folks decided to step up and start talking about “the right way” to test software. While I’ll be one of the first to admit that many of these folks published “right ways” that were far superior to the state of the practice of the time, none of these “right ways” were right for everyone or every situation.

So, in an attempt to help people and organizations be aware that there are many ways to approach testing, and that the way that is “best” depends on a wide variety of factors, none of which involve which entrepreneurial person has the best “sales pitch” for their favorite approach.

Regardless of what the buzz may say, context-driven isn’t about Exploratory Testing, isn’t against test scripts, and doesn’t dictate who do what testing when. The way I talk about being driven by context is to regularly stop and ask yourself, “What can I do, right now that will be of most value to my team/project/company?” If you make decisions that way, instead of simply doing what someone who knows nothing about your situation recommends, I’d say that is at least the first step to being driven by context.

uTest: We’re always interested in how people got involved in software testing. What made you choose this path – what lit the spark?

SB:  The short story is that I didn’t choose software testing, it chose me.

I was dabbling in various technology and software related jobs after my time in the Army, when the company I was working for got bought by a big company who changed our compensation plan to something I couldn’t live with. While complaining to my friend, he said “Forget them, come work for us, we need a Performance Test Engineer.” In response, I asked “What’s that?”  He said “Don’t worry, you’ll like it.”

I started the job a few weeks later, and he was right, I did like it. I liked it so much, in fact, that I’m still happy being a tester over 10 years later. And to think, when I started that job I wasn’t even aware there were people who tested software for a living.

uTest: Follow-up question:  If you weren’t in testing, or anything related to software, what would you be doing right now?

SB: That’s hard to say. I went to college to become a Civil Engineer, and while I did very well in school and love the field, I was already too far removed from it by the time I left the Army to have a viable path back in. I’ve considered teaching secondary school or at a university, but there are aspects to those careers that I don’t think I would end up being very good at or enjoying. I’ve considered becoming the editor for a technical magazine, and becoming a conference coordinator for a technical conference. I’ve even considered going to work on the administrative side of a non-profit whose work inspires me.

My friends who know me well are always telling me that I should “retire into” sports announcing and/or coaching. I don’t know if that’s even vaguely possible, but I do know that there is nothing I seem to enjoy doing more than coaching and impressing people sitting around me by constantly pre-empting sports casters while watching a game.

uTest: Say we choose a software testing project at random. You don’t anything about it, except for the fact that there’s a major bottleneck somewhere in the process that is preventing quality software from getting released on-schedule. If you had to guess, what would the source of that bottleneck most likely be in a typical 2010 company? Management? Testers? Technology? Metrics? Give us your best educated guess and why.

SB: I believe it was in his book Secrets of Consulting that Jerry Weinburg stated “Every problem is a people problem.” Throughout my career, I have found this to be a great place to start. People make decisions, people devise and implement process, people write and test code, and people manage projects. If there is something preventing software from being released on time, it’s because someone made a mistake, a poor decision, an inaccurate estimate, or simply isn’t doing their job. It may or may not be relevant to figure out who did what that lead to the problem, but if you trace a problem far enough, eventually you end up at a person. The way I see it, people problems fall under management.

Add to this that quality education for managers and executives overseeing software development is at least as sparse as quality education for testers, and it makes “Management” a pretty high percentage place to start.  Of course, it’s also the most difficult to do anything about, so frequently organizations try to address their software development challenges by attacking symptoms rather than the cause.

uTest: Often times, testers will report bugs they deem “critical” (or high priority, P1, or whichever vernacular you choose) that are subsequently rejected by the client, product manager or dev lead as being ‘out of scope’ or ‘works as designed.’ What advice do you have for testers trying to make the case that a particular bug MUST be addressed before launch? Has this ever happened to you?

SB: It used to happen to me all the time, until one day I realized that I’m a tester, not an executive. My job is to provide information that my team, my employer, and whatever other stakeholders there are for my project want or need to know to make informed decisions. My job is to help them understand the state of the software and the possible implications of what I’ve found.

If, as a tester, you feel that a bug you have found is “extra important” to fix prior to launch, I recommend getting an audience with a decision maker and explaining the bug and its implications to items that matter to them (such as profitability, public relations, etc.) until you are convinced they can accurately represent your side of the argument to their peers and supervisors. After that you have two choices, as I see it, accept the decision that is made as being best for the business based on information that you’ll probably never have, or decide that your values and your company’s values are not in line and start looking for a job in a company with values closer to your own.

uTest: Favorite book that taught you an important testing lesson (even if the book wasn’t about testing per se)?  TV show?  Haiku?  Parable?  Off-Broadway play?  Shoot, just share any source of testing wisdom that has inspired you.

SB: General Systems Thinking by Jerry Weinberg, as I mentioned earlier. To be honest, it validated the way I have always thought about testing more than actually teaching me something new. That said, I’ve been teaching testers and other members of software development teams for almost 10 years and I can say with high degree of confidence that if all of those folks were reasonably skilled in systems thinking, software systems and their development, it would be in a much better place than it is today.

On a more pop-culture note, the television series House, M.D. is absolutely the best example of testing I’ve seen on a T.V. or movie screen. This is a show about diagnostic medicine, not software, but these people know testing. They understand the difference between positive and negative testing. They use scripted and exploratory testing. They aren’t satisfied with minimizing symptoms, they have an inner need to determine and eliminate root causes of unacceptable symptoms or behaviors. In fact, I’ve been influenced enough by the show that in one of my training classes, I have a segment titled “If Dr. House were a Performance Tester”, I’ve considered starting a blog under the same title, and I’ve seriously considered trying to popularize “Software Testing and Diagnostics Department” as a replacement for “Software Quality Assurance Department”.

uTest: It looks like you’ve hired your fair share of testers – what skills, experience and characteristics do you look for when making these critical hiring decisions?

SB: The most important things I look for in an individual are aptitude to learn whatever technologies, business information, processes, etc. that they will be expected to test – that and investigative curiosity.  Beyond that, I’m always looking to build a well-rounded and diverse testing team. I find that a team full of testers “like me” leads to us all missing the same kinds of bugs, so I’m always looking for a breadth of skills and experiences across the team. I’ve hired aircraft mechanics, medical billing specialists, call-center representatives, technical sales representatives, subject matter experts, developers… oh yeah, and even a few testers.

I know how to teach people technologies, the basics of a new programming language, how to report bugs, how to document process, and how to generate test ideas from whatever information we have.  What I don’t know how to teach people is how to be curious, the desire to investigate their curiosity, or to be excited by the thought of investigating things that strike them as odd.

So I guess I’m always on the lookout for someone with a tester’s attitude who fills a gap in perspective of the current team whose own knowledge or skill gaps fall in areas I know how to teach them or get them trained in.

uTest: Hypothetical: All testers are barred from practicing their craft for an entire year. What does the software industry – nay, the modern world – look like without their services? Please be as dramatic and hyperbolic as possible.

SB: Sadly, I think most of the world wouldn’t even notice. I think for the period of a year, non-testers would pick up enough of the slack to keep things moving, and enough individuals would perform enough “heroics” to keep the software world running. Sure, more bugs would make it into production, more software would ship late, and there would be more evening news reports of software failures, but I’m not sure the general public would even notice.

In fact, if software producers were smart, they’d pass some of the savings they were realizing by not having to pay testers on to the general public, making them even less likely to mind a few extra bugs.

But eventually, some kind of tipping point would be reached and the public would demand less buggy software. That is when the disaster would actually happen.  By the time companies re-engaged with testers, re-integrated them into their new (clearly, more streamlined, process), started getting results, realizing just how much re-work needed to be done and how long it would take to get software back up to the “old” standard, the public would be so furious that the “old” standard would no longer suffice.  Of course, achieving the new, higher, quality standard would take more time, and make the software cost more, and thus make the public demand yet *more* for their money.

Interestingly, if the software industry managed to survive for long enough to find a balance point at the tail of this downward, then upward spiral in software quality, I think it would be the best thing that could happen for testers. The publicity would lead to higher quality training for testers, more education about testing for developers and managers in school and on the job, better hiring methods and practices, and more respect for testing as a career. As the public became aware that software doesn’t have to be bug-infested (sure, all software will always have some bugs, but a couple of lady bugs are far more acceptable than a cockroach farm) more buyers will become more willing to spend a little extra for better quality, leading to more jobs for skilled testers, leading to more people wanting to become skilled testers, which would increase enrollment into education programs, leading to more research and development, and on, and on.

Maybe this isn’t exactly how it would go, but I do believe that it would take far too long for the public to notice and far too long for companies to start losing more money than they save by not paying testers. I also believe that unless something like this does happen, the public may never realize that they have been conditioned to accept far lower quality from software than they do from non-software items of equivalent cost. Think about it, how many times in the last year has a software bug ruined your day?  If the building materials used to construct your home were of the same quality of the software you use, how would you feel about your home?  What would you think if you had to patch a wall every time you patched your software?

Testers can’t solve the quality problem on our own. Higher quality has to be driven by conscientious business owners or disgruntled buyers. The best testers can do is try to make those executives as aware as we can of those bugs we can find while operating with not enough training, not enough time, not enough people, not enough equipment, and not enough support.

uTest: Are great testers born or built?  And if they’re built, then how the heck does that happen?  Share a little wisdom with the newbie testers and college kids who will join the ranks in the next year.

SB: I think that the potential to be a great tester is a part of a person’s personality that more or less exists or doesn’t by the time that person reaches college or the job market, but I don’t think that potential alone is enough to make someone a truly great tester.  Becoming a great tester is like becoming a great anything-else. It takes some natural ability, some opportunity, some good coaching, some relevant experiences to draw from, a lot of self-motivation, and a lot of practice.

Every now and again, someone might have so much of some of these items that they can get away with less of the others, but that is the exception. If you want to be a great tester, just like if you want to be a great musician, writer, or athlete, practice all you can, find a good coach or mentor, learn everything you can about technologies, designing experiments, systems, psychology, human interfaces, anything you can imagine being related to what you might end up testing or might help you better understand what might matter to the people who use what you might end up testing.

When you think you’re getting really good at something, start blogging it, then teaching it to others.  Don’t be discouraged when you find that it doesn’t really work for them at first.  Keep refining and revising until you believe those you are teaching are at least as good as you thought you were when you first started trying to teach them. When your students reach that level, pick the thing you are next best at and do the same thing.  You’ll know you’ve become a “great tester” when your current students have been referred by your former students, and your former students are trying to hire you instead of just trying to pick your brain.

uTest: [In the voice of deep, gravel-voiced movie preview guy] The year is 2020.  And in a world gone mad… where only one man knows what really happened… only one man can face the danger, beat the odds and reveal the truth… (insert pyrotechnic explosion here).  Okay, I don’t know precisely where I’m going with that, but what do you think testing look like in 2020?  What needs to change ASAP in the world of software design, development or testing in order to ensure a future of safe, reliable apps?

SB: The first thing that needs to happen is that testers, developers, managers, and testers need to learn what a “test” is.  Sounds silly, right?  It’s not.  Almost every day I encounter someone who wants to know “What test can I conduct to prove that…?”  While the answer (“There is no single test to prove that…”) it demonstrates a fundamental misunderstanding of what a test is.

Think about all the tests you have taken during your days as a student.  Do you believe that any of them proved that you had learned what the instructor had intended?  How about your driver’s test?  Do you believe that passing your driver’s test proved that you were going to be a good driver?  What about the last time you or someone you knew had a blood test?  Did you believe that blood test was going to prove the person was healthy?  Be honest now.

Of course you didn’t.  But somehow, most folks seem to think that software tests can prove that something will work in production.  The best we’re ever going to be able to do is demonstrate ways in which the software already isn’t working, demonstrate that the software doesn’t currently meet some measure of “goodness”, demonstrate that it is possible that the software may work in production, and/or collect information of potential value to someone who matters.  Even that depends on our tests being reliable and accurate.

Before testing can truly advance, folks need to recognize that testing is an application of the scientific method, that testing involves experimental design, and that tests don’t pass or fail, they provide useful information or they don’t.

Most of what happens today under the label of testing is nothing more than simple validation of expectations.  Testing is a search for information.  Testing is most valuable when expectations are not met.

Don’t get me wrong.  Someone had better be validating that expectations are being met or no one is going to want the software we’re developing, no matter how much we test it, or how few bugs are in it.  But we can’t stop there.  Technologies evolve too fast, users do things the most creative of us would never imagine before our software is retired for us to think that validation in some non-production environment is enough.

10 years ago, I heard a lot of managers and executives saying “What do you need testers for?  If you’d just develop it right in the first place, they’d have nothing to find.”  I don’t hear that very often anymore.  Today, executives understand that is an unreasonable sentiment.  Now they seem to think that testing can prove and/or control quality.  Maybe, 10 years from now, they will have figured out how to get the most value out of testing by asking them to actually test for the purposes of helping developers build better software and helping executives make better decisions.  I hope that is the case, but if there isn’t a change in the way “a test” is perceived, I don’t think it’s very likely.

uTest: We’ve been fortunate enough to interview the likes of Whittaker, Heusser, Bach, Bolton and Bach in the past for our ‘Testing The Limits’ series… who should we interview next?  Give us a few names who would continue this great (albeit young) tradition and blow our minds?

SB:

Rob Sabourin
Mike Kelly
Cem Kaner
Elizabeth Hendrickson

Interview of Michael Bolton

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-michael-bolton-part-i/2010/01/)

We’re thrilled to have Michael Bolton as the latest victim of our Testing the Limits series. As the founder of DevelopSense, Michael has traveled the world teaching the craft of software testing to businesses and individuals alike. Since 2005, he has specialized in courses on Rapid Software Testing – which he co-authored with James Bach. Michael is also a prolific writer, and his publications include hundreds of articles, essays and columns. Aside from his blog, you can keep tabs on his latest work through Twitter.

In Part I of the “trilogy” we discuss the Weekend Testers, testing abroad, how numbers can enslave managers, and of course, his pop-star namesake.

uTest: You’ve been a thought leader in the testing space for a while now, but people still seem to get you confused with Michael Bolton (the singer) on Twitter. Ever thought about creating a tester alias? Or have you considered asking him to change his name since “he’s the one that sucks.” Assuming you (and our readers) have seen Office Space, I bet this joke never gets old.

MB: Yeah, it never gets old.  Try renting a car with this name.

A couple of things on that. First, Office Space captures very well what it’s like to have my name. Second, it’s not his real name; he changed it already. Way back when, before Office Space, I was working in tech support at Quarterdeck Canada.  American callers would occasionally turn north when there were long phone queues in Santa Monica. On one call, when I introduced myself to the customer, he laughed. “Really? That’s your real name?” “Yes, really,” I said, expecting one of the usual jokes. He said, “You know, it isn’t his real name. I used to be his bass player.”  The singer’s real name is Bolotin, but according to the bass player, there was no hope that radio DJs would ever pronounce “Bolotin” right, so he changed it.

uTest: We recently interviewed your friend and colleague James Bach, who had high praise for a group called the Weekend Testers. Can you give our readers a quick recap of what this group does, and whether or not you’re on board with their testing philosophy?

MB: James hadn’t heard about Weekend Testers (or maybe he had heard, but it hadn’t registered) until I raved about them to him. When I was in India in November 2009, I attended a talk by Ajay Balamurugadas, who told an amazing story. They’re a bunch of fairly young testers from Bangalore. The core organizers (Ajay plus Sharath Byregowda, Manoj Nair, and Parimala Shankaraiah) are students of our colleague Pradeep Soundararajan. They gather online every week to spend an hour or so testing some freeware or some Web service, and then they spend an hour or so debriefing each other.

There’s a weekly leader who sets a mission for the session. Sometimes the focus is on finding bugs quickly; sometimes it’s on choosing a particular approach, sometimes it’s on note taking. In the talk, Ajay described testers taking responsibility for their own skills development; testers building a self-critical community; testers overcoming obstacles like power cuts and buggy messaging tools; testers ignoring meaningless certifications; testers providing service to the open source development community; testers discovering testing for themselves, owning it.

At the end of the presentation, I whooped, stood up and applauded like I was at a rock concert. At Q & A time, I raised my hand to say that I didn’t have a question, but that this was the most exciting talk on testing I had seen in years, maybe ever.  At the time of the presentation, there were already chapters forming in other cities in India.  At the beginning of this year, Markus Gartner and Anna Baik announced a European chapter.  It’s spreading!

Am I on board with their philosophy?  Hell yeah!  These people, and people like them, are the future of skilled testing.

uTest: It looks like you’ve been “testing abroad” for quite some time now. What’s the biggest thing you’ve learned about testing in locales other than the US and Canada (your two primary residences)?

MB: Scandinavia—Sweden in particular—and New Zealand seem to be percolating more excellent testing than other places.  Meanwhile, I observe that many Western firms—mostly the Americans—are making life difficult for testers in India and other developing countries. These firms didn’t know much about testing to begin with. They viewed it as a rote, clerical activity, piecework, checking work, commodity work that delivered little value. They knew how to do checking, sort of, very slowly and very expensively. But they didn’t know how to do testing, or how to increase its value, so they focused on cost and outsourced it.

The developing countries have millions of intelligent people who want to develop skills, but the West is still requiring these overprescribed, expensive, low-value, confirmatory approaches in which smart human testers are being asked to behave like slow, dumb machines. Confirmatory tests do find problems, but to a great degree, programmers should be pairing and low-level automated tests to squash those problems before testers ever see them. Then free up the testers to look for higher-level problems and previously unanticipated risks.

uTest: Your testing philosophy seems to draw a lot of heat from some circles. For instance, a lot of people seem to think you’re an anti-numbers guy (as you mentioned recently on our blog). What is it these people are so opposed to? Or are they simply misinterpreting your approach to testing?

MB: Excellent testing, the way we teach it, involves thinking critically about the product and developing a story about it. In order to do that expertly, we have to think critically about how we’re getting that story and how we’re telling it in ways that are important to our clients. Testing is about trying to make sure that our clients aren’t being fooled. If we’re fooling  ourselves, we’re likely miss important problems.

When I object to people using numbers badly or using numbers excessively, some people perceive that I object to using numbers at all – not so. It’s just that I love numbers so much that I hate to see the poor things abused.

Some people enslave numbers.  They make numbers work too hard, and too often. I’ve seen organizations collect piles of data about defect escape ratios and defect detection percentages. They hire market research firms and calculate the ratio of happy customers to unhappy customers. But the aggregated data doesn’t tell you anything specific on how to make things better for the unhappy customers.

When you enslave numbers, they eventually rise up, revolt, and enslave you. These organizations spend so much time collecting the data and talking about it and justifying it and trying to duck blame that they don’t seem to have time to do anything about the actual problems, which generally fall into two categories. One:  the organizations are trying to do more work than they can handle with the approaches they’re using. Two:  they’re not listening to people that matter—neither to their customers, nor to their own front-line staff, many of whom are closest to the customers. VPs could learn a ton of useful business information from their own customer service and technical support reps, and they could learn plenty about the project by listening to their programmers and their testers. Product and project knowledge gets mediated by middle managers and numbers; it turns from information into data. When your car is about to go off a cliff, it’s a weird time to be thinking about gas mileage and drag coefficients; better to take the right control action—look out the window and steer or use the brake until you’re back on course. Once you’re back to being productive, then you can start thinking about optimizing, if you think you need to. I wrote about that here and here .

Besides, people don’t decide things based on numbers anyway. They decide based on how they feel about the numbers.  On their own, numbers don’t tell you what is or isn’t okay. Your judgment does that. Your judgment is governed by the synthesis of observations and inferences and facts and feelings, in your head and in your gut.  Effective decisions require both head and gut. Each can mislead the other, which ends up with someone being fooled, or worse. Neuroscientists and psychologists and economists (people like Antonio Damasio, Dan Ariely, Daniel Kahneman and the late Amos Tversky, Daniel Gilbert, Gerd Gigerenzer, Daniel Goleman) have been looking into each others’ fields to discover fascinating stuff about the links between emotion and intelligence. (If you want to do research in that area, apparently it helps to be named Dan.)

Some people seem genuinely scared by the human issues and the uncertainty that’s inherent in software development.  They want testing to be this simple technical problem, a confirmatory thing. Questions related to exploration and discovery and investigation and learning emphasize the fact that we don’t know everything from the outset when we’re dealing with complicated systems. We can’t. We prepare what we think are the requirements, but we often don’t understand what we want until we can compare it to something we’ve got. We have to live in this messy, uncertain, continuously changing world that we can’t control very well.

In that kind of world, repeatability isn’t that big a deal; computers are good at that. The big deal is adaptability; can our software help people to solve their problems, not only reliably but also flexibly?  Designing software to do that, putting all the design work up front, is difficult—I’d say impossible. But we can do a bit of focused work, gather information (that is, test), and then tune things where they need tuning, add things where they need to be added, drop them where they’re not needed any more. Then we repeat the cycle often, with lots of little variations.  That’s what Agile development is about. That’s what evolution is all about, really. In that kind of world, testing is an important service, and the kind of testing that James and I teach is designed to make that service sophisticated and powerful and fast. I think the need for testing annoys people who think everything should be known in advance. Alas, I believe that there are lots of people, even many testers, who think like that.

uTest: There’s a lot of passion amongst testing thought leaders about the best way to test, or the best way to manage or train testers.  Often that passion overflows into heated debates.  How can this passion best be channeled to improve the state of testing?

MB: First of all, we should welcome debate, and we should welcome skilled argumentation as part of the art of construction and practice of persuasion. I’ve found, though, that it helps to remember that we’re exploring and challenging ideas. That means it’s good not to get too personally invested in certain ideas, because we’re always learning more, and because changes in context can mean big changes in what needs to be done.

That said, there are some ideas that seem robust for me. I believe that it’s unethical to dumb down people or the work that they do. I believe that we should focus our craft on learning, and learning how to learn rapidly. How can we improve the state of testing? By recognizing that software development is a web of people who are related in service to each other. That means putting people and social issues first. Get that right, and everything else will follow.

Suggestions are cool.  Standards are something else.  No group should be dictating to other people how they must test unless there are compelling human health and safety reasons for it. Do you really believe that the standards people know anything useful about your business? That the force of government-supported regulation, created by busybodies, should weigh on how you do your daily work? And if your answer is No, what are you going to do to get it stopped?

I talked to one fellow at a conference who said that managers would have no way to filter candidates without certifications.  “No way?” I asked  “Really?”  “Well, what other way is there?” he asked. “What other ways can you think of?” I asked him. He was stumped.  He didn’t seem to know about personal references, well-crafted resumes, networking, interviewing, auditioning, hiring from within and training people.  He did know about those things, but he was so fixated on certification that he couldn’t come up with a single one of them on the spot. Now:  do you really want someone like that, with that kind of limited mindset, setting the standards for how you test your product? Big money and people’s livelihoods are riding on your answer.

uTest: We sat in on your STPCon presentation in October, where you said “Pay attention to your emotions while testing. If something strikes you as being off, then chances are it probably is.” How did you come up with this piece of advice? And can you give testers a brief real-world example?

MB: I fly a lot, and so I often have to book travel using buggy airline Web sites. A couple of years ago, I had to book a flight to Europe. In the course of five minutes, I went through surprise (the site kept revising the dates I had chosen), confusion (why was it doing that?), frustration (the airline’s own Web site didn’t offer flights directly from Toronto to Amsterdam, even I knew though travel agency sites and personal experience that there were two such flights a day), impatience (why is this so hard and taking so long, for crying out loud?) and annoyance (I don’t liked being stymied and having my time wasted). I wrote an article about the experience; you can find it here.

We all want to do testing better and faster, right? Tests of the business rules and the calculations are really important—no question about that—but emotional responses are important too, and they have the advantage of being immediate, both in the “right now” sense and the “no intermediary” sense; they’re direct experience.  Our customers will have emotional responses to their experiences too, and you never get a second chance to make a first impression.

In the Rapid Testing course that James Bach and I write and teach, we say that a bug is “something that bugs somebody who matters”.  People, including testers, are bugged by stuff that doesn’t work.  Being bugged is a feeling that starts with an emotional reaction.  The feeling is a signal.  It’s data, but not information; you still need to interpret the signal’s meaning and significance.  So instead of saying “If something strikes you as being off, then chances are it probably is,” I’d prefer not to be too certain about that.  Look into what you’re feeling, for sure, but consider the possibility that something else is the source of the feeling.  Maybe what you’re seeing is okay, but you need to learn something about the business rules.  Maybe it’s not consistent with an older version of the product, but the new way of doing things is really better and you’re just not used to it. Maybe it feels like a big deal for you, but it’s okay for your client or your end users.

We teach people to pay attention not only to the product, but also to their testing. If you’re bored while testing, use that feeling as a trigger heuristic to prompt a question:  why am I bored? Is it because my testing isn’t revealing anything new? Is it because I’m doing repetitive and un-engaging work that I could delegate to a machine? Is it because this is useless work that I shouldn’t bother with at all? Am I missing the point or paying attention to the wrong things?  If you’re uncomfortable for some reason—if you feel like you’re being pressured, for example—there’s a possibility that you’ll be distracted, or that you’ll rush, or that you’ll be biased towards noticing some kinds of problem at the expense of others. But maybe the pressure is telling you something about the significance of risk, or maybe it’ll remind you to work rapidly, or something else that’s good in your context.

In Jerry Weinberg’s Quality Software Management Vol. 1, Systems Thinking, he points out that decisions about quality are always political and emotional, but people want to appear to be rational. In fact, that in itself causes an emotional reaction in people who want to preserve their own illusions that they’re rational. At a conference recently, a tester approached me and said that she had been intrigued by the role of emotion in software development. She mentioned this to her boss. He replied, “There’s no room for emotion in software development.”  She said, “Really?  No room at all?” He got very agitated, and repeated, shouting this time, “THERE’S NO ROOM FOR EMOTION IN SOFTWARE DEVELOPMENT!” She told me he was quite serious. Naturally, we laughed about that. Our emotional reaction was a pointer to the incongruence between what he was claiming and how he was feeling.

uTest: What are the greatest threats right now to the software testing discipline?  What are the greatest hopes for a brighter testing future?

MB: The greatest threat at the moment, I think, is a systemic set of misconceptions about testing, especially at the middle management level. We don’t do quality assurance; people with the power to make decisions about the nature of the product and the project do that. We call those people managers. Software testing is not like manufacturing quality assurance, where we’re trying to make a zillion identical copies of something, and then checking to make sure that each one is just like the prototype. Every software development project is an attempt to build something new—otherwise we’d use a product that had been built already. So software testing has more in common with design quality assurance, where we’re trying to figure out the relationships between a product and its stakeholders, and trying to understand the product as we’re designing and building the first instance of it. But even then, we don’t assure quality; we question it on behalf of our clients. The people who are building and managing the product assure quality.

When we model software testing on manufacturing quality assurance, we emphasize checking at the expense of testing.  I’ve written a lot on my blog about the distinction between the two since August of 2009.  (You can see it at http://www.developsense.com/blog.shtml). Checking is important, but it’s mostly a confirmatory activity, verifying what we knew to be true before, or what we hope is still true, making sure that we’re getting the right answers. Testing is a bigger deal, focused on finding new information, identifying new risks, asking new questions.

Lots of people—managers, programmers, and even many testers—conflate testing and checking. Again, checking is really important, and it’s important to do it skillfully.  In particular, I think it’s important to automate it skillfully.  But when people confuse testing and checking, they don’t think clearly about cost, value and risk, and they tend to dumb testing down.  That’s a big problem.

uTest: Looking back over your testing career, you’ve seen a lot of changes to tools and techniques and trends.  What do you think software testing will look like in 2020?

MB: 2020 the year, or 20/20 the vision? I predict that, in 2020, there will have been a lot of changes to tools, techniques, and trends, and that someone will ask a tester to predict what testing will be like in 2030.

At this point in history, no sensible person should make specific predictions about technology beyond dinner time; read The Black Swan for more about that. But as for testing, the fundamentals will still be the same. People will want stuff, and programmers and engineers will design it. There will be miscommunication, misunderstanding, unexpected incompatibilities, delays, mistakes, emergent properties. So we’ll still need to investigate requirements and designs and products and problems skillfully.

It’s like writing. After a few thousand years of development on paper, we’ve got people writing for print and online media and television and radio. People’s work can be prepared and consumed anywhere in the world, 24/7, in any number of languages. But we still have people to manage the writers, and we still have people to help the writers—editors and research assistants and designers and so forth.

uTest: Hypothetical: You’ve been banished from testing – nay, ALL software-related activities – for the rest of your days. What will you to earn a living?  What hobbies would you pick up to fill the intellectual void?

MB: Who knows?  For fun, I’d keep playing mandolin, probably. Teach, maybe. Write. I’ve worked in theatre stage management, been a book-keeper, tended bar, worked in a comedy club. In high school I worked in mail rooms during the summer. Whatever I’ve picked up in life, it was because something needed to be done and I was there to do it.  If it didn’t seem like much at first, I started to learn about it quickly. When you invest a little bit of effort to figure out your job, you learn how to makes it faster and better and more interesting. It turns into this great feedback loop. Any job can be more fun when you set out to master it.

uTest: Tell our testing community something about you that your most avid readers don’t know.

MB: While walking through the woods on an island near Vancouver recently, I found myself being quiet and brief, which I like from time to time. Practically nobody knows that.

Lots of people probably don’t know how much I’m eager to help people out. All of my work—courses, articles, conference presentations, this interview—comes with lifetime free technical support. Have a question? Just ask. I might not answer right away—supporting the family with paying work takes precedence over supporting the community—but I’ve never knowingly turned anybody down, so if I don’t answer right away, be persistent. James Bach makes the same offer, by the way. We’ve found that it’s a great way not only to help people, but also to explore problems and come up with solutions and learn things that can help our clients.

uTest: If you were talking to a newbie tester, what advice would you give him or her to set their professional journey off on the right foot?  How about for a 10-year veteran tester?

MB: To both: learn, and don’t stop learning. Practice in your work, but also on open source projects, and in your life generally. Study testing, but note that the great ideas are likely to come from outside the field—at least the narrow vision of the field that the process enthusiasts and “certificationists” present. Testing is a marvelously interdisciplinary craft.  One implication is that whatever you bring to the table from your life and your experience and your education can inform new ideas about testing.  Join or start communities of skilled testers in your area. Join online groups; I like the Software Testing Mailing List and the Software Testing Club particularly. Learn about your craft and build your reputation by asking questions and seeking answers. As with testing itself, the really excellent work starts with asking questions.

uTest: In your opinion, what characteristics, skills or experience separate a good tester from a great tester?  How about in a testing manager?

MB: My colleagues and I have been talking about skills of great testers lately.  (In term of public postings, James and Jon Bach have podcast a nice pair of conversations which you can find here. James also did a CodingQA podcast here. One key skill is the ability to identify context quickly, which helps to avoid all kinds of pathologies—wasting time, failing understand the mission, testing the wrong stuff, investigating unimportant risks, and so forth.  Get the context right, and you’ve got a better shot at doing excellent testing. We teach people to consider and ask questions about the test lab, the test team, the development model and the programmers, the requirements, and the mission to help with that.

Another skill—which we also teach—is something that James has named test framing, which is the same kind of thing in the other direction. It’s the high-level skill—a skill set, really—associated with linking the context, your motivation, the actions you perform, the observations you make, the inferences you draw, and the story you tell about all that stuff.  It’s the ability to follow and describe the threads of logic forward and backward, from the mission to the outcome of the test and back again. It’s the ability to competently address and answer the questions “How, specifically, does this test relate to your mission?” as you prepare to test and “How does the outcome fulfill your mission?” after you’re done.

We’ve seen some good testers whom we can’t quite call great, because they have a hard time with some aspect of that skill set. They seem to lose the ability to link premises and conclusions at some point.

I think there’s a strong relationship to the skills of a great test manager there. A great test manager needs to be able to do the same kind of framing with respect to the test project and the test strategy. A manager that can frame the project can negotiate for people and resources more effectively, can train and deploy her people more efficiently, can explain the work of her team, can defend herself skillfully, can adapt to rapidly changing circumstances. That’s really important, because most projects are messy and confusing. If they weren’t we wouldn’t need testers.

Here’s an example: Test managers get asked questions like “When is testing going to be done?” That question is hard for some test managers because it’s not the test manager’s job to make the decision. They feel pressured to provide an answer, but they can’t frame the problem. A great test manager can deal with that pressure. First: Problems in a product get introduced almost entirely by people other than the testers. Second: Neither those people nor the testers know what problems are there, or where they are. Third: Testing is done in cycles that reveal as-yet unknown information about problems that may or may not need to be fixed. Fourth: The decision to fix the problems depends on the programmers and on the product owner. Fifth: Testing isn’t done until the problems in the product are fixed to the product owner’s satisfaction.

If those things are true (and they are), the amount of investigation and reporting and bug verification testers have to do is unpredictable until you’ve actually tested the product, made decisions about what’s to be fixed, and fixed the problems. Well… it’s predictable, but the prediction doesn’t have any real validity, because there are too many variables that could throw the whole prediction off. “Accurate” predictions in this business come mostly self-fulfilling prophecy—and from ignoring changes in scope or staffing or schedules or budgets or tools that just happened to make the prediction come true. If you set only a couple of data points for your targets, all the other stuff can change without people noticing much.

So testing cycles can be scheduled, but testing cannot be done until the last problem deemed important by the product owner is fixed, by the programmers, to his satisfaction. The great test manager can help the product owner to identify and decide upon what information he needs to know about the product. His goal is to be satisfied in his technical awareness of the product—in relation to the business needs—such that it makes business sense to ship or deploy the product. The test manager and the product owner collaborate in identifying the testing tasks that might help in providing that information.  That turns into the mission for that cycle of testing, test manager and her team can then set about fulfilling the mission—and adapting it when the product owner changes his mind, or when other things don’t go according to plan.  But in the end, the question of when testing will be done depends very little upon the test manager, and far more upon the programmers, the product owner, and a bunch of information that is presently unknown.  A great test manager can explain that in a way that satisfies the product owner, without being trapped into an unachievable commitment.

uTest: At STPCon, you also said that “Testing can be assisted by automation, but automation is not the arbiter of value. We are.” Does this mean that testing will never be replaced by machines? And does this include cyborgs?

MB: Automation is a tool; it’s a means to an end.  It’s a medium, in the McLuhan sense—it extends or enhances or enables or accelerates or intensifies some human capability, but it doesn’t replace that capability. Writing and email and wikis extend our ability to communicate, but they don’t replace talking.

Machines can’t make decisions about what’s best for us, because they’re only extensions of some human faculty. Value is about what humans want, what they like, what they’re willing to pay or do. People don’t even like other people making those decisions for them.

uTest: You’re an active blogger, teacher and speaker – do you enjoy these things more than testing itself?

MB: That’s a little like asking about someone’s favorite Beatle album. I love testing, I love teaching it, I love talking about it. It’s a feedback loop. I wouldn’t be able to do the other stuff if I didn’t test, and the more testing and research about testing that I do, the more I find that’s interesting to talk about. When we teach Rapid Testing, we offer a free day of hands-on testing and coaching. We offer that separately too, and it’s always a source of great learning and great fun.  I’d be delighted for clients to involve me more directly more often.

uTest: Who do you read in the world of testing (blogs, sites, journalists, et al)?

MB: I have a very expansive view of the world of testing. To me, the world of testing is the world of human knowledge and inquiry, of critical thinking, of science. That includes software-related stuff, but it includes a ton of other disciplines too:  medicine (Jerome Groopman, Atul Gawande), aviation (Patrick Smith on Salon.com), education and other social issues (David Cayley), advertising and marketing (Terry O’Reilly), economics (James Surowiecki, John Cassidy, Herbert Simon), security (Bruce Schneier), risk (Nassim Taleb).  All those folks and their specialties are Googleable. I really like Malcolm Gladwell’s writing, too; he’s harder to pigeonhole, but he’s good at finding intriguing sides to the basic story. I first encountered most of the writers via The New Yorker and most of the broadcasters via CBC Radio. I like starting with very accessible stuff, and then if something interests me, I follow the lines of references, bibliographies and notes.

A couple of testing books that came out recently have few notes and no bibliography. That’s like publishing malpractice, so far as I’m concerned. You’ll notice that I’ve mentioned a lot of names already. That’s specifically so that people who are reading this interview can check out the stuff that’s fascinated me and helped to spark new ideas. I hope it helps to inspire people to talk about their own discoveries. Thank you for this opportunity to chat about mine!

Editor’s note: We hope you’ve enjoyed our three-part chat with Michael Bolton. If there’s a question you wanted us to ask that we didn’t get to, you can ask Michael directly at: michael@developsense.com.

Interview of James Whittaker

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-james-whittaker-part-one/2009/06/#more-907)

Once a month, we’re going to “test the limits”, interviewing a leading thinker in the world of testing and quality.  It could be a journalist, an industry analyst or an exec from a top software company.  To kick this program off, we could think of no better person than our good friend, Dr. James Whittaker.  So we recently interviewed James by bouncing emails back & forth over the course of a few days.

Several of these questions came directly from our community of testers.  The whole exchange is fairly lengthy, so we’re splitting it into two posts.  Come back and check out the 2^nd half later this week.

uTest:  So the news is out about your move to Google. What prompted you to make this move?

JW:  I didn’t so much leave Microsoft and I did join Google. I was attracted by all the Googlers I met at conferences and what I read on their blogs about the way they test. When they offered me the opportunity to be a part of it, one might even argue an important part of it, I found it impossible to decline.

uTest:  Is there something about Microsoft you’ll miss the most?

JW:  Yes, the breadth of both products and expertise. You literally have every type of software imaginable and a chance to collaborate with the people who make that software. From an intellectual standpoint, Microsoft is mind-blowing.

uTest:  What specific work at Microsoft did you enjoy the most?

JW:  Mentoring – making other people better. I love working with smart people and finding their passions. When you take a smart person and help them find and focus their passions, you have a powerful mix. Smart people who aren’t passionate creates a bad vibe. I think I was pretty good at helping people find that combination and enabling them to become the company’s top performers.

uTest:  That’s interesting. What is your approach to mentoring? Can you give us an example?

JW:  To attract passionate people, you have to be a passionate person. I don’t go out of my way to select mentees, we just discover each other. When I was at Florida Tech I supported a lot of students. Some came looking for the wages or to improve their chances of landing a job at Microsoft or Google, but the ones who were drawn by their intellectual curiosity where the ones that saw the most success. It’s been the same at Microsoft. I mentored folks who came and went and others that really got it and became stars.

uTest:  Can you tell us a little about the interview process at your former and current employers? They are both famously difficult and, now that you’ve experienced them both, do you have any tips for our readers?

JW:  During my Noogler orientation I saw stats on the accept rate of Google applicants — it’s astoundingly low. And although I never saw such stats about Microsoft, I have to imagine its low too. But the interviews themselves were fairly similar.

A lot of hard technical questions, both from a coding and testing perspective and also problem solving. Google kept me in a single room and brought the interviews to me. Microsoft shuffled me around from office to office. I think I prefer Microsoft’s approach from that standpoint, but the actual interviews were fairly similar. I think Google took a page from Microsoft’s playbook on that one.

Anyone out there who plans to apply at either place should be ready for problems that test your knowledge and your creativity. Good luck!

uTest:  With this new role, will you still be able to write, teach and evangelize testing?

JW:  I wouldn’t accept a job where I didn’t have to teach! Googlers will have every reason to get sick of me and I am still doing my STAR tutorial and occasional keynotes.

uTest:  So where can everyone find the soothing, savvy writings of Doc JW?

JW:  On the Google Testing Blog, of course! http://googletesting.blogspot.com/

uTest:  And when all is said and done what will be the professional accomplishment you’ll look back on with the most pride?

JW:  Creating an actual discipline around software quality. Note I said quality and not testing. I want software projects as a whole to run more smoothly and more predictably. I really think that’s what software testing is all about — reducing the uncertainty of software development and finding ways to muscle errors out of the process. A process in which mistakes are harder than doing the right thing is the ultimate goal. We can’t eliminate them, but we can make doing the right thing to be the easiest thing to do.

uTest:  What’s your first assignment at Google?

JW:  To raise the level of testing precision and diligence. Google has a lot of smart testers, my job is to help mold them into a serious fighting force and let our bugs beware. But this isn’t so much an individual commitment. Google has a culture of collaboration that I am fascinated by as a Noogler.

We share offices (which might explain their interview strategy), inhabit common areas, collaborate constantly and work as a community. If I am successful, there will be many people who can take credit and if I fail, I won’t go down alone! I think the whole free food thing is at the heart of this as food is often the centerpiece for bringing people together. Lots of work gets done while your mouth is full. I hope to succeed before I have to buy bigger clothes.

uTest:  Rumor has it that you have a new book coming out.  What’s it about and when will it hit Amazon’s shelves?

JW:  It’s in production now. I hope it’s available later this summer. The title is Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Manual Testing.

[uTest note:  Since James is too shy to promote it, we'll do it for him.  The new book is available for pre-order at Amazon... get your advance copy today!]

uTest:  Will you be implementing test tours at Google? If so, which ones in particular?

JW:  That’s going to be up to Google’s engineers. I will most certainly teach them everything I know and be there to work alongside them. Good ideas tend to stick, so this little Google adventure will help decide whether the tours work here or not!

uTest:  In your opinion, what does the future of software testing look like?

JW:  I actually just wrote it up. It’s chapter 8 of my new book. But I am working on a pre-publication version of this particular topic.  I’m not yet sure whether it will be out as an eBook form or a presentation that I do separately. Watch the Google test blog for updates on this!

uTest:  How do automated and manual testing coexist in the future – are the compliments or substitutes?

JW:  They will co-exist better than they do today, but this question requires a longer answer.  Suffice it to say that every good automated test I have ever seen started out as a manual test. How’s that for coexistence!

[uTest note:  this would probably make a good topic for a future webinar with James]

Interview of Matthew Heusser

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-matt-heusser-part-1/2009/11/)

In this month’s installment of “Testing The Limits”, we sit down with Matt Heusser (@mheusser) — prolific blogger for STPCollaborative, thought leader and testing extraordinaire.  We’ll discuss the state of software testing, SpeedGeeking, the role of chaos in testing software, and the lack of fistfights at STPCon 2009

uTest:  We loved the SpeedGeeking session you led at STPCon, so we’re going to flip it on you – If you had just five minutes to teach, motivate or inspire the uTest audience about software testing, what would you say?

MH: Well, I’d start by asking the audience what they are doing today – what’s the greatest point or opportunity they feel – and asking what options they see to improve. Most of the time, I hear that testing is “too slow” or “the bottleneck” or something like that.

So I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting – it is very important the team stop the time tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the numbers for improvement. For example, many of the people I talk to can spend 80% of their time or more in meetings, working on documentation, working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from 80/20 to 60/40 will double the amount of time the team spends actually doing testing.

Another thing to look at is the amount of time spent trying to reproduce defects, document defects, file bug reports, “verify” fixes, and so on. We think of these activities as testing, and they can take a substantial chunk of that 20% – but they are really accidental. That’s not a testing bottleneck – it is a development bottleneck. If test can work with development to improve the quality of the software prior to code complete, that will improve the speed of the whole system. Realizing this, and having a little bit of data to “prove” it, can help the entire system improve.

So if I had five minutes, I would say start with measuring how you track your time, and ask yourself if this is the best use of your time and what can change. Sometimes, the big boss will say “no, we absolutely need you to fill out all seven pages of documentation per test run”, and you can say “ok.”  Six months from now, when someone asks why the big project is late, you can point out that the business made an explicit decision to pay the full price of defined process. You presented options and those were not accepted.

That won’t save this project — but it might save the next.  It also turns out that actually testing tends to be much more fulfilling than documentation and compliance activities. Who could have guessed?

Lots of contrasting opinions at last month’s STP Conference. While there were no fist fights (that we heard about anyway), what did you see as the most contentious issue? And where do you fall on this issue?

MH: I’m a little sad that we keep talking about best practices vs. context, scripting vs. exploratory. Of course we need a balanced breakfast of approaches to testing, and of course, “best practices” are a marketing term that do not exist in the engineering literature. Personally, I’m most embarrassed that you could get the crib notes from any conference from 2004 – or possibly 1999 – and see the same arguments.

Some of the problems are due to personalities. There are a few people who just don’t give past credit for ideas, or make wild claims without having actually done much software testing. Where do I fall on the issues? Well, let me first say, if anyone claims to be an expert on software testing, one place to start is to go to LinkedIn, look at what they actually claim to have done, and send out a few emails and try to verify those claims. If you can’t verify those claims, or realize they are written in a very specific way as to be non-falsifiable, well, that tells you a lot.

I was impressed that we managed to not fight about certification at the conference. There was no certification course before, after, or during, and we didn’t have to spend our time debating it’s merits or lack thereof – we mostly talked about, you know, actually testing and stuff. In that, I was pleased.

Congrats on getting your blog “Testing at the Edge of Chaos” exclusively featured on the STP Collaborative. What does it mean to you to test at the edge of chaos? What ideas are you most interested in getting across to the tester community?

MH: Uh, I think my blog has something to do with testing. And, um, like, something to with Chaos, or something.  Seriously, started blogging in 2001 or so to express myself and my ideas. The last iteration of that was my “Creative Chaos” Blog, where I talked covered creativity and innovation in the software process. “Testing at the Edge” is a little more tester-focused, with the goal of covering skills and innovation in the software testing space.

It seems to be that every year’s a new batch of graduates come out of MIT, Carnegie-Mellon, and the University of California at Berkely that know how to automate defined business processes – and take a look at testing and say “gee, there is a defined process, we should automate it.”

Four or five years later, they’ve learned a bit and turn around and say things like “gee, some of testing can be automated, but part of testing an investigative, feedback-oriented process.”  Then every May a new batch graduates and we start all over again.

I think “Creative Chaos” did a good job in reaching out to that audience, and with the publication of “Beautiful Testing”, we may be finally out-growing it. With the new blog “Testing at the Edge”, I hope to move into specific examples of good testing, how to get better at it, and how to set some appropriate boundaries around the testing activity – and discuss what those should be.

What’s the deal with your Testing Challenges? Are they still on-going, or have you simply run out of apps to test?

MH: I generally run test challenges in private, sometimes electronically, as part of a mentoring or training program. I do this both commercially and non-commercially, as part of my zero-profit “Miagi-Do School of Software Testing.” Recently, I’ve been asking people for what they want, and the idea of running test challenges publicly – and sharing the answers – keeps coming up. I ran one in October (link) that was well-recieved. Sure, I can do more of them if there is interest.

As I work full-time for software product company, I don’t see us running out of apps to test anytime soon. (Laughs)

A quick hypothetical: You’re banished from the software testing industry for five years. What do you do during that time? And don’t say developer.

MH: I’d probably challenge the authority of who it is that is trying to banish me! But I will take your question in the spirit you intended it, and answer what my next career choice would if, for some reason, I chose not to test. That would likely be writing about technology or business.

When you think about it, the investigative journalist shares a lot with the tester. Journalists are paid to look around, find something that seems correct on it’s surface but has a problem, and uncover information and evidence. They take careful notes, they work on projects that are different every time, think critically, and are guided by rules of thumb.

A surprising number of the leaders in the testing industry have an education of background in journalism – both Karen Johnson and Jonathan Bach were trained as journalists before they became testers, and Dr. Cem Kaner has a law degree as well as his PhD in Psychology.

What would the Matt Heusser from ten years ago think about today’s software testing landscape?

MH: Well let’s see … ten years ago Extreme Programming was a crazy idea that Kent Beck was trying at Chrysler, “agile” was spelled lower case and was an adjective that meant ‘bendy.’ The software development landscape was full of RUP and patterns and generalizations and abstractions, and testers were counting test cases. James Bach and Cem Kaner were using the term exploratory testing, but it was far from popular. Today, I think testers have more voices to listen to – the standard school, the context-driven school, and the Agile School all understand each other a bit better and have more clear comparisons. Also, test managers have more options, like the crowdsourced approach or perpetual beta.

Overall, there are more options, the profession is taken more seriously, and we are finally starting to evaluate ideas based on consequences and outcomes instead of ideology. It’s a great time to be a tester. I wouldn’t want to go back.

What other blogs, sites or message boards do you read to stay on the leading edge of all things testing?

MH: I’ve got a bit of a bias there, as I write a monthly column for Software Test and Performance Magazine. You can get the PDF version for free every month. I also subscribe to Better Software Magazine. For communities, I like softwaretestingclub.com and the agile-testing discussion list. For blogs, well, there’s James Bach, Cem Kaner, and Michael Bolton. Adam Goucher has been doing a lot of writing lately, including editing the new book, Beautiful Testing, which I contributed a chapter to. And I spent a fair amount of time working on my own blog, “Testing At The Edge of Chaos.”

Recently I’ve been getting to know people by working on projects with them; my friend Chris McMahon started a Google Group on Writing About Testing which I found to be a blast. Through that group (and the Miagi-Do School) I’ve met quite a few new bloggers: Markus Gaertner, Lanette Creamer, Marlena Compton, and Ajay Balamurugadas come immediately to mind.

What OS are you running right now? What’s your browser of choice? Anti-virus? Inquiring minds would like to know.

MH: Max OS X, and I have to support all of them, so I run all browsers. Anti-Virus? Dude, I told you, I use a mac, and I try to avoid the questionable websites that host the viruses. What have you been browsing lately, Mr. Johnston?  (Ed. Note: I’ll ask the questions around here.)

As more companies develop mobile apps, mobile testing is becoming a greater concern. Are you seeing a convergence of mobile app testing with traditional software testing? Or is mobile testing an separate entity altogether?

MH: My initial thought is that testing mobile apps is a nice little niche inside of web-based software testing. But mobile devices keep getting more and more standards-compliant, and thanks to sites like www.iphonetester.com, you can test handhelds without physically owning a dozen different makes and models. So while I believe there is value in identifying yourself as a mobile device tester, I suspect that value is going to decrease over time.

What attributes and skills make for the best software testers?

MH: My experience lately is with web-based and database-backed commercial and transactional applications, and so I’m going to give an answer that assumes that world view. First off, I think good testers understand the business itself – what the basic transactions are, and what happens if they go wrong. The strong tester will use this knowledge and seek out complexity and undefined behavior; bugs love to hang out there. Strong testers are good systems thinkers; they can perform modeling of the system to figure out where the weak points are. They have a strong memory; they can track a large number of variables in their head at one time. They may play chess, soduku, or maybe just board games by Avalon Hill, or possibly Steve Jackson Games in it’s prime. James Bach pointed me to curiosity, and I think he’s right – if you never ask “what happens when I click this?” it will be hard for you to click this. They’ll want to know common failure modes across all software, and will want to study “quick attacks”, or the “How to break … Software” literature. They will be solid critical and systems thinkers – the kind of skills you can get from listening to politicians and asking what they are not saying, or reading articles in the press and looking for the bias. They will have solid middle school math skills, but be able to figure out a 15% tip quickly, combined with a little bit of statistics and probability. An ability to do some programming, to understand HTML and javascript and such really helps.

I would suggest influence skills combined with a healthy dose of skepticism. It is very hard to find someone with both of these, but to find the problem you need to be skeptical of glib answers and false confidence – and to get anything done about it, you’ll need influence skills. Influence is more than conversation, it is written communication and situational awareness.

Another hard to measure skill is creativity, but it is so crucial. The industry may like to believe that test cases can be generated automatically by some standardized process, but the reality is someone has to think of them, and a creative mind can find all kinds of possible failures that a dull one will miss.

Finally, there’s discipline, which is an interesting one. I have to admit, I have a hard time denying myself as much as a cup of coffee. I’m not very good at forcing myself to testing – I actually want to test. If you’ll permit me just one brag, I think it is the actually wanting to test part that makes me good at it, not the discipline.

Of course it’s going to be different by the job, government and regulated work is very different than mass-market commercial software. And some organizations care a lot about technical skills like dotNet or Java or whatever. You might have a very effective boss who covers for a tester who lacks influence skills. If I had to pick one tester attribute to spark the rest, it would be curiosity. If you just don’t care “what would happen if?” you wouldn’t imagine it, and you won’t press the button to find out.

You seem to be big on tester training lately. From what you’ve read and observed, what’s the best way for a new tester to get up and running?

MH: I’m afraid I can’t give a simple answer like “just go to a conference for a week” – it would have to customized by the environment. So you’d start with some sort of exercise to look at where a tester is according to a skills inventory, where they fit in with the team, and what to work on next.

Likely, that would be a combination of job shadowing, directed readings, brown bag lunches, mentoring, and, possibly, external training. I do think sending a new tester to a conference is a good idea, mostly because of the rich variety of training options and idea exposure they will have that week. A conference offers an opportunity to customize training just by selecting which sessions to attend. If I didn’t have the budget for training, and wanted something local and online, I would consider something like the black box software testing course run by the Association for Software Testing.

Whatever the new tester does, I suggest it be some combination of theory and practical exercise, with the students aware of the exercise and built-in time for reflection on what happened.

Now, what do I recommend if you have a certain budget and want to train an entire team? Something customized in a similar way. So, for example, I am getting more and more excited about the idea of bringing the trainer on-site for a day or two of consulting by walking around. Based on what he observes, the trainer develops a day or two of custom training using similar technologies to those used in the actual workspace; then he comes back for a two days of actual testing, followed by a retrospective. The trainer might then come back every month or two for several brief engagements.

The idea there is to actually tie the training to the work being done, to move it from “Drive-by training” to “helping to collaboratively explore how to improve our work.” Or, in other words, to do outcome-based training. I know. Crazy.

Are great testers born or bred – asked differently is being a world-class tester something that anyone can learn or are you born with it in your blood?

Can I answer that by telling a story? Malcolm Gladwell has a new book out, called Outliers: The Story of Success. In the book he discusses the nature of language – that some cultures, such as South Korea, are high-ceremony cultures. These are the places where the conversations between superiors and subordinates is very formal and scripted. He also discusses low ceremony cultures, like the United States, that have more of a cowboy culture. Gladwell points out that, while human beings make mistakes, the high ceremony culture offers no way for a subordinate to point the mistake out and attempt to correct it. While an American co-pilot might say “we’re flying into a storm you ninny”, in other cultures that behavior is not allowed. That’s no big deal when you’re talking about how many lumps of sugar you’ve counted out, but when it comes to flying airplanes – or mission and life critical software – it matters. So I would say that culture and values or ideology, plays a huge part in the effectiveness of the testing. So much of our work is shining the light of truth when it would be more politically palatable to not notice.

So on the one hand we’ve got cultural values, and on another critical thinking, modeling skills, and curiosity. Certainly critical thinking and modeling can be strengthened through exercise. Curiosity? That’s a good question. You might make an argument that curiosity is inherent, that it “comes with the genes” so to speak — but I’ve seen enough children have their curiosity sucked out by schooling to believe that it must be possible, somehow, to create an environment to encourage it to come back in.

Interview of Rosie Sherry

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-rosie-sherry/2009/07/)

In the second installment of our “Testing the Limits” series, we sat down with Rosie Sherry, founder of the Software Testing Club, to discuss the state of the profession, advice for QA beginners, “flash mob” testing and much more.

uTest: Where did you get the idea for the Software Testing Club?

RS: It was fairly simple really: I wanted a place to meet like-minded software testing professionals. Everything else out there at the time (this was 2 years ago) didn’t meet this need at all. I was doing a fair amount of blogging on software testing then, and so starting a community seemed like the next logical step.

uTest: Considering the growth of the community over the past few years, have your expectations changed?

RS: My expectations have definitely changed – they had to. For instance, I have stopped new members from joining for free since that model is now completely unsustainable. Like I said, I never expected to have created a community this large. I’d prefer something much smaller where we can give members the attention they deserve, but we’re obviously not going to turn anyone away. If you’re interested in software testing, you’re welcome in our community.

uTest: Based on your observations, what is the most contested debate in the field of software testing today?

RS: The biggest ongoing debate right now is software testing certification. A lot of people maintain that some companies/people put too much emphasis on the certification process, and that much of what is learned deals with how to pass an exam, rather than learning true testing skills. It’s definitely a lively debate with many viewpoints, so I’m not going to get too deep into it right now.

uTest: A recent poll on your website found that most members saw budgetary pressures as the greatest challenge facing QA managers. Would you agree with this?

RS: I think a lot of it depends on the industry people work in. Much of my experience has been in small web companies, where budgets are always an issue and where testing routinely gets cut (if it’s even considered at all).

uTest: What are the keys to building and launching high-quality applications?

RS: There are obviously many factors to consider. From the testing angle, building strong and respectful relationships between testers, developers, project managers and others is crucial. The respect part is important – and we all have to work hard to earn it. That means as testers, we need to prove our worth.

uTest: How does the testing profession keep up with a set of trends that makes assuring quality increasingly complex (agile, tighter budgets, mobile apps, etc.)?

RS: Developers need to specialize in niche areas. Testers need to do the same. Define a niche like e-commerce, social media,  mobile apps, or even  iPhone apps. Be ‘the person’ (or company) to hire for your niche.

uTest: What’s the biggest challenge for new testers who are just now entering the profession? Any words of advice for them on how to become a world-class tester?

RS: You have to be willing to stand out from the crowd. Hard work scares a lot of people off, but becoming a world-class tester isn’t difficult if you have a passion for it and make the commitment early on. I’d also advise anyone pursuing this career to use the social web to their advantage – participating in communities, keeping a blog, Tweeting, etc. Create your own personal brand that would make your momma proud.

uTest: So how did you get your start as a software tester?

RS: I wanted to get into IT (and out of a really boring administration job at a bank). And so when I was offered a job as a tester in a local web company, I took it. I’ve enjoyed the profession so much, I’ve stuck with it ever since.

uTest: What’s the deal with “Flash Mob” testing?

RS: The idea is that – like a real Flash Mob – testers come together over a short period of time, test something, log their bugs and then leave. It is all very experimental at the moment. I find the whole process (that is, bringing a crowd together) to be very interesting and fun for me personally. I figured if there was a way to coordinate testers and make use of their skills in a positive way, then we should do it.

uTest: So besides Software Testing Club, what are some of the other sites you visit to stay on top of testing and QA news?

RS: The news is no longer what the big guys are talking about, so over the last two years I have lovingly collected as many testing blogs as I could find and posted them to the Testers Feed on the website. I’m a big fan of the li’l guys and do what I can to support them. I also follow many testers on Twitter.

Interview of Andrew Muns

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-andrew-muns-president-of-stp-part-1/2009/08/)

In the latest installment of our “Testing the Limits” series, we sat down with Andrew Muns (@amuns) the President of Software Test & Performance (of STP Magazine and STPCon fame), to discuss how testers are perceived by execs and developers, the future of media companies, and the changes that are underway at STP.

uTest:  STPCon is being held this October in Cambridge, MA… what do you have in store for the attendees this year?

Andrew: This is the first conference that will have been planned start to finish by Redwood Collaborative Media and was designed according to our very simple philosophy: “ask your audience what they want and give it to them.”  The show’s program was designed largely  based on a survey in which we asked two things, what topics are most important to you and who do you want to hear from.

The most requested topics among our readers were Test Automation, Performance Testing, Test Management and Agile.  We’ve built a five track program with specialized training and workshops for each of these four areas, plus a track we call “FutureTest.”  The concept of FutureTest is to take a look ahead to emerging tools, technologies and practices – to help our members stay on the cutting edge of the testing industry.

We’ve got only all-stars here (check out the full roster) plus a keynote by a NASA astronaut, Mike Mullane, who will talk about leadership and the organizational culture that led to one of the most tragic QA mistakes in history: the O-ring of the space shuttle Challenger.  Michael Bolton, will then use this story as a launching point (pardon the pun) to talk about test leadership.  It’s going to be a phenomenal event.

uTest:  You recently launched STPCollaborative.com. Tell us the purpose of this site and what’s so different about it.

A: The media business is undergoing a dramatic shift, and the STPCollaborative site was built as a platform for our member-focused model.  Somewhat dramatically perhaps, I call this a Copernican shift because today’s media companies now realize things don’t revolve around them.  Our focus is on members of the software testing community, in which we’re just one of many participants.

In a way, this isn’t new to us and is based on the very successful model developed by Ron Muns, my father, our Chairman, and founder of HDI (previously the Help Desk Institute). HDI was a membership organization for technical support professionals founded before the current trendiness of social networking and web-based communities emerged.  His company built a vibrant community for a group of professionals that didn’t get much respect, and really helped to create professional standards and be a vocal advocate for the profession.  We are working to build STP Collaborative into a membership organization that provides that same sense of community, that facilitates knowledge sharing and education, and that helps testers reach their professional goals.

STP Collaborative is the information resource platform for our members.  It brings the previously static content from Software Test & Performance into a more interactive web-based format, and informs members of new educational content, whether in the form of online content or live events and conferences.  Of course the best is yet to come, and we’re working diligently behind the scenes to increase the interactivity of our site and to roll out some truly world-class online and offline educational resources for testers.

uTest: By the way, who tested the site before launch?

A: First off, uTest was extremely helpful to us in preparing for launch (thanks!).  We’re a start-up in many ways and as such have an extremely lean team with limited resources, so having the uTest community assist us was invaluable.  In fact, two of the bugs you guys found would have been real show stoppers!

Before this, however, the testing was done largely by Igor Balos at Wildbit (our developer team), our own staff (everyone participated), and a few members of the testing community who volunteered to help.  Our contributing editor Matt Heusser was also particularly helpful.  Best of all, I actually got to test software myself!

uTest: So based on what you’ve seen so far at the helm of STP Collaborative, how close-knit is the software testing community compared to other professions?

A: Well, as a newcomer to the space, my first impression is that the profession is fairly Balkanized.  People talk a lot to me about “schools of software testing” and many of the consultants and thought leaders in the industry feel very passionately about their “school.”  This is partly because software testing is by definition concerned with a very fuzzy and philosophical concept called “quality,” and partly because “software” itself has so many incarnations that people in different fields can have radically different perspectives.

In spite of this, I believe that very tight bonds exist among testers.  There is a strong community here that identifies with common professional and interpersonal issues.  I attended Jerry Weinberg’s session at the recent CAST conference and we did a role-playing exercise where a tester was arguing for equal pay with the developers.  The outpouring of support from the group for the woman playing the role of the tester was incredible.

uTest: What’s been the biggest challenge in creating STP Collaborative?

A: Let’s see, we’re a start-up in a highly-cyclical industry undergoing dramatic change, in one of the most challenging economic situations of our lifetime.  Where should I start?

Joking aside, I’m amazed at what we’ve accomplished in the face of some very strong headwinds. In the last eight months, we’ve reinvented our business from a one-to-many traditional media model to a member-centric, highly interactive listening and learning model. We’ve built a world-class strategic advisory board. We’ve launched a terrific new website focused on building a community; and we’ve redesigned our magazine.  The biggest challenge has been managing the chaos associated with this pace of change!

uTest: Your background has included stints in finance, teaching, as well as the Peace Corps. Ever think you’d end up in the world of software testing?

A: Yeah, I’m pretty sure I’m the only person on earth to have worked teaching math in Central Africa, done IT consulting in Venezuela, worked on a distressed debt trading floor in New York, helped Carl Icahn shake up corporate boards, and run a new media company.  Looking back, though, any role I’ve ever been good at was really just teaching.  Sales done properly and honestly is an educational endeavor, and business is really relationship building and sales no matter what field you’re in.

In fairness, no one would hire me to be a software tester.  That said, I have an open mind, I’m very humble, I listen to people, and I love to learn.  I’m told the best software testers bring in diverse perspectives, and learn and adapt constantly.  If that’s the case, my involvement with the testing world should be a great fit!

uTest: Testers and developers: Friends or foes?

A: I’m sure this varies from company to company and team to team, but I’d wager a sum of money that teams where the two are friends dramatically outperform teams where they’re foes.

uTest: Testing is often viewed as a behind-the-scenes profession. What can testers do to bring their craft to light and make sure others understand the value?

A: Upper management at most companies may never truly understand what a test department contributes, especially since a contribution by definition goes unnoticed (i.e., something worked as expected.)  To me this sounds like a cultural issue: how to translate the value of testing into manager-speak.  Managers like things they can measure, so speaking their language means associating a measurable value on something vital but difficult to observe.

Software Test & Performance magazine has written many features on this question, but as a manager more than a tester, here is one argument I like (that applies more to consumer-facing applications): explain QA as a marketing function.  How much does your company spend on marketing?  Why would testing merit less investment?  I bet your company would spend a lot to spread positive word-of-mouth from users.  Shouldn’t management be willing to spend the same amount or more to avoid negative word-of-mouth?  As United Airlines learned after breaking a customer’s guitar, negative word of mouth can be viral.

Critically, neither this argument, nor any other, will be made if testers themselves don’t make it!

uTest: Is James Bach really as smart as we think he is?  Who would win in a fight between him and his brother, Jon?

A: James is a phenomenal, self-taught polymath, and is one of the many testers I’ve met in the last year whose intelligence has impressed me.  It’s a profession that attracts some creative, analytical and multi-disciplinary individuals.

As for the fight, I have two young sons at home, a one-year old and a two-and-a-half-year old. The two-and-a-half-year-old’s twice the size, but his baby brother’s fearless!  In short, I’d never underestimate the tenacity of a younger brother.  I’m going with Jon.

uTest: Much has been made of the “fall” of traditional print media companies.  So what attracted you to enter this space?

A: One of my previous jobs was to conduct research and valuation analysis on companies in financial distress and bankruptcy, so I’m a contrarian by training.  I’m actually not a believer in the traditional print media model in general—newspapers as we know them, for example, are doomed.

That said, I believe that niche B2B media markets have the potential for economies of scale in specialty content and are well positioned to take advantage of new communications community building tools.  B2B markets that perform well will be those that embrace the new role of media which is to provide educational and community building resources.  Software testers have a strong common sense of identity and a high demand for learning, due to the complex, constantly changing and technological skills required by the profession.

uTest: Put your prognosticating hat on… what trends do you think we’ll be talking about in software and testing one year from today?  10 years from today?

A: One impact of working in corporate finance during the credit boom had on me was to erode my faith in prognosticating!  While any attempt (especially by a layman such as myself) to project specifics would be pretty useless, I will make one obvious observation based on thermodynamics: I am a strong believer in entropy.  Complexity increases with time, especially in software.  Wrestling with exponential increases in complexity and interconnectedness will be one the primary challenges of those in the software business or any business going forward.  If STP Collaborative is successful, it will be because we will have found a way to simplify this complexity enough to help testers navigate a landscape of increasing entropy.

uTest: Other than STPCollaborative.com, what other sites do you read to learn more about QA and testing?

A: I read blogs of notable testers primarily, and also follow testers of every ilk on Twitter.  In the blogosphere, two of my favorites are Creative Chaos by Matt Heusser and DevelopSense by Michael Bolton, but there are many more.

Twitter is also a great listening tool that’s been a great way for me to get to know what the test community is saying about their profession, new products and technologies, about each other, about us or about the world in general.  It’s invaluable.

uTest: Any advice for new testers who are just starting out? What should they do/read/know in order to be successful in the world of professional testing?

A: I can only give generic advice here, which is to build a network of peers that includes a mentor.  I doubt there’s any single book that can make you into a great tester (or a great anything), but if you get to know others in your field, treat them with respect and help them whenever you can, you’ll have a network that will be more effective at helping you solve professional problems than any static written resource could ever be.  If that fails, just read Software Test & Performance magazine

uTest: Are you a Mac or a PC?

A: I’m a recovering PC aspiring to be a Mac.  My suits have been hung, my jeans are getting worn, and I spend two days a week working from home and putting my kids to bed.  I’m well on my way!

uTest: Finish this sentence: The software testing industry is on the verge of______________.

A: I’m an optimist, so I’ll say: “gaining the respect that the profession deserves.”  This won’t happen quickly without a strong unified voice from the community, however.

uTest: What’s the worst bug you’ve seen or heard of since you got involved in the world of software testing?

A: We misspelled the word “Testing” in an issue of Software Test & Performance magazine.  That’s certainly the most embarrassing “bug” from my point of view, even if it’s not really a software bug.  The test crowd spotted this one in about 0.2 nanoseconds and promptly turned my face beet red. I guess that shows the skills our audience has learned from the magazine!

Thanks to Andy for taking the time to answer questions from us and our testing community.  If you have a question that we missed about testing, STP or any other subject, drop it in the comments and we’ll try to compel Andy to answer it (as you can tell, he’s quite shy)!

Interview of Rex Black

This interview was conducted by WhatIsTesting.

Rex Black is founder and principal consultant of Rex Black consulting services. He has written two books Managing the Testing Process and Critical Testing Processes. He has provided many templates for download on his site http://www.rexblackconsulting.com.

WhatIsTesting interviewed him recently. We hope that this interview will be useful for you.

Q1. Rex, please tell something about your background, your experience, and non-testing interests.

I have spent over 20 years in the software, hardware, and systems engineering business now. Most of that time has been as a tester and test manager, though I have also worked as a programmer and system administrator. I?ve worked with shrink-wrapped software for start-ups and big Information Technologies projects for banks. I?ve worked with the Department of Defense and with university research facilities. I?ve worked on PCs (starting back in the CP/M days), Linux, SGI, HP/UX and Solaris workstations, Unix, VMS, and AS/400 minicomputers, and even big mainframes.

In my varied experience, I?ve learned that, in testing, common themes run through projects that involve different technologies, different business problems, and different teams. The specific context matters, of course, but the commonality of problems?and solutions?is what strikes me. When I?m not working, I try to spend time around the Texas Hill Country with my wife and daughters, out hunting with my dog, or fixing up our house.

Q2. What are your comments on the state of software testing in industry today?

Software testing?like programming?is in a state of intense change. We are seeing new technologies in both; testing tools and programming languages are evolving rapidly. We are seeing new project methodologies in both; quality risk analyses and solid unit testing are gaining ground. We are seeing new approaches to organizing projects in both; outsourcing is big for testing and development. As is the case in all industries throughout history, times of intense change are times of what the economist Schumpeter called ?creative destruction.? These changes will destroy many existing organizations, approaches, and accepted ideas, but those changes will also create amazing opportunities and growth for the educated, the clever, the disciplined, and the industrious.

Q3. Even after decades of practice software testing still remains a much misunderstood process in industry. Everybody agrees that it is important but nobody agrees on exactly how or what to be done. Your comments?

Indeed, there is a large gap between known best practices and typical practices in software testing. I believe this is true for software engineering in general. Think of how many projects are managed without even a cursory work-breakdown-structure. Read Brook?s The Mythical Man-Month: You?d think he was describing software engineering projects being done today, but he wrote that book based on experiences in the 1960s and early 1970s!

Education is part of the answer. People who want to prosper in this time of change will need to master the known best practices. There are plenty of books, articles, and courses available for those committed to doing so, in software testing in particular and in software engineering in general.
Just knowing the ideas is not enough. You must be clever enough to adapt these ideas to your circumstances. As I wrote in Critical Testing Processes, there is no such thing as the One Right Way to do anything. However, there are general ideas that clever people can fine-tune to fit their circumstances.

Discipline and hard work are also part of the answer. The lazy and the undisciplined cannot expect to survive in tough economic periods, particularly when major changes are occurring. Many of these best practices require patiently gathering data, incrementally improving, sticking with hard work, over long periods of time.

Q4. Do you think that software testing profession is maturing as an engineering discipline?

Yes. We are clearly seeing more people understanding the need for specialization in testing. We are seeing more books, educational offerings, and consulting services that specialize in testing. That said, we still have a long way to go, especially in terms of the gap between known best practices and typical practices that we spoke of earlier.

Q5. What is the future of software testing in your opinion?

It?s hard to say right now. I see software as in a position very similar to the automotive industry in the 1970s and 1980s. Up to that point, the US carmakers dominated the global market for automobiles, but, frankly, they built and sold low-quality cars. By applying ideas from people like Juran and Deming, the Japanese carmakers showed the world that it was possible to have high quality cars without high price tags. The Japanese helped people understand that quality does not mean luxury and ostentation, bells and whistles; quality means satisfying reasonable customer and user expectations again and again and again.

In the Japanese automotive revolution, quality control and quality assurance lead the way. They were key. There could be parallels now in software, if users and customers are ready to demand affordable software that simply works and works simply.
Certainly the economic situation and the rise of outsourcing is driving another possible parallel: The off-shore software houses, if they seize the moment, could lead the way, just as the Japanese took over the leadership of the auto industry in the 1980s.

Q6. How should a tester prepare oneself for the upward movement in management ladder as a test team lead, test manager and beyond? What are the essential qualities that they must develop?

There are many essential qualities and skills. A tester who aspires to management must understand the underlying business considerations. Technology by itself is not by the point. Technology makes business innovations and improvements possible. Technology solves problems. And technology involves costs and benefits, risks and opportunities. Management decisions revolve around such considerations.
So, a tester must grow the skills needed to effectively consider and manage these factors. Estimation and budgeting. Risk analysis. The pertinent business domains. The effective tester knows testing and technology, but transcends those areas to understand management.

Q7. Given that consumers are increasingly becoming conscious of quality, how do you see that impacting the software testing process? Or maybe consumers are not as demanding of quality as they should be?

Consumers are still not as demanding of software quality as they should be. Many users of software and systems assume that, if their computer screws up, they must be stupid. They say things like, ?Well, I wouldn?t have this problem if I knew how computers work.? Hogwash.
First, this statement is not true. I know how computers work, and I have plenty of problems with them. I call those problems ?bugs? rather than calling myself ?stupid.? Second, no one would ever make such an assumption about any other work-a-day implement or tool. Can you imagine someone saying, ?Stupid me, my car wouldn?t be stalling if I just understood internal combustion engines and could adjust my own timing belt?? Third, it is the job of those who build tools to make those tools fit for use. If the users find them clumsy or cumbersome, they should reject the tools, not insult themselves.

Sooner or later, as with cars, refrigerators, and washing-machines, computers will become common place. Then, as consumers did with other common-place appliances, tools, and conveniences, consumers will start to take them for granted. I think that day might be sooner than we think. At that point, consumer tolerance for lousy software will end.

I think we?re already seeing that in terms of IT software. Business customers?the CIOs, the IT managers, the project sponsors?are unwilling to tolerate big-budget projects that deliver poor quality, often late and sometimes not even at all. I think this intolerance of poor quality, along with a desire to save money, is part of what drives the outsource and off-shore boom. Perhaps consumers are not fair behind.
Companies?and professionals?who position themselves to thrive in a new environment of higher expectations for software will, I think, do well in the future, quite possibly the very near future.

Q8. Rex, you have written two books on software testing process management. What prompted you to write these books?

There have been books written on test design and testing itself, but not on test management. There have been books on software project management, but those books did not address test managers. So, my books are addressed to those previously-underserved professionals who are managing testing projects. Based on the fact that my first book has, in the last four years, sold over 15,000 copies around the world, I think the need is clearly there.

Q9. Why did you need to write two books? Why did you think one was not enough? How are the two books related? How should one read these books- Independently or one after the other? What is the overlap between two books?

Managing the Testing Process is about tools and techniques for test managers. Critical Testing Processes is about fitting those tools and techniques together into a successful test project, and fitting that test project into the larger project. They are distinct but related topics.
They overlap slightly, in that I can?t assume that everyone who reads Critical Testing Processes has already read Managing the Testing Process. So, I have to describe some of the tools covered in Managing the Testing Process, such as the test tracking spreadsheet when I?m talking about the test execution process. If you’re planning on reading both, then you should read Managing the Testing Process first, then Critical Testing Processes.

Q10. What are the future trends in software testing that we are going to witness?

Clearly, international outsourcing is one key trend. The diverse set of tools for programmer testing will, I think, consolidate into tools for early testing in general, including static testing. Focusing test resources via risk mitigation is another trend.

Q11. Do you think outsourcing of testing is a good idea? What are the factors that prompt consideration of outsourcing?

Outsourcing is a good idea under many circumstances.

Certainly cost can be a factor. Geographical convenience can be a factor when the outsource test facility is close to the development facility. Specialized skills?including test expertise?in the outsource test facility can be a factor. Short-term spikes in work that can?t be handled in-house can also be a factor.

Q12. What are the main pain points of outsourcing?

It?s important to be very careful in planning and organizing distributed and outsource test efforts. Poorly planned and disorganized test efforts, run in-house, can sometimes succeed through brilliant and inspirational management and leadership. However, under such circumstances, an outsource project will fail.

Q13. What are the advantages of outsourcing?

Cost-saving, the access to specialized test professionals, and the ability to leverage strengths possessed by the outsource test facility.

Interview of Elfriede Dustin

This interview was conducted by WhatIsTesting.

Elfriede Dustin is author of various books on testing – “Automated Software Testing,” “Quality Web Systems,” and her latest book “Effective Software Testing.”

Her website http://www.effectivesoftwaretesting.com was created to create a community of software professionals in the Washington DC area.

If you want to ask her more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com

Q1. Elfriede please tell us something about yourself, your interests and things that you do in your free time.

Originally I am from Germany where I’ve received my business degree and then went on to get a Computer Science degree at KSU. In 1984 at one of my first jobs working at a US Government law center, we received a number of Wang computers and I lead the automation of the claims processes. Ever since then I have been hooked on computers, automation, and testing. Having lived in the suburbs of Washington, DC over a decade, I am now a naturalized American citizen. I enjoy spending any of my free time getting smarter in my chosen field of testing. Additionally, I enjoy spending time with my two daughters Jackie and Erika.

Q2. What prompted you to write the books that you wrote? What kind of response do you think these books have received?

In 1997 at a Rational users conference I presented the topic “Introducing Automated Testing to your Project.” After the presentation I received uncountable repetitive inquiries from numerous sources throughout the world (even Belgium) about this same topic. It made me realize that there is a need for a book to put out this information and it prompted me to write the book “Automated Software Testing,” together with my co-authors and former co-workers Jeff Rashka and John Paul. The books have been very well received, for example the book “Automated Software Testing” is in its 9th printing and has been translated into German, Russian, Japanese, Chinese, and is also available as a less expensive version for the testing audience in India.

Q3. What were your sources of experience that your book “Effective Software Testing” drew on?

“Effective Software Testing” is based on my many years of testing experience combined with the many years of software development experience of my former co-worker Douglas McDiarmid. The book also draws on the experiences of my former co-workers and co-authors Jeff Rashka, Program Manager, and John Paul, Developer and now CEO, as it also draws on the content of our books “Automated Software Testing” and “Quality Web Systems.”

Q4. Are you working on any other book?

I am in the process of writing a book on Security Testing for QA professionals, published by Symantec Press, written with two Symantec security experts. I feel there is a need for such a book that’s specifically geared towards the testing and QA professionals. Not enough is known about how to approach security testing as a QA professional, often it’s just a band-aid and too late in the development life-cycle.

My latest article Teamwork Tackles the Quality Goal was written for Software Test and Performance magazine, see http://www.stpmag.com for a free download. A summary of one of my five “Software Test and Performance conference” (see http://www.stpcon.com) presentations “Introducing Automated Software Testing” recently appeared in the SDMagazine newsletter. I am also preparing for a presentation at the ICSTest (see http://www.icstest.com) conference in Duesseldorf, Germany in April, while working full-time as an employee, i.e. an internal SQA consultant to GSS, Symantec. (Please note all opinions expressed here are my own and not those of my employer.)

Q5. Do you think there is a gap between what testing consultants and trainers teach and what the testers and test managers in the trenches face in their daily work? In what ways?

Yes, I definitely see a gap in what testing consultants and trainers teach and what testers and test managers face in the trenches. Theories are nice to have, but some theories are just not realistic and too time consuming or expensive to implement. In the trenches we constantly face ever shrinking budgets and schedules, and go-live dates and production deadlines which can’t be moved, and often the development schedule slips and cuts into the testing time.

In the trenches, so much depends on the developers’ skills. The most efficient and knowledgeable testers cannot succeed if developers implement bad software and/or ineffective software development life-cycles and processes are in place. If testing is the only quality phase implemented as part of a QA realm, it can at most be considered a band-aid often too late in the system development lifecycle to make much of a quality difference. Testing is only one piece of the Quality puzzle.

I don’t see this here at Symantec, but I have seen it at other companies where there is still this general confusion by some management about what testers do. The one side of the management camp thinks that “anyone” can do the testing and no special skills are required and the other side of the management camp expects testers to “test quality into a system” and blames the tester if a defect makes it into production. When it comes to testing salaries and job requirements, generally a tester is paid less and technically less is expected of him. Generally there is still a gap between testers’ technical knowledge vs. the developer technical knowledge. This gap also rears itself in still little or no respect for the tester profession and testing still often being seen as the underdog and necessary evil. Based on this general mindset, I wrote my article Teamwork Tackles the Quality Goal on “integrating the “independent” testing team where I make the point that the tester has to be every bit as technical as the developer, working integrated with the development team, while maintaining the independent testing frame of mind. The testing profession has so much to offer, but so little about how it can really effectively be implemented is known or used.

Q6. Who in the testing community has influenced your thinking most?

Actually, my Computer Science professor Dr. Gustafson at Kansas State University has influenced my thinking about testing and quality in general. I took his graduate level course on “Quality software engineering processes” which peaked my interest in Quality and testing and I learned about great quality concepts and influential thinkers such as Tom DeMarco, Capers Jones, Boris Beizer, and Gerald Weinberg.

Q7. According to you what are top three things that one should learn/practice/be good at to be an effective and efficient software tester?

To be an efficient software tester the person really has to enjoy testing, have a knack for it, be analytical, detail oriented, and be versatile, such as a) be technically savvy, b) understand the business, and c) never stop learning and improving on the testing efforts

Q8. What do you think is the current state of automation? Where do you see automation heading towards? Do you think a paradigm shift is required in order to have more automation? Do you think there are other areas where tool vendors need to focus in addition to what they focus on right now?

Current state of automation. During a recent presentation at the Software Test and Performance conference (http://www.stpcon.com) I surveyed and learned that 50 percent of my audience had test automation tools that have become “shelf-ware.” Those people are now tasked with resurrecting the automation tools and efforts, which can be an exercise in futility for many reasons. They will have to keep in mind whether the tool is still compatible with the technology being used, i.e. has the technology changed since the tool was purchased. Also, if the automation didn’t succeed in the first go around did they note the reason for failure and have those “lessons learned” been evaluated so they wouldn’t be repeated on the second automation attempt, etc.

This inofficial survey result is just another example that shows not much progress is being made with the use of vendor provided automated testing tools, an issue we’ve already discussed in our book “Automated Software Testing” published in 1999, “When inefficient automated testing implementation hasn’t shown significant ROI, and budget pressures materialize, planned expenditures for test tool licenses and related tool support may be scratched. Often the tools end up as shelf-ware.”

Q9. Where do you see automation heading towards?

Granted vendor provided tools continue to improve, vendors still have and will continue to have a difficult time to keep up with all the latest and greatest technologies and third party controls and widgets, which are usually the source for the biggest compatibility issues and user frustrations. People will continue to struggle with the vendor provided tools, others will turn to in-house developed automation efforts or to automation freeware. Here are some important tips before “jumping on the test automation bandwagon.”
• Know the types of tools available along with the problem they are trying to solve. There is no one best tool on the market.
• Safeguard the integrity of the automated test scripts by using a configuration management tool to baseline them.
• Consider that automated testing is software development.
• Educate stakeholders about the test automation efforts, especially developers who need to understand the impact any code changes could have on it.
• Manage the expectations by not overselling test automation’s return on investment.
• Do not let automated testing become a “side activity” to work on when time allows; ideally use technical testing talent whose sole focus is on test automation.
• Not all testers need to be “automators,” since development expertise is required.
• Decide whether to buy or build a tool or whether to use freeware (or all of the above)

Q10. Do you think a paradigm shift is required in order to have more automation?

Unit test automation is the most effective defect detection/test automation technique. Too many companies still focus all of their test automation efforts on system testing without realizing that unit test automation can have the most ROI. Wishful thinking: Wouldn’t it be nice if self-testable development languages existed, such as components are developed automated related unit tests were automatically self-generated.

Q11. In absence of licenses most people can’t learn using tools from most of the tool vendors. But employers generally want tools knowledge. How do you think testers can break this vicious circle and gain automation knowledge?

I think employers need to be educated to understand that someone with a development background can easily pick up any automated testing tool, no matter which one. When I hire automated testers and they can show extensive experience in one tool’s scripting language or any scripting language for that matter, chances are they will be able to pick up and learn another tool’s scripting language. Testers need to remember that automation is software development, and it is important to have experience in one or more programming or scripting languages.