Interview of Matt Evans

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-matt-evans-part-i/2010/12/)

What better way to end a great year of Testing the Limits interviews than to pick the brain of Matt Evans, QA Director of Mozilla.His 20+ years of software testing experience include stints at Palm, where he managed the quality program for the WebOS Applications and Services of the Palm Pre smartphone, as well as Agitar Software, where he helped pioneer automated test generation from Java source code. Today, Matt is recognized as one of the foremost experts on open-source development and crowdsourced testing.

In Part I of our must-read interview, we get his thoughts on the diversity of the testing profession; the importance of developer-written unit tests; the evolution of Mozilla’s testing community; the biggest myths of crowdsourcing; the unique challenges of mobile testing and more. Be sure to check back in tomorrow morning for Part II of the interview.

********

uTest: You’ve been all over the tech spectrum throughout your career: You’ve worked at web companies and mobile companies, startups and enterprises, open source and closed source. But during that whole time, you’ve always been involved in testing and QA. What’s kept you in the testing space for the last 20 years?

ME: There are several reasons. First of all, software testing is a huge challenge and it takes a lot of different intellectual skills to do it right. It really boils down to asking the right questions about the application under test and continuing to do so. Testing an application well requires you to look at functionality from a lot of different perspectives: What are the different types of users? How will they go about using your product? In what different environments and conditions will users expect your application or product to work in? Drilling down on these questions and ultimately coming up with the test cases and test data to ensure you are adequately covering these conditions has always been very stimulating and rewarding to me throughout my career in the testing field.

Secondly, the exposure you get in the testing field is incredible. You can typically explore various technologies incorporated in an application and get well-versed with each. In fact, to do the job of a tester well, you are required to get a solid understanding of the technologies used in creating the application and the influences of the environment where the application is intended to operate. The more you understand these technical aspects of the application under test, the more you will know what are functional dependencies and environmental conditions that you must test the application under. In addition, you also must interact with the many players and stakeholders of a project. Obviously, at the top of the list are the users and customers of the application. You will need to understand their expectations and usage of the product, and translate those into testable use cases. Your relationship with development is also key. Providing the developers timely, contextual, and actionable feedback on the health of the application is critical to any software release. The exposure to technology and the various players on software projects have been key to my continued passion with software testing.

Lastly, there have always seemed to be good opportunities in the software testing area, whether it be traditional black box testing, test automation, or testing tools development. In my experience, the need for good qualified test professionals at every level has always been pretty consistent in good or bad economies.

uTest: For a mainstream web app in 2010, what’s the appropriate mix/interplay between automated functional testing and manual testing (both test case execution and/or exploratory testing)?

ME: It really depends on where the state of the software project is at. Hopefully, testing and test development are done at an early stage of the product life cycle and you have the time to develop test cases and write them as automated tests. With respect to automated tests, in my experience most of the new bugs are found at the point of developing the test cases and the initial runs of the automated scripts. Once these tests are running correctly, their future value is directly related to how often they are run on the updated code base. Ideally, they are run on the developer’s desktop before check-in as well as part of continuous integration. Actually, these days I think you are at a great competitive disadvantage if you don’t have a robust practice of developer-written unit tests and functional automated tests, all under the control of a continuous integration system. If you don’t have that in place, you need to invest in that now.

Having said that, I do also feel on any software project that there is and always should be some level of manual testing. The ultimate users of the program are not robots or automated test frameworks. They are live human beings with lots of different skill sets, backgrounds, geographic location, and expectations. You need to experience the product with your own fingers and eyes on the keyboard and screen. Most automated tests are pretty superficial and have rather myopic validation methods for determining a pass or fail. Because of that, good manual validation–especially with a fully configured and integrated product–is a must in my book. So, what’s the appropriate mix? I would target 70% automated and 30% manual.

uTest: You were recently one of the keynote speakers at the Google Test and Automation Conference (GTAC) in India, where you presented on crowdsourced testing. How much easier has it gotten to speak on this topic in the last few years? Is it starting to register with most people that crowdsourcing (particularly crowdsourced testing) is a viable method for helping to get important work done?

ME: I am relatively new to the concept of crowdsourcing and crowdsourced testing. It was a topic I started to get very interested in a couple of years ago. When I was at Agitar, we started a free service for JUnit test generation called JUnit Factory. It was a pretty successful open project that attracted more than ten thousand users. It was a win-win in that users received pretty good unit tests for their Java projects, and we got crowdsourced testing on our latest code base. Since then, I’ve kept an eye on crowdsourcing progress and I am pleased that it has made it to the software testing field. I do believe it is the right model for a lot of testing tasks, especially in the web and mobile space. So, I’m glad that uTest is making a go of it. I think the explosion of social networking over the past years has really made it apparent that the virtual crowd is a very powerful thing. It can solve lots of interesting problems we know about now, and I’m sure we will be discovering areas where crowdsourcing can be very effective in the future. So, I think crowdsourcing is a practice that is here to stay and will be with us in some form for as long as the web stays running.

uTest: The Mozilla community is unmatched in terms of sheer size, at nearly a million members. How can you possibly manage that much feedback on a regular basis? How do you separate the valuable data from the junk (aka: signal-to-noise ratio)? In short, how do you make something that sounds so chaotic seem so orderly?

ME: It certainly is a challenge and is a constant battle to stay on top of it. And the honest answer is that we do the best we can. As you have surmised there is a mountain of feedback that is received by Mozilla on a daily basis. Primarily the feedback we receive on Firefox is in the form of crash reports, Bugzilla bug reports, beta feedback and twitter. That list is in priority order of what we continuously monitor every day. We are constantly searching for any trends where a particular crash or reported bug starts to become and issue within the Firefox user population. Even a bug that affects a relatively small percentage of users say 0.5%, we are still talking about two million users. So there’s a constant vigilance to detect of there are any emerging issues within the code base.

In the latest beta releases of Firefox 4, there is the feedback feature at the top right of the browser. This allows users to quickly and with limited interaction give a thumbs up or down whether they are happy with the current behavior of the browser. This along with an optional comment is sent to a collection server for categorization and display: http://input.mozilla.com/en-US/. This has been a pretty successful method to get instantaneous feedback, although it does suffer from  a lot of noise contain within the data. But given the sorting and filtering tools we have certainly have received some key insights where we are weak and need more attention with our testing. We are exploring ways to allow users to participate more directly in the test effort through these types of feedback tool features.

uTest: People often (wrongly) view crowdsourcing as a passing fad, or something that only involves unskilled labor. In your experience, what’s been the biggest misconception when it comes to crowdsourcing or community-driven projects?

ME: I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard.  The reality is that crowdsourcing and community based projects are as diverse as the people that are involved.  The interaction and participation of the project members can vary drastically from project to project. The mechanisms for interaction can foster high communication and collaboration among the members or complete isolation and independent input.  There is such a big spectrum of crowd-sourced project implementations and it isn’t correct to have crowd-sourced and community-driven projects under the same umbrella. Strict crowd-sourced projects tend to be targeted towards tasks that are discrete and distributed in nature and is better suited for projects that require a number of paralleled tasks. In strict crowd-sourced projects members tend not to collaborate together to get a consensus results and may even be in a competitive situation to get the best result.

Community based approaches tend to be longer lived projects and are typically for public benefit. Membership is considered a privilege and betterment of the community as whole is usually at the top of the list of shared goals amongst the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Mozilla certainly is an example of of a community-based project. Many people would be surprised at our mission statement: “to promote openness, innovation and opportunity on the web”. It just so happens that building a world-class competitive browser helps us achieve the mission. Community citizenship and passion for the project has been the key success factors for Mozilla success in my opinion.

uTest: You were the QA Director for Palm when they launched the Palm Pre Smartphone, as well as the WebOS apps and services. What’s been the biggest difference (if any) between launching a mobile product and a web product?

ME: The biggest difference between a web product and mobile device is the amount of testing and certification that must precede the launch of a mobile product. A smartphone such as the Pre is an incredibly complex and highly integrated piece of technology–much more so than a typical web application. First off, a smartphone contains a fully-functional OS, usually based on some variant of Linux running on very constrained hardware. It must perform all of its concurrent services utilizing limited memory and limited CPU horsepower. The smartphone must also respond correctly to the multitude of many current events, from those generated from the environment–like switching from wifi to a WAN internet connection–to handling data input from the user, as well as handling events from the onboard applications.

Launching a mobile product requires exhaustive certification of individual hardware components such as the CPU, modems, codecs, and displays. Even then, the finished product is really launched by the carrier and must go through their exhaustive certification tests as well. Testing an onboard mobile application is also a much harder testing task. There are so many conditions and constraints that are involved in testing a mobile application. A typical mobile application is nearly functionally equivalent to any counterpart desktop client-side web application. Take, for example, a mobile email application. It must behave and interact with the server-side application in nearly the same way as a desktop web client. The established protocols were designed for a stable communications environment, but this is just not the case in a mobile environment. The internet connection may be lost and reconnected very rapidly. The connection may even be lost for long periods of time. The application may, at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of system conditions happen in a smartphone that would rarely or never happen in the context of a desktop web client application. However, a mobile application must perform its main functional operations of retrieving and sending messages flawlessly with no loss of data and full operational integrity. Testing mobile applications under these environmental scenarios is a huge challenge. In short, testing a web application is no easy task, but mobile applications and products represent a much tougher and larger testing challenge.

uTest: There’s been a lot of bold predictions about the future of native apps, the mobile web, tablets, and just about anything that relates to mobile. Having lived (and worked) through the dotcom boom and bust of the last decade, do you see any similarities with regard to mobile?

ME: Not really. In the dotcom boom and bust era, the problem was such rampant speculation about how we as consumers would utilize the web for just about every type of activity and commerce transaction. It seemed at that time that anyone with a slick PowerPoint presentation and an outrageous business plan could get startup funding. The main problem was that the technology of the day just couldn’t deliver the user experience that all the web startup companies promised. Here we are, nearly a decade after the bust, and I believe we are just beginning to fulfill some of the wild web usage predictions and capitalization by users and consumers. That’s quite a gap between the vision that was promised and when it was actually delivered. No wonder there was such startup company roadkill during the bust.

Mobile computing, like any other surge in adoption of technology, will have winners and losers. However, there seems to be no end in sight for all things mobile. There is such innovation happening in this space. Take, for instance, the iPad. It is a game-changer, in my opinion. It has defined a whole new computing platform between what we consider a mobile handset and a laptop, and surely is adding to Apple’s vast bank vault. Smartphones continue to wow us with their capabilities and the ability to combine consumer-based technologies into such a convenient package, and we are quickly seeing how these devices are changing how we interact and communicate with each other. Mozilla encourages people to think big thoughts about the future of mobile computing. Just check out this video about the concept mobile phone called seabird and you will see what I mean. I sure would like the phone for Christmas.

So here’s the big difference I see between the dotcom boom/bust period a decade ago: In this mobile era we are leading with innovation and seeing how the marketplace utilizes the technologies. Where as, in the dotcom/bust era it was a lot of empty promises about technology and investor riches that ultimately were never delivered. The current stage in mobile and web computing in general is set for an explosive boom possibly much bigger and broader than than the dotcom period.  But this time I think it will be sustainable. Mobile computing and platforms are here to stay and will be the magnet for innovation for the foreseeable future. The current biggest hurdles are battery life and communication bandwidth. The latter is being addressed as we speak. 4G networks will have broadband connectivity. Hopefully battery life will also be improved shortly. A bright future ahead is my prediction.

uTest: In many ways, mobile is still playing catch up to the web. Is there one area in particular where you see the most room for improvement? If so, where?

ME: Well, there are some obvious platform deficiencies around inconsistent UI and whether Flash is going to be fully supported across mobile devices or not. But this is a testing blog, so let’s talk about that. As I mention elsewhere in this interview, mobile is just a really tough testing challenge. The big problem is that there is very little support for cross-platform mobile device test automation. I suspect most of mobile device and application testing is done 100% manually. If any environment needed more test automation, it is mobile. At Palm, we rolled our own test harness that ran on the Pre. This became extremely important for endurance testing and finding memory leaks in the Pre applications.

Mobile software companies have an uphill battle since developing automated system tests for every platform is very costly, both in time and resources. However, reliance on mostly manual testing has lots of quality risks. If the quality of mobile devices and software is to rise about what it is now, we need automated test tool support that works well across all device platforms.

uTest: What’s the best (and by that, we mean the worst) bug ever submitted by one of your community members?

ME: Recently, Alex Miller, a Mozilla community member, discovered a very critical security bug and was awarded $3000 for finding and reporting the bug. He’s been hard at work finding and discovering other security flaws in Firefox, too, and was even given clearance access to all Mozilla security-related bugs reported in Bugzilla. Very few people have this access.  Oh, I forgot to add a little fact about Alex: he’s only 12 years old. That’s an awesome accomplishment by a really smart kid. This exemplifies the opportunity Mozilla provides to the community: an incredible technology playground where anyone that spends the time to learn can participate at any level no matter who you are or what your background is. The more you prove what you can do, the more you will be encouraged and acknowledged for that effort. Finding bugs is a good place to start for anyone who wants to participate. Certainly, not everyone is going to develop the expertise to discover deep level security bugs, but believe me there is plenty of testing folks can really help us out with. If you are so inclined, we will welcome you with open arms. Please visit us here.

uTest: We keep reading about the skills shortage in Silicon Valley. Are you seeing this at all, particularly when it comes to software testers? If so, what do you suspect is the reason?  And how do you overcome this dearth of top-shelf talent?

ME: It’s always been hard to find good software testers. Good or bad economy, a seasoned and talented software tester is rarely unemployed. Typically, many good testers wind up in development because it often appears to be the growth path for a software technology career. Software testing as a career path is still viewed as something lesser than development. However, there has been a lot of gained respect for the software testing career path in recent years. The industry and practice has definitely matured since I started as a QA engineer when breaking software meant dumping your card deck all over the floor. So, I think the pool of quality testing folks is better now than ever, but it is still very hard and time consuming to finally hire someone. How do you fix this? We are always in search mode. We never stop looking for good QA engineers. Also having a team of great recruiters that can poach well is very key as well.

uTest: Which blogs/sites/magazines/conferences do you frequent to stay on top of the latest trends and tools in testing?

ME: I haven’t been to many conferences in the last ten years, except of course the recent GTAC conference in Hyderabad. Mostly, this is because I’ve been involved in startups which didn’t have the budget for sending folks to conferences, and I couldn’t afford the time as well. I have followed James Bach’s blog over the years; he’s as much an entertainer as he is a software tester. I also read a lot from Michael Bulton, mostly because I think he is the E. F. Hutton of the testing world. He is definitely worth listening to. Last year, I took some time off between jobs. It was a great time to have the luxury to really do a lot of web-surfing and getting caught up on things. I became rather fascinated with Yvette Francino. She’s a wonderful writer/blogger, and she chronicled her story of getting laid off as software test manager. It was quite the feel-good story to see how her use of blogging and online networking landed her a job as editor of SearchQuality.com, a site I still frequent today. Lastly, it was great to finally meet James Whittaker in person at GTAC this year. He’s quite a speaker, and I’ve learned a lot from his posts and books. I can’t wait to see how he will avenge being called the “octomom of testing books” at the GTAC conference .

uTest: What could software companies (eg: Palm/HP) learn from the open source movement about launching great products?  And vice versa?

ME: Traditional software companies are skeptical of open source tools and projects. They generally think that they have lesser quality that commercial counterparts. That may have been true some years ago, but I don’t think it is true today. Open source projects have one great advantage over commercially developed ones. The rate of updates and fixes is typically is much higher than commercial projects. I’m mostly talking about the bigger projects such as the popular linux distributions, apache tools and even Mozilla. We are unparalleled in our transparency and response to reported security flaws for instance. In fact being more transparent across the development cycle is a competitive advantage to open source. It encourages user feedback and therefore course corrections can be done incrementally and continuously and ultimately deliver a product a user is expecting. I think that software companies should be more transparent and allow their customers to participate more in the development process. What can open source projects learn from commercial companies? If there is anything it would probably be discipline around scope of the project. Too often, I see that open source projects get feature bloat or accumulate features that are cool to build but are not really needed by the user. Since, most open source developers are working on the projects for the pure enjoyment of it, they are often attracted to working on new stuff and are less interested in improving or fixing regression bugs.

uTest: Put on your prognosticator’s hat – what’s the next big “wow” feature or innovation in the world of browsers?

ME: I think Aza Raskin of Mozilla Labs fame has it right. Identity and browser identity awareness are the next big wow browser features in my opinion. Let me try to explain, although I’m sure Aza does a much better job of it. Your browser knows or could know all about you: your browsing habits, where you browse, what you buy, where you login to, who you communicate with, etc. etc. Currently, much if not all of this data is being captured on the web in many, many places. I won’t go into any of the arguments about whether your captured browsing habits and history are truly anonymous data or whether this data is being used only for commercial benevolence. But one thing is clear: you don’t own this data, and you can’t access or alter it either. And for many, that is a real problem. However, this data can be very useful to you. It is mostly used for a better experience on the web–or at least a merchant’s version of what it thinks will be a better experience for you. Well, your browser could do a much better job of it. It could capture this data directly and it would only be about you. This data could be stored locally, or even in the cloud using services such as Firefox Sync. It could do many cool things like real targeted pre-fetching of your favorite blogs, and then scan them for keywords that are important to you and arrange them in priority order. It could sense what tabs are used more often and arrange for a better workflow experience. I think there is a lot of assistance a browser could do either in the context of the browser interaction, e.g., automate tasks or perform operations based on the web content. Well, there is one huge obstacle, and that is security. How do we ensure that this personal data that absolutely defines you is kept safe and secure? And how do we prove it to you so that you feel safe in allowing a browser to continue capturing this data and making operational decisions based on this data? That may take awhile, but little by little we are making inroads towards this goal such as the Firefox Sync feature that securely stores your profile data in the cloud across instances of Firefox running on desktops and mobile devices. We are getting there, but I suspect it will be a while before you let your browser do your online banking for you.

uTest: What’s the greatest challenge that tech execs will face around app quality in 2011?

ME: The biggest challenge I see is around compatibility testing and maintaining a consistent user experience across the landscape of platforms. To be competitive in today’s web-enabled software market, you need to support a variety of platform interfaces. It is now expected that any web-accessible service will include feature-rich clients for iPhones, iPads, tablets, smartphones, desktops, and kiosks. The compatibility matrix starts to explode when you add in device models, OS versions, and other configuration-dependent attributes. In my experience, you do need to test on real hardware and configurations. Emulation capabilities do help, but at the end of the day you can not be sure of software behavior unless you test it on live hardware. Its a real problem that doesn’t seem to get better with the plethora of new platforms available to users.

Advertisement