Interview of Patrick Copeland

This interview was conducted by uTest (http://blog.utest.com/testing-the-limits-with-patrick-copeland-part-i/2010/02/)

In this month’s Testing the Limits interview, we’ll put Patrick Copeland on the hot seat. Patrick is the Senior Engineering Director for a promising young upstart named Google (we’re not familiar with them ourselves, but we’ve heard good things) where he oversees a global team of about 800 engineers. But this isn’t his first rodeo –  prior to Google, Patrick spent a decade at Microsoft, where he specialized in all things related to software engineering.

So what do you ask someone who’s probably forgotten more about software than we’ll ever know? Well, in this installment, we’re going to get his views on catering to a global base of users; his criteria for evaluating testers based on their “tester DNA”; the recent addition of our good friend James Whittaker; the challenges of launching new products like the Nexus One, as well as other tidbits from inside the GooglePlex. Stay tuned for Parts II and III in the days ahead.

uTest: What are some of the challenges that come with having a global base of customers and users? Are certain products noticeably more popular in some areas rather than others? And how does this affect your future planning?

PC: Yes, of course some products and features do better than others. Our approach is to do lots of experimentation and to release and iterate. We push bits to customers early and often, and then we listen and watch usage. Customers help us by “voting with their feet.” Popular features and products are improved, and poorly performing products are deprecated. With a big focus on innovation, we also need to “fail fast” and customer feedback helps us make those decisions.

Not surprising, our global customers have different demands of our products. We want products to “feel local” and we need to support features that may be unique to specific markets. For instance, in Indic based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we can’t find internal talent, community-based testing is an interesting solution to this challenge.

We base staffing and planning decisions on several criteria:

• Strategic: Maybe a new feature, but in a market with existing competition (like Android).
• Financial: Obviously Ads and Search, but we have several emerging businesses that are also getting important.
• Customer usage: For example, popular high-traffic applications like GMail.
• Legal or Compliance: Certain areas need to be prioritized high for legal reasons. For example, SOX compliance for CheckOut.
• Ability to Impact: We look at our capability and decide if investing testers in an area would have a significant impact.

uTest: A few years back, you were the keynote speaker at GTAC, where you said something to the effect that “the longer I’ve been in the business, the less I know about it.” How important is it for testers and developers (and those who manage them) to maintain this student-for-life mindset?

PC: Very. When I hire people I look for folks with a “testing DNA.” These are people who are great computer scientists at their core, but also are very curious, love software, and are passionate about test engineering. People who have those characteristics tend to pursue challenges and continue to learn.

By the way, what I meant is that the surface area of the challenges we face in software has grown exponentially over time. Just in the past few years we’ve witnessed a paradigm shift to cloud computing that stretches our imagination and challenges the limits of software. Our process for developing that software is going through an equally dramatic revolution. We are reconsidering the appropriateness of the lessons we’ve taken from traditional software development companies. Testing teams need to continue to evolve quickly and innovate just keep up with what’s happening in the industry.

uTest: Not too long ago, you added testing guru and author, James Whittaker to the Google team, who is well-known to the uTest community. Why was he such an important addition, and has there been anything that’s surprised you since he came on board?

PC: James has been a great addition to the team. His passion for testing is incredible. The biggest surprise is that he’s having fun being a manager. James accepted our offer without knowing exactly what he was going to do. This is a pretty common situation for new senior people at Google. I remember giving him our offer and he said repeatedly, “I’m not interested in being a manager.” I kept saying, “ok, ok, no manager position for you.” But he’s decided to run the Kirkland and Seattle Test Teams, in addition to several groups in Mountain View. He loves it. I think that might even be a surprise even to him.

uTest: Google seems to break into another market every week, with the new Nexus One phone being the most recent example. How much of your time is spent looking forward to new apps and categories, as opposed to concentrating on existing products?

PC: At Google we have a 70-20-10 rule for investing in technology. While the numbers are not carved in stone, the basic concept is that we invest ~70% on our existing core products, ~20% in related but new categories and ~10% on riskier bets or areas outside our existing core competencies.

Innovation is a high order bit of course. We’re having to do a lot of innovation on supportive infrastructure to keep up with the pace. For instance, we’ve created an infrastructure that allows us to innovate very rapidly while maintaining high quality despite a huge and complex code base. On any given day, there are thousands of projects under development with over 5000 CPU cores producing 65K compilations of build targets, and over 100M executed test suites. The build and test system accommodates the load, and an annual growth rate of 75%, using distributed execution, caching, and work avoidance techniques. The average build and test cycle take just 4 minutes and we are quickly moving toward a model where developers can receive results before they formally check-in code changes (‘pre-submit checking’).

In our model, product teams maintain a build that’s always green (compiling and testing without error). We built the supporting system based on three priorities:

• Speed: All test and analysis systems need to return results very fast. If it takes too long, engineers will either ignore or not bother looking for that data.
• Feedback: The focus of test systems must be on high quality feedback. We want engineers to keep code at production quality at all times, not fixing code that is broken.
• Simplicity: Engineers should not have to understand how the underlying systems work. All data and feedback must be easy to understand, integrated into commonly-used productivity tools, and presented in a workflow that allows them to take appropriate action.

uTest: What are some of the challenges that come with managing teams in dozen (or more) countries, as you’re currently doing? How difficult is it to maintain control over the people, processes and products? And when do you sleep?

PC: “Maintaining control over people” <smiling and laughing like Dr. Evil>.

But that’s not how it works at Google. The truth is…our team structure is atypical in the industry. For one, we are a flat company with many Nooglers being a few steps below senior executives. The expectation is that people and teams are semi-autonomous. In this model it’s impractical for managers to be controllers. And regardless, I’d rather set up teams that are made of great people who can run their areas themselves. My focus is on helping teams to be effective. Managers at Google are generally judged on their ability to enable smart people to get things done. Many have 15 or more direct reports, introducing some chaos and reducing the time available to micromanage.

One way we get everyone moving in a similar direction is to use OKRs, it came to Google thanks to board member John Doerr back in 2000. John stressed the importance of setting overall company Objectives and Key Results that would help develop departmental objectives; in turn, individual OKRs for every employee would support achievement of team and company wide goals. In Q1 of 2000, we rolled out our first company-wide OKRs, which included “8 million searches/day” and “Select CEO.” We’ve come a long way since then.

uTest: A lot’s been made of the unique and friendly work environment Google offers its employees. Does this also apply to your engineers? Or are they handcuffed to their desks and given food pellets for every line of code written (like we do at uTest)? Seriously though, how does an open atmosphere lend itself to better software?

PC: We have a pretty open culture where engineers have a good degree of freedom to invest in areas that interest them. We have an idea called “20% time” where engineers are encourage to invest a portion of their time in projects outside of their primary area.

Does culture help to make better software? I think so, but it’s a tough thing to quantify. On occasion the idea of collecting and measuring “productivity metrics” has come up (ex. lines of code, # of check-ins). We uniformly resist this because similar metrics can result in undesirable outcomes as people try to “succeed” in the system. Our biggest measure of performance, besides the velocity of innovation being delivered to customers, is quarterly peer reviews. This type of peer system reinforces ideas like teamwork, making sure you are having an impact, and building respect from peers. It’s more subjective, but harder to “game” and, ultimately, much more valuable in reflecting the real value and contribution of an individual.

BTW, “Food-pellets” wouldn’t work because our engineers already have free access to gourmet food.

uTest: What’s the difference between a good tester and a great tester?

PC: Good testers can be trained. Among many others, the basics you need to be good are: depth in computer fundamentals, an understanding of the application domain, an strong understanding of the use case, empathy of the user, mastery of metrics, and a focus on the development process.

Great testers, meaning the 90th percentile, are rare and special. Not everyone can be truly great. In my experience great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. From the 100s of interviews I’ve done, “great” boils down to: 1) a special predisposition to finding problems and 2) a passion for testing to go along with that predisposition. In other words, they love testing and they are good at it. They also appreciate that the challenges of testing are, more often than not, equal or greater than the challenges of programming. A great “career” tester with the testing gene and the right attitude will always be able to find a job. They are gold.

uTest: What’s the greatest barrier to designing, developing and launching high-quality products on a consistent basis (deadlines, prioritizing, competing visions, etc)? Said different, why can’t every company launch world-class apps every time?

PC: Every product I’ve worked on required a unique approach. There are so many variables in play and most of them are situational. Some of them you can control almost completely (e.g. technology choices). Many others you can control somewhat. Others still are completely outside of your control (ex. what the competition decides to do).

Some companies try to normalize their development process. One thread common to formal development models are that they focus on a few of the many variables: improving efficiency, predictable process, estimation of quality, or others. Process-heavy development models may work well for manufacturing airplanes and have been successfully applied by some companies, they have been viewed by many developers as burdensome and contrary to the creative nature of writing innovative software. Conversely, “process-less process”, can lead to a heroic culture that’s unable to repeatedly deliver. There needs to be balance.

Consider the physics of flight as an analogy to software process. In addition to reasonable flying conditions and an experienced pilot, the key to getting airborne is having an appropriate balance of factors that match the situation: too much weight or too little thrust can be disastrous depending on the situation. Similarly, teams, products and process all have virtual physics. For instance, adding engineers late in a product cycle doesn’t necessarily provide more lift. Adopting a new process may give a team more thrust momentarily, but may also incur a longer term drag that makes them incapable of innovation.

The popularity of Agile, while not a wholesale rejection of more rigid processes, indicates that developers desire more balance and creativity. Whatever we do to make software higher quality and more predictable to build, we must maintain a balance that encourages the innovative aspects of the art form. We need to motivate smart minds to solve hard problems and deliver rich features to customers. In other words, we need to be adaptable to stay airborne for the long term.

uTest: What advice would you give to young software turks who aspire to become a senior tech exec at a global powerhouse?

PC: Here’s my advice:

1. Be technical: At Google all of our engineering execs have very strong technical backgrounds. They were programmers and many of them can – and do – still program today. Advice: remember your computer science and stay sharp.
2. Be a connector and innovator: One exec commented that his job could be described as being “a very expensive email router.” Get connecting within your company and establish a network of strong peers.
3. Drive your career: Don’t wait for someone to recognize you. Inform people about your plans and successes. You are your own best billboard. People will create a myth about you, like it or not. If you are smart you’ll help them create the myth you want them to have.
4. Play to your strengths: Pick projects that you are passionate about and where you can rock worlds.
5. Bite the hand that feeds you (a little): Don’t be a jerk, but make sure you don’t go passively along with bad ideas. A weak willed tester is a bad thing. Make your concerns known…and know when to fold.
6. Go where there’s opportunity for growth: Not all projects or companies can offer you an ability to grow. Move when you think you are topping out. The biggest mistake I’ve seen is staying too long in a place that hinders growth.

uTest: Google is booming in the mobile apps space; what QA testing challenges do mobile apps present that web or desktop apps do not?

PC: Where do I begin?

A good starting place is the huge diversity of devices. For instance, significant variations in HW and SW: OS, memory, processors, screen size, and input methods. Just having access to a fraction of the devices on the market for testing purposes is a challenge.

On top of that, if you want any automation, you have the problem of simulating input and capturing the actual output on the device. We needed to develop tools that allow up to test and build against multiple platforms. Normalizing the testing is tough. We debated record and playback vs writing abstraction layers vs other methods. All can be brittle in the extreme. We also needed to think about the long term cost of maintaining tests suites on fast moving SUTs.

A couple other issues are the carrier networks and the numerous languages. As I said, I could go on for a long time, but I believe I hit some of the major pain points. Even with all of the automation in the world, a community based test approach can be complimentary.

uTest: Is there a particular programming language to which you are loyal (PHP, Java, Ruby, etc)?

PC: Nope. Not really. At Google the most common ones are Java, JavaScript, Python, Perl, and C++. Perhaps a personal failing, but I still think in C and my code ends up looking kind of C-ish even if it’s in another language. My preference also depends on the thing I’m trying to do, obviously each has strengths and weaknesses.

On a similar topic, I recently met famed computer language innovator, Niklaus Wirth at the last GTAC conference in Zurich. He had in interesting comment on testing and quality, “The experience, judgment, and intuition of programmers who have survived the rigors of testing are what make programs of the present day useful, efficient, and correct.” In other words, you can’t test quality in and quality comes from experience.

uTest: Any new year’s resolutions for Google’s testing & QA efforts?

PC: Yes. Every year I write up a “one pager” describing what I think we need to do. I try to stay focused on general productivity and not specifically testing. Testing is one tool among many we use to make software high quality and delivery faster. We are doing well on many levels, but we aspire to do even better. Here are a few of the high level bullets:

• Greater release velocity. We apply appropriate use of automation, tools, process, and information that solve Google challenges. We introduce and use metrics that fit naturally into the product cycle. We are introducing key metrics that are actionable and representative of the health of the project that allow team members to understand issues and take action quickly.
• Cohesive tools. We’ve built an impressive set of tools that have made significant impact. In our next stage, we need to better integrate some of the data and tools and focus on accelerating the end to end developer workflow. This includes identifying bottle necks that slow people down and keeping an eye on innovative ideas coming from product teams. And of course, continual improvement of build and test speed.
• Driving improvement upstream. We will take advantage of our ability to make good ideas useful across the company. We use our broad view of projects to push adoption of ideas and methodologies that work. We look for ways to compliment existing product strategy by identifying changes where the adoption is in a project’s self interest. We constantly evaluate our own practices by looking at their effectiveness in relation to customer and business needs.

uTest: You went to school at University of Arizona, and USC; you’ve worked at Microsoft and Google; are you a lifetime west coaster or has it just worked out that way? Also, Dodgers or Giants (or D-backs)?

PC: Are there any good East Coast teams? I heard it was so foggy the other day that the Washington Nationals couldn’t even see who was beating them.

They may not be sports teams, but I’m a fan of the testing teams we have on the east coast: NYC “Bubble Sorts”, Boston “Red SOX”, and Pittsburgh “Open Sourcers”. BTW, we just hired a Test Director on the East Coast named John Turek who’s looking for a few good players.

BTW, I actually have lived in other places. My family lived in Maryland and Tallahassee for a bit. I also lived in Madagascar, which is definably on the East Coast (of Africa).

uTest: What sites, blogs or magazines do you read to stay ahead of the curve in the areas of business, technology, etc?

PC: Track and Field, Cosmo and, occasionally, Monster Truck Weekly . Seriously, I have a bunch of custom news feeds on certain topics, like Google, our competitors, and testing. I also read wsj on line and at least one interesting industry technical paper per quarter.