Summary

The first thing that jumped out at me, was how few chapters there are.  This is the second “google” book I have read and just like the first it starts with many people jumping on the bandwagon about how great google is.  My opinion is that google is great, but not worthy of the pedestal that it is put on sometimes, as if it can do no wrong and even when it does wrong see rule #1.  It just gets a little old.  The book does not introduce earth shattering concepts (theory based items) but does have many innovative (shoot I’m starting to use their words — must …. not …. be …. assimilated) ideas.  As the book progresses, you understand why there are so few chapters.  They go into great detail about the roles and responsibilities of testing (not as boring as that summary statement might sound) and as the book draws to a close, they move up the food chain in the testing world.  So they emphasize what is happening in the trenches, not the board room.  But they shift modes when they move up the food chain to interviews and observations so that the content remains interesting and applicable.  I like the interviews more than I thought I would.  It was a good way to bring out opinions and ideas without those people appearing to have come down from Mount Olympus to spread knowledge to us mortals.  I did not get much out of the case studies, but I might have just been too tired by that point.

Top 3 Things I learned

1. Three types of tests, small (functions, like unit tests), medium (interaction between classes, like integration tests), and large (like end-to-end tests).
2. I liked the ACC (Attribute – Component – Capability) model for developing testing cases.
3. The last big thing I learned was at the end of the book and almost overlooked.  This was a comment by James Whittaker on why Google testing was so successful.  His keys are:

• Tester skill set
• Scarcity of testing resources
• Automation first
• Ability to iterate
• Rapidly integrate user feedback

---

 NOTES: Chapter 1: Introduction to Google Software Testing

SUMMARY: A very good introduction to what is different at google.  I think if you only read this one chapter you would not need to read the book because it does a very good job of introducing the key differences and why they matter.

IDEA: Testers are aligned under an organization called “Engineering Productivity” not the project or product to give them the final independence they need.

QUOTE: Scarcity brings clarity — Larry Page

IDEA: Don’t hire too many testers

IDEA: Don’t treat test and development as separate disciplines, test isn’t a separate practice

TERM: Software Engineer (SWE) has a feature focus

TERM: Software Engineer in Test (SET) has a quality focus.

TERM: Test Engineer (TE) has a user first focus

IDEA: Testers are on loan so that they move around a lot

IDEA: Different channels

• Canary: Daily builds.  If it fails to survive then the process has gotten chaotic
• Dev.  developers use for their day-to-day build  used for real work and sustained testing
• Test. best build of the month, passes the most sustained testing can be picked up and used for dogfood
• Beta/Release: survived internal usage and pass every quality bar.  First build to get external exposure

CONCEPT: Types of tests

• SMALL:
  • Automation: Mostly
  • Exercises: Single function or module
  • Written by: SWE (mostly)
  • Other: Requires Mocks and Fakes
  • Answers: Does this code do what it is supposed to do?
• MEDIUM:
  • Automation: Usually
  • Exercises: Two or more interacting features, nearest neighbor
  • Written by: SET drive, SWE heavily involved
  • Other: TEs may execute later
  • Answers: Does a set of near neighbors functions interoperate with each other the way they are supposed to?
• LARGE
  • Automation: Yes, but exploratory testing too
  • Exercises: Three or more features, represent real user scenarios, real user data sources
  • Written by: SWE, SET, TE
  • Other: Can take hours to run, results driven
  • Answers: Does the product operate the way a user would expect and produce the desired result

---

NOTES: Chapter 2: The Software Engineer in Test

SUMMARY: Explores everything about a system engineer in test (SET) including candidate interviews and interviews with some key SETs.  Introduces a few ideas and concepts then goes into great length on the interface protocol used.  A long discussion on the different tiers of testing: small, medium, and large.  A long discussion on what it means to be test certified, how they setup the process and the different levels they have.

IDEA: feature developer is about creating code, test developer is about breaking the code

• SWE: Feature developer
• SET: Test developer
• TE: User developer

IDEA: Bug in code and a bug in testing is treated no differently

IDEA: Google’s protocol buffer language.  Language-neutral, platform-neutral extensible mechanism for serializing structured data.  Written often by SETs

IDEA: SETs create mocks and fakes early when implementing interface protocols.

IDEA: Overinvesting in end-to-end automation often ties you to a product’s specific design, which isn’t useful until it is stable.

IDEA: Make the common case fast

CONCEPT: Test sizing

• Small
  • single unit of code
  • single class or a small group of related functions
  • “unit tests”
  • external must be mocked or faked.
  • Run quickly
  • Lead to cleaner code
  • mocking leads to well-defined interfaces between subsystems
  • catch bugs early
  • run reliably in all environments due to mocks
  • tighter, more focused scope
  • don’t exercise integration between modules
  • mocking can be challenging
  • mocked or faked can get out of sync with reality quickly
  • 70% of your tests
  • Execution time: 100ms to 1 minute
• Medium
  • One of more application modules
  • run longer and have a larger scope than small tests
  • test interaction across a limited subset of modules
  • integration tests
  • mocking is encourages but not required
  • stepping stone between small and large
  • relatively fast
  • run in standard development environment
  • account for behavior of external subsystems
  • can be nondeterministic
  • 20% of your tests
  • 1 sec to 5 minutes
• Large (and enormous)
  • any or all application subsystems from UI to backend
  • ultimately tests what is important
  • nondeterministic because of dependencies
  • when failure, might be difficult to understand why
  • data setup can be time consuming
  • impractical to exercise specific corner cases
  • 10% of your tests
  • up to 1 hour

IDEA: Tests must run independently and must have no persistent side effects.

IDEA: Randomize the order the test run in to catch dependencies you didn’t know existed.

IDEA: Running fewer tests at check-in helps isolate what has been broken.

CONCEPT: Test Certified

• Level 1
  • Tests can be red
  • Tests not executed before submit
  • Continuous build is in place
  • Build a smoke test suite
• Level 2
  • Beginning of integration testing
• Level 3
  • Tests for all nontrivial changes
• Level 4
  • Testing all submissions
• Level 5
  • Tests for all non-trivial bug fixes

IDEA: Challenges in adoption / who adopts best

• Challenges
  • Inertia
  • Bad Test
  • No Tests
  • Time
• Adoption
  • Those with Interest
  • Not too much legacy code
  • Testing Champion

---

NOTES: Chapter 3: The Test Engineer

SUMMARY: This section is all about the Test Engineer, someone who performs the tests and develops the test plans. This is the largest chapter in the book and is almost the entire book by itself.  There is a lot about test planning.  The chapter introduces and discusses Google’s ACC (Attribute, Component, Capability) concept.  The ACC concept is useful structure for test planning. There is a long discussion on Risk Analysis and how they go about mitigating the risk.  The chapter spends a lot of time on the various Google tools, Google Test Analytics (GTA), Google Test Case Manager (GTCM), Buganizer, Bots, and Browser Integrated Test Environment (BITE).  There is a good discussion on testing an application once it has gone into maintenance mode.  Some interesting stuff on how to interview a test engineer.  The chapter ends with an interview of two Google folks.

IDEA: Test Plans should have the briefest actual lifespan of any test artifact.  Things about a test plan

• Always up to date
• Intent of the software
• Snapshot of how the software is structure
• Describes what the software should do and how it does it
• Cannot take long to create
• Describe what must be tested
• Useful during testing
• Must lead directly to tests

CONCEPT: ACC – Attribute

• A – Attribute
  • Adjectives
  • What a marketing person would say
• C – Component
  • Nouns
  • What you are working on
• C – Capability
  • verbs
  • what you might look to do in the software
  • Really should have a test case, sometimes more than 1
• Captured in Google Test Analytics (GTA)

CONCEPT: Risk Analysis

• Process of determining risk. Comes down to:
  • Frequency of failure
  • Impact
• GTA has 4 frequency values
  • Rarely: hard to imagine
  • Seldom: low complexity or low usage would make such occurrences rare
  • Occasionally: somewhat complex, capability is one we expect to be popular
  • Often: high-use, high-complexity, experience failure on a regular basis
• Impact:
  • Minimal: user might not notice
  • Some: annoy a user
  • Considerable: block usage scenarios
  • Maximal: permanently damage the reputation of the product

IDEA: If you can’t test everything, test the most important stuff first and the most important stuff is the riskiest stuff.

IDEA: Write a test plan in 10 minutes, maybe you get 80 percent in 30 minutes.

IDEA: Nothing technical should ever be in a user story

TERM: Tours: High-level guidance for exploratory testing.

CONCEPT: Severity levels used by google for bugs:

• S0 = System Unstable
• S1 = High
• S2 = Medium
• S3 = Low
• s4 = No effect on system

CONCEPT: Life of an issue

• Unassign
• Needs clarification
• Not feasible
• Working as intended
• Not Repeatable
• Obsolete
• Duplicate
• Accept
• Fix now
• Fix later

IDEA: Bugs are like over-parented kids.  They get a lot of attention

CONCEPT: Maintenance mode

• Reduce manual tests
• Try to add some automation
• Remove flaky tests

IDEA: BITE (Browser Integrated Test Environment) is an experiment to bring everything testing to the browser and the cloud

---

NOTES: Chapter 4: The Test Engineering Manager

SUMMARY: The content in this chapter is very small.  It talks about the life of the TEM (Test Engineer Manager)  and then goes into interviews with many different TEMs.

IDEA: A TEM must:

• Know the product better than anyone
• Know their people

IDEA: Scarcity brings

• Clarity to execution
• Sense of ownership
• Forced to optimized

IDEA: Rotate every 18 mos avoids dependency on a person

CONCEPT: Transition: 1st quarter 80% old project, 20% new; 2nd quarter 20% old project, 80% new

IDEA: Focus on impact

INTERVIEW: Ankit Mehta (Gmail)

• Listen then do
• Ask Why
• Get right people with right skills
• Write tests in the same language the app is written in
• Avoid test being ignored by linking the person who wrote the feature writes the tests
• 20% of the use cases account for 80% of the use
• Holistic
• Focus on bug prevention not bug detection

INTERVIEW: Hung Dang (Android)

• Add value
• Be the product expert
• Be skeptical of automation
• Focused manual testing
• Everyone should be exploratory tester
• I can whine or provide value, choose the later

INTERVIEW: Joel Hynoski (Chrome)

• Application compatibility is key for browser based apps
• Test is the last frontier of engineering

INTERVIEW: Shelton Mar (Search and Geo)

• Key is deciding what to focus on
• Ask “What is really critical for the system under test?”
• Feels you should automate as much as possible

INTERVIEW: Ashish Kumar (Engineering Tools)

• Types of tools they build: Source, Development, and Build Infrastructure
• Continuous integration at scale
• Felt that remote pair programming tool did not succeed
• When starting, make it dead easy
• Define dependencies cleanly, make them explicit, set up a continuous integration system

INTERVIEW: Sujay Sahni (India Testing)

• Follow the sun, meaning working in different time zones and having a good method for handing off code
• Crowd testing

INTERVIEW: Brad Green (Engineering Manager)

• Things that have changed: the average developer has become much more involved and they have been able to attract top notch talent
• Metrics are hard to generalize because context is important
• Best tool to get a team to write tests is competition
• Automated clustering of similar problems reported through feedback

INTERVIEW: James Whittaker (The Man)

• Key ingredients to google’s testing magic
  • Tester skill set
  • Scarcity of testing resources
  • Automation first
  • Ability to iterate
  • Rapidly integrate user feedback

---

NOTES: Chapter 5: Improving How Google Tests Software

SUMMARY: A very short chapter, almost self-deprecating.   Talks about fatal flaws in their process (not keen on the separation between test and development), the future of the SET role (they feel it will go away), the future of the TE (move to more specialized roles).

IDEAS: 

• Google is simply no longer best served by keeping development and testing separate
• Testing is often seen as a surrogate for quality
• Testers have become a crutch for developers
• Fatal flaw to have testing and engineering in two separate organizational boundaries
• Fatal flaw: testers worship artifacts over the software itself.  The value of testing is in the activity not the artifacts
• They have seen the life span of a bug decline from months to minutes because of better software delivery and better user feedback.
• An army of actual users is always better than a handful of professional testers
• Fail fast

IDEA: Things that have influenced software development

• Agile development
• Continuous builds
• Early user involvement
• Crowd-based testing
• Online software delivery
• NOTE: Sticking to decades-old testing dogma is a recipe for irrelevance

Appendix A: Chrome OS Test Plan: A case study

Appendix B: Test Tours for Chrome: A case study

Appendix C: Blog Posts on Tools and Code: A case study