Thursday, November 10, 2011

“Whole-team” Test Automation

At my company, we practice the art of “whole-team test automation.”  Very simply, this means that test automation is not a specialized skill on our test team.  For us, the ability to develop automated tests is a required skill for all members of our test team.  At our company, highly skilled automated test engineers are common as dirt.

If you thinking that this seems like a lot of work, you are right.  It takes an effort.  So, why would we invest in this idea?
  • In agile development, rapid development of automated tests is critical.  While we still struggle with keeping up with automation during sprint work, the option is always there for the testers and the scrum teams to do what is best for the project.  This may mean aggressively developing regression tests throughout the project.  It is not acceptable for one project to have good coverage just because they happen to have the good automator – all projects deserve technically skilled testers.
  • At many companies, automated tests are developed, executed, and owned by a separate group within the test team. This creates a two-tiered test team.  It creates a situation where the most technical members become more technical because they have the most practice and the least technical members have no opportunities to grow their skills.  That sucks if you not technical and have no opportunity to grow.
  • The more technical a tester is, the more tools the tester has in his toolkit, and the better the tester will adapt to new situations.  The new and changing situations range from new technologies such as services testing to a fundamental change in how software testing is done.  Adapt or die.
  • Agile testers are developers.  As members of a software development team, everyone should know how to code.  It is good for your self esteem, and it helps you gain and keep the respect of the other developers on your team.
If you are convinced that we are on to something, how do you go about it?
  • Build or select a suitable test framework and implementation methodology, and develop a transparent approach to testing.  For this, we chose HP’s BPT framework with QTP as the underlying automation engine.  Using BPT and our “Archetype” implementation technique, all members work together and take advantage previous work and each other’s skills.  With a good framework, new team members can build tests, and experienced automators have opportunities to work on challenging work.
  • Good recruiting.  Regular, run-of-the-mill “QAs” are not right people for the team.  Be prepared for this to take a long time.  Because of the history of testing (a combination of “QA” process types and non-technical manual script runners), many good people stay away from testing jobs.  My only advice here is to start with high standards, don’t let your standards drop, and keep hope.  Great people are out there.  And in my opinion, most recruiters will not be your friend.  They will pressure you to take conventional candidates.  Be aware of recruiters.
  • Be prepared to train people .  Some of the most obviously experienced candidates will take the most time.  Many experienced automators do not work in a whole-team automation environment.  As a result, they tend to build tests for themselves and not for the team.  There is often “unlearning” that has to happen.  
  • Promote technical skills. Learn new programming languages and new technologies.  This year when we studied Ruby together, we spent some time looking at Rails projects.  By looking at and studying Rails, we accidentally learned about MVC frameworks.  
  • Devote enough time. In addition to the sprint work that we do (with or without other testers), we devote an hour each day to work together on automation.  This way, we share ideas, learn from each other, set high expectations for each other, and continue to practice our art.  There is a bonus with this approach when you are working on projects that are not interesting.  If your project stinks, you can work with other team members on their automation needs.  Everyone wins.
As you live with the “whole-team” automated test philosophy, be prepared for things you do not predict.  Train your team, give them the tools and support, give up control, and let them surprise you.


  1. Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.

  2. Hi Bob,
    I read the long version of this in Stickyminds. Question: how you manage the long-term vision for test automation in your organization? Our management only has product owners that keep backlogs for 'product' issues. Everything else is (theoretically) supposed to belong to the developers. That the developers should be free to create and organize whatever tools they need to succeed. But we have no structures at all for making any of that happen. And all the people with any organizational skills (ie. managers) are either product owners, scrum masters or sit outside of the scrum/sprint process altogether.

    So tell me how you organize the long term thinking in your organization.
    Larry Fast

    1. Hi Larry, sorry for not seeing this sooner. I suppose my answer to this is that I won the organizational jackpot -- I have a lot of support. Even with the support, results win the day. We run our regression tests every day and the analysis of our tests (done in Europe) determines the start of the day for all our development teams. The early successes wet the appetite for more automation and technical testing.
      I have a couple other things in our favor. As a development organization, we allocate time for maintenance -- maintenance means development prioritized work as opposed to business prioritized work. With this resource, we improve infrastructure, refactor code, and improve testing. The other thing in our favor is that testing is on par with programming -- we are all developers. With that mindset, we have the ability to develop and to manage to our own backlog.
      We have been doing this for several years now, and now both product owners and other members of the development teams expect us to work this way.
      The only other thing that comes to mind now about how we got this started is just getting started. The first successes (the ones that helped get everything started) came from after hours work. I don't think I would have been as successful if I would have asked for resources to build our first tools. But already built and able to provide value, the tools were well received and it was easier to build the structure around them.
      Hope this helps...Bob

  3. Test automation in its true sense is writing and maintaining code. You invest thousands of dollars in modern test automation tools.STC Technologies|STC Technologies