Regular Expressions: A Tester’s Best Friend

Okay, I feel I am entitled to a bit of hyperbole, but I think regular expressions are great.  I like them for two reasons.  One is they are a programming language, and many programmers do not know regular expressions very well.  It is always great when the testing team has a technical advantage over the application developers.

The first reason is a bit lame.  The real reason I like regular expressions is that they enable you to have some much power in your automated tests while keeping the test framework very simple.

For those unfamiliar with regular expressions, regular expressions enable you to match patterns in strings of text using a simple syntax of reserved characters and patterns.  Regular expressions range from something simple like “Bob.*Jones” where “.*” is essentially a wildcard where matches include “Bob the wildman Jones” and “Bob Jones”  You can also make regular expressions to make sure that a value matches a date format or social security format or use a regular expression match an application validation message where the message changes in some predictable way when a test runs.

In our test framework, we use regular expressions for all automation components involved with verification.  For example, we use many “verify data” components.  In these components, you enter the expected value for a field in the application under test.  All our fields are regEx enabled, and typically this means that you enter an exact string value or a regular expression pattern to match in the test.  This means you have a very simple test with a lot of power.

Implementation note:  There was a point of the development of our automation framework, we had to make decisions about whether to use regular expressions and how to use them.  As much as it pains me to say, the biggest factor was whether the use of regular expressions would be too hard for testers.  Fortunately, we decided that testers are smart enough to handle regular expressions (and of course, they are!)

Let’s Write an Automated Test Framework

I am a huge one-time fan of HP’s BPT framework.  Test execution may be slow, but it is the best tool I have found for a growing test team to use in a whole-team test automation environment.  BPT has powerful abstraction of test scripts to make it easy for business-focused people and new test team members to build tests.  It enables automators to build tests using simple keyword-driven techniques and more powerful and sophisticated test scripts when the situation demands it.  As soon as I learned to ignore the misguided documentation from HP on how to use BPT, it became a very powerful tool.

Over the past year, I have been struggling with the new (and supposedly improved) version of QC/BPT.  The past year has been a mess.  What was a promising enterprise solution for long-term use has, in my mind, crumbled into a pile of what-could-have-been.  As it looks now, it is time to write an open-source automated test framework.

There are many others who have gone down this path.  Most have mixed results (we have looked for other frameworks that meet our needs and haven’t been satisfied with anything yet).  I am still convinced that BPT (as we implemented it) is the best test automation framework out there.  And since so few other people have found BPT (or found it to be a suitable solution for them), I think my team is uniquely (I am sure we are not alone) suited to drive the development of this project.  We have seen what works well.  We have solved the key problems of automation Fire Swamp -- Remember the movie the Princess Bride (1987) when Westley and Buttercup escaped from the Lightning Quicksand?

Westley: No, no. We have already succeeded. I mean, what are the three terrors of the Fire Swamp? One, the flame spurt - no problem. There's a popping sound preceding each; we can avoid that. Two, the lightning sand, which you were clever enough to discover what that looks like, so in the future we can avoid that too.

Of course, in the next moment, he learns about the Rodents of Unusual Size (ROUS) -- the third of the problems.

In the case of test automation, the three terrors are lack of readability, lack of maintainability, and lack of flexibility.  By using our implementation of BPT, we know what problems to avoid.

Look out HP, I think we might write a framework.

“Whole-team” Test Automation

At my company, we practice the art of “whole-team test automation.”  Very simply, this means that test automation is not a specialized skill on our test team.  For us, the ability to develop automated tests is a required skill for all members of our test team.  At our company, highly skilled automated test engineers are common as dirt.

If you thinking that this seems like a lot of work, you are right.  It takes an effort.  So, why would we invest in this idea?

• In agile development, rapid development of automated tests is critical.  While we still struggle with keeping up with automation during sprint work, the option is always there for the testers and the scrum teams to do what is best for the project.  This may mean aggressively developing regression tests throughout the project.  It is not acceptable for one project to have good coverage just because they happen to have the good automator – all projects deserve technically skilled testers.
• At many companies, automated tests are developed, executed, and owned by a separate group within the test team. This creates a two-tiered test team.  It creates a situation where the most technical members become more technical because they have the most practice and the least technical members have no opportunities to grow their skills.  That sucks if you not technical and have no opportunity to grow.
• The more technical a tester is, the more tools the tester has in his toolkit, and the better the tester will adapt to new situations.  The new and changing situations range from new technologies such as services testing to a fundamental change in how software testing is done.  Adapt or die.
• Agile testers are developers.  As members of a software development team, everyone should know how to code.  It is good for your self esteem, and it helps you gain and keep the respect of the other developers on your team.

If you are convinced that we are on to something, how do you go about it?

• Build or select a suitable test framework and implementation methodology, and develop a transparent approach to testing.  For this, we chose HP’s BPT framework with QTP as the underlying automation engine.  Using BPT and our “Archetype” implementation technique, all members work together and take advantage previous work and each other’s skills.  With a good framework, new team members can build tests, and experienced automators have opportunities to work on challenging work.
• Good recruiting.  Regular, run-of-the-mill “QAs” are not right people for the team.  Be prepared for this to take a long time.  Because of the history of testing (a combination of “QA” process types and non-technical manual script runners), many good people stay away from testing jobs.  My only advice here is to start with high standards, don’t let your standards drop, and keep hope.  Great people are out there.  And in my opinion, most recruiters will not be your friend.  They will pressure you to take conventional candidates.  Be aware of recruiters.
• Be prepared to train people .  Some of the most obviously experienced candidates will take the most time.  Many experienced automators do not work in a whole-team automation environment.  As a result, they tend to build tests for themselves and not for the team.  There is often “unlearning” that has to happen.  
• Promote technical skills. Learn new programming languages and new technologies.  This year when we studied Ruby together, we spent some time looking at Rails projects.  By looking at and studying Rails, we accidentally learned about MVC frameworks.  
• Devote enough time. In addition to the sprint work that we do (with or without other testers), we devote an hour each day to work together on automation.  This way, we share ideas, learn from each other, set high expectations for each other, and continue to practice our art.  There is a bonus with this approach when you are working on projects that are not interesting.  If your project stinks, you can work with other team members on their automation needs.  Everyone wins.

As you live with the “whole-team” automated test philosophy, be prepared for things you do not predict.  Train your team, give them the tools and support, give up control, and let them surprise you.

Migration from BPT 10 to 11: Changes to Data Tables

In standard QTP, data tables are the work horses for managing test data.  They are used for data driving tests, passing data from one action to another, storing large volumes of test results, and for iterating tests.  In BPT, data tables take a back seat to other means of managing test data, most notably, test parameter data.  But even in BPT, there is a place for data tables.  For example, test data read from associated Excel files can be read in at run time and table data from the application under test can be stored in the component data table and reviewed in the test logs.

In BPT 10, each component has a local datasheet.  Because each component runs independently (remember the slow execution speed of BPT 10 – one component is loaded, it runs, the component is closed, the next component is loaded, it runs, …), there is no global data sheet.

There is a key benefit in BPT 10 to the way tests are constructed.  In a component, you can create data tables named “Actual” and “Expected.”  If this component is used more than once in a test, you don’t have to worry about name contention – the scope of data table names is in a single instance of a component.  In standard QTP, all data tables are exposed to the entire test and you have to make sure that no data tables share the same name. In BPT 11, tests share much more in common with standard QTP tests, and data tables must now be unique.

In my migration from BPT 10 to 11, I had to rethink some of my key assumptions about how I use data tables.

1. Because of the appearance of the global data sheet in the business component, I changed from using the DataTable.Import method to DataTable.ImportSheet.  The Import method loads the Excel sheet into the first data table. In BPT 10, the first data table is a local sheet; in BPT 11, the first data table is the global sheet. 

A surprise change for me when I ran my components migrated from 10 in 11 is that the imported data was not where I expected it to be.  This caused verifications to fail.  I was further surprised when some of my tests ran extra iterations.  Tests ran extra iterations because the import method added rows to the global datasheet, and the global datasheet determines the number of iterations a test runs.

ImportSheet enables you to specify which sheet to import and where to place it.  When using ImportSheet, you have to know the name of the data sheet you want to overwrite – if your test iterates, this must be a unique name.

2. Because the scope of data sheet names is not longer local to business components, you must manage the names of the sheets and make sure the names are unique.  As you do this, consider analysis of test results.  If your test has 10 iterations, there may be 10 instances of the “Expected_Result” sheet.  Getting the iteration value of component at run time is difficult so I chose to make the sheets unique by adding part of a time stamp to the name.  Ten instances of a sheet with similar names is difficult to analyze.  To give clarity, I added a line to the log file identifying which data sheet goes with which iteration.

Here is a summary of the changes I made:

BPT 10:

DataTable.Import GetCurrentTestPath() & "\" & strQCExcelSheet

BPT 11:

strUniqueSheetName = “Reference_” & Minute(Now) & Second(Now)

Reporter.ReportEvent micDone, strUniqueSheetName, “Look for reference data at “ & strUniqueSheetName

DataTable.AddSheets(strUniqueSheetName)

DataTable.ImportSheet GetCurrentTestPath() & "\" & strQCExcelSheet, 1, strUniqueSheetName

 

Generally, the migration from BPT 10 to 11 did not require changes to the automation code.  Underlying BPT is QTP, and QTP code did not change.  The change in how BPT handles data tables may be unnoticed in many implementations, depending on which features of BPT are used.  For those who push the limits of how BPT handles data, this undocumented change to data tables will be a stumbling block.  I suppose there is a price to pay to get BPT to perform at an acceptable speed.  I’ll pay this one.

How HP Failed BPT

HP failed Business Process Testing (BPT) when it rolled the automated test framework out several years ago, and it has not really done much since.

HP marketed BPT as a way for non-technical content experts to build automated tests.  The original tutorials showed an implementation where the non-technical content experts designed and built components (the reusable building blocks of tests) by simply stringing together objects and keywords.  Then, the content experts drag the components together to build tests.  That tutorial demonstrated the functionality of BPT, but it failed to even give a hint about how to implement it successfully.

Non-technical content experts are the wrong group to use when building the infrastructure for a test system.  The system is good, but it is not automagic.

Non-technical content experts should never be the ones to decide the scope and purpose of components.  The components are way too important, and the approach is way too random.  There are too many ways to define a component, and if you don’t have a pattern or model in mind, you will end up with a bunch of components that are redundant, inefficient, and hard to maintain.  The strategy for designing components should be treated like a software development activity.  An automator should be involved here.

In addition to confusing the issue of who should design components, HP provided no advice on how to design them, and there is still a fair amount of confusion about this.  For example, should we build components that perform a business process?  That makes sense based on the name, “Business Process Testing.”  But, when you have a component called “Register a New Member,” you are limited to using the same positive flow.  How would you use that component if you wanted to verify negative tests such as what happens when you attempt to register a new member with an invalid social security number?  Business processes like this would make sense if you only wanted to test “happy path” cases.

After this confusing rollout, BPT languished without a lot of attention from HP or the testing community.  This has been really frustrating to those of us who use and see the value of BPT.  As I see it, BPT is the perfect tool for agile testing.  It is a great tool for getting the power of automation to all members of the team without being limited by the level of technical skills.