When I was first employed at Gilmond as a Test Automation engineer, I was brought in to lead the charge on getting our current products under some form of automation. Having just come from an environment where this is done as standard, I was very excited to get stuck in. However, when I realised the size of the task ahead, my optimism was quickly dashed.

Our systems had been developed in a way that didn’t consider how to test them properly or even consistently, let alone factor in the hooks needed to retro-fit automation. I wouldn’t recommend retro fitting automation if you can get away with it, because it’s expensive to do and usually very high maintenance. However, if you wish to be ‘Agile’ and release after every sprint (every two weeks in our case), then the sprint needs to include regression which is automated.

We had also only recently changed from using a Waterfall methodology to an Agile methodology. This is usually painful for many teams/companies because there are a few things which need to be addressed to make ‘Agile’ work.

To provide a very brief illustrative summary of our ‘old world’ methods, we would gather requirements and create documents which developers would use to create the code and testers would use to write test scripts. However this creates a suite of test cases which need to be manually executed to regress the system before a release. As features are developed, regression tests increase which has a time impact. Automation is usually an after thought (as in this case) and to do automation well you need to design the product with it in mind.

One of the reasons for changing to our newer methods, in summary, is the idea to have a releasable build/product at the end of each sprint which can then be shipped. This allows us to deliver value add to the product and our customers sooner, and react more quickly to changes in the market. Timing is key - if you have a product which takes two weeks to regress and you only have a two week sprint, there’s not a lot of time for developing new features. If you continue with manual testing, eventually regression will take longer than two weeks and the whole approach falls down a bit. Simply reducing the number of tests isn’t a viable option for me as if you decide not to test features then it’s the equivalent of saying that it is ‘OK’ for that code not to work in production. So the only real option is to automate.

There are many different automation approaches - unfortunately most of them are expensive, high maintenance and can sometimes be more trouble than they are worth. However, in my experience, Test Driven Development or Behavioural Driven Development is an effective way of automating and keeping maintenance costs down. There is also the bi product of living documentation, which reduces the need for other requirement documents (which usually end up out of date).

Given the current situation with testing tools, and legacy systems we had to work with I took the approach to try to change practices and attitudes towards these approaches. As many QA’s will know, it is a challenge as you need to tell developers to change practices to incorporate the techniques, and many don’t like to listen to QA engineers telling them to code in a different way to how they have for the past 10 or 20 years…

When trying to integrate this approach into the teams, I found that only by pairing QA and development, going through examples that teased out good questions, and consolidating the teams understanding of the story did we start to get any movement. Some saw the upfront value, however not all were convinced. Fortunately, a few key people did create new modules of features which were now automated at an API level. It did take a little longer to create the functionality, but this was the start of living documentation and would mean if I needed to know what the system behaviour was meant to be, I wasn’t just told to ‘look at the code’. This really is a very difficult task, and only really strong development professionals with business and industry knowledge would be able to attempt a task like this. The code only tells you what the system currently does, not what it is meant to do or why it’s doing it. Living documentation does.

Then…our breakthrough came just a few months later. A separate team was investigating an issue in an area we had living documentation for. It turned out we had missed a scenario which became a live issue, however once we added it in and created a failed test the developer could try to fix it. It did take a while to fix as every time that scenario worked, it broke another. However it was immediately picked up by the test automation performed on each build. In the time allowed, with the tools available, I wouldn’t have been able to manually test all the scenarios which broke as the result of a fix. One of our fiercest opponents to the approach appreciated this and was won over at last!

Now a few years on, I’m pleased to report all new products we create have no manual regression suites. We now spend our time developing new features during the sprints, and at the end of it, have confidence in our automation that we can release that build. The new techniques have finally enabled us to be Agile.

Key Learning for me:

  1. Find the approach which works best for you and your company
  2. Early engagement with developers is key – pairing up QA and development was invaluable to our process
  3. Use examples that tease out good questions and consolidate understanding of the story
  4. Automation is where it’s at – manual testing is a thing of the past, costs more money and time in the long term
  5. If you truly wish to be ‘Agile’ and release after every sprint, the sprint needs to include regression which is automated