The signs of automation code require restructuring. [PART-1]
Many English lexicons describe the word “Assert” as “Insist on having one’s opinions, rights recognized, or to postulate positively.” Through this article, I am going to ask every test automation developer to architects to have added more diligence towards the software they develop. I’ll be explaining the most general problem, which may be automatically and left unchecked in the test automation suite due to a fast-paced development environment. An inexperienced, over committing, or maybe the overconfident developer can introduce such plights in automation code.
A substandard code review introduces plights in an enormous amount. A legacy automation code automatically develops plights over a long period of development, no matter what an efficient and talented your team is. Throughout this article, I will be discussing accounts of signs and symptoms that you can find in your test automation code to make it better, faster, and efficient than ever before.
Objectives of Test Automation
Automation testing starts with the developers when they write a piece of automation code to test their changes in or to test a new functionality as a unit. Every developer knows this thumb rule but very few follow it religiously. In a way, a developer writing a fewer test code will be less productive than one who is writing them.
Investment to a good automation test and framework that eventually save efforts and time in development of product is also a key objective of test automation.
Economics as a motivation for better test automation is very well described in “Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity written by Ramler, Rudolf & Wolfmaier, Klaus”.
ROI on Automation code easily visualized by figure-1.
From figure-1 we start the automation test project with final result depicted in blue line. But as the project grows in live environment at some point to time manual test cost goes less than live automation tests and start making losses. To prevent any possible loss occurrence in future, we need refactoring of automation code and review of test cases regularly.
Now from engineering perspective an automated test code should be able achieve following objectives:
1. Automated tests should understand system under test.
2. Automated tests should easy to run.
3. Automated tests reduce risk and improve quality of the product.
4. Automated tests should be easy to write and maintain.
5. Automated tests require minimum maintenance when product evolve around it.
6. Automated tests should produce easily understandable results.
Logically test a product should qualify on quality derived as an answer to questions like has software built correctly or from user perspective have a team built a right software. These questions gets answered when a development team comes with an executable specifications formulated in form of tests. If these tests gets automated to discover bugs and localize them then a QA person can states “Automated tests understand the system under test”.
Now, your tests should be easy to execute. If automated test user (e.g. Software developer) find 10 page manual or some time even less, may not be interested in reading it and may bypass various important aspect of configuring a system under test and end up with wrong result. If test automation developer make following criteria part of there test automation, then only an automation tests can call easily runnable.
1. Fully automated test should run with zero effort.
2. Fully automated test should be self-sufficient to check and report the errors with any manual intervention.
3. These tests should be able to run multiple iterations.
4. Every tests should be independent of earlier tests.
An easy to write and maintainable automation code should have key features like one test per condition and to test them there should be an independent path. No doubt this will make large test code base but will keep the easily maintainability of the code. This kind of test code will helps in documentation and user acceptance tests too. This kind of test development practice may need a manual test mindset for a person who is automating the test. Domain specific test will require very expressive test. Therefore, the test may be collected as library that may be called and express the idea of test case.
Revisit the fundamentals of your Automation test suite
Software industry is very open two new ideas. Every decade comes with a new theme and areas of development in core technologies and so as its application.
And once this happened a test adapts it a first. Now words like Test driven development or Agile tests are more common words than last decade. There are some fundamental ideas that people comes with when a development start. This fundamental idea becomes the foundation stone of future development. for instance if a team starts developing a new product with TDD methodology, soon they have a good bunch of test as part of their Automation suite. But if you are working with a Legacy software, you may think of Behavior driven development (BDD) is best fit. Anyway it is just a need how a team handles their testing need. In addition, the agreement they come with before starting development becomes the fundamental of Automation test suite. But one can point out following rules of engagement for their Automation test suite, when it comes to restructuring
1. Test should not alter the system under test state than the documented one. One can understand this fundamental as unit test of a product should complement testing as user acceptance level.
2. A test code that may present in the source tree, but build system should be able to compiled them out.
3. Keep different test suite for different level of testing needs. A developer may have a test suite with lots of test code and mock test enabled. Which may not be a good idea for a System test team. Using techniques like Fault injection or mock test is never a good idea for QA team responsible for General Availability of product.
4. Test should talk to developers/testers and not to make them suffer in debugging of test itself.
5. An automated test should use a independent fixture to make any one particular test independent of another as much as possible. Using a share fixture may cause interaction between tests, which make defect localization very hard compared to independent tests.
6. Always try of find out the level of context awareness of code under tests. As the context awareness goes up, so the tests start showing thin level of consistency. To reduce such situation try to make System under test and software both loosely coupled. Writing independent test also help to ease this situations. To test such scenario in automation one can resort on concepts like dependency injection or dependency lookup.
7. One test should cover only once condition. It has seen that test automation is not actually increasing the test coverage with new test addition, instead testing same code path more and more. This may seems achievable for fresh product but for legacy product is require lot of understanding, dedication and patients.
8. Always review your Automation source code against Dynamic and static code coverage data to minimize the code coverage. Some of the best product in industries had achieved 95% of code coverage within test automation.
9. Last but most important of test Automation rule ONE TEST COVERS ONE CONDITION.
10. Test should work in phases. for example a standard procedure should be look like this
a. Test environment setup
b. execute tests
c. verify results
d. Teardown the test environment.
11. A complex application runs on smaller component of different behaviors. It may be possible that these behaviors are provided by same component. In such situation if test fails we may require manual debugging, which may be time consuming. These type of situation can be reduced by taking out from automation with careful investigation. This will need the refactoring of automation code, so new code can test same behavior in automation.
12. Cost of writing a new test case or modifying an existing one should not be more than implementing a new functionality in the code.
Reference:
- xUnit Test Patterns Refactoring Test Code (Addison-Wesley Signature Series (Fowler)) Author: Gerard Meszaros — 2007
