RIGHT Blogs                                                              

Can 100% test automation be achieved?

Some of the challenges commonly seen with automation are:

1.When the first build comes out in a sprint/iteration, it will have defects. These defects may be blocking and may not allow for automation script development to proceed. The most critical issue is to find defects and report so developers can fix them quickly and complete the functionality.

2.In the case of sprints, the continuous integration builds will not have fully developed features initially. There may be significant changes with subsequent builds, including naming conventions for UI elements which will require a lot of rework from the automation team. This can make the automation team frustrated after a few rounds of changes to the same set of scripts.

3.Testers need to validate that all the requirements are implemented as designed and there are no gaps. This will require a manual testing effort for first time.

4.Testing of screen/UI layout is nearly impossible with automation and requires at least one round of testing manually. This will verify that there are no inconsistencies/issues and to certify that everything works fine.

5.Manual test cases are more granular and detailed, they include a lot of positive and negative test cases so that the feature can be thoroughly tested. Automating all of them as is does not provide any business value. Automation testing requires automation scenarios (which are more like user/business scenarios) developed from these granular manual test cases. The granular manual test cases then need to be covered in regression testing and only then should they be used for automation.

6.Automation can only be implemented selectively and the shelf life of a product should be considered before investing. If a simple business application is being developed with little shelf life (say 1-2 years with no potential changes to the application), then it does not make any sense to invest in automation.

This throws up two important points to consider for Test Automation:

1. Automation cannot be attempted unless the application is automation ready. It has been my experience that it is very important to have the application or functionality stable before automation is attempted.

2.For 99% of the applications, it is not possible to achieve 100% automation. Even if it is possible, it does not make any sense from a business case and ROI standpoint.

Studies have proven that more than 70% of Test Automation projects fail to deliver their objectives and some of the common reasons are:

• Objectives are not clearly defined
• Lack of automation architecture expertise
• Automate everything
• Automate very little
• High cost of automating new functionality

At Alliance, we have defined the testing process such that we follow lagging sprint concepts and create an automation backlog for automation purposes. This means that when the functionality is first developed, business scenarios and test scenarios are designed, followed by test cases. Test cases are manually executed in current sprint/iteration. At the same time, the test scenarios suitable for automation are marked for automation with the participation of an automation team and put in an automation backlog along with their priority. At the start of a new sprint/iteration, the automation team picks the automation scenarios from the automation backlog based on the bandwidth available which takes into account maintenance of existing scripts to run for current sprint/iteration. This also means that sprint/iteration 0 will have no automation and only manual testing is used for testing the sprint/iteration.

This approach has proven to be very successful and is implemented in our RightLine^TMdelivery methodology. There is a white paper on our web site which outlines some of these objectives.

http://www.allianceglobalservices.com/white-paper/test-automation-framew...

More than 60% of our customers today, from large enterprises to small ISVs, come to us for automation after they have had bad experiences with automation. They have spent thousands of $$$ but have seen no business value at all. When we propose the approach that we follow for automation, we get feedback of, “The previous vendor told us that automation is very easy, 100% automation is possible, and they will do JIT automation within each sprint/iteration.” All of these efforts have failed. Some have even suffered with the quality of releases decreasing significantly because of these issues. Some do not even want to attempt further automation but ask us to improve the quality of their deliverables first.

In summary, automation is critical in today's environment. Companies are struggling with accelerated time to market and cost reduction. It is also critical that automation is implemented correctly as almost 70% of automation projects fail to meet business value.

I would like to hear what my peers in industry think about this topic and what are the best practices and methodologies they use to make automation succeed.

-Sandeep