RIGHT Blogs                                                              

From Code Coverage to Quality

Code Coverage and Quality - Are these directly proportional?

There are different views on this and since this blog is not geared towards evangelizing any specific tool, the emphasis is more on the fact that including this as a best practice in software engineering will do more good than harm from a Quality perspective. The audience is both development and test teams but primarily geared towards the test discipline.

A popular concept in measuring Test Coverage is often termed as ‘Traceability Matrix’. This flows from Business Requirements to Functional Specifications and finally to Test Cases. Engineers typically create a matrix against the Functional Specifications and (or) Business Requirements and map to each test case to accomplish this task. The one’s missing an intersecting value is marked as incomplete and additional test cases and scenarios are written for these gaps to measure the test comprehensiveness.

While this is a standard approach that many test engineering groups follow and quantify it with a number in terms of % per release, this approach is still half baked and appears like accommodating a metric thereby provoking a thought to re-visit our test strategy.

Let’s take a hypothetical example of a Product or a feature that you are responsible for formally signing off for production purposes that has a large set of customer base who have reported several problems in the past. As a goal to fix these issues, Product Management team has taken an approach to prioritize the bugs based on business impact and have decided to fix 3 major issues for every release cycle and reduced the release cycle from a quarter to a month.

This is further complicated by addition of new requirements to compete with the products that are already available in the market. The lines of code get larger and larger in the absence of a true re-factoring spirit and the black box approach to track test coverage via a spreadsheet sounds bizarre as you are getting into a mode of satisfying a metric for the sake of reporting the same. You still go ahead and update the spreadsheet and there are no gaps as every business requirement or functional specification was mapped to a test case. You have used automation and were informed by your development manager that a lot of automated unit testing was already done and the assumptions were validated. The handoff to production is completed and there have been some bugs reported for features that got added or the bugs that got fixed.

 There are angry customers and the management is not happy with the quality despite the automation of the test cases that map to the new requirements or regression test cases.

There is two school(s) of thoughts in this scenario. One that believes YES, a white box approach to traverse through the code could have helped in designing additional test cases and NO; test coverage could not have saved from additional bugs in the software as reported by the customer. Needless to say, organizations should lean towards the former thought as this is a quality issue and one could run into the risk of losing customers to the competitors. It is a simple and an obvious business reason to retain customers via quality.

A popular belief that this is a task by itself and needs additional resources or manpower to implement the tools facilitating code coverage is not completely correct. There is a slight learning curve in understanding a given tool or the setup aspects; this should be fairly simple once done for a single release cycle. The core of the activity is focused on the analysis portion of what is not covered and mapping it to a business requirement or a functional specification and design additional test cases.

Let’s take a deep dive and understand the steps involved to achieve this using some industry standard tools. Rational (now a part of IBM), has a tool called PureCoverage and for Java based code there is a plethora of open source tools available such as Clover, Cobertura, Emma etc… There are other benefits these tools offer as well. Microsoft has built this utility into IDE such as Visual Studio 2008 as well as Visual Studio 2010. Oracle has been providing a DBMS Package since quite some time and these could be exploited for PLSQL coverage. For t-SQL, there are internal tools available from Microsoft and these might have been incorporated into Visual Studio 2010 by now which are worth exploring. For any language that has no associated tool, great value can be leveraged by building a custom aspect. For instance, any x-Path expression could be traced by writing your own utility that measures the coverage.

These following easy to use steps will help an engineer in accomplishing this task. Please do note that I am specifying what my experience has been with Microsoft tools and the implementation might vary across other tools. These steps can be used to measure t-SQL, C# and C++

1)Any non-SQL code needs a debug build. So get extremely friendly with your development lead OR Application outsourcing vendor who is providing the development services. Most of the engineering organizations build components with a secured key and make sure your debug build is also using the same key. This is usually skipped during the code signing process which usually defeats the purpose. It does not matter if we are dealing with managed or un-managed code. The differences could exist in how the debug bits are created.

2)Instrument the binaries or SQL objects. This step will put tool specific markers in the code for identification purposes. Do not get worried about these insertions in the code base

3)Start monitoring of the code which means the tool starts to build its data repository with information on user actions on components that were instrumented which are now being traversed by the application and its associated logic

4)Run your tests which could be automated or manual. The automation would help the execution of all scenarios quicker for understanding the gaps in test case design

5)Stop monitoring of the code.

6)Use the reporting feature available with the tool or the IDE itself displays the color coded report depicting what was traversed and what was not.

7)Analyze the portions that are not covered to design additional test cases.

Some caution may be required while analyzing the blocks of SQL that use GOTO statements. Given that fact that this is a bad practice, might still be worth paying extra attention on how the tool interprets a block and a GOTO statement. Caution is also required when the application needs to hook or latch on to the process which initiates the tool itself. With these simple and easy to use steps, one could report on what is covered Vs not covered for further analysis.

To conclude, code coverage utilities whether you are testing PLSQL, t-SQL, Java, C++ or C# based applications, are a great way to be assertive about the Quality of the software before releasing it for production.  The caveat however is that these tools can traverse through a line of code that exists and not for corner case scenarios (ex: an exception that was not handled OR a catch all exception) that resulted in defects. This technique is not biased towards product based companies, but also useful for IT consulting organizations or application software maintenance companies operating in a global delivery model .