Coverage – Pros and Cons

Code coverage, the goal of companies try to attain in software development. To understand how much of the code is exercised and better understand how and what needs to be tested when a change occurs. This is a great thing to have in any software development that will help teams become as efficient as possible to get to a quality product for customers.

As this is something has extremely high value to an organization how could I put “Pros and Cons” in my blog title? You’re right the concept of coverage are highly valuable, when I talk about “Pros and Cons” it is more to do with what is needed to get done to get that understanding of coverage.

The reason I am bringing this up stems from a vendor cold call I got for third party testing services. In my position I get a lot of those calls, and for the most part I do not have an issue to start a conversation to see what they provide. As long as they are able to differentiate from all other third party vendors. In this one call there was one statement that came out that peaked my interests and I pushed back to get more details. That comment was “We can have 100% code coverage through automation.” Sounds too good to be true, right? Well in my experience it is. I have never seen any software shop that had 100% code coverage through automation. There are too many variables out there where to do the work for that statement to be true it would take a lot of work.

Before I continue on with the above comments lets go through what I see as the pros and cons.


  • Automation coverage can target specific components
  • No need for full regression testing with a change
  • Better estimation to changes, more accurate
  • Shortened test timeframes

There are more and there is plenty of literature out there that will go into details. Here I just wanted to list a few and foster some conversation, as always.


  • A lot of effort needed
  • Tedious
  • May not be completely accurate
  • May not have full understanding of code

Again the list of the cons can be a lot longer. The good news is the cons are issues that can be worked on and resolved within the company to get to what is needed. There is no technical reason or roadblock. It is about getting the work culture changed to tackle these cons.

There are software packages out there that will give you the code coverage an organization desires, Like all tools it is all about how the tool is used to get the most out of it.

Do I think an organization can have 100% coverage? Yes. In automation? Maybe. Technology is getting better and better over time that it is something that could be achieved. There are a lot of other variables that need to fall into place that will allow it to happen. Consistent coding practices, full knowledge sharing within the teams, and full cooperation between everyone within the organization to name a few.



Helping others in the discipline

This week I had the opportunity to talk with someone within the QA discipline and provide some career advice. We talked from helpful hints on how to get a foot in the door with the resume to a good discussion of QA as a whole.

This was the first instance where someone took up on my offer to have these types of discussions. It was a great conversation and fun to have this type of discussion with someone outside of someone who directly reports to me.

This individual had some great questions from what changes could be done with the resume, how to approach an interview and what could be done to improve knowledge within the industry. The conversation went well over an hour, and that time flew by.

I think these types of conversations are needed to help the community stick together and improve the value that everyone can bring. As I say in every one of my appearances in SPaM Cast “feel free to reach out to me to further the discussion or start a new one”.

Keeping it short this week. If there are any topics you would like to hear, read or discuss about let me know.

Being self reliant without depending on 3rd parties

Keeping with the recent theme about freedoms I want do discuss testing with 3rd parties,  we can even add different departments within an organization as well. In the past 20 years more and more applications and back end systems have required to be more integrated than ever. Most of this is due to basic business logic of identifying value and whether it is better to buy (or subscribe to service) or to build.

Prior to Y2K (I just twitched as I wrote that) most organizations built what they needed to succeed. Now time to market was fairly long and Agile was not as widespread as it is today so it made sense to build. Also different departments were rarely integrated. Then the internet came into play and things changed. People were getting access to stuff they never thought possible and wanted more. Exponentially the demand for information grew. Along with that the demand to do transactions online as well instead of going to a store. It snowballed to something quite amazing because it forced organizations to adjust to meet those demands and also become innovated to catch a bigger piece of the market and improve client satisfaction.

So here is where things started to get tricky, to build what was needed would take a lot of work and speed to market is everything. What started to happen was companies started to form to only focus on key components to sell to larger organizations. What this did was  make the “Buy vs Build” decision making a key process to help improve value. Why spend time building something when somebody else already did and we can get it at a fraction of the costs? Also we don’t have to support it. Sometimes it makes sense to do that.

This also happens within an organization as well. The best example I can come up with is the banking industry and the use of the internet. Online banking has become such a big part of most peoples lives they take it for granted the amount of work that went into a site that shows your account balance and do almost all transactions you can do at an actual branch and you can do it sitting at the dinning room table with a nice cup of coffee in hand. There are so many departments and divisions that need to work together it is amazing.

Wow that took a little longer than what I wanted, none the less if you were not part of the growing pains myself and a lot of the experienced (that sounds better than old) individuals were we can now move on to the topic at hand.

I have worked and seen other groups focus so much effort on working with other groups or organizations that they become obsessed with testing with them to get work done. Now I do agree that there is a need to ensure that integration between systems are tested it is a must. The question that really needs to be asked is; do we really need to wait for someone to exercise our code? My answer is no, and I am sure that most of you that do any sort of SOA or transaction testing would agree.

For me as long as the Format, Content, and Layout have been defined (doesn’t have to be fully defined at first it can be done in Agile or Iterative fashion) teams can test their code before integration starts. Why not? Does the code know where the information is coming from? No, it is not like asking someone to hand you something from the other end of a table. Code expects to see something in a specific location and in something that it is coded to understand and process. Here is an example:

System flow A

Here is a very high level view of sending information between systems. Now I know most people I talk with understand the simulation of data and sending transactions through, yet this concept is still foreign to others even with a lot of literature out there about it. In the end it is a very simple process to exercise all the code you can without the wait.

System flow B

Now teams are in more control of the data that comes and goes. It is hard to do and not very fun to get started. Once you do the quality of what comes out should improve. Development can start doing some quick integration tests to make sure nothing goes awry. If you look at it from a certification perspective where sometimes you have to pull use specific data to run through the system, now you can.

Taking control of what you do as a team to get code out is important, relying on others outside of your control could cause delays or even frustration if issues arise and now it turns into a blaming session as to who’s fault it is. Stuff like that will put the progress of the product to a complete stop.

I am sure that this blog entry is what some of you and your teams are doing. My hope is that for those that don’t can use this to foster conversations here or other areas to help them get to where they need to go.

Listen to my segments and others on SPaMCast