Code coverage, the tool to reduce errors in the digital code
Regardless of his or her level of experience, any programmer can make mistakes, caused by various factors such as carelessness, unclear specifications, or suboptimal knowledge of the programming environment or language used. That is why it is crucial to set up a system to detect and correct these errors as quickly as possible, avoiding loss of time and possible disruption. This becomes even more relevant for high-level projects, where even a small bug can have a significant impact and create inconvenience and disruption, affecting the experience of a large number of customers. In this context, code coverage proves to be an invaluable ally, because it tests and measures the quality of software code, allowing us to take the first step toward the success of our site.
What is code coverage
In computer science, code coverage or code coverage is a method of measuring the amount of code in a program that was actually executed during a test, so as to determine whether the code was tested completely and effectively. If a portion of the code was never executed, in fact, there is a greater chance that it will contain errors that will never be discovered.
It is therefore an invaluable tool for anyone developing softwares, from large teams of developers to small startups to freelancers, and it also serves websites to ensure that the code underlying the site is robust and reliable, able to run smoothly in any situation. In particular, if we own or operate a website, such as an e-commerce or web application that handles sensitive data, a bug could not only cause problems for users but also result in legal or image repercussions.
From a more technical point of view, code coverage is a white-box testing technique that is used to verify the extent to which code has been executed; performing code coverage requires static instrumentation in which instructions that monitor code execution are inserted at critical points in the code, and, as a helpful guide by web.dev developers explains, recording these metrics usually looks like this:
Each addition of instrumentation code also results in an increase in execution time and code length, but it is more than justified in light of the information the tester gains from this extra code.
Code coverage is generally expressed as a percentage value-for example, a code coverage of 90 percent means that 90 percent of the code was executed during the test-and this parameter essentially helps us understand whether the tests we have written are sufficient and whether they are checking all parts of our code. In other words, and to simplify, code coverage allows us to understand if there are parts of our code that might behave unexpectedly because they were not properly tested.
How code coverage works
As anyone involved in software development et similia knows, every customer wants a quality software product, where this term includes high level of performance, functionality, behavior, correctness, reliability, effectiveness, security and maintainability of the product itself.
The Code Coverage metric precisely helps determine the performance and quality aspects of any software and works by monitoring code execution during testing to determine which parts actually executed and which did not. This process is usually automated using code coverage tools, which are able to detect which lines, functions, branches or blocks of code were executed during testing.
More specifically, code coverage scripts generate a report detailing the amount of application code that was executed, expressing as mentioned a percentage value that defines the metric. If, for example, 70% of our code was executed during testing, we would say that we have 70% code coverage, and this percentage is an indicator of the completeness of our tests.
A high percentage suggests that our tests are complete and cover most of the possible executions of our code, while a low percentage may indicate that there are parts of our code that have not been tested. However, it is important to note that code coverage only examines code execution, not code correctness. In other words, 100% code coverage does not mean that our code is error-free, only that every part of the code was executed at least once during testing. For this reason, code coverage should be used in conjunction with other testing techniques to ensure software quality.
Why perform code coverage on the site: the benefits
It is already clear from what has been written that code coverage is a tool that should never be missing from the toolkit of those who work in digital, and all developers, from the most experienced to those new to the world of coding, should not underestimate the importance of this metric. If we are commissioning the creation of a website or application, asking about code coverage can help us understand whether the developer is doing a good job; if, on the other hand, we are engaged in programming ourselves, understanding and applying the concept of code coverage can take us a significant leap forward.
The bottom line is that untested code is potentially flawed, while well-written, well-tested code can take us the first step toward success in any digital project.
Indeed, it is a fact that many projects fail because of the lack of quality in software systems, and so having an eye on code quality control, during the various stages of development, is no longer an optional choice but a critical factor.
The first and most obvious benefit of code coverage is the ability to identify parts of our code that have not been tested: this allows us to write more effective tests and reduce the likelihood of unexpected errors occurring. In addition, good code coverage can increase trust in our products from customers and users, who will see in it a sign of professionalism and attention to quality.
In addition, code coverage can help identify parts of the code that may be redundant or unnecessary, allowing us to optimize our software and make it more efficient. Finally, it is a great tool for monitoring the evolution of software over time, so we can also understand whether we are improving or deteriorating in quality.
The importance of testing for software and sites
Code coverage is part of the broader work of computer testing, which is the process of evaluating a software system (or one of its components) based on its behavior while running under controlled conditions.
The specific purpose of the testing activity is to identify possible problems and defects in the program before delivery to the user, rather than focusing on proving their absence. This is a direct reference to the lessons of Wybe Dijkstra, one of the pioneers of structured programming and data theory, who precisely sanctioned how “testing cannot prove the absence of errors, but can only prove their presence.”
This statement underscores a fundamental aspect of testing: even if we test our software intensively and find no errors, this does not mean that the software is free of bugs; it may simply highlight that our tests were unable to detect them.
This concept is particularly relevant when we talk about code coverage: as mentioned above, even if we achieve 100% coverage, we do not have a guarantee of the absence of errors in the software. However, high code coverage, combined with other testing techniques, can help us detect and correct more errors, improving the quality of our software.
While not, therefore, a definitive solution to the problem of software bugs, the testing phase is nonetheless crucial in detecting the presence of glitches and facilitating the subsequent debugging process, which will take care of removing these errors from the code. Testing also should not alter the environment in which the program operates in any way, as it could affect the repeatability property of the test case.
White Box Testing, Black Box Testing and Code Coverage
It is appropriate at this point to open a quick parenthesis on the two main two different approaches to testing that can be used in software development, namely “white box test” and “black box test.”
In white box testing, the tester has access to the source code and can therefore verify the internal behavior of the software. In black box testing, on the other hand, the tester does not have access to the source code and can only verify the external behavior of the software, that is, whether it responds correctly to inputs and produces the expected results.
Code coverage falls within the scope of white box testing, in that it requires access to the source code to verify which parts were executed during testing, although this does not necessarily imply the absence of bugs, as mentioned earlier. It may therefore be important to combine code coverage with other testing techniques, such as black box testing, to get a complete picture of the quality of our software.
Then there is a further distinction to be made, namely between test coverage and code coverage, which are often confused while being profoundly different.
Test coverage is a qualitative metric that measures how well the test suite covers the functionalities of the software, thus helping to determine the level of risk involved. It thus refers to the effectiveness of the tests in covering all possible cases that the software might encounter during its operation, including, for example, not only the execution of all lines of code, but also the testing of all possible combinations of inputs, all possible execution paths, and all possible conditions.
As mentioned, on the other hand, code coverage is a quantitative metric that measures the proportion of code executed during the test and thus concerns the amount of code covered by the tests. In this sense, code coverage is a more specific measure that is part of the more general test coverage; this means that we could have 100% code coverage (all lines of code were executed during testing), but still have low test coverage if, for example, we did not test all possible input combinations or all possible execution paths.
If we think of a web application as a house, test coverage measures how well the tests cover the rooms in the house, while code coverage measures the part of the house traversed by the tests.
How to do code coverage: the four main methods of measurement
There are different options for creating coverage reports, depending on the code language and the testing tools we use; however, the basic process is similar in many cases and involves the use of a code coverage tool that monitors code execution during testing and generates a report that shows which parts of the code executed and which did not.
Here are some examples of code coverage tools for different programming languages:
- JaCoCo is one of the leading code coverage tools for Java; it integrates with most development environments and testing frameworks and generates detailed reports in various formats, including HTML and XML.
- Coverage.py is a code coverage tool for Python that can be used with any testing framework; it generates detailed reports showing which parts of the code executed and which did not.
- Istanbul is a code coverage tool for JavaScript that can be used with various testing frameworks, including Mocha and Jasmine; it generates reports in various formats, including HTML and JSON.
- C#. DotCover is a code coverage tool for .NET that integrates with the Visual Studio development environment and generates detailed reports.
To use these tools, it is usually necessary to configure them to work with our development environment and testing framework, run the tests with the code coverage option enabled, and then generate the report; the specific instructions may vary depending on the tool and programming language used. In addition, some of these tools show the results directly in the terminal, while others can generate a full HTML report that allows us to explore what part of the code is missing coverage.
Going back to the guiding article by Ramona Schwering and Jecelyn Yeen on web.dev, we currently recognize four common ways to collect and calculate code coverage, namely function, line, branch, and instruction coverage.
Here is a brief description of their characteristics and implementation methods.
- Function coverage
Function coverage is a simple metric that captures the percentage of functions in the code that are called by tests. In the code example, there are two functions: calcCoffeeIngredient and isValidCoffee. The tests call only the calcCoffeeIngredient function, so the function coverage is 50%.
- Line coverage
Line coverage measures the percentage of executable lines of source code that are actually executed by the test suite. If a line of code remains unexecuted, it means that some parts of the code have not been tested.
The code example has eight lines of executable code (highlighted in red and green) but the tests do not execute the American condition (two lines) and the isValidCoffee function (one line). This results in a line coverage of 62.5%.
The line coverage does not account for statement statements (declaration statements) such as function isValidCoffee(name) and let express, water;, because they are not executable.
- Branch coverage
Branch coverage measures the percentage of executed branches or decision points in the code, such as if statements or loops, and determines whether tests examine both the true and false branches of conditional statements.
There are five branches in the code example:
- Call calcCoffeeIngredient with only coffeeName (check).
- Call calcCoffeeIngredient with coffeeNameecup (check).
- Coffee is espresso (check).
- Coffee is American (uncorrected).
- Other coffee (check).
The tests cover all branches except the condition Coffee is American, so the branch coverage is 80%.
- Statement coverage
Instruction coverage measures the percentage of instructions in the code executed by the tests. At first glance it might seem similar to line coverage, but it takes into account individual lines of code that contain multiple instructions.
In the code example there are eight lines of executable code, but nine instructions, because the line espresso = 30 * cup; water = 70 * cup; contains two instructions. The tests cover only five of the nine statements, so the statement coverage is 55.55%.
If we always write only one instruction per line, the line coverage will actually be the same as the instruction coverage.
How to choose the most suitable method
Most code coverage tools include these four common types of code coverage, and so we can say that choosing which code coverage metric to prioritize depends on your specific project requirements, development practices, and testing goals.
In general, statement coverage is a good place to start because it is a simple and easy-to-understand metric; unlike statement coverage, branch coverage and function coverage measure whether tests call a condition (branch) or a function and thus represent a natural progression after statement coverage.
From a practical point of view, if the test shows high statement coverage we can move on to analyze branch coverage and function coverage.
Beware of data: a wrong metric is worse than no metric at all
We mentioned that there is an inherent risk in code coverage, namely getting 100% in the test and thinking that the software is error-free. In reality, besides not being true in a general sense (this value only informs us that every part of the code was executed at least once during testing) it can also be the result of a “hallucination,” as in the example shown again by web.dev where the test achieves 100% coverage of functions, lines, branches, and instructions, but it is meaningless because it does not actually test the code really.
As experts summarize, incorrect metrics can give us a false sense of security, which is worse than having no metrics at all. For example, if we have a test suite that achieves 100 percent code coverage but the tests are all meaningless, we may have a false sense of security that the code is well tested; if we accidentally delete or break some of the application’s code, the tests will still pass, even if the application no longer works properly.
To avoid this scenario, we have two paths:
- Test revision. We can write and review tests to make sure they are meaningful by testing the code in a variety of different scenarios.
- Use code coverage as a guideline, not as the only measure of test effectiveness or code quality.
Ultimately, code coverage can be a useful metric for measuring the effectiveness of testing and can help us improve the quality of applications by ensuring that the crucial logic in the code is well tested.
However, we must keep in mind that it remains only one metric, that other factors, such as test quality and application requirements, must also be integrated or supported, and most importantly, remember that our goal is not to achieve 100 percent code coverage. Instead, it is more useful to use code coverage in conjunction with a comprehensive testing plan, incorporating a variety of methods, including unit testing, integration testing, end-to-end testing, and manual testing, to make sure we always deliver a positive, error-free experience.
