are key for a Test Manager to understand, monitor and control the test phases across test levels. Proper use of software testing metrics is critical during any project and they are one of the test manager’s best allies in understanding where the testing is and which way it is heading within the project and across the organization.
Defining and using Test Metrics
It is important that the proper set of metrics be established for any endeavor, including testing. Software testing metrics can be classified as belonging to one or more of the following categories:
- Project metrics – measure progress toward exit criteria, such as the percentage of test cases executed, passed, and failed
- Product metrics – measure some attribute of the product, such as the extent to which it has been tested or the defect density
- Process metrics – measure the capability of the testing or development process, such as the percentage of defects detected by testing
- People metrics – measure the capability of individuals or groups, such as the implementation of test cases within a given schedule
Any given metric may belong in two, three, or even four categories: a trend chart showing the daily arrival rate of defects can be associated with:
- an exit criterion (zero new defects found for one week)
- the quality of the product (testing cannot locate further defects in it)
- the capability of the test process (finds a large number of defects early in test execution)
The Test Manager role is to use of metrics to measure the progress of testing. Some of the project metrics used for test progress also relate to the product and process. Overall, bu using these metrics, this enables testers to report results in a consistent way and enables coherent tracking of progress over time.
Test Managers are frequently required to present metrics at various meetings which may be attended by multiple levels of stakeholders, ranging from technical staff to executive management. Because metrics are sometimes used to determine the overall success of a project, great care should be taken when determining what to track, how often to report it, and the method to be used to present the information.
Test Managers consider test metrics
Definition of metrics
- Limited set of useful metrics should be defined
- defined based on specific objective(s) for the project, process and/or product
- defined for balance (a single metric may give a misleading impression of status or trends)
- interpretation agreed upon by all stakeholders to avoid confusion when they are discussed
- avoid to define too many metrics, the focus should be on the most pertinent ones
Tracking of metrics
- Reporting and merging metrics should be as automated as possible to reduce the time spent in taking and processing measurements
- Variations of measurements over time for a specific metric may reflect information other than the interpretation agreed upon in the metric definition phase
- to carefully analyze possible divergence in measurements from expectations, and the reasons for that divergence
Reporting of metrics
- The objective is to provide an immediate understanding of the information
- Presentations may show a snapshot of a metric at a certain time or show the evolution of a metric over time so that trends can be evaluated
Validity of software testing metrics
- verify the information that is being reported
- The measurements taken for a metric may not reflect the true status of a project or may convey an overly positive or negative trend
- Before any data is presented, it must be reviewed for both accuracy and for the message that it is likely to convey
This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.
In late 2019 we have launched A Test Manager’s Guide eBook that stands as the base for this article. You can check out more useful test management lessons by enrolling for free to view Chapter 1 – Back to the basics.