Defect management and Incident Reports

Software testing Defect Management (or Incident Management) must happen across all test levels and it is critical for understanding where the quality of the product/project. Software testing basics consider the dealing with the software testing issues or defects required to reach the test levels exit criteria. A large part of the time of a Test Manager is spent performing Defect Management.

Defect Management is the process of recognizing and recording defects, classifying them, investigating them, taking action to resolve them, and disposing of them when resolved.

An organization’s defect management process and the tool used to manage this work are of critical importance:

• Information gathered by effective management of defects allows to gain insight on the state of a project throughout the development life cycle
• By collecting and analyzing data over time, this can help locate areas of potential improvement for testing and development processes

Defect management tools – Defect Report

Defect report = Documentation of the occurrence, nature, and status of a defect.

Whatever the specific information determined as necessary for defect reports, it is critical that testers enter information that is complete, concise, accurate, objective, relevant and timely. When a software testing defect is detected (static testing), or a failure is observed (dynamic testing) data should be gathered by the person(s) involved and included in the defect report.

Information from a defect report should suffice for Management of the report; Assessment of project status (terms of product quality and test progress) and Assessment of process capability.

Core information gathered should be consistent across the software testing life cycle and ideally across all projects in order to allow for meaningful comparison of process defect data within and across all projects. The software test manager knows that when handling software testing defect reports we have the following test objectives:

• Provide developers and other parties with information about any adverse event that occurred
  • enables to identify specific effects and to isolate the problem with a minimal reproducing test
  • enables to correct the potential defect(s) as needed or to otherwise resolve the problem
• Provide test managers a means of tracking:
  • the quality of the work product
  • the impact on the testing
• Provide ideas for development and test process improvement

When logging defects, you should consider: the context or component/system under test; the test level; and the software development life cycle. Software defects may be reported during: coding, static analysis, reviews, dynamic testing, and use of a software product.

Testing defects may be reported for issues in code or working system, documentation, requirements, user stories, development or test documents, and user manuals or installation guides. Any defects identified should be investigated and should be tracked from discovery and classification to their resolution. This is not just a target, but a software testing goal.

• Dynamic defect reports may sometimes differ. Defects found during static testing (particularly reviews) will be documented in a different way.
• An example of contents of a defect report can be found in ISO standard (ISO/IEC/IEEE 29119-3) and refers to it as incident reports
• The defect management tools used may automatically manage and fill in some details

Information from a software defect report should suffice for:

1. Management of the report
2. Assessment of project status (terms of product quality and test progress)
3. Assessment of process capability

Core information gathered should be consistent across the life cycle and ideally across all projects in order to allow for meaningful comparison of process defect data within and across all projects.

Defect data may also include the following (on top of the already presented data for a Defect Report):

• The role of the author (end user, tester, business analyst, technical support, etc.)
• Type of testing performed (usability, performance, regression, etc.)
• Sub-system or component where the issue lies besides the System under test
• Project activity occurring when the issue was found
• Type of defect (based on defect taxonomy)
• Quality characteristics affected by the defect
• Project or product in which the problem lies
• Current owner (assignee)
• Business Impact
• Risks, costs, opportunity and benefits for fix/no fix
• Description of how the defect was resolved

Defect information should support

• defect density analysis
• trend analysis of defects detected and resolved
• average time from defect detection to resolution
• failure intensity like mean time between failures
• Even though the collection of data can assist in:
• test progress monitoring
• control
• exit criteria evaluation

There can be a lot of challenges in properly assessing project status, test progress and process capability when defect report data is of low quality and not properly managed. Various standards as ISO 9126 being replaced by ISO 25000, IEEE 829, IEEE 1044 and Orthogonal Defect Classification exist to help the Test Manager determine which information to gather for defect reporting.

When a defect is detected (static testing), or a failure is observed (dynamic testing) data should be gathered by the person(s) involved and included in the defect report.

Software Defect management workflow and states

A defect report typically progresses through a workflow and moves through a sequence of states as it continues through the defect life cycle. In most of these states a single defect participant owns the report and is responsible for carrying out a task which will cause the defect report to be moved to the next state.

Once the defect report reaches a terminal state like closed, cancelled, could not reproduce or deferred then it does not have an owner anymore as no further action is required.

1. The INITIAL state

• One or more testers gather the information necessary for the person responsible for resolving the defect to reproduce the anomaly
• Also be referred to as the “open” or “new” state

2. The RETURNED state

• The receiver of the report has rejected the report or is asking the tester to supply further details
• May indicate a shortfall in the initial information-gathering process or of the testing itself.
• This may also be referred to as the “rejected” or “clarification” state

3. The CONFIRMATION state

• The tester will run a confirmation test to determine whether the fix has solved the problem.
• The defect can then be “Closed” or “Re-Open”
• This may also be referred to as the “resolved” or “verification” state

If you liked this article and want to learn more about software test management, then I would recommend that you check out our eBook – A Test Manager’s Guide.

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.