Software Testing Reviews are key for a test manager, but knowing the art of managing test reviews is key to reducing the cost of quality through these tools. Knowing how to manage informal and formal testing reviews helps the Test Manager across projects and products.
The software testing review strategy must be coordinated with the test policy and the overall test strategy. Reviews should be planned to take place at natural break points or milestones:
- typically after requirement and design specification
- start with business objectives and work down to low level design
- often as part of a verification activity before, during, and after test execution
Before establishing an overall review plan at the project level, the review leader (may be a Test Manager) should take into account:
- What should be reviewed (product and processes)
- Who should be involved in specific reviews
- Which relevant risk factors to cover
The review leader should also:
- identify the items to be reviewed
- select the appropriate review type and level of formality
- identify a budget (time & resource) for the review
- create a business case and include a risk evaluation and return on investment calculation
The optimal time to perform reviews depends on:
- availability of the items to review in a sufficiently final format
- availability of the right personnel for the review
- time when the final version of the item should be available
- time required for the review process of that specific item
The objective of the review must be defined during test planning and should also include:
- conducting effective and efficient reviews
- reaching consensus decisions regarding review feedback
A good tip in case of audit or inspections on big projects is to use brief inspections/audits conducted at the author’s request as document fragments are completed. This will help have initial and iterative checks rather than a big bang approach when all is taken in at once. There are also options to have an advance audit prior the main certification audit.
We should not forget about the project reviews which are frequently held for the overall system and may also be necessary for subsystems and even individual software elements.
Based on project complexity or product risks we should adjust:
- the number of reviews
- the types of reviews
- the organization of reviews
- the people involved in reviews
Participants in reviews must:
- Have the appropriate level of knowledge, both technical and procedural
- Be thorough and pay attention to details
- Have clarity and the use of a correct prioritization
- Understand their roles and responsibilities in the review process
Review planning should address:
- The risks associated with technical factors, organizational factors and people issues
- The availability of reviewers with sufficient technical knowledge
- Ensure that each team is committed to the success of the review process
- Ensure that each organization is allocating sufficient time for required reviewers to prepare for and participate in the reviews
- Time allocation for required technical or process training for the reviewers
- Backup reviewers should be identified in case key reviewers become unavailable due to changes in personal or business plans
During execution, a review Leader must ensure:
- Adequate measurements are provided by the participants in the reviews to allow evaluation of review efficiency
- Checklists are created, and maintained to improve future reviews
- Defect severity and priority evaluation are defined for use in defect management of issues found during reviews
After each review, the review Leader should:
- Collect the review metrics and ensure that the issues identified are sufficiently resolved to meet the specific test objectives for the review
- Use the review metrics as input when determining the return on investment (ROI) for reviews
- Provide feedback information to the relevant stakeholders
- Provide feedback to review participants
A software testing review effectiveness is evaluated by comparing results from the review with the actual results found in subsequent testing.
In case a work product is reviewed, but later found defective, then the review leader should consider ways in which the review process might have allowed the defects to escape. Some likely causes:
- include problems with the review process (like poor entry or exit criteria
- improper composition of the review team
- inadequate review tools (checklists)
- insufficient reviewer training and experience
- too little preparation and review meeting time
It is possible that reviews lose their effectiveness over time and this can be noticed in project retrospectives. In such cases, the review leader should investigate the cause.
If a pattern of escaped defects is consistent across projects this is another indicator that there are significant problems with the review process which have to be assessed and addressed.
By using metrics we must focus on the review process and never use them to punish or reward individuals.
This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.