Risk based software testing techniques

Risk based software testing techniques

There are a number of risk based software testing techniques, both formal and informal. Such approaches are subjective and dependent on the skills, experience and preferences of the tester and are closely linked together with software risk based testing.

Attempting to capture the benefits of software risk based testing while minimizing the costs, many practitioners employ lightweight risk based approaches:

  • Pragmatic Risk Analysis and Management (PRAM)
  • Systematic Software Testing (SST)
  • Product Risk Management (PRisMa)

In addition to the usual attributes of software risk based testing, these software testing techniques typically have the following attributes:

  • Evolved over time based on industry experience (especially in industries where questions of efficiency are important)
  • Extensive involvement of a cross-functional team of stakeholders during the initial risk identification and assessment (both business and technical)
  • Optimized when introduced in the earliest phases of a project:
    • options to mitigate quality risks are maximized
    • risk analysis can help to influence the product specification and implementation in a way that minimizes risk
  • Use the generated output (risk analysis table) as the basis for the software test plan and the test conditions and all test management and analysis activities
  • Support the reporting of test results in terms of residual risk to all levels of testing stakeholders

Systematic Software Testing (SST)

  • require requirements specifications as input into the risk analysis
  • cannot be used except when requirements specifications are provided

PRisMa and PRAM

  • Encourage using a blended risk and requirements based strategy
  • Can function entirely based on stakeholder input
  • Requirements and/or other specifications as an input to the risk analysis
  • The use of requirements as an input helps ensure good coverage of the requirements
  • Test Manager must ensure that important risks which are not suggested by the requirements (like non-functional) are not missed
  • When good, prioritized requirements are provided as an input, one typically sees a strong correlation between risk levels and requirement priority

Many of these software testing techniques encourage the use of risk identification and assessment process as a way to create consensus across stakeholders on software testing and increase the business value of testing.

Insufficient stakeholder participation leads to gaps in the risk analysis. Stakeholders do not always agree on the level of risk, so the person leading a quality risk analysis effort must work creatively and positively with stakeholders to achieve the best possible degree of agreement

Risk based testing techniques 3

Lightweight risk based testing techniques

Lightweight techniques allow the use of weighting of the likelihood and impact factors to emphasize business or technical risks. They use:

  • only two factors: likelihood and impact
  • simple, qualitative judgments and scales

And these approaches provide:

  • flexibility
  • applicability in a range of domains
  • accessibility across teams of all experience and skill levels

Risk based testing techniques 4

Heavy-weight risk based testing techniques

At the formal, heavy-weight end of the scale, the Test Manager has a number of options available:

  • Hazard analysis which
    • extends the analytical process upstream
    • attempting to identify the hazards that underlie each risk
  • Cost of exposure which involves determining three factors
    • likelihood (expressed as %) of a failure related to the risk item
    • cost of a loss (expressed as a financial quantity) associated with a typical failure related to the risk item, should it occur in production
    • cost of testing for such failures

Risk based testing techniques 6

Failure Mode and Effect Analysis (FMEA)

  • identify quality risks, their potential causes and their likely effects
  • then severity, occurrence and detection ratings are assigned in order to establish priority
  • strength on precision and meticulousness
  • document heavy and hard to learn
  • should be considered for high-risk or conservative projects

Risk based testing techniques 5

Quality Function Deployment (QFD)

  • a quality risk management technique with testing implications
  • concerned with quality risks that arise from an incorrect or insufficient understanding of the customers’ or users’ requirements
  • focus on the function of execution of a quality plan

Fault Tree Analysis (FTA)

  • observed failures (from testing or from production) or potential failures (quality risks)
  • then failures are subjected to a root cause analysis
    • starting with defects that could cause the failure
    • then with errors or defects that could cause those defects
    • continuing on until the various root causes are identified

Risk based testing techniques 7

Risk based software testing techniques – conclusion

The specific software testing techniques that should be used for risk-based testing, and the degree of formality of each technique depend on project, process, and product considerations.

In Agile projects, the quality risk analysis is fully integrated into the early period of each sprint and the documentation of risks is blended into the user story tracking.

Systems of systems require risk analysis on each system as well as on the overall system of systems and safety-critical and mission-critical projects require higher levels of formality and documentation.

The most critical success factor for risk-based testing is the involvement of the right team of stakeholders in the risk identification and assessment.

All stakeholders have their own understanding of what constitutes quality for the product and their own set of priorities and concerns about quality. They also fall into two broad categories:

  • Business stakeholders
  • Technical stakeholders

Risk Based Testing Techniques by Lucian C on Scribd

For a broader article on this topic, you can check our Test Manager School or further browse our software testing blog.

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.

In late 2019 we have launched A Test Manager’s Guide eBook that stands as the base for this article. You can check out more useful test management lessons by enrolling for free to view Chapter 1 – Back to the basics.