Risk based testing

Risk based testing

Risk Based Testing focuses mainly on risk. This is a software testing technique and covers risk management in software testing.

Risk based testing 3

As per ISTQB, a Risk = A factor that could result in future negative consequences

  • possibility of a negative or undesirable outcome or event
  • exist whenever some problem may occur which would decrease customer, user, participant or stakeholder perceptions of product quality or project success
  • the level of risk is determined by the likelihood of the event and the impact from that event

ISTQB book classified risks as follows:

  • quality risks/product risks/product quality risks = main effect of the potential problem is on product
  • project risks/planning risks = main effect of the potential problem is on project

Product Risk

Involves the possibility that a work product may fail to satisfy the legitimate needs of its users and/or stakeholders. When these risks are associated with specific quality characteristics of a product like: functional suitability, reliability, performance, usability, security, compatibility, maintainability and portability, then product risks are also called quality risks.

Some examples of product risks are:

  • Software might not perform its intended functions according to specification, user, customer, and/or stakeholder needs
  • A system architecture may not properly support some non-functional requirement(s)
  • A particular behavior may perform incorrectly in some circumstances
  • A loop control structure may be coded incorrectly
  • Response-times may be inadequate
  • User experience feedback might not meet product expectations

Project Risk

Involves situations that may have a negative effect on a project’s ability to achieve its objectives. This may affect both development activities and test.

In some cases, project managers are responsible for handling all project risks, but it is not unusual for test managers to have responsibility for test-related project risks

Some examples of project risks are:

  • Project issues
    • Delays may occur in delivery, task completion, or satisfaction of exit criteria or definition of done
    • Inaccurate estimates, reallocation of funds or general cost reduction across the organization may result in inadequate project/product funding
    • Late changes may result in substantial re-work
  • Organizational issues
    • Skills, training, and staff may not be sufficient
    • Personnel issues may cause conflict and problems
    • Users, business staff, or subject matter experts may not be available due to conflicting priorities
  • Political issues
    • Testers may not communicate their needs and/or the test results adequately
    • Developers and/or testers may fail to follow up on information found in testing and reviews
    • There may be an improper attitude toward testing or incorrect expectations from testing
  • Technical issues
    • Requirements may not be defined well enough
    • The requirements may not be met, given existing constraints
    • The test environment may not be ready on time
    • Data conversion, migration planning and their tool support may be late
    • Weaknesses in the development process may impact the consistency or quality of project work products such as design, code, configuration, test data & test cases
    • Poor defect management and similar problems may result in accumulated defects & other technical debt
  • Supplier issues
    • 3rd party may fail to deliver or go bankrupt
    • Contractual issues may cause problems to the project

Risk based testing 4

Risk based testing

A universal test management challenge is the proper selection, allocation and prioritization of tests as a software testing technique:

  • practically infinite number of test conditions and combinations of conditions that could be covered
  • the test team must select a finite set of conditions
  • determine the appropriate amount of effort to allocate in order to reach the intended coverage
  • sequence the resulting test cases in a prioritized order that optimizes the effectiveness and efficiency

The identification and analysis of risk, along with other factors, can be used by the Test Manager to help solve this problem as interacting constraints and variables may require a compromised solution.

In risk based testing:

  • quality risks are identified and assessed during a product quality risk analysis with the stakeholders
  • test team then designs, implements, and executes tests to mitigate the quality risks

Quality includes the totality of:

  • features
  • behaviors
  • characteristics
  • attributes that affect:
  • customer
  • user
  • stakeholder

Quality risk is a potential situation where quality problems might exist in a product like:

  • incorrect calculations in reports
  • slow response to user input
  • difficulty in understanding screens and fields

Testing has mitigated quality risk when:

  • defects are revealed (deal with them before release)
  • no more defects are found (system operates correctly)

Risk bases testing uses product risks to:

  • select test conditions
  • allocate test effort for those conditions
  • prioritize the resulting test cases

Risk based testing consists of:

Risk based testing 5

1.      Risk Identification

Stakeholders can identify risks through:

  • Expert interviews
  • Independent assessments
  • Use of risk templates
  • Project retrospectives
  • Risk workshops
  • Brainstorming
  • Checklists
  • Calling on past experience

By involving the broadest possible sample of stakeholders, the risk identification process is most likely to identify most of the significant product quality risks.

Project risks are also often identified as a by-product of quality risk but are not the main focus of risk-based testing.

2.      Risk Assessment

Once risk identification has occurred, risk assessment can begin. It involves categorizing each risk and determining the likelihood and impact associated with each risk. It may also involve other risk properties lie owner.

Categorization of risk is the placing each risk into an appropriate type like: performance, reliability, functionality, etc.). It usually occurs during identification when using a checklist.

Some organizations use the ISO 9126 which is being replaced by ISO 25000 quality characteristics for categorization, but many organizations use other categorization schemes.

Generic quality risk checklists exist and some organizations use the same checklist for risk identification to categorize the risks.

Risk based testing 6

Factors influencing likelihood of risk include:

  • Complexity of technology and teams
  • Personnel and training issues among the business analysts, designers and programmers
  • Conflict within the team
  • Contractual problems with suppliers
  • Geographically distributed team
  • Legacy versus new approaches
  • Tools and technology
  • Weak managerial or technical leadership
  • Time, resource, budget and management pressure
  • Lack of earlier quality assurance activities
  • High change rates
  • High earlier defect rates
  • Interfacing and integration issues

Factors influencing impact of risk include:

  • Frequency of use of the affected feature
  • Criticality of the feature to accomplishing a business goal
  • Damage to reputation
  • Loss of business
  • Potential financial, ecological or social losses or liability
  • Civil or criminal legal sanctions
  • Loss of license
  • Safety

If likelihood and impact can be ascertained quantitatively, one can multiply the two values together to calculate a risk priority number.

In most cases, the level of risk can only be evaluated qualitatively:

  • Likelihood being very high, high, medium, low or very low, but cannot be express as a % with any real precision
  • Impact being very high, high, medium, low or very low, but cannot be expressed in financial terms in a complete or precise fashion.

Qualitative assessments of risk levels are often combined, through multiplication or addition, to create an aggregate risk score. This may be expressed as a risk priority number but should be interpreted as a qualitative, relative rating on an ordinal scale.

Risk based testing 7

Unless the risk analysis is based upon extensive and statistically valid risk data, the risk analysis will be based on the stakeholders’ subjective perceptions of likelihood and impact.

Project managers, programmers, users, business analysts, architects and testers typically have different perceptions and this results in different level of risk for each item. This is why risk analysis process should include some way of reaching consensus or should establish an agreed upon level of risk.

Risk levels should be checked for good distribution through the range, to ensure that the risk rates provide meaningful guidance in terms of test sequencing, prioritization, and effort allocation.

3.      Risk Mitigation

After risks are identified and assessed, tests are designed, implemented and executed in order to cover the risks.

The effort associated with developing and executing a test is proportional to the level of risk:

  • more meticulous test techniques (such as pairwise testing) are used for higher risks
  • less meticulous test techniques (such as equivalence partitioning or time-boxed exploratory testing) are used for lower risks

The level of risk should influence decisions such as:

  • the use of reviews of project work products
  • the level of independence
  • the level of experience of the tester
  • the degree of confirmation testing (re-testing)
  • the degree of regression testing performed

The test team should remain aware of additional information that changes the set of quality risks and/or the level of risk associated with them.

Periodic adjustment of the quality risk analysis should occur at least at major project milestones. These adjustment may include:

  • identify new risks
  • re-assess the level of existing risks
  • evaluate risk mitigation effectiveness

For example:

  1. If risk identification occurred based on requirement specification, then a re-evaluation of risk should occur once the design specification is done
  2. If a component is found to contain considerably more defects than expected, then adjust the likelihood and overall level of risk upward

Product quality risks can also be mitigated before test execution starts. If problems with the requirements are located during risk identification, then the project team can implement thorough requirements specification reviews as a mitigating action. This can reduce the level of risk, which can mean that fewer tests are needed to mitigate the remaining risk.

Risk based testing 8

4.      Risk Management

Risk management occurs throughout the entire software testing life cycle. If an organization has a test policy document and/or test strategy document, these should cover:

  • the general process by which product and project risks are managed in testing
  • how risk management is integrated into testing

In a mature organization where risk awareness pervades the project team, risk management takes place at many levels, and not just for testing.

  • Important risks are not only addressed earlier in particular test levels, but also in earlier test levels
  • Sources of risk and the consequence of those risks should also be identified
  • For defects that do occur, root cause analysis is used to understand sources of risk and to implement process improvements that prevent defects
  • Mitigation exists throughout the software development lifecycle
  • Risk analysis transcends testing, with the test team participating in a program-wide risk analysis
  • Risk analysis is fully informed, considering:
    • related work activity
    • analysis of system behavior
    • cost-based risk assessment
    • product risk analysis
    • end user risk analysis
    • liability risk analysis

Conclusion

Most risk-based testing methods also include software testing techniques to use the level of risk to sequence and prioritize testing. This ensures early coverage of the most important areas and discovery of the most important defects during test execution.

In some cases all the high-risk tests are executed first and tests are run in strict risk order (depth-first). In other cases sampling approach is used to select tests across all the identified risk items using risk to weight the selection. (Breadth-first)

As it is possible that the time allocated for testing might be consumed without all tests being run. This is why risk based testing allows:

  • testers to report to management in terms of the remaining level of risk at this point
  • management to decide whether to extend testing or transfer the remaining risk onto the users, customers, help desk/technical support and/or operational staff

During test execution the most sophisticated risk-based testing allows project participants, project and product managers, executives, senior managers, and project stakeholders to monitor and control the software development life cycle. This requires the Test Manager to report test results in terms of risk in a way each test stakeholder can understand.

Risk Based Testing by Lucian C on Scribd

For a broader article on this topic, you can check our Test Manager School or further browse our software testing blog.

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.

In late 2019 we have launched A Test Manager’s Guide eBook that stands as the base for this article. You can check out more useful test management lessons by enrolling for free to view Chapter 1 – Back to the basics.