Risk based testing

Risk Based Testing focuses mainly on risk. This is a software testing technique and covers risk management in software testing. As per ISTQB, a Risk = A factor that could result in future negative consequences; possibility of a negative or undesirable outcome or event. This exist whenever some problem may occur which would decrease customer, user, participant or stakeholder perceptions of product quality or project success. The level of risk is determined by the likelihood of the event and the impact from that event.

ISTQB book classified risks as follows:

• quality risks/product risks/product quality risks = main effect of the potential problem is on product
• project risks/planning risks = main effect of the potential problem is on project

Product Risk

Involves the possibility that a work product may fail to satisfy the legitimate needs of its users and/or stakeholders. When these risks are associated with specific quality characteristics of a product like: functional suitability, reliability, performance, usability, security, compatibility, maintainability and portability, then product risks are also called quality risks.

Some examples of product risks are:

• Software might not perform its intended functions according to specification, user, customer, and/or stakeholder needs
• A system architecture may not properly support some non-functional requirement(s)
• A particular behavior may perform incorrectly in some circumstances
• A loop control structure may be coded incorrectly
• Response-times may be inadequate
• User experience feedback might not meet product expectations

Project Risk

Involves situations that may have a negative effect on a project’s ability to achieve its objectives. This may affect both development activities and test. In some cases, project managers are responsible for handling all project risks, but it is not unusual for test managers to have responsibility for test-related project risks.

Some examples of project risks are:

• Project issues
  • Delays may occur in delivery, task completion, or satisfaction of exit criteria or definition of done
  • Inaccurate estimates, reallocation of funds or general cost reduction across the organization may result in inadequate project/product funding
  • Late changes may result in substantial re-work
• Organizational issues
  • Skills, training, and staff may not be sufficient
  • Personnel issues may cause conflict and problems
  • Users, business staff, or subject matter experts may not be available due to conflicting priorities
• Political issues
  • Testers may not communicate their needs and/or the test results adequately
  • Developers and/or testers may fail to follow up on information found in testing and reviews
  • There may be an improper attitude toward testing or incorrect expectations from testing
• Technical issues
  • Requirements may not be defined well enough
  • The requirements may not be met, given existing constraints
  • The test environment may not be ready on time
  • Data conversion, migration planning and their tool support may be late
  • Weaknesses in the development process may impact the consistency or quality of project work products such as design, code, configuration, test data & test cases
  • Poor defect management and similar problems may result in accumulated defects & other technical debt
• Supplier issues
  • 3rd party may fail to deliver or go bankrupt
  • Contractual issues may cause problems to the project

Risk based testing

A universal test management challenge is the proper selection, allocation and prioritization of tests as a software testing technique:

• practically infinite number of test conditions and combinations of conditions that could be covered
• the test team must select a finite set of conditions
• determine the appropriate amount of effort to allocate in order to reach the intended coverage
• sequence the resulting test cases in a prioritized order that optimizes the effectiveness and efficiency

The identification and analysis of risk, along with other factors, can be used by the Test Manager to help solve this problem as interacting constraints and variables may require a compromised solution.

In risk based testing quality risks are identified and cryptocurrency privacy techniques assessed during a product quality risk analysis with the stakeholders and the test team then designs, implements, and executes tests to mitigate the quality risks.

Quality includes the totality of features, behaviors, characteristics, attributes that affect customer, user, and stakeholder. Quality risk is a potential situation where quality problems might exist in a product like:

• incorrect calculations in reports
• slow response to user input
• difficulty in understanding screens and fields

Testing has mitigated quality risk when defects are revealed (deal with them before release) and no more defects are found (system operates correctly).

Risk bases testing uses product risks to select test conditions, allocate test effort for those conditions and prioritize the resulting test cases. Risk based testing consists of:

1. Risk Identification

Stakeholders can identify risks through:

• Expert interviews
• Independent assessments
• Use of risk templates
• Project retrospectives
• Risk workshops
• Brainstorming
• Checklists
• Calling on past experience

By involving the broadest possible sample of stakeholders, the risk identification process is most likely to identify most of the significant product quality risks. Project risks are also often identified as a by-product of quality risk but are not the main focus of risk-based testing.

2. Risk Assessment

Once risk identification has occurred, risk assessment can begin. It involves categorizing each risk and determining the likelihood and impact associated with each risk. It may also involve other risk properties lie owner. Categorization of risk is the placing each risk into an appropriate type like: performance, reliability, functionality, etc.). It usually occurs during identification when using a checklist.

Some organizations use the ISO 9126 which is being replaced by ISO 25000 quality characteristics for categorization, but many organizations use other categorization schemes. Generic quality risk checklists exist and some organizations use the same checklist for risk identification to categorize the risks.

Factors influencing likelihood of risk include:

• Complexity of technology and teams
• Personnel and training issues among the business analysts, designers and programmers
• Conflict within the team
• Contractual problems with suppliers
• Geographically distributed team
• Legacy versus new approaches
• Testing Tools and technology
• Weak managerial or technical leadership
• Time, resource, budget and management pressure
• Lack of earlier quality assurance activities
• High change rates
• High earlier defect rates
• Interfacing and integration issues

Factors influencing impact of risk include:

• Frequency of use of the affected feature
• Criticality of the feature to accomplishing a business goal
• Damage to reputation
• Loss of business
• Potential financial, ecological or social losses or liability
• Civil or criminal legal sanctions
• Loss of license
• Safety

If likelihood and impact can be ascertained quantitatively, one can multiply the two values together to calculate a risk priority number. In most cases, the level of risk can only be evaluated qualitatively:

• Likelihood being very high, high, medium, low or very low, but cannot be express as a % with any real precision
• Impact being very high, high, medium, low or very low, but cannot be expressed in financial terms in a complete or precise fashion.

Qualitative assessments of risk levels are often combined, through multiplication or addition, to create an aggregate risk score. This may be expressed as a risk priority number but should be interpreted as a qualitative, relative rating on an ordinal scale.

Unless the risk analysis is based upon extensive and statistically valid risk data, the risk analysis will be based on the stakeholders’ subjective perceptions of likelihood and impact.

Project managers, programmers, users, business analysts, architects and testers typically have different perceptions and this results in different level of risk for each item. This is why risk analysis process should include some way of reaching consensus or should establish an agreed upon level of risk.

Risk levels should be checked for good distribution through the range, to ensure that the risk rates provide meaningful guidance in terms of test sequencing, prioritization, and effort allocation.

3. Risk Mitigation

After risks are identified and assessed, tests are designed, implemented and executed in order to cover the risks. The effort associated with developing and executing a test is proportional to the level of risk:

• more meticulous test techniques (such as pairwise testing) are used for higher risks
• less meticulous test techniques (such as equivalence partitioning or time-boxed exploratory testing) are used for lower risks

The level of risk should influence decisions such as:

• the use of reviews of project work products
• the level of independence
• the level of experience of the tester
• the degree of confirmation testing (re-testing)
• the degree of regression testing performed

The test team should remain aware of additional information that changes the set of quality risks and/or the level of risk associated with them. Periodic adjustment of the quality risk analysis should occur at least at major project milestones. These adjustment may include: identify new risks, re-assess the level of existing risks and the evaluate risk mitigation effectiveness. For example:

1. If risk identification occurred based on requirement specification, then a re-evaluation of risk should occur once the design specification is done
2. If a component is found to contain considerably more defects than expected, then adjust the likelihood and overall level of risk upward

Product quality risks can also be mitigated before test execution starts. If problems with the requirements are located during risk identification, then the project team can implement thorough requirements specification reviews as a mitigating action. This can reduce the level of risk, which can mean that fewer tests are needed to mitigate the remaining risk.

4. Risk Management

Risk management occurs throughout the entire software testing life cycle. If an organization has a test policy document and/or test strategy document, these should cover the general process by which product and project risks are managed in testing and how risk management is integrated into testing.

In a mature organization where risk awareness pervades the project team, risk management takes place at many levels, and not just for testing.

• Important risks are not only addressed earlier in particular test levels, but also in earlier test levels
• Sources of risk and the consequence of those risks should also be identified
• For defects that do occur, root cause analysis is used to understand sources of risk and to implement process improvements that prevent defects
• Mitigation exists throughout the software development lifecycle
• Risk analysis transcends testing, with the test team participating in a program-wide risk analysis
• Risk analysis is fully informed, considering:
  • related work activity
  • analysis of system behavior
  • cost-based risk assessment
  • product risk analysis
  • end user risk analysis
  • liability risk analysis

Risk based software testing techniques

There are a number of risk based testing techniques, both formal and informal. Such approaches are subjective and dependent on the skills, experience and preferences of the tester and are closely linked together with software risk based testing.

Attempting to capture the benefits of software risk based testing while minimizing the costs, many practitioners employ lightweight risk based approaches:

• Pragmatic Risk Analysis and Management (PRAM)
• Systematic Software Testing (SST)
• Product Risk Management (PRisMa)

In addition to the usual attributes of software risk based testing, these software testing techniques typically have the following attributes:

• Evolved over time based on industry experience (especially in industries where questions of efficiency are important)
• Extensive involvement of a cross-functional team of stakeholders during the initial risk identification and assessment (both business and technical)
• Optimized when introduced in the earliest phases of a project:
  • options to mitigate quality risks are maximized
  • risk analysis can help to influence the product specification and implementation in a way that minimizes risk
• Use the generated output (risk analysis table) as the basis for the software test plan and the test conditions and all test management and analysis activities
• Support the reporting of test results in terms of residual risk to all levels of testing stakeholders

Systematic Software Testing (SST)

• require requirements specifications as input into the risk analysis
• cannot be used except when requirements specifications are provided

PRisMa and PRAM

• Encourage using a blended risk and requirements based strategy
• Can function entirely based on stakeholder input
• Requirements and/or other specifications as an input to the risk analysis
• The use of requirements as an input helps ensure good coverage of the requirements
• Test Manager must ensure that important risks which are not suggested by the requirements (like non-functional) are not missed
• When good, prioritized requirements are provided as an input, one typically sees a strong correlation between risk levels and requirement priority

Many of these software testing techniques encourage the use of risk identification and assessment process as a way to create consensus across stakeholders on software testing and increase the business value of testing.

Insufficient stakeholder participation leads to gaps in the risk analysis. Stakeholders do not always agree on the level of risk, so the person leading a quality risk analysis effort must work creatively and positively with stakeholders to achieve the best possible degree of agreement

Lightweight risk based testing techniques

Lightweight techniques allow the use of weighting of the likelihood and impact factors to emphasize business or technical risks. They use:

• only two factors: likelihood and impact
• simple, qualitative judgments and scales

And these approaches provide:

• flexibility
• applicability in a range of domains
• accessibility across teams of all experience and skill levels

Heavy-weight risk based testing techniques

At the formal, heavy-weight end of the scale, the Test Manager has a number of options available:

• Hazard analysis which
  • extends the analytical process upstream
  • attempting to identify the hazards that underlie each risk
• Cost of exposure which involves determining three factors
  • likelihood (expressed as %) of a failure related to the risk item
  • cost of a loss (expressed as a financial quantity) associated with a typical failure related to the risk item, should it occur in production
  • cost of testing for such failures

Failure Mode and Effect Analysis (FMEA)

• identify quality risks, their potential causes and their likely effects
• then severity, occurrence and detection ratings are assigned in order to establish priority
• strength on precision and meticulousness
• document heavy and hard to learn
• should be considered for high-risk or conservative projects

Quality Function Deployment (QFD)

• a quality risk management technique with testing implications
• concerned with quality risks that arise from an incorrect or insufficient understanding of the customers’ or users’ requirements
• focus on the function of execution of a quality plan

Fault Tree Analysis (FTA)

• observed failures (from testing or from production) or potential failures (quality risks)
• then failures are subjected to a root cause analysis
  • starting with defects that could cause the failure
  • then with errors or defects that could cause those defects
  • continuing on until the various root causes are identified

What risk based testing technique to choose?

The specific software testing techniques that should be used for risk-based testing, and the degree of formality of each technique depend on project, process, and product considerations. In Agile projects, the quality risk analysis is fully integrated into the early period of each sprint and the documentation of risks is blended into the user story tracking. Also, the proper test effort estimation is required if you want a successful project.

Systems of systems require risk analysis on each system as well as on the overall system of systems and safety-critical and mission-critical projects require higher levels of formality and documentation. This risk based testing technique can be used at all stages, even as a user acceptance testing approach.

The most critical success factor for risk-based testing is the involvement of the right team of stakeholders in the risk identification and assessment. All stakeholders have their own understanding of what constitutes quality for the product and their own set of priorities and concerns about quality. They also fall into two broad categories: Business stakeholders and Technical stakeholders

Conclusion

Most risk-based testing methods also include software testing techniques to use the level of risk to sequence and prioritize testing. This ensures early coverage of the most important areas and discovery of the most important defects during test execution. More risk based testing techniques or test estimation techniques are also shared within the blog.

In some cases all the high-risk tests are executed first and tests are run in strict risk order (depth-first). In other cases sampling approach is used to select tests across all the identified risk items using risk to weight the selection. (Breadth-first)

As it is possible that the time allocated for testing might be consumed without all tests being run. This is why risk based testing allows:

• testers to report to management in terms of the remaining level of risk at this point
• management to decide whether to extend testing or transfer the remaining risk onto the users, customers, help desk/technical support and/or operational staff

During test execution the most sophisticated risk-based testing allows project participants, project and product managers, executives, senior managers, and project stakeholders to monitor and control the software development life cycle. This requires the Test Manager to report test results in terms of risk in a way each test stakeholder can understand.

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.