Gaining business value from software testing

A central responsibility of a software test manager is to secure and utilize resources (people, software, hardware, infrastructure, etc.) to carry out value-adding processes in software testing. These processes are often part of a project or a Program aimed at delivering software or a system for internal or external use. This is why we need software test management in context of the project or product under work in order to improve the business value of testing.

For this, the Test Manager role is to arrange the software testing process, associated activities, and work products according to the needs and circumstances of stakeholders, activities (ex: software development life cycle) and work products (ex: requirements, specifications).

While most organizations consider testing valuable in some sense, few managers, including Test Managers, can quantify, describe, or articulate the business value of testing. Many test managers, test leads or testers focus on the tactical details of testing (aspects specific to the task or test level) while ignoring the larger strategic (higher level) issues related to testing that other project participants care about.

The Test Manager must work to optimize testing so that good business value is delivered:

• Testing excessively does not deliver good business value, because the testing will impose unreasonable delays and cost more than it saves.
• Testing too little does not deliver good business value because too many defects will be delivered to users

The optimum business value of testing lies between those two extremes and the Test Manager must help testing stakeholders understand this and the value delivered by testing.

Delivering value in software testing

Testing delivers value to the organization, project, and/or operation in both quantitative and qualitative ways:

Qualitative values such as:

• improved reputation for quality
• smoother and more-predictable releases
• increased confidence, protection from liability
• reducing risk of loss of whole missions or lives

Quantitative values such as:

• include finding defects that are prevented or fixed prior to release
• finding defects that are known prior to release (not fixed but documented, with workarounds)
• reducing risk by running tests
• delivering information on project, process, and product status

Testing value versus cost of quality

The Test Manager should understand which of these values apply for their organization, project, and/or operation, and be able to communicate about testing in terms of these values.

A well-established method for measuring the quantitative value and efficiency of testing is called cost of quality (or, sometimes, cost of poor quality how many followers on facebook to get paid 2021). It involves classifying project and operational costs into four categories related to product defect costs:

• Costs of prevention training developers to write more maintainable or secure code
• Costs of detection writing test cases, configuring test environments, and reviewing requirements
• Costs of internal failure fixing defects detected during testing or reviews, prior to delivery
• Costs of external failure support costs associated with defective software delivered to customers

Sometimes it is not enough to deliver value or improve the cost of quality, but in times you need to think about distributed testing across in-sourced and out sourced organizations.

Understanding test management stakeholders

People are stakeholders of software testing when they have an interest in the testing activities, the testing work products, or the quality of the final system or deliverable. Such interests can be direct or indirect involvement in the testing activities, direct or indirect receipt of testing work products or direct or indirect effect by the quality of the deliverable produced by the project or program. Below is an example (not comprehensive) of stakeholders and their roles:

• Developers, development leads, and development managers
  • implement the software under test
  • receive test results
  • must take action based on those results
• Database architects, system architects, and designers
  • design the software
  • receive test results
  • must take action on those results
• Marketing and business analysts
  • determine the features and the level of quality inherent in those features
  • involved in defining needed test coverage
  • reviewing test results
  • making decisions based on test results
• Senior management, product managers and project sponsors
  • often involved in defining needed test coverage
  • reviewing test results
  • making decisions based on test results
• Project managers
  • responsible for managing their projects to success
  • balance quality, schedule, feature, and budget priorities
  • often procure the resources required for the test activities
  • collaborate with the Test Manager in test planning and control
• Technical support, customer support and help desk staff
  • support the users and customers
• Direct and indirect users
  • use the software directly
  • receive outputs or services produced or supported by the software

Test Managers must identify the specific testing stakeholders for their project or Program and understand the precise nature of the stakeholder relationship with testing and how the test team serves the needs of the stakeholders. They should identify the other software development life cycle activities and work products that affect testing and/or are affected by testing.

Software Delivery Life cycle Activities and Work Products

Software testing exists in the context of a larger set of software development lifecycle activities.

The Test Manager must plan and guide the testing activities with an understanding of how these other activities and their work products affect testing and identify the other software development life cycle activities and work products that affect testing and/or are affected by testing. 

Software testing is closely interconnected and related to:

Requirements engineering and management

• The Test Manager must:
  • consider requirements during the scoping and estimation of test effort
  • remain aware of changes to the requirements
  • exercising test control actions to adjust to those changes
• Technical Test Analysts and Test Analysts should participate in requirements reviews

Configuration management, release management, and change management

• The Test Manager must:
  • establish the test object delivery processes and mechanisms
  • capture test object delivery processes and mechanisms in the test plan
• and may ask Test Analysts and Technical Test Analysts to create build verification tests and to ensure version control during test execution

Project management

• The Test Manager must:
  • provide schedule and resource requirements to the Project Manager
  • work with the Project Manager to understand changes in the project plan and exercise test control actions to adjust to those changes

Software development and maintenance

• The Test Manager should:
  • work with Development Managers to coordinate the delivery of test objects, including content and dates of each test release
  • participating in defect management

Technical support

• The Test Manager should:
  • work with the Technical Support Manager to ensure proper delivery of test results during test closure to make sure that the support team is aware of aware of known failures and workarounds
  • work with the Technical Support Manager to analyze production failures in order to implement test process improvement

Production of technical documentation

• The Test Manager should:
  • work with the Technical Documentation Manager to ensure delivery of documentation for testing in a timely fashion as the management of defects found in those documents

Aligning testing with Life cycle Activities

Testing should be an integral part of the project, regardless of the software development models used. The Test Strategy may capture general information about how to align testing with other life cycle activities and The Test Manager should perform project-specific alignment for each test level and for any selected combination of software development life cycle and test process during the planning stage.

Depending on the needs of the organization, project and product, additional test levels beyond those defined may be required, such as: Hardware-software integration testing, System integration testing, Feature interaction testing, and Customer product integration testing. Each of these test levels should have the following elements clearly defined within the test plan.

Sequential models: Waterfall, V-model and W-model

• All of the work products and activities for a given phase are completed before the next phase begins
• Test planning, test analysis, test design and test implementation proceed in an overlapping fashion with project planning, business/requirements analysis, software design and development
• Test execution proceeds sequentially according to the test levels

Iterative or incremental models: Rapid Application Development (RAD) and Rational Unified Process (RUP)

• The features to be implemented are grouped together and then the project phases and their work products and activities occur for each group
• The phases may be done either sequentially or can overlap
• The iterations themselves may be sequential or overlapping
• During project initiation, high-level test planning and test analysis occurs in parallel with the project planning and business/requirements analysis
• Detailed test planning, test analysis, test design, and test implementation occurs at the beginning of each iteration, in an overlapping fashion
• Test execution often involves overlapping test levels
• Each test level begins as early as possible and may continue after subsequent, higher levels started

Agile: SCRUM and Extreme Programming (XP)

• These are iterative life cycles where the iterations are very short (often two to four weeks)
• The work products and activities for each iteration are concluded before the next iteration starts
• Testing is similar to iterative models, but with a higher degree of overlap of the various testing activities with the development activities, including considerable overlap of test execution (at various levels) with the development activities
• All of the activities in an iteration, including the test activities should be complete before the next iteration starts
• In an Agile project, the role of the Test Manager usually changes from a direct managerial role to a technical authority/advisory role

Spiral

• Prototypes are used early in the project to confirm feasibility and to experiment with design and implementation decisions
• The order of prototyping is based on the level of business priority and technical risk
• These prototypes are tested to determine what aspects of the technical problems remain unsolved.
• Once the main technical problems are resolved, the project proceeds according to either a sequential or iterative model

In order to properly align testing activities within the life cycle, the Test Manager must have a detailed understanding of the life cycle models used.

In V-model we have

• System Test planning activities occur concurrently with project planning
• System Test control continues until system test execution and closure are complete
• System Test analysis and design activities occur concurrently with requirements specification, system architectural and component design specification.
• System Test implementation activities might start during system design and would typically occur concurrently with development and unit test. Work on system test implementation activities stretch often until just days before the start of execution
• System Test execution begins when system test entry criteria is met and execution continues until the system test exit criteria are met.
• Evaluation of the system test exit criteria and reporting of system test results occur throughout test execution, generally with greater frequency and urgency as project deadlines approach.
• System Test closure activities occur the system test exit criteria is met and system test execution is declared complete. Sometimes it can be delayed until after acceptance testing is over

In Iterative or incremental we have

• rather than being able to implement the entire test environment at the beginning of the project, it may be more efficient to implement only the part needed for the current iteration
• the farther ahead the planning occurs, the farther ahead the scope of the test process can extend
• test execution and reporting may also be influenced by the lifecycle being used by the team
  • effective to produce complete reports and to conduct post-iteration review sessions before the start of the next iteration
  • by treating each iteration as a mini-project, the team is given an opportunity to correct and adjust based on what occurred before
  • as iterations may be short tasks might be conducted in a manner in which testing can be reported based on task completion
  • process issues experienced in one iteration can easily affect and recur in the next iteration if corrective measures are not taken

Distributed, outsourced testing or insourced testing

Some or perhaps even all of the software testing effort is done by people in different locations, employed by different companies, or separated from the project team. This is why a test manager role is to know how to handle distributed, outsourced or in-sourced testing.

Distributed testing = test effort occurs at multiple locations

Outsourced testing = test effort is carried out at one or more locations by people who are not employees of the company and who are not co-located with the project team

Insourced testing = test effort is carried out by people who are co-located with the project team but who are not fellow employees

Common across all such software test efforts is the need for clear channels of communication and well-defined expectations for missions, tasks, and deliverable:

• Project team must rely less on informal channels such as hallway conversations and colleagues spending social time together
• Important to have defined ways in which communication should occur, including topics as:
  • escalation of problems
  • types of information to be communicated
  • methods of communication
• Everyone, on all sides of the team relationships, must have a clear understanding of their roles and responsibilities as well as the other parties’ roles
• Location, time zone, cultural and language differences make communication and expectation issues even more likely to arise

Common across all such test efforts is the need for alignment of methodologies:

• miss-alignment of methodologies is more likely to arise in situations where the work is distributed and/or performed by outside entities
• Significant problems, especially during test execution can occur if different testing groups use different methodologies, but also if different methodologies are used across development and testing.

Example of distributed outsourced testing

For distributed testing, the division of the test work across the multiple locations must be explicit and intelligently decided:

• Without such guidance, the most competent group may not do the work for which they are qualified
• If each team doesn’t know what they are responsible for, they may not do what they are supposed to do
• Expectations for each of the teams must be clearly communicated
• Without careful management, the test work as a whole may suffer from gaps and overlap

For all such test efforts, it is critical that the entire project team develop and maintain trust that all of the test team(s) will carry out their roles properly in spite of organizational, cultural, language, and geographical boundaries.

Lack of trust leads to inefficiencies and delays associated with verifying activities, assigning blame and playing organizational politics. If a client is using Agile development, while the testing services provider has a predefined software testing methodology that assumes a sequential software testing life cycle, the timing and nature of delivery of test items to the testing services provider will be a point of contention.

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.