The software test tools basics

A Test Manager knows that it is not sufficient to adapt a tool to improve the software testing process and optimize testing for a product or project. There are key principles for software testing tools management that raise from a tool life cycle, range from test tool selection up until return on investment for that testing tool.

Throughout the software testing tool life cycle, it is the responsibility of the Test Manager to ensure the smooth functioning and graceful continuation of tool, ranging from acquisition and up until retirement.

Testing tool life cycle

1. Acquisition of testing tool

• specific software testing tool selection approach has to be used (more details to follow)
• the Administrator of the tool should be assigned by the Test Manager. This can be a test analyst or a technical test analyst and should:
  • define how and when the tool will be used
  • where created artifacts will be stored
  • define naming conventions, etc.
• ROI (return on investment) can be improved by taking these decisions versus allowing them to just happen in an ad-hoc basis
• training will be required and should be done here

2. Support and maintenance

• Ongoing processes for tool support and maintenance of the tool will be required
• The responsibility for maintaining the tool may go to the administrator of the tool, or it might be assigned to a dedicated tools group
• If the tool is to work with other tools, then data interchange and processes for should be considered
• Decisions on backup and restore of the tool and its outputs need to be considered

3. Evolution of software test tool

• Conversion must be considered
• As time goes by, the environment, business needs, or vendor issues may require that changes to the tool or its use are made like:
  • an update to the tool may cause issues with cooperating tools
  • a necessary change to the environment for business reasons may cause issues with the tool
• The more complex the operating environment for a tool, the more evolutionary change may disrupt its use.
• the Test Manager may need to ensure that the organization has a way to ensure continuity of service

4. Retirement

• The time will come when the software testing tools had outlasted their useful lifetime and they will need to be retired gracefully.
• The functionality supplied by the tool will need to be replaced and data will need to be preserved and archived.
• This can occur because the tool is at the end of its testing tool life cycle, or simply because it has reached a point where the benefits and opportunities of conversion to a new tool exceed the costs and risks.

Software test tool selection

A test tool selection is something that any Test Manager has to be familiar with in order to adapt the required tools for the project or organization. There are a set of principles that apply regardless of performance testing tools, test management tools, security testing tools, manual testing tools, etc. When selecting software testing tools for your organization you should:

• Assess the maturity of the organization, its strengths and weaknesses
• identify opportunities for an improved test process supported by tools
• Evaluate the tool versus clear requirements and objective criteria
• Consider whether or not the tool is available for a free trial period
• Evaluate the vendor (training, support, commercial aspects) or evaluate the support if open source
• Identify internal requirements for coaching and mentoring in the use of the tool
• Evaluate training needs by considering testing (and test automation) skills of those who will be working directly with the tool(s)
• Balance the pros and cons of various licensing models
• Estimate a cost-benefit ratio based on a concrete business case
• Execute a proof of concept evaluation
• Establish if the tool performs effectively with the software under test and within the current infrastructure
• Identify changes needed to that infrastructure to use the tool effectively

After completing the test tool selection and a successful proof-of-concept, introducing the selected software testing tool into an organization generally starts with a pilot project with the objective to:

• Gain in-depth knowledge about the tool, understanding both its strengths and weaknesses
• Evaluate how the tool fits with existing processes and practices, and determine the required change
• Decide on standard ways of using, managing, storing, and maintaining the tool and the test assets
• Assess the benefits (can be achieved at a reasonable cost?)
• Understand the metrics that you wish the tool to collect and report, and configuring the tool to ensure their capture and reporting

A Test Manager role is to purchasing the tool from a vendor and using an open source or custom tool, and must investigate the total cost of ownership through the expected lifecycle of the tool and perform the cost-benefit analysis.

Some benefits and challenges of a software testing tool are also listed in the cheat sheet below.

Software testing tools success factors

The software testing manager must know which test tool success factors should be assessed for each new test tool added within the project or organization, but to also consider the importance of integrating the software testing tools in the software development life cycle within the organization. Besides the success factors of a tool, it is key to have a proper management of the testing tool life cycle if you want a test tool return on investment. Some of the most common software testing tools success factors are:

• Rolling out the tool to the rest of the organization incrementally
• Adapting and improving processes to fit with the use of the tool
• Providing training, coaching, and mentoring for tool users
• Defining guidelines for the use of the tool
• Implementing a way to gather usage information from the actual use of the tool
• Monitoring tool use and benefits
• Providing support to the users of a given tool
• Gathering lessons learned from all users

This can be achieved by defining reporting requirements for the tools during the selection process. These requirements must be properly implemented during tool configuration to ensure that the information tracked by the tools can be reported in ways that will be understandable to the stakeholders. Simply acquiring a software testing tool does not guarantee success. Each new tool introduced into an organization will require effort to achieve real and lasting benefits. In order to have a smooth &successful implementation, there are a number of things that should be considered when selecting and integrating test execution and test management tools into an organization.

Potential test tool benefits

Potential benefits of using tools to support test execution (test automation) include:

• Reduction in repetitive manual work (save time)
  • running regression tests
  • environment set up/tear down tasks
  • re-entering the same test data
  • checking against coding standards
• Greater consistency and repeat-ability
  • test data is created in a coherent manner
  • tests are executed by a tool in the same order with the same frequency
  • tests are consistently derived from requirements
• More objective assessment
  • static measures
  • coverage
• Easier access to information about testing
  • statistics and graphs about test progress
  • defect rates and performance

Potential testing tool risks

Potential RISKS of using tools to support test execution:

• Expectations for the tool may be unrealistic
• The time, cost and effort for the initial introduction of a tool may be under-estimated
• The time and effort needed to achieve significant and continuing benefits may be under-estimated
• The effort required to maintain the test assets generated by the tool may be under-estimated
• The tool may be relied on too much (seen as a replacement for test design or execution, or use of automated testing where manual would be better)
• Version control of test assets may be neglected
• An open source project may be suspended
• Relationships and interoperability issues between critical tools may be neglected, such as requirements management tools, configuration management tools, defect management tools and tools from multiple vendors
• The tool vendor may go out of business, retire the tool, or sell the tool to a different vendor
• The vendor may provide a poor response for support, upgrades, and defect fixes
• A new platform or technology may not be supported by the tool
• There may be no clear ownership of the tool

Software testing tools return on investment

The Test Manager must ensure that all tools introduced in the organization add value and have a positive test tool return on investment. This is mainly done by performing a cost-benefit analysis before acquiring or building a tool and considering both recurring and non-recurring costs when calculating return on investment ROI. These are covering at least monetary, resources, time and risks that may impact the value of the tool.

The test team usually used more than one tool and the ROI will be obtained by evaluating the entire software testing tool set and for larger projects or for organizations, a long term and comprehensive software testing tool strategy is recommended.

Nothing comes without risk and a Test Manager should also consider the following when assessing ROI: Immaturity of the organization (not ready); Artifacts created by the tool may be difficult to maintain, requiring multiple revisions when the software under test is changed and reduction of Test Analysts involvement in testing tasks may reduce the value of testing. 

Besides this, the Test manager role is to also look at the benefits that may accrue from the use of the tool and consider a prospective tool from several different viewpoints like: business; project; user.

Non-recurring costs

• Defining tool requirements to meet the objectives and goals
• Evaluating and selecting the correct tool and tool vendor
• Purchasing, adapting or developing the tool
• Performing the initial training for the tool
• Integrating the tool with other tools
• Procuring the hardware/software needed to
• support the tool

Recurring costs

• Owning the tool (licensing, maintenance costs of the tool and of the tool artifacts, ongoing training, etc.)
• Porting the tool to different environments
• Adapting the tool to future needs
• Improving the quality and processes to ensure optimal use of the selected tools

Opportunity costs

• usually inherent in the tool
• The time spent on acquiring, administering, training and using the tool could have been spent on actual testing tasks
• More testing resources may be needed up front until the tool goes on line

Benefits that may accrue can be a reduction in

• Repetitive work
• Test cycle time (automated regression)
• Test execution costs
• Human error in different phases of testing because:
  • Test data may be more valid using data generation tools
  • Test result comparisons may be more accurate using comparison tools
  • Test data entry may be more correct by entering with a scripting tool
• Effort required to access information about tests:
  • Tool generated reports and metrics
  • Reuse of test assets such as test cases, test scripts and test data

Other viewpoints for a tool prospect

When looking from other viewpoints at a tool prospect, the ones we should most likely consider are:

• Business viewpoint (Positive ROI is required)
  • we need to assure that the existing and non test tool work together with the new test tool
  • look at improving the processes and inter connectivity between tool usage
• Project viewpoint (Tool must be effective)
  • The tool may require an appreciable amount of time in order to start earning positive ROI
  • In many cases, ROI may occur in the second release or during maintenance rather than during the initial project when it was implemented
• Tool user viewpoint (must support in doing tasks)
  • learning curve must be considered to ensure the users will be able to learn the tool quickly with minimal stress
  • When first introduced, test tools require training and mentoring for the users

This article is based on the ISTQB Advanced Syllabus version 2012 and it also references the ISTQB Foundation Syllabus version 2018. It uses terminology definitions from the ISTQB Glossary version 3.2.