Business Value-based Requirement For Test Case Prioritization Using Reinforcement Learning

Abstract

Testing is a cost-intensive process whereas regression testing is used to find out if there is a regression in a post-evolution or post-maintenance scenario. It is expensive to run all test cases for regression testing and therefore we require a test case selection strategy so that we may select a subset of test cases from a previously available set of all test cases. Test case prioritization is a mechanism that can be used as a test subset selection strategy for regression testing. We find the use of reinforcement learning to optimize test case prioritization adopting reward strategies considering test case failure, failure count, time, and weighted historical results, etc. We investigate how requirements priority and importance as well as test case parameters such as fault severity, fault count, and their inter-dependency can be used for yielding better results. We propose a new reward-based system to optimize the prioritization scheme, which does not rely on prior techniques or limited knowledge. We use a deep reinforcement learning approach to optimize or prioritize the test cases. This approach uses an off-policy, model-free, and value-based approach Deep Q-Network (DQN) agent and a reward system based on a business value of test cases that prioritize the test cases against those business values. We consider case studies of online web systems for test case extraction. Every test case assigned a reward based on their business value requirement. Release 1, release 2, and release 3 of test cycles are created. Every release consists of a varied number of test cases and fault counts. We have created our dataset relevant to 3 releases of test cases. The dataset consists of test cases, their attributes, and the business requirement attributes. The business value reward function distinguishes the faulty test cases from the correct ones by assigning them a specified business value as a reward which is based on the dependency, fault severity, fault count, and business requirement attributes of the test cases. We use different performance metrics to measure the performance and effectiveness of our research. We have compared our results with the published research and conclude that the results achieved from our approach outperform the previously published results of state-of-the-art studies.