Search

Industrial robotic systems (IRS) consist of industrial robots that automate industrial processes. They accurately perform repetitive tasks, replacing or assisting with dangerous jobs like assembly in the automotive and chemical industries. Failures in these systems can be catastrophic, so it is important to ensure their quality and safety before using them. One way to do this is by applying a software testing process to find faults before they become failures. However, software testing in industrial robotic systems has some challenges. These include differences in perspectives on software testing from people with diverse backgrounds, coordinating and collaborating with diverse teams, and performing software testing within the complex integration inherent in industrial environments. In traditional systems, a well-known development process uses simple, structured sentences in English to facilitate communication between project team members and business stakeholders. This process is called behavior-driven development (BDD), and one of its pillars is the use of templates to write user stories, scenarios, and automated acceptance tests. We propose a software testing (ST) approach called automated acceptance testing for industrial robotic systems (AAT4IRS) that uses natural language to write the features and scenarios to be tested. We evaluated our ST approach through a proof-of-concept, performing a pick-and-place process and applying mutation testing to measure its effectiveness. The results show that the test suites implemented using AAT4IRS were highly effective, with 79% of the generated mutants detected, thus instilling confidence in the robustness of our approach. [ABSTRACT FROM AUTHOR]

Human–Machine Pair Programming (HMPP) is a promising technique in the software development process, which means that software construction can be done in the manner that humans are responsible for developing the program while computer is responsible for monitoring the program in real-time and reporting errors. The Java runtime exceptions in the current version of the software under construction can only be effectively detected by means of its execution. Traditional software testing techniques are suitable for testing completed programs but face a challenge in building a suitable testing environment for testing the partial programs produced during HMPP. In this paper, we put forward a novel technique, called Program Segment Testing (PST) for automatically identifying errors caused by runtime exceptions to support HMPP. We first introduce the relevant involved in this technique to detect index out of bounds exceptions, a representative of runtime exceptions. Then we discuss the methodology of this technique in detail and illustrate its workflow with a simple case study. Finally, we carry out an experiment to evaluate this technique and compare it with three existing fault detection techniques using several programs to demonstrate its effectiveness. [ABSTRACT FROM AUTHOR]

The rapid proliferation of network applications has led to a significant increase in network attacks. According to the OWASP Top 10 Projects report released in 2021, injection attacks rank among the top three vulnerabilities in software projects. This growing threat landscape has increased the complexity and workload of software testing, necessitating advanced tools to support agile development cycles. This paper introduces a novel test prioritization method for SQL injection vulnerabilities to enhance testing efficiency. By leveraging previous test outcomes, our method adjusts defense strength vectors for subsequent tests, optimizing the testing workflow and tailoring defense mechanisms to specific software needs. This approach aims to improve the effectiveness and efficiency of vulnerability detection and mitigation through a flexible framework that incorporates dynamic adjustments and considers the temporal aspects of vulnerability exposure. [ABSTRACT FROM AUTHOR]

Software testing is an effective means of ensuring software quality. The cost of software testing is the main component of the total cost of software development. The generation of test data is very important in testing, because the efficiency of testing depends on the test data used. A significant part of software testing automation is the test data generation. How to automatically generate test datasets is still an open problem. Search-based software testing (SBST) is a process of generating test data that employs meta-heuristic methods to resolve tough NP-hard problems. The test data generation problem belongs to NP-hard problems. This paper focuses on the automatic generation of test data for path coverage based on control flow criteria in structural testing. Some meta-heuristic algorithms have been used to search for software test data. To advance the efficiency and effectiveness of path test data generation, an adaptive simulated annealing particle swarm optimization (ASAPSO) algorithm is proposed. The probabilistic jumping property of simulated annealing (SA) algorithm is introduced into the particle swarm optimization (PSO) algorithm to make the particle accept bad solution at a certain probability during the searching process, and therefore boost the capacity of the proposed algorithm to bounce from the local optimum. The fitness function is constructed by the branch function superposition to properly guide the search process. In addition, to improve the convergence speed of the algorithm, an adaptive adjustment scheme based on inertial weight and an adjustment scheme based on learning factor are proposed as well. The experimental results show that the proposed method can effectually avoid premature convergence and effectively upgrades the efficiency of generating test data automatically. It is also competitive in solving other complicated optimization problems. [ABSTRACT FROM AUTHOR]

The purpose of this study is to identify and classify studies published on software testing techniques applied to blockchain systems. Previously published reviews in related areas have a narrow focus and/or do not follow a systematic review protocol. We conducted a systematic mapping based on an initial selection of 1025 studies. A rigorous selection process resulted in a final pool of 17 primary studies. These studies are categorized with respect to the employed testing methods, considered quality attributes, and functionality. We observe that most of the publications focus on testing functional correctness or security, whereas the testing of runtime performance attracts less attention. Existing approaches mostly employ fuzz testing or mutation testing. Search-based testing is usually combined with these techniques. The application of model-based testing is rare. The adaptability of fuzz testing and model-based testing techniques to changing blockchain platforms and languages remains a concern. On the other hand, performance and scalability issues are noted for search-based techniques and mutation testing. The use and integration of multiple testing techniques also stand out as a viable research direction. [ABSTRACT FROM AUTHOR]

Software applications (apps) have been playing an increasingly important role in various aspects of society. In particular, mobile apps and web apps are the most prevalent among all applications and are widely used in various industries as well as in people's daily lives. To help ensure mobile and web app quality, many approaches have been introduced to improve app GUI testing via automated exploration, including random testing, model-based testing, learning-based testing, and so on. Despite the extensive effort, existing approaches are still limited in reaching high code coverage, constructing high-quality models, and being generally applicable. Reinforcement learning-based approaches, as a group of representative and advanced approaches for automated GUI exploration testing, are faced with difficult challenges, including effective app state abstraction, reward function design, and so on. Moreover, they heavily depend on the specific execution platforms (i.e., Android or Web), thus leading to poor generalizability and being unable to adapt to different platforms. This work specifically tackles these challenges based on the high-level observation that apps from distinct platforms share commonalities in GUI design. Indeed, we propose PIRLTest, an effective platform-independent approach for app testing. Specifically, PIRLTest utilizes computer vision and reinforcement learning techniques in a novel, synergistic manner for automated testing. It extracts the GUI widgets from GUI pages and characterizes the corresponding GUI layouts, embedding the GUI pages as states. The app GUI state combines the macroscopic perspective (app GUI layout) and the microscopic perspective (app GUI widget) and attaches the critical semantic information from GUI images. This enables PIRLTest to be platform-independent and makes the testing approach generally applicable on different platforms. PIRLTest explores apps with the guidance of a curiosity-driven strategy, which uses a Q-network to estimate the values of specific state-action pairs to encourage more exploration in uncovered pages without platform dependency. The exploration will be assigned with rewards for all actions, which are designed considering both the app GUI states and the concrete widgets, to help the framework explore more uncovered pages. We conduct an empirical study on 20 mobile apps and 5 web apps, and the results show that PIRLTest is zero-cost when being adapted to different platforms, and can perform better than the baselines, covering 6.3–41.4% more code on mobile apps and 1.5–51.1% more code on web apps. PIRLTest is capable of detecting 128 unique bugs on mobile and web apps, including 100 bugs that cannot be detected by the baselines. [ABSTRACT FROM AUTHOR]

Natural language processing (NLP) has gained widespread adoption in the development of real-world applications. However, the black-box nature of neural networks in NLP applications poses a challenge when evaluating their performance, let alone ensuring it. Recent research has proposed testing techniques to enhance the trustworthiness of NLP-based applications. However, most existing works use a single, aggregated metric (i.e., accuracy) which is difficult for users to assess NLP model performance on fine-grained aspects, such as LCs. To address this limitation, we present ALiCT, an automated testing technique for validating NLP applications based on their LCs. ALiCT takes user-specified LCs as inputs and produces diverse test suite with test oracles for each of given LC. We evaluate ALiCT on two widely adopted NLP tasks, sentiment analysis and hate speech detection, in terms of diversity, effectiveness, and consistency. Using Self-BLEU and syntactic diversity metrics, our findings reveal that ALiCT generates test cases that are 190% and 2213% more diverse in semantics and syntax, respectively, compared to those produced by state-of-the-art techniques. In addition, ALiCT is capable of producing a larger number of NLP model failures in 22 out of 25 LCs over the two NLP applications. [ABSTRACT FROM AUTHOR]

The increasing number of software applications incorporating machine learning (ML) solutions has led to the need for testing techniques. However, testing ML software requires tremendous human effort to design realistic and relevant test inputs and to judge software output correctness according to human common sense. Even when misbehavior is exposed, it is often unclear whether the defect is inside ML API or the surrounding code and how to fix the implementation. This article tackles these challenges by proposing Keeper, an automated testing and fixing tool for ML software. The core idea of Keeper is designing pseudo-inverse functions that semantically reverse the corresponding ML task in an empirical way and proxy common human judgment of real-world data. It incorporates these functions into a symbolic execution engine to generate tests. Keeper also detects code smells that degrade software performance. Once misbehavior is exposed, Keeper attempts to change how ML APIs are used to alleviate the misbehavior. Our evaluation on a variety of applications shows that Keeper greatly improves branch coverage, while identifying 74 previously unknown failures and 19 code smells from 56 out of 104 applications. Our user studies show that 78% of end-users and 95% of developers agree with Keeper's detection and fixing results. [ABSTRACT FROM AUTHOR]

Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of LLMs instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime-generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present LLMEffiChecker, which can work under both white-box setting and black-box setting. In the white-box scenario, LLMEffiChecker develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario, LLMEffiChecker employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally unreachable threshold. To demonstrate the effectiveness of LLMEffiChecker, we conduct a systematic evaluation on nine publicly available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT, and Salesforce CodeGen. Experimental results show that LLMEffiChecker can increase on average LLMs' response latency and energy consumption by 325% to 3,244% and 344% to 3,616%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by LLMEffiChecker significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs). [ABSTRACT FROM AUTHOR]

Metamorphic testing is a valuable approach to verifying machine learning programs where traditional oracles are unavailable or difficult to apply. This paper proposes a technique to prioritize metamorphic relations (MRs) in metamorphic testing for machine learning and deep learning systems, aiming to enhance early fault detection. We introduce five metrics based on diversity in source and follow-up test cases to prioritize MRs. The effectiveness of our proposed prioritization methods is evaluated on three machine learning and one deep learning algorithm implementation. We compare our approach against random-based, fault-based, and neuron activation coverage-based MR ordering. The results show that our data diversity-based prioritization performs comparably to fault-based prioritization, reducing fault detection time by up to 62% compared to random MR execution. Our proposed metrics outperformed neuron activation coverage-based prioritization, providing 5–550% higher fault detection effectiveness. Overall, our approach to prioritizing metamorphic relations leads to increased fault detection effectiveness and reduced average fault detection time. This improvement in efficiency can result in significant time and cost savings when applying metamorphic testing to machine learning and deep learning systems. [ABSTRACT FROM AUTHOR]

Software testing is a critical phase due to misconceptions about ambiguities in the requirements during specification, which affect the testing process. Therefore, it is difficult to identify all faults in software. As requirement changes continuously, it increases the irrelevancy and redundancy during testing. Due to these challenges; fault detection capability decreases and there arises a need to improve the testing process, which is based on changes in requirements specification. In this research, we have developed a model to resolve testing challenges through requirement prioritization and prediction in an agile-based environment. The research objective is to identify the most relevant and meaningful requirements through semantic analysis for correct change analysis. Then compute the similarity of requirements through case-based reasoning, which predicted the requirements for reuse and restricted to error-based requirements. Afterward, the apriori algorithm mapped out requirement frequency to select relevant test cases based on frequently reused or not reused test cases to increase the fault detection rate. Furthermore, the proposed model was evaluated by conducting experiments. The results showed that requirement redundancy and irrelevancy improved due to semantic analysis, which correctly predicted the requirements, increasing the fault detection rate and resulting in high user satisfaction. The predicted requirements are mapped into test cases, increasing the fault detection rate after changes to achieve higher user satisfaction. Therefore, the model improves the redundancy and irrelevancy of requirements by more than 90% compared to other clustering methods and the analytical hierarchical process, achieving an 80% fault detection rate at an earlier stage. Hence, it provides guidelines for practitioners and researchers in the modern era. In the future, we will provide the working prototype of this model for proof of concept. [ABSTRACT FROM AUTHOR]

Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. FROGa attempts to estimate suspiciousness more accurately using logistic regression analysis. FROGb attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that FROGa has very high accuracy, and FROGb can drastically reduce time cost for targets that have been difficult to complete by the conventional method. [ABSTRACT FROM AUTHOR]

The internet key exchange (IKE) protocol, integral to the authentication and key negotiation process within the Internet Protocol Security (IPSec) framework, has been utilized for the protection of IP communications. Given the complex protocol logic, security vulnerabilities in the implementation of the IKE protocol are inevitably present. Fuzz testing, recognized as an effective means of detecting potential vulnerabilities in protocol implementations, has been conventionally applied. However, the direct application of existing fuzzing tools to the IKE protocol has been found to present limitations, such as the generation of low-quality test cases and difficulty in exploring deep states. To address these issues, a mutation strategy based on the grammar of the IKE protocol was designed, aiming to reduce the generation of invalid test cases while increasing the diversity of generated test cases. Additionally, an evolutionary strategy-based mutation scheduling scheme was introduced, which automatically optimized the probability distribution of mutation operators, further increasing the likelihood of generating high-quality test cases. A message handler was designed to maintain protocol interaction context information and perform cryptographic operations, thereby supporting testing of the IKE protocol under black-box conditions. This enabled the exploration of deep protocol interaction behavior and state space. Utilizing the aforementioned methods, an IKE protocol stateful fuzz testing tool named IKEChecker was implemented, supporting testing of both IKEv1 and IKEv2 protocols. Testing was conducted on two widely used open-source IKE protocol implementations, strongSwan and Libreswan, resulting in the revelation of 4 undisclosed vulnerabilities. By comparing IKEChecker with other fuzz testing tools, its efficiency in vulnerability detection was evaluated.

In this research, authors give a literature analysis of the methods used to detect and anticipate security risks in software testing by using a number of deep learning models. The purpose of this study is to conduct a literature review on the use of deep learning models in software testing for the purpose of detecting and predicting security issues. Moreover, the motive of this work is to analyze the exiting techniques related as software testing model and its application oriented performance. A thorough search of the available literature across several databases and resources forms the basis of this evaluation. There are a total of 69 publications found via the search, 34 of which are original research. Convolutional neural networks, long short-term memory networks, generative adversarial networks, and capsule networks are only some of the state-of-the-art methods for identifying and predicting security risks that are the topic of this research study. Datasets, performance indicators, and assessment criteria are all dissected for this examination. Finally, the benefits and drawbacks of using deep learning models to detect and anticipate software security flaws during testing are outlined in this article. Software testing experts and academics will benefit from this study since it will shed light on the present state of the art in the area and provide direction for the creation of innovative approaches to the detection and forecasting of security risks in software testing. [ABSTRACT FROM AUTHOR]

Few mobile robot developers already test their software on simulated robots in virtual environments or sceneries. However, the majority still shy away from simulation-based test campaigns because it remains challenging to specify and execute suitable testing scenarios, that is, models of the environment and the robots' tasks. Through developer interviews, we identified that managing the enormous variability of testing scenarios is a major barrier to the application of simulation-based testing in robotics. Furthermore, traditional CAD or 3Dmodelling tools such as SolidWorks, 3ds Max, or Blender are not suitable for specifying sceneries that vary significantly and serve different testing objectives. For some testing campaigns, it is required that the scenery replicates the dynamic (e.g., opening doors) and static features of real-world environments, whereas for others, simplified scenery is sufficient. Similarly, the task and mission specifications used for simulation-based testing range from simple point-to-point navigation tasks to more elaborate tasks that require advanced deliberation and decision-making. We propose the concept of composable and executable scenarios and associated tooling to support developers in specifying, reusing, and executing scenarios for the simulation-based testing of robotic systems. Our approach differs from traditional approaches in that it offers a means of creating scenarios that allow the addition of new semantics (e.g., dynamic elements such as doors or varying task specifications) to existing models without altering them. Thus, we can systematically construct richer scenarios that remain manageable. We evaluated our approach in a small simulation-based testing campaign, with scenarios defined around the navigation stack of a mobile robot. The scenarios gradually increased in complexity, composing new features into the scenery of previous scenarios. Our evaluation demonstrated how our approach can facilitate the reuse of models and revealed the presence of errors in the configuration of the publicly available navigation stack of our SUT, which had gone unnoticed despite its frequent use. [ABSTRACT FROM AUTHOR]

The enhancement of software system quality is achieved through a process called software testing, which is a time and cost-intensive stage of software development. As a result, automating software tests is recognized as an effective solution that can simplify time-consuming and arduous testing activities. Generating test data with maximum branch coverage and fault discovery capability is an NP-complete optimization problem. Various methods based on heuristics and evolutionary algorithms have been suggested to create test suites that provide the most feasible coverage. The main disadvantages of past approaches include inadequate branching coverage, fault detection rate, and unstable results. The main objectives of the current research are to improve the branch coverage rate, fault detection rate, success rate, and stability. This research has suggested an efficient technique to produce test data automatically utilizing a hybrid version of Olympiad Optimization Algorithms (OOA) in conjunction with genetic algorithm (GA) operators theory. Maximum coverage, fault detection capability, and success rate are the main characteristics of produced test data. Various experiments have been conducted on the nine standard benchmark programs. Regarding the results, the suggested method provides 99.92% average coverage, a success rate of 99.20%, an average generation of 5.76, and an average time of 7.97 s. Based on the fault injection experiment's results, the proposed method can discover about 89% of the faults injected by mutation testing tools such as MuJava. [ABSTRACT FROM AUTHOR]

Quality and productivity needs are considered together in software. For this reason, any existing software should be tested automatically with test automation. Software test automations is automated software testing activities. Automating constantly manually written tests, on the other hand, saves time, reduces error rates, produces better quality software, and reduces costs. This study aims to produce an automatic unit testing framework that is planned to work in run-time on software products. This developed application performs unit test transformations that can respond to the desired test scenarios on the product being studied.. Java agent is used as the basis of all these transformations. All information about the objects, methods, and variables of the sample java classes to be worked on is converted into data in run-time using byte code. During this transformation, information is saved in the database, and unit tests are created automatically through the template engine. Compared to the products developed on automatic unit test generation in the literature, the opcode parsing method was developed for this study. This method reads a byte code at run-time, uses the properties of the java class it belongs to, and automatically creates the unit test class and test methods. The study can also examine different object definitions and conditional and loop structures within a method and produce alternative test scenarios. The automatic unit test scenario produced has been turned into a flexible framework that can encounter minimum errors at run-time. Considering the scarcity of studies in the field of national software testing; It is thought that the automatic unit test generation product developed within the scope of this study, using byte code, will contribute to the work area. [ABSTRACT FROM AUTHOR]

As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power. [ABSTRACT FROM AUTHOR]

Summary: To ensure the quality of software systems, software engineers can make use of a variety of quality assurance approaches, for example, software testing, modern code review, automated static analysis, and build automation. Each of these quality assurance practices have been studied in depth in isolation, but there is a clear knowledge gap when it comes to our understanding of how these approaches are being used in conjunction, or not. In our study, we broadly investigate whether and how these quality assurance approaches are being used in conjunction in the development of 1454 popular open source software projects on GitHub. Our study indicates that typically projects do not follow all quality assurance practices together with high intensity. In fact, we only observe weak correlation among some quality assurance practices. In general, our study provides a deeper understanding of how existing quality assurance approaches are currently being used in Java‐based open source software development. Besides, we specifically zoom in on the more mature projects in our dataset, and generally we observe that more mature projects are more intense in their application of the quality assurance practices, with more focus on their ASAT usage, and code reviewing, but no strong change in their CI usage. [ABSTRACT FROM AUTHOR]

In software that is free from defects or errors, a fixed method and technique of software testing is required. For this step, an Information System is needed to help eliminate software defects, so that testing or testing work is not a burden on software costs but is a perfection of software development work so that Zero Defect Application Software can be realized. From several software testing methods and techniques, a strategy suitable for the size of a software development project is required. So even though the software development project is small, it still requires a suitable test. And for software development testing to have reliable quality, it is necessary to measure and criteria for each software testing practice, so that a software development project can be determined in quality. To support software testing work, an Information System is needed that supports the need for processors to assess a quality software development test. With the Software Defect Elimination Information System, an objective quality of software development testing can be produced. [ABSTRACT FROM AUTHOR]

Copyright of Ordnance Industry Automation is the property of Editorial Board for Ordnance Industry Automation and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Techno.com is the property of Universitas Dian Nuswantoro, Fakultas Ilmu Komputer and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

As software vulnerabilities can cause cybersecurity threats and have severe consequences, it is necessary to develop effective techniques to discover such vulnerabilities. Fuzzing is one of the most widely employed approaches that has been adapted for software testing. The mutation-based fuzzing approach is currently the most popular. The state-of-the-art American Fuzzy Lop (AFL) selects mutations randomly and lacks knowledge of mutation operations that are more helpful in a particular stage. This study performs a systematic review to identify and analyze existing approaches that optimize the selection of mutation operations. The main contributions of this work are to draw attention to the importance of mutation operator selection, identify optimization algorithms for mutation operator selection, and investigate their impact on fuzzing testing in terms of code coverage and finding new vulnerabilities. The investigation shows the effectiveness and advantages of optimizing the selection of mutation operations to achieve higher code coverage and find more vulnerabilities. [ABSTRACT FROM AUTHOR]

Copyright of Chinese Journal of Network & Information Security is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Software is a part of most complex systems used for different purposes in today's world. The reliability of these systems must be verified before putting them to use, because their malfunction may lead to catastrophic failure, the loss of human lives, or huge costs. Software reliability can be verified by performing black‐box testing, without knowledge of the complex internal structure of these systems. In this work, an embedded IoT alarm system for detection of intruders and potential robbery is proposed. It is composed of three subsystems: embedded IoT, web‐server, and mobile phone subsystems. Cause‐effect graphing technique was applied for generating three types of black‐box test suites, by using the forward‐propagation and two different minimization algorithms. These test suites were used for black‐box testing the system, which was performed after injecting a maximum of 12 faults into the system in 100 iterations. The resulting software quality and reliability metrics showed that the forward‐propagation algorithm always achieves maximum fault detection rate (FDR), and this algorithm should be used when the highest demand for reliability must be met. The optimized minimization should only be used in cases where testing is very expensive, because it did not find faults in all testing iterations. The basic optimization retained high FDR, while reducing the impact of the minimized test suite and achieved a significant increase of test effect coverage values. [ABSTRACT FROM AUTHOR]

Testing Autonomous Driving Systems (ADSs) is crucial to ensure their reliability when navigating complex environments. ADSs may exhibit unexpected behaviours when presented, during operation, with driving scenarios containing features inadequately represented in the training dataset. To address this shift from development to operation, developers must acquire new data with the newly observed features. This data can be then utilised to fine tune the ADS, so as to reach the desired level of reliability in performing driving tasks. However, the resource-intensive nature of testing ADSs requires efficient methodologies for generating targeted and diverse tests. In this work, we introduce a novel approach, DeepAtash-LR, that incorporates a surrogate model into the focused test generation process. This integration significantly improves focused testing effectiveness and applicability in resource-intensive scenarios. Experimental results show that the integration of the surrogate model is fundamental to the success of DeepAtash-LR. Our approach was able to generate an average of up to 60× more targeted, failure-inducing inputs compared to the baseline approach. Moreover, the inputs generated by DeepAtash-LR were useful to significantly improve the quality of the original ADS through fine tuning. [ABSTRACT FROM AUTHOR]

Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs) that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this article, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers. [ABSTRACT FROM AUTHOR]

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria. [ABSTRACT FROM AUTHOR]

Voice-based virtual assistants are becoming increasingly popular. Such systems provide frameworks to developers for building custom apps. End-users can interact with such apps through a Voice User Interface (VUI), which allows the user to use natural language commands to perform actions. Testing such apps is not trivial: The same command can be expressed in different semantically equivalent ways. In this article, we introduce VUI-UPSET, an approach that adapts chatbot-testing approaches to VUI-testing. We conducted an empirical study to understand how VUI-UPSET compares to two state-of-the-art approaches (i.e., a chatbot testing technique and ChatGPT) in terms of (i) correctness of the generated paraphrases, and (ii) capability of revealing bugs. To this aim, we analyzed 14,898 generated paraphrases for 40 Alexa Skills. Our results show that VUI-UPSET generates more bug-revealing paraphrases than the two baselines with, however, ChatGPT being the approach generating the highest percentage of correct paraphrases. We also tried to use the generated paraphrases to improve the skills. We tried to include in the voice interaction models of the skills (i) only the bug-revealing paraphrases, (ii) all the valid paraphrases. We observed that including only bug-revealing paraphrases is sometimes not sufficient to make all the tests pass. [ABSTRACT FROM AUTHOR]

In today's rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application's code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications. [ABSTRACT FROM AUTHOR]

Summary Natural language inference (NLI) is a task to infer the relationship between the premise and hypothesis sentences, whose models have essential applications in the many natural language processing (NLP) fields, for example, machine reading comprehension and recognizing textual entailment. Due to the data‐driven programming paradigm, bugs inevitably occur in NLI models during the application process, which calls for novel automatic testing techniques to deal with NLI testing challenges. The main difficulty in achieving automatic testing for NLI models is the oracle problem; that is, it may be too expensive to label NLI model inputs manually and hence be too challenging to verify the correctness of model outputs. To tackle the oracle problem, this study proposes a novel automatic testing method hybrid mutation driven testing (HMT), which extends the mutation idea applied in other NLP domains successfully. Specifically, as there are two sets of sentences, that is, premise and hypothesis, to be mutated, we propose four mutation operators to achieve the hybrid mutation strategy, which mutate the premise and the hypothesis sentences jointly or individually. We assume that the mutation would not affect the outputs; that is, if the original and mutated outputs are inconsistent, inconsistency bugs could be detected without knowing the true labels. To evaluate our method HMT, we conduct experiments on two widely used datasets with two advanced models and generate more than 520,000 mutations by applying our mutation operators. Our experimental results show that (a) our method, HMT, can effectively generate mutated testing samples, (b) our method can effectively trigger the inconsistency bugs of the NLI models, and (c) all four mutation operators can independently trigger inconsistency bugs. [ABSTRACT FROM AUTHOR]

With the rapid development of information technology, software has become a part of social life. However, for complex software systems, traditional design and testing methods face a series of problems such as low testing coverage and low efficiency. The main purpose of this article is to study the optimization of software design and testing processes using intelligent algorithms, in order to improve the quality and efficiency of software. This article established a real-time optimization and feedback adjustment mechanism during the software design and testing phase, enabling intelligent algorithms to respond promptly to the environment and requirements. Monitoring and control strategies were designed to ensure the algorithm runs stably for a long time. Test method 1 (traditional method) had 100 test cases, a total execution time of 1200 seconds, and an average execution time of 12 seconds per test case; test method 2 (Intelligent Algorithm Optimization) had 100 test cases, a total execution time of 800 seconds, and an average execution time of 8 seconds per test case. The research results of this article play a good promoting role in improving software testing coverage and software structure optimization, and also provide new ideas and methods for the promotion of intelligent algorithms in software engineering. [ABSTRACT FROM AUTHOR]

Test Case Generation (TCG) generates various types of tests, including functional tests, performance tests, security tests, and reliability tests to ensure software quality, while Test Case Prioritization (TCP) prioritizes the generated tests. However, the previous studies had challenges, including resource constraints, detecting crucial requirements, and automating the Test Case (TC) process efficiently. Additionally, the process is costlier and takes a maximum time duration that affects the effective performance. Therefore, an effective framework is proposed to overcome such issues by optimizing TCG and TCP processes effectively. The proposed work starts with the generation of a Unified Modeling Language (UML) diagram from historical project source code, which is then converted into a Comma-Separated Value (CSV) format. Then, the feature extraction is performed on this CSV file, followed by optimal TCG using the Entropy-based Locust Swarm Optimization Algorithm (Ent-LSOA). Additionally, factors are extracted and reduced from the historical project source code using Pearson Correlation Coefficient-Generalized Discriminant Analysis (PCC-GDA). Finally, the optimal TCs and selected factors are prioritized with the highest accuracy and recall of 96.89% and 96.92%, respectively using an Interpolated Multiple Time scale Recurrent Neural Network (IMTRNN). Thus, the proposed work outperformed the existing techniques by providing an efficient solution for TCG and TCP in software testing. [ABSTRACT FROM AUTHOR]

Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness. [ABSTRACT FROM AUTHOR]

Machine translation systems have been widely adopted in our daily life, making life easier and more convenient. Unfortunately, erroneous translations may result in severe consequences, such as financial losses. This requires to improve the accuracy and the reliability of machine translation systems. However, it is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models. To tackle these challenges, we propose a novel metamorphic testing approach by syntactic tree pruning (STP) to validate machine translation systems. Our key insight is that a pruned sentence should have similar crucial semantics compared with the original sentence. Specifically, STP (1) proposes a core semantics-preserving pruning strategy by basic sentence structures and dependency relations on the level of syntactic tree representation, (2) generates source sentence pairs based on the metamorphic relation, and (3) reports suspicious issues whose translations break the consistency property by a bag-of-words model. We further evaluate STP on two state-of-the-art machine translation systems (i.e., Google Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs. The results show that STP accurately finds 5,073 unique erroneous translations in Google Translate and 5,100 unique erroneous translations in Bing Microsoft Translator (400% more than state-of-the-art techniques), with 64.5% and 65.4% precision, respectively. The reported erroneous translations vary in types and more than 90% of them are not found by state-of-the-art techniques. There are 9,393 erroneous translations unique to STP, which is 711.9% more than state-of-the-art techniques. Moreover, STP is quite effective in detecting translation errors for the original sentences with a recall reaching 74.0%, improving state-of-the-art techniques by 55.1% on average. [ABSTRACT FROM AUTHOR]

Software-defined networks (SDN) enable flexible and effective communication systems that are managed by centralized software controllers. However, such a controller can undermine the underlying communication network of an SDN-based system and thus must be carefully tested. When an SDN-based system fails, in order to address such a failure, engineers need to precisely understand the conditions under which it occurs. In this article, we introduce a machine learning-guided fuzzing method, named FuzzSDN, aiming at both (1) generating effective test data leading to failures in SDN-based systems and (2) learning accurate failure-inducing models that characterize conditions under which such system fails. To our knowledge, no existing work simultaneously addresses these two objectives for SDNs. We evaluate FuzzSDN by applying it to systems controlled by two open-source SDN controllers. Furthermore, we compare FuzzSDN with two state-of-the-art methods for fuzzing SDNs and two baselines for learning failure-inducing models. Our results show that (1) compared to the state-of-the-art methods, FuzzSDN generates at least 12 times more failures, within the same time budget, with a controller that is fairly robust to fuzzing and (2) our failure-inducing models have, on average, a precision of 98% and a recall of 86%, significantly outperforming the baselines. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Cryptologic Research (2097-4116) is the property of Editorial Board of Journal of Cryptologic Research and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Test smells, detrimental coding practices that hinder high-quality test code development, pose a significant challenge in software testing and maintenance. Software refactoring, traditionally a powerful approach for addressing code smells and improving software quality without changing functionality, often focuses on production code, leaving test code overlooked. Despite extensive research in test smell refactoring, understanding the efficacy of existing refactoring operations on test code quality remains limited. Investigating real-world developer refactoring practices is crucial to bridge this knowledge gap. In this study, we investigate refactorings performed by developers to address test smells, resulting in a comprehensive catalog of test smells and their corresponding test-specific refactorings. Two test-specific refactorings closely tied to JUnit5 and seven version-agnostic refactorings for various JUnit versions have been identified. While many of these test-specific refactorings are documented in the literature, this analysis unveils new test-specific refactorings aimed at dealing with the "Inappropriate Assertion" test smell. This research provides insights into the challenges faced by developers and prevailing practices for effectively refactoring test code, thereby enhancing software testing and maintenance. [ABSTRACT FROM AUTHOR]

Boundary value analysis (BVA) is a common technique in software testing that uses input values that lie at the boundaries where significant changes in behavior are expected. This approach is widely recognized and used as a natural and effective strategy for testing software. Test coverage is one of the criteria to measure how much the software execution paths are covered by the set of test cases. This paper focuses on evaluating test coverage with respect to BVA by defining a metric called boundary coverage distance (BCD). The BCD metric measures the extent to which a test set covers the boundaries. In addition, based on BCD, we consider the optimal test input generation to minimize BCD under the random testing scheme. We propose three algorithms, each representing a different test input generation strategy, and evaluate their fault detection capabilities through experimental validation. The results indicate that the BCD-based approach has the potential to generate boundary values and improve the effectiveness of software testing. [ABSTRACT FROM AUTHOR]

Pairwise testing is a method for identifying defects through combinatorial analysis. It involves testing all possible combinations of input parameters in pairs within a system, ensuring that each pair is tested at least once. The field of test case generation is highly active in the realm of combinatorial interaction testing. Research in this area is particularly encouraged, as it falls under the category of non-deterministic polynomial-time hardness. A big challenge in this field is the combinatorial explosion problem. It is about finding the best test suite that covers all possible combinations of interaction strength. In this paper, we present the task of discovering a pairwise test set as a search problem and introduce an innovative testing tool referred to as pairwise test case generation in harmony search algorithm with seeding and constraint mechanism (PHOSC). Experimental results show that PHOSC performs better compared to some existing pairwise strategies in terms of test suite size. Additionally, PHOSC provides a comprehensive framework and serves as a research platform for the generation of pairwise test sets employing the harmony search algorithm. It adopts an approach that focuses on one parameter at a time (OPAT) and incorporates seeding and constraint mechanisms at the same time, thereby enhancing the efficiency and effectiveness of the testing process. [ABSTRACT FROM AUTHOR]

With the continuous development of Internet technology, the functional requirements of all kinds of business software are increasing, and their complexity is gradually increasing. The reliability and security of software have attracted more and more attention. Software testing is the key technology for software quality assurance. Due to the frequent changes in requirements and rapid version iteration of modern business software products, manually writing test cases for them will consume a lot of manpower costs, especially in agile development processes where regression testing and other tasks require a large number of repeated use cases. A test case automatic generation model based on genetic algorithm and LightGBM model is proposed using machine learning technology. The innovative contribution lies in: 1 Abstracting the test steps into a directed graph model, the test case data were simplified; 2 The genetic algorithm was used to solve reachable paths in directed graphs, replacing manually generated test paths; 3 Using the LightGBM model to accelerate the convergence speed of the genetic algorithm, the effectiveness of the proposed method was experimentally verified, meeting the testing coverage criteria. This model can reduce the workload of testers, accelerate testing speed, and is of great significance for improving project quality and accelerating project progress. [ABSTRACT FROM AUTHOR]

Industrial robotic systems (IRS) consist of industrial robots that automate industrial processes. They accurately perform repetitive tasks, replacing or assisting with dangerous jobs like assembly in the automotive and chemical industries. Failures in these systems can be catastrophic, so it is important to ensure their quality and safety before using them. One way to do this is by applying a software testing process to find faults before they become failures. However, software testing in industrial robotic systems has some challenges. These include differences in perspectives on software testing from people with diverse backgrounds, coordinating and collaborating with diverse teams, and performing software testing within the complex integration inherent in industrial environments. In traditional systems, a well-known development process uses simple, structured sentences in English to facilitate communication between project team members and business stakeholders. This process is called behavior-driven development (BDD), and one of its pillars is the use of templates to write user stories, scenarios, and automated acceptance tests. We propose a software testing (ST) approach called automated acceptance testing for industrial robotic systems (AAT4IRS) that uses natural language to write the features and scenarios to be tested. We evaluated our ST approach through a proof-of-concept, performing a pick-and-place process and applying mutation testing to measure its effectiveness. The results show that the test suites implemented using AAT4IRS were highly effective, with 79% of the generated mutants detected, thus instilling confidence in the robustness of our approach.