Search

This article sets out to improve upon the current state of design principles and implementation guidance for software safety and security by focusing on the developer's ecosystem to prevent software defects. The article emphasizes a continuous assurance method focusing on the implementation of safe coding and secure design for all applications and tools used by software developers.

The article explores the potential for language-based testing bots to conduct software testing. First, the authors detail the requirements of a software testing bot including input generation as well as output evaluation. Then, an investigation of the incorporation of grammars with constraints in order to satisfy both syntax and semantics of inputs and outputs are presented utilizing the authors' open source tool ILSa (Input Specification Language).

The article focuses on the improvement of test effectiveness and quality of test data in deep learning systems. Topics of discussion include the combination of mutation testing using DeepMutation and differential testing using DeepXplore, iterations of models using data from the MNIST database, and the discovery of errors in deep neural network (DNN) models.

This article discusses the concept of fuzzing as a learning process, using Shannon's entropy to quantify the efficiency of a fuzzer in discovering new behaviors of a program. The authors propose an entropy-based power schedule called "Entropic" for greybox fuzzing, assigning more energy to seeds that reveal more information about a program's behaviors. This approach is implemented in the popular greybox fuzzer LibFuzzer and has been integrated into Google and Microsoft's fuzzing platforms. The paper highlights that the efficiency of a fuzzer is determined by the average information each generated input reveals about a program's behaviors. The authors conducted experiments with over 250 open-source programs, demonstrating a substantial improvement in efficiency and confirming their hypothesis that an efficient fuzzer maximizes information.

The article presents information on software fuzzing, a process used for automated bug and vulnerability testing. Several papers are cited and discussed by Stefan Nagy, an assistant professor at the University of Utah. Software fuzzing uses new and unexpected inputs which are systematically generated in order to test programs. The cited papers provide information on topics such as specific classes of bugs and finding bugs in compilers, demonstrating the advantage of combining traditional testing methods with innovative techniques and machine learning in order to identify bugs in complex software systems.

The article examines the design of software for quantum computing. The application of quantum computing for drug discovery and vaccine development is described along with efforts at European research centers to build ion-trapped and 100-qubit supercomputer and alleged idiosyncrasies associated with testing and debugging quantum software.

The article discusses the significance of fuzzing as an effective method for testing software. Fuzzing involves feeding random or invalid test data to programs to discover potential crashes or vulnerabilities, often at scale. Fuzzing is categorized into black-box, grey-box, and white-box approaches, each with varying levels of information about the system under test. Unlike traditional test generation papers that focus on theory and code coverage, fuzzing emphasizes practical applications and bug discovery in real systems. The article introduces a a paper that presents a novel twist to grey-box fuzzing, proposing a power schedule that selects seeds for mutation based on their potential to reveal new program behavior. This approach is underpinned by information theory and measures entropy as an efficiency metric for fuzzers. The article highlights the potential of fuzzing to bridge the gap between traditional test generation and practical testing methods, offering a valuable contribution to the field of software testing.

The article discusses the use of metamorphic testing to test driverless cars, specifically their untestable software, detecting fatal errors in autonomous vehicles' onboard computer systems. Topics include metamorphic testing's cost-effective ability to address the oracle problem and the automated test-case-generation problem, and the use of both LiDAR obstacle perception to determine an object's shape and position and radar to estimate the velocity of moving obstacles in autonomous vehicles.

The article discusses the authors' views about the need for the distributed systems research community to improve the state of the art in fault tolerance testing as of 2018, and it mentions how ordinary users need the tools to automate the selection of custom-tailored faults to inject. Cloud software applications, Chaos Engineering, and computer failures are assessed, along with the Jepsen fault-injection testing technique and the superusers that are known as Chaos Engineers.

The article discusses the use of open source software-defined networking (SDN) in enterprise networks. Topics include the use of the Faucet network by the organization Open Networking Foundation, the testing and deployment of networking software, and the role of a control-plane network (CPN) in network security.

The article discusses the use of satisfiability modulo theory solvers in formula-based software debugging as form of computer-assisted debugging. Topics include the use of symbolic techniques to extract candidate specifications, the use of an Address Resolution Protocol (ARP) implementation, and the use of interpolants to find error root-causes.

The article on the Veritesting dynamic symbolic execution and path merging or static symbolic execution testing technique for automating software testing as a method of solving the "path explosion" problem is presented. The bug identification rate of MergePoint software with veritesting is compared to MergePoint without veritesting using Debian binaries and verification-based algorithms are presented as a technique to enhance symbolic execution.

The article discusses the verification and validation of distributed systems. Particular focus is given to partial failure and asynchrony. Details on formal specification languages such as TLA+ and Coq, and on the formal verification method known as model checking, are presented. Informal methods such as monitoring, canary tests, and unit and integration tests are also discussed, along with random model checkers and fault-injection testing.

The article focuses on debugging high-performance computing (HPC) applications. It is noted that these applications can run long numerical simulations which model the real world. The numerical correctness and software reliability of these applications is very important for scientists. Debugging parallel programs is much more difficult compared to debugging serial programs. According to the authors, dynamic analysis tools and algorithms which match an application's input and number of compute nodes allow programmers identify the root cause of bugs. INSET: Real-World Debugging Case.

The article discusses the importance of finding security-critical computer bugs, which are software errors, flaws, or failures in a computer program or system, before they are exploited by computer hackers and cyberterrorists. Topics include the prevalence of computer bugs, the automatic exploit generation (AEG) research program, which challenges researchers to both automatically find bugs and generate working exploits, the author's verification approach to AEG, and the advantages of a verification-based approach, such as the guarantee of an automatically generated exploit that provides proof that the reported bug is security-critical. INSET: History of AEG.

The article presents an overview of modern and classical symbolic execution techniques which are used in software testing to explore program paths and to find errors in complex software applications. The article notes the input values for symbolic execution are symbolic values, the values of a computer program's variables are symbolic expressions, and the computed output values represent a function of the input symbolic values. The challenges of symbolic execution such as path explosion, constraint solving, and memory modeling are also mentioned. Topics include execution-generated testing (EGT) and directed automated random testing (DART).

The article discusses resiliency in computer systems and Internet infrastructure engineering, arguing that systems should be tested before they reach production stages and focusing on "Game Day" fault testing processes employed at the Internet merchant Etsy. Topics include systems integration and degradation, fault integration as a business continuity plan (BCP), risk, and the potential decrease of anxiety that can result from testing a system before its launch. The process of testing an Internet payment system, including its security and processing components, is described.

Virtual testbeds model them by seamlessly integrating physical, simulated, and emulated sensor nodes and radios in real time. INSET: Physical Testbeds vs. Simulation vs. Emulation. [ABSTRACT FROM AUTHOR]

The article discusses Satisfiability Modulo Theories (SMT) solvers, which are said to be the core of many tools for program analysis, testing, and verification. The topics of constraint satisfaction, propositional satisfiability, and Boolean variables are addressed. SMT solvers are able to check the satisfiability of logical formulas on a scale orders of magnitude beyond custom ad hoc solvers. Their use in software engineering tasks such as case analysis, static analysis, and modeling is also described.

Windows Error Reporting (WER) is a distributed system that automates the processing of error reports coming from an installed base of a billion machines. WER has collected billions of error reports in 10 years of operation. It collects error data automatically and classifies errors into buckets, which are used to prioritize developer effort and report fixes to users. WER uses a progressive approach to data collection, which minimizes overhead for most reports yet allows developers to collect detailed information when needed. WER takes advantage of its scale to use error statistics as a tool in debugging; this allows developers to isolate bugs that cannot be found at smaller scale. WER has been designed for efficient operation at large scale: one pair of database servers records all the errors that occur on all Windows computers worldwide. [ABSTRACT FROM AUTHOR]

The article discusses automated test methods for finding bugs in computer programs which negatively affect usability. Several ways in which automated tests can help identify problems in web-based applications are described. Automated tests can provide information about the software as it is being developed, helping to quickly identify potential problems. Used as an adjunct to traditional human usability testing, it can offer software developers fast and consistent feedback. Examples of automated testing methods are described.

The article presents a discussion of methods for testing complex computer programs by modeling subsets of their architecture. It is noted that this is a useful method for detecting problems of scaling before extensive resources are committed to systems development, and is also helpful in terms of debugging complicated systems. Examples are provided involving Windows software by Microsoft. The advantages of constructing dummy modules, which can be used to duplicate the performance of a sub-system under various conditions without generating unnecessary data, are described.

The article, the second of a series which began in the September 2010 issue of the journal, analyzes principles of software design. It focuses on high-performance software, and heuristics which can help identify methods of improving performance. Specific factors discussed include risk minimization, efficiency of execution, and queuing and coherency delays caused by increased system loads. The concept of a knee is explained, as a section of a curve where response times and throughput are optimally balanced. The pernicious aspects of surrogate measures and the benefits of performance instrumentation are also discussed.

The article presents a discussion of improvements in computing which can be obtained by combining performance evaluation methods with model checking techniques. The idea of jointly working on quantitative verification and performance is described as a useful way of combining complementary functions. An overview of the history of the mathematical concepts involved is presented, from the first half of the 20th century to the early 21st century. Rules of correctness proofs, queuing and temporal logic are discussed.

Static program analysis techniques cannot know certain values, such as the value of user input or network state, at analysis time. While such unknown values need to be treated as nondeterministic choices made by the program's execution environment, it is still possible to glean very useful information about how such statically unknown values may or must influence computation. We give a method for integrating such nondeterministic choices with an expressive static analysis. Interestingly, we cannot solve the resulting recursive constraints directly, but we give an exact method for answering all may and must queries. We show experimentally that the resulting solved forms are concise in practice, enabling us to apply the technique to very large programs, including an entire operating system. [ABSTRACT FROM AUTHOR]

The article discusses model driven engineering techniques in computer software design, and the difficulties of testing automatic model transformations. A fault in one model system can propagate into another during this process, but the changes induced by the transformation often make it difficult to systematically isolate the source of the problem. An example is provided which involves flattening hierarchies and meta-models. The difficulties which arise from the translation of complex data between heterogeneous computer languages is discussed. Potential solutions to some of these problems are also presented, involving the construction of oracles, meta-heuristics, and the use of white-box testing.

The trend towards processors with more and more parallel cores is increasing the need for software that can take advantage of parallelism. The most widespread method for writing parallel software is to use explicit threads. Writing correct multithreaded programs, however, has proven to be quite challenging in practice. The key difficulty is non-determinism. The threads of a parallel application may be interleaved non-deterministically during execution. In a buggy program, nondeterministic scheduling can lead to nondeterministic results—where some interleavings produce the correct result while others do not. We propose an assertion framework for specifying that regions of a parallel program behave deterministically despite nondeterministic thread interleaving. Our framework allows programmers to write assertions involving pairs of program states arising from different parallel schedules. We describe an implementation of our deterministic assertions as a library for Java, and evaluate the utility of our specifications on a number of parallel Java benchmarks. We found specifying deterministic behavior to be quite simple using our assertions. Further, in experiments with our assertions, we were able to identify two races as true parallelism errors that lead to incorrect nondeterministic behavior. These races were distinguished from a number of benign races in the benchmarks. [ABSTRACT FROM AUTHOR]

The article discusses the number of personnel needed to test the usability of software products. The authors discusses different methods of usability evaluation, including think aloud (TA), in which the reactions of test users to products are recorded in a laboratory, heuristic evaluation (HE), in which test users evaluate user interfaces based on established usability principles, and cognitive walkthrough (CW), in which the steps for the performance of a task are examined for usability problems. The use of meta-analysis and regression analysis to determine sample sizes for usability studies is discussed.

The article discusses the use of static analysis to identify bugs in computer software. Specifically, the authors describe what they learned in the course of commercializing their Coverity Static Analysis bug-detecting software for industrial use. Among topics covered are the unwillingness of some firms to subject all of their code to the author's software, conflicts between their product and other software programs, the necessity of being able to parse code that is to be checked, and firms' occasional indifference to bugs that are discovered.

The authors discuss their formulation of the process of model checking and its historical background. Mathematical logic is used in model checking to confirm that computer program behavior conforms to the correct specifications. While the Floyd-Hoare style deductive verification method was used since the 1960s, it was not useful for large programs. Model checking was devised using temporal logic (TL) that describes change over time. Debugging, correctness, expressiveness, efficiency and accounting for architectures and their features in model checking are detailed.

The article presents questions and answers related to computer science. Topics include the costs or benefits of creating a new wireless Internet transport protocol for a business network rather than relying on existing Internet protocols such as transmission control protocol (TCP) or user datagram protocol (UDP); and a discussion of how to test software.

Presents the author's opinions on methods that are used for testing compilers. Information on the "Orthogonal Latin Squares" method for testing compilers; Discussion on the development of test data for testing compilers and other computer programs; Views on compiler testing through experimental design; Information about pairs or sets of orthogonal Latin squares.

This paper describes the design and implementation of a program testing assistant which aids a programmer in the definition, execution, and modification of test cases during incremental program development. The testing assistant helps in the interactive definition of test cases and executes them automatically when appropriate. It modifies test cases to preserve their usefulness when the program they test undergoes certain types of design Changes. The testing assistant acts as a fully integrated part of the programming environment and cooperates with existing programming tools such as a display editor, compiler, interpreter, and debugger. [ABSTRACT FROM AUTHOR]

Describes the symbolic execution of programs. Provision of symbols representing arbitrary values instead of supplying the normal inputs to a program; Symbolic execution of conditional branch type statements; Description of the EFFIGY system which provides symbolic execution for program testing and debugging; Execution of programs written in a simple PL/I style programming language.

The article considers the merits of integrated development environments (IDEs) in computer software development. A brief history of software development and computer programming is also provided. According to the author, IDEs enable the development and testing of complex systems in a matter of days or hours. He explains the difference between experimenting and thinking in relation to systems development, as well as the effect of faster development tools on the cost of programming tasks. The goal of systems development is also discussed.

Forced errors focus attention on neural network quirks. [ABSTRACT FROM AUTHOR]

The author discusses the difference between finding computer errors or software testing and ensuring that a computer program or software product works correctly. Topics include the challenge of determining when to stop testing a program as there is no way of verifying all bugs have been found, and the importance of giving testers a goal rather than an unlimited amount of time to continue testing a program.

The article discusses the importance of motivation for pursuing a career in computer science, focusing on efforts to increase the diversity of computer science students. A program called Glitch Game Testers, run by the authors, is described, which recruits predominantly African-American male high school students to test pre-release versions of video games. The effectiveness of this approach for increasing interest in computer science among the target population is described.

The author presents his views on the use of axioms for computer programming. He reflects on the ways in which the field has changed, on the occasion of the 40th anniversary of his first published paper on the subject. He notes some of the ways in which software verification using axioms can help to guard against malware, and observes that software testing is typically more of a test of the programmer than a test of the code.

The article discusses the reliability of computer hardware and software, examining the manner in which developers can create more reliable designs. A number of high-visibility design flaws are presented in the article including a 1994 circuitry error in the Pentium I microprocessor made by Intel Corp. Topics include the degree to which people depend on computers, the costs of faulty software on the economy, and flaws that arise in higher-level design process. Also discussed are programs debugged by random testing and model testing.

Presents several letters to the editor. Clarification of the Pascal Standard developed as a joint project by the X3 committee; Analysis of temporary benefits of unionism and other similar forms of coercion; Discussion of error messages as the neglected area of the man/machine interface.

The article presents an answer to a question about convincing a Web development team that an implementation has problems in relation to the software design review process.

The article discusses software testing, looking at how testing can be used as a knowledge acquisition process instead of as a quality assurance activity. The author discusses how defects which are found during testing can be constructively applied to software and examines the concept of orders of ignorance. Information is provided on managing test complexity, what constitutes a successful test, and the transmission of information.

The article presents blogs which discuss computer science education and computer software testing.

The article presents general advice on debugging a network problem by using packet-capture programs such as tcpdump. Capturing data is essential, especially for bugs that may involve timing issues. Network traffic should be recorded as soon as possible once a problem is identified. Due to privacy concerns it is a good idea to obtain permission first before capturing data. Also, try to record only as much data as is necessary, and separate it into portions no larger than one megabyte apiece. Never record captured data on a network's file system.

An article on the Veritesting dynamic symbolic execution and path merging or static symbolic execution testing technique for automating software testing as a method of solving the "path explosion" problem is introduced.

In this article the author discusses computer science, examining how to manage computer software projects that are behind schedule and the manner in which computer code is patched, compiled, and tested. When a software development project continues to be extended, or there is a loss of control over the project, one should consider stopping the project, the author indicates.

An introduction is presented to the article "Reasoning About the Unknown in Static Analysis," by Isil Dillig, Thomas Dillig, and Alex Aiken.

The article presents several abstracts from the Association of Computing Machinery's Digital Library including "Choc'late: A Framework for Specification-based Testing" by Pak-Lok Poon, Sau-Fun Tang, T.H. Tse, and T.Y. Chen, "Designing for Collective Intelligence" by Dawn Gregg, and "Individual Resistance to IT Innovations" Rhoda C. Joseph.