Industrial Evaluation of Test Suite Generation Strategies for Model-Based Testing

We report on a case study on model based testingfor a commercially available telecom software system. A mainpurpose is to investigate how different strategies for test suitegeneration affect quality attributes of the generated test suites, in a realistic industrial environment. We develop a functionalmodel in the form of an extended finite state machine, from whichwe generate test suites using several different (model) coveragecriteria, alongside with randomly and manually generated testsuites. We compare test suites with respect to fault-detectioncapability, incurred (source) code coverage, and test generationand execution time. The system under test is a commerciallyreleased version, not seeded with any faults, implying thatexposed faults are "real" faults that passed previous testing. Wedid not find clear difference between coverage-based and randomtest suites. Test suite generation and execution is performed usingthe tool E RLY M ARSH, developed by the first author.