Search

Deep Neural Networks (DNNs) are used in many critical areas, such as autonomous vehicles, generative AI systems, etc. Therefore, testing DNNs is vital, especially for models used in critical areas. Mutation-based testing is a very successful technique for testing DNNs by mutating their complex structures. Deep Mutation Module was developed to address mutation-based testing and the robustness challenges of DNNs. It analyses the structures of DNNs in detail. It tests models by applying mutation to parameters and structures using its fault library. Testing DNN structures and detecting faults is a highly complex and open-ended challenge. The method proposed in this study applies mutations to DNN parameters to expose faults and weaknesses in the models, thereby testing their robustness. The paper focuses on mutation-based tests of an Reinforce Learning (RL) model developed for electric vehicle routing, a Long Short-Term Memory (LSTM) model developed for prognostic predictions, and a Transformer-based neural network model for electric vehicle routing tasks. The best mutation scores for the LSTM model were measured as 96%, 91.02%, 71.19%, and 68.77%. The test results for the RL model resulted in mutation scores of 93.20%, 72.13%, 77.47%, 79.28%, and 55.74%. The mutation scores of the Transformer model were 75.87%, 76.36%, and 74.93%. These results show that the module can successfully test the targeted models and generate mutants classified as “survived mutants” that outperform the original models. In this way, it provides critical information to researchers to improve the overall performance of the models. Conducting these tests before using them in real-world applications minimizes faults and maximizes model success.

Verification and validation (V&V) are complex processes combining different approaches and incorporating many different methods including many activities. System engineers regularly face the question if their V&V activities lead to better products, and having appropriate criteria at hand for evaluation of safety and cybersecurity of the systems would help to answer such a question. Additionally, when there is a demand to improve the quality of an already managed V&V process, there is a struggle over what criteria to use in order to measure the improvement. This paper presents an extensive set of criteria suitable for safety and cybersecurity evaluation of cyberphysical systems. The evaluation criteria are agreed upon by 60 researchers from 32 academic and industrial organizations jointly working in a large-scale European research project on 13 real-world use cases from the domains of automotive, railway, aerospace, agriculture, healthcare, and industrial robotics.

Due to the fast-paced development of technology, the software has become a crucial aspect of modern life, facilitating the operation and management of hardware devices. Nevertheless, using substandard software can result in severe complications for users, putting human lives at risk. This underscores the significance of error-free and premium-quality software. Verification and validation are essential in ensuring high-quality software development; software testing is integral to this process. Although code coverage is a prevalent method for assessing the efficacy of test suites, it has some limitations. Therefore, mutation testing is proposed as a remedy to tackle these limitations. Furthermore, mutation testing is recognized as a method for directing test case creation and evaluating the effectiveness of test suites. Our proposed method involves autonomously learning mutations from faults in real-world software applications. Firstly, our approach involves extracting bug fixes at the method-level, classifying them according to mutation types, and performing code abstraction. Subsequently, the approach utilizes a deep learning technique based on neural machine translation to develop mutation models. Our method has been trained and assessed using approximately $\sim $ 588k bug fix commits extracted from GitHub. The results of our experimental assessment indicate that our models can forecast mutations resembling resolved bugs in 6% to 35% of instances. The models effectively revert fixed code to its original buggy version, reproducing the original bug and generating various other buggy codes with up to 94% accuracy. More than 96% of the generated mutants also demonstrate lexical and syntactic accuracy.

The “Tailored Mutation-based Software Fault Injection Tool (IM-FIT)” is a versatile tool developed for evaluating the robustness of safety-critical system software, specifically Python-based and ROS-based systems. IM-FIT employs mutation-based tests to simulate different conditions under which the system operates. This paper introduces IM-FIT, a tool that contributes significantly to the field of safety-critical systems by creating comprehensive fault libraries for ROS-based and Python-based systems and implementing mutation-based testing methods. IM-FIT leverages a customizable RegEx and AST-based software structure, making it a valuable asset for system verification and validation (V&V). The tool not only enhances system robustness but also optimizes the V&V process by reducing effort, cost, and time. This paper outlines the key features, applications, and benefits of IM-FIT in the context of safety-critical system software.

Günümüzde, gelişen görüntü işleme teknikleri ile birlikte kamera tabanlı robotik inceleme sistemleri oldukça popülerlik kazanmıştır. Bu tür sistemler gıdadan, askeriyeye birçok sektörde yoğun olarak kullanılmaktadır. Bu sistemler geliştirilirken gerekli olan doğrulama ve onaylama süreçleri oldukça uzun ve maliyetli olmaktadır. Bu çalışma, kamera tabanlı endüstriyel robotik sistemler üzerinde doğrulama ve onaylama faaliyetlerini gerçekleştirmek ve iyileştirmek amacıyla geliştirilmiştir. RGB ve TOF kameralara farklı türlerde (Open, Close, Dilation, Erosion, Gradient, Motionblur, Tuz&Biber, Gaussian ve Poisson) hata enjeksiyon yöntemleri kullanılmasını mümkün Kamera Hata Enjeksiyon Aracı (CamFITool) ile gerçekleştirilmiş testler ve sonuçlar açıklanmıştır. Yapılan çalışma, VALU3S projesi kapsamında, OTOKAR’ın ROKOS robotik sistemine, CamFITool ile gerçek ortamdan alınmış kamera görüntülerinden oluşan kitaplıklara, çeşitli konfigürasyonlarda hatalar enjekte edilip, bu enjeksiyonun sisteme etkilerinin incelenmesine odaklanmıştır. Bu kapsamda 49 farklı test komfigürasyonunda hata enjeksiyonu gerçekleştirilmiştir. Sonuç olarak, kamera tabanlı endüstriyel robotik sistemlerin daha güvenli ve stabil çalışmalarının sağlanması için, bu sistemlerin hataya dayanıklı olup olmadıklarını test eden açık kaynaklı bir hata enjeksiyon aracı olan CamFITool önerilmiştir.

Fossil studies are of great importance in order to observe the change of living species over the years, to make inferences by using the information provided by the observed species, and to understand the developing and changing structure of the world we live in over the years. However, the examination and interpretation of fossil specimens is a complex and long process. Artificial intelligence studies have begun to be applied to this field in order to facilitate the working methods of paleontologists. The detection and classification of fossil specimens with the aid of computers simplifies this process as much as possible compared to manual classification processes and reduces foreign dependency for fossil assemblages for which paleontologists are not experts. To achieve this, 9 benthic foraminiferal species and non-foraminiferal sample photographs from a selected dataset were used. In this study, a new method developed for the classification of benthic foraminifera using deep convolutional neural networks, reaching higher accuracy than the results in the literature, is presented. With this method, at least 70% accuracy rates were achieved in the test results of the trained system. This study, which reached high accuracy rates with a new method, has created a successful development for the branch of paleontology in the use of artificial intelligence in microfossil identification.

Nowadays, camera-based perception is most popular topic in robotic systems. Verification of camera-based perception systems are crucial and difficult with current tools and methods. This study proposes Camera Fault Injection Tool (CamFITool), which enables different kind of fault injection methods to RGB and TOF cameras in order to perform verification and validation activities on robotic systems. Besides, Fault Injected Image Database which is created by CamFITool is introduced. In addition, the study guides to readers to create new datasets by injecting faults into existing image libraries or camera streams with CamFITool. As a result, CamFITool, an open-source fault injection tool, which is a critical tool for assessing of fault tolerant systems’ safety and security, is proposed. Also, a fault injected image dataset created by CamFITool for verification of camera-based perception studies in robotic systems is given.