Search

As artificial intelligence (AI) is becoming increasingly integrated into critical infrastructures, it brings about both transformative benefits and unprecedented risks. AI has the potential to revolutionize the efficiency, reliability, and responsiveness of essential services, but it can also offer these benefits along with the vulnerability to a growing array of sophisticated adversarial attacks. This paper explores the evolving landscape of adversarial threats to AI systems, highlighting the potential of nation-state actors to exploit these vulnerabilities for geopolitical gains. A range of adversarial techniques is examined, including dataset poisoning, model stealing, and privacy inference attacks, and their potential impact on sectors such as energy, transportation, healthcare, and water management is assessed. The consequences of successful attacks are substantial, encompassing economic disruption, public safety risks, national security implications, and the erosion of public trust. Given the escalating sophistication of these threats, this paper proposes a comprehensive security framework that includes robust incident response protocols, specialized training, the development of a collaborative ecosystem, and the continuous evaluation of AI systems. The findings of this study 11 underscore the critical need for a proactive approach to AI security in order to safeguard the future of critical infrastructures in an increasingly AI-driven world.

Adversarial attacks pose a significant threat to real-world applications based on deep neural networks (DNNs), especially in security-critical applications. Research has shown that adversarial examples (AEs) generated on a surrogate model can also succeed on a target model, which is known as transferability. Feature-level transfer-based attacks improve the transferability of AEs by disrupting intermediate features. They target the intermediate layer of the model and use feature importance metrics to find these features. However, current methods overfit feature importance metrics to surrogate models, which results in poor sharing of the importance metrics across models and insufficient destruction of deep features. This work demonstrates the trade-off between feature importance metrics and feature corruption generalization, and categorizes feature destructive causes of misclassification. This work proposes a generative framework named PTNAA to guide the destruction of deep features across models, thus improving the transferability of AEs. Specifically, the method introduces path methods into integrated gradients. It selects path functions using only a priori knowledge and approximates neuron attribution using nonuniform sampling. In addition, it measures neurons based on the attribution results and performs feature-level attacks to remove inherent features of the image. Extensive experiments demonstrate the effectiveness of the proposed method. The code is available at https://github.com/lounwb/PTNAA. [ABSTRACT FROM AUTHOR]

Copyright of Romanian Journal of Information Technology & Automatic Control / Revista Română de Informatică și Automatică is the property of National Institute for Research & Development in Informatics - ICI Bucharest and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Recent studies show that neural networks are vulnerable to backdoor attacks, in which compromised networks behave normally for clean inputs but make mistakes when a pre-defined trigger appears. Although prior studies have designed various invisible triggers to avoid causing visual anomalies, they cannot evade some trigger detectors. In this paper, we consider the stealthiness of backdoor attacks from input space and feature representation space. We propose a novel backdoor attack named sparse backdoor attack, and investigate the minimum required trigger to induce the well-trained networks to make incorrect results. A U-net-based generator is employed to create triggers for each clean image. Considering the stealthiness of the trigger, we restrict the elements of the trigger between −1 and 1. In the aspect of the feature representation domain, we adopt an entanglement cost function to minimize the gap between feature representations of benign and malicious inputs. The inseparability of benign and malicious feature representations contributes to the stealthiness of our attack against various model diagnosis-based defences. We validate the effectiveness and generalization of our method by conducting extensive experiments on multiple datasets and networks. [ABSTRACT FROM AUTHOR]

Many educators and professionals in different industries may need to become more familiar with the basic concepts of artificial intelligence (AI) and generative artificial intelligence (Gen-AI). Therefore, this paper aims to introduce some of the basic concepts of AI and Gen-AI. The approach of this explanatory paper is first to introduce some of the underlying concepts, such as artificial intelligence, machine learning, deep learning, artificial neural networks, and large language models (LLMs), that would allow the reader to better understand generative AI. The paper also discusses some of the applications and implications of generative AI on businesses and education, followed by the current challenges associated with generative AI. [ABSTRACT FROM AUTHOR]

Various studies have been conducted on Multi-Agent Reinforcement Learning (MARL) to control multiple agents to drive effectively and safely in a simulation, demonstrating the applicability of MARL in autonomous driving. However, several studies have indicated that MARL is vulnerable to poisoning attacks. This study proposes a 'locality-based action-poisoning attack' against MARL-based continuous control systems. Each bird in a flock interacts with its neighbors to generate the collective behavior, which is implemented through rules in the Reynolds' flocking algorithm, where each individual maintains an appropriate distance from its neighbors and moves in a similar direction. We use this concept to propose an action-poisoning attack, based on the hypothesis that if an agent is performing significantly different behaviors from neighboring agents, it can disturb the driving stability of the entirety of the agents. We demonstrate that when a MARL-based continuous control system is trained in an environment where a single target agent performs an action that violates Reynolds' rules, the driving performance of all victim agents decreases, and the model can converge to a suboptimal policy. The proposed attack method can disrupt the training performance of the victim model by up to 97% compared to the original model in certain setting, when the attacker is allowed black-box access. [ABSTRACT FROM AUTHOR]

Deep neural network (DNN) -based object detectors have achieved remarkable success, but recent research has revealed their vulnerability to backdoor attacks. The attacks cause the poisoned model to output results normally on benign images, but outputs results specified by the attacker on images inserted with a trigger. Although backdoor attacks have been extensively investigated on image classification tasks, their exploration in object detection tasks remains limited. With the increasing application of object detectors in safety-sensitive fields such as autonomous driving, backdoor attacks on object detection tasks may have serious consequences. Currently, strategies for object disappearance attack scenarios exhibit certain limitations. First, these strategies typically exhibit a one-to-one correspondence, implying that the insertion of one trigger can only result in the disappearance of one object. Second, these strategies typically necessitate the attacker’s knowledge of the object’s precise location information to achieve its disappearance, thereby rendering real-time insertion of triggers unfeasible. Finally, these strategies exhibit diminished attack success rates when applied to two-stage detectors. The paper presents a global object disappearance attack scenario and proposes a simple, covert, and highly effective attack strategy. Experimental evaluations are conducted on four widely-used object detection models (Yolov5s, Yolov8s, Faster R-CNN, and Libra R-CNN) using two benchmark datasets (PASCAL VOC $07+12$ and MS COCO2017) to validate the effectiveness of the proposed strategy. The results demonstrate that the success rate of this attack strategy exceeds 96%, while the poison rate is only 10%.

Education in cybersecurity is crucial in the current society, and it will be extended into the artificial intelligence (AI) area, called AI security, in the near future. Although many video games for education in cybersecurity have been designed, we have two problems for education in AI security: a helpful design of a video game for users to learn cybersecurity is still unclear, and there is no game for AI security, to the best of our knowledge. In this paper, we design a video game for education in AI security, REN-A.I., to address the above problems. In designing REN-A.I., we built some hypotheses: simulating damage caused by attacks on AI and the effectiveness of their countermeasures through a video game helps a user to improve awareness of AI security with the episodic memory of the user itself. We focus on game scenarios and game functionalities to learn AI security with episodic memory in accordance with the above hypothesis. We conducted a questionnaire survey with 48 users to evaluate REN-A.I.. As a result, we confirm that both game scenarios and game functionalities are effective for learning with episodic memory. Specifically, 74% of users consider game scenarios effective, and 81% of users consider game functionalities effective. Our survey results have revealed two suggestions for beneficial design aspects in video games for education in cybersecurity. In particular, users who read game scenarios in REN-A.I. can learn AI security by the game more effectively than the other users. Furthermore, the functionality for accuracy deterioration due to attacks in REN-A.I. is effective even for users who do not read the game scenario. REN-A.I. is publicly available (https://www-infosec.ist.osaka-u.ac.jp/software/ren-ai/REN-AI(EN).html).

Deep neural networks (DNNs) are vulnerable to adversarial examples that fool the models with tiny perturbations. Although adversarial attacks have achieved incredible attack success rates in the white-box setting, most existing adversaries often exhibit weak transferability in the black-box setting, especially for models with defense mechanisms. In this work, we reveal the cross-model channel redundancy and channel invariance of DNNs and thus propose two channel-augmented methods to improve the transferability of adversarial examples, namely, the channel transformation (CT) method and the channel-invariant Patch (CIP) method. Specifically, channel transformation shuffles and rewrites channels to enhance cross-model feature redundancy in convolution, and channel-invariant patches distinctly weaken different channels to achieve loss-preserving transformation. We compute the aggregated gradients of the transformed dataset to create adversaries with higher transferability. In addition, the two proposed methods can be naturally combined with each other and with almost all other gradient-based methods to further improve performance. Empirical results on the ImageNet dataset demonstrate that our attack methods exhibit higher transferability and achieve higher attack success rates than state-of-the-art gradient-based attacks. Specifically, our attack improves the average attack success rate from 86.9% to 91.0% on normally trained models and from 44.6% to 68.3% on adversarially trained models. [ABSTRACT FROM AUTHOR]

Abstract Intrusion detection systems are increasingly using machine learning. While machine learning has shown excellent performance in identifying malicious traffic, it may increase the risk of privacy leakage. This paper focuses on implementing a model stealing attack on intrusion detection systems. Existing model stealing attacks are hard to implement in practical network environments, as they either need private data of the victim dataset or frequent access to the victim model. In this paper, we propose a novel solution called Fast Model Stealing Attack (FMSA) to address the problem in the field of model stealing attacks. We also highlight the risks of using ML-NIDS in network security. First, meta-learning frameworks are introduced into the model stealing algorithm to clone the victim model in a black-box state. Then, the number of accesses to the target model is used as an optimization term, resulting in minimal queries to achieve model stealing. Finally, adversarial training is used to simulate the data distribution of the target model and achieve the recovery of privacy data. Through experiments on multiple public datasets, compared to existing state-of-the-art algorithms, FMSA reduces the number of accesses to the target model and improves the accuracy of the clone model on the test dataset to 88.9% and the similarity with the target model to 90.1%. We can demonstrate the successful execution of model stealing attacks on the ML-NIDS system even with protective measures in place to limit the number of anomalous queries.

Benefiting from the high-speed transmission and super-low latency, the Fifth Generation (5G) networks are playing an important role in contemporary society. The accessibility and friendly experience provided by 5G results in the generation of massive data, which are recklessly transmitted in various forms and in turn, promote the development of big data and intelligent decision support systems (DSS). Although AI (Artificial Intelligence) can boost DSS to obtain high recognition performance on large-scale data, an adversarial sample generated by deliberately adding subtle noise to a clear sample will cause AI models to give false output with high confidence, which increases concerns about AI. It is necessary to enhance its interpretability and security when adopting AI in areas where decision-making is crucial. In this paper, we study the challenges posed by the next-generation DSS in the era of 5G and big data. To build trust in AI, the saliency map is adopted as a visualization method to reveal the vulnerability of neural networks. The visualization method is further taken to identify imperceptible adversarial samples and reasons for the misclassification of high-accuracy models. Finally, we conduct extensive experiments on large-scale datasets to verify the effectiveness of the visualization method in enhancing AI security for 5G-enabled DSS. [ABSTRACT FROM AUTHOR]

As an essential research task in artificial intelligence (AI), the estimation of 3D human poses has important application value in virtual reality, medical diagnosis, athlete training and other fields. However, human pose recovery and retargeting require the acquisition of detailed visual data containing private information, which has led to increasing concerns about user privacy and security. Therefore, we build a lightweight framework, called Human Motion Parameters Prediction (HMPP), which can infer the 3D mesh and 3D skeletal joint points of the human body while protecting the privacy of the user. The proposed method successfully reduces or suppresses privacy attributes while ensuring important features to perform human pose estimation. The 2D and 3D joints are used for supervision to improve the interpretability of the model at each stage. In addition, the prediction of the camera's internal parameters is added so that the model can be augmented with projection supervision, thereby using more 2D datasets for training and improving the generalization ability of the model. Finally, the predicted motion parameters are used for 3D reconstruction and motion retargeting. Experiments show that our approach can achieve excellent evaluation results on multiple datasets and avoid inadvertently compromising private and sensitive data. [ABSTRACT FROM AUTHOR]

The rapid expansion of artificial intelligence poses significant challenges in terms of data security and privacy. This article proposes a comprehensive approach to develop a framework to address these issues. First, previous research on security and privacy in artificial intelligence is reviewed, highlighting the advances and existing limitations. Likewise, open research areas and gaps that require attention to improve current frameworks are identified. Regarding the development of the framework, data protection in artificial intelligence is addressed, explaining the importance of safeguarding the data used in artificial intelligence models and describing policies and practices to guarantee their security, as well as approaches to preserve the integrity of said data. In addition, the security of artificial intelligence is examined, analyzing the vulnerabilities and risks present in artificial intelligence systems and presenting examples of potential attacks and malicious manipulations, together with security frameworks to mitigate these risks. Similarly, the ethical and regulatory framework relevant to security and privacy in artificial intelligence is considered, offering an overview of existing regulations and guidelines. [ABSTRACT FROM AUTHOR]

The traffic classification model is vulnerable to the interference of data pollution in the update process and reduces the performance of the model. The existing defense methods based on data cleaning need to rely on expert experience and manual screening, and cannot effectively deal with the poison attack constructed by using unknown distributed samples. In view of the above problems, inspired by out-of-distribution detection and discrimination active learning, the model design a data poisoning prevention method based on sample distribution characteristics, and the binary classification discriminator can screen out the known and unknown distribution samples in each new round of samples. For the new known distribution samples, the concordant rate of prediction and annotation can evaluate the data quality of the new samples and determine whether to update the model. For the new unknown distribution samples, the small sample sampling based on the labeling accuracy can evaluate the sample availability. The experimental results show that this method can guarantee the accuracy of the model while resisting the data poisoning attack, and effectively identify the data poisoning attack constructed by using unknown distribution samples. [ABSTRACT FROM AUTHOR]

In recent years, the number of parameters of deep neural networks (DNNs) has been increasing rapidly. The training of DNNs is typically computation-intensive. As a result, many users leverage cloud computing and outsource their training procedures. Outsourcing computation results in a potential risk called backdoor attack, in which a well trained DNN would perform abnormally on inputs with a certain trigger. Backdoor attacks can also be classified as attacks that exploit fake images. However, most backdoor attacks design a uniform trigger for all images, which can be easily detected and removed. In this paper, we propose a novel adaptive backdoor attack. We overcome this defect and design a generator to assign a unique trigger for each image depending on its texture. To achieve this goal, we use a texture complexity metric to create a special mask for each image, which forces the trigger to be embedded into the rich texture regions. The trigger is distributed in texture regions, which makes it invisible to humans. Besides the stealthiness of triggers, we limit the range of modification of backdoor models to evade detection. Experiments show that our method is efficient in multiple datasets, and traditional detectors cannot reveal the existence of a backdoor. [ABSTRACT FROM AUTHOR]

The vulnerability of deep-learning-based image classification models to erroneous conclusions in the presence of small perturbations crafted by attackers has prompted attention to the question of the models' robustness level. However, the question of how to comprehensively and fairly measure the adversarial robustness of models with different structures and defenses as well as the performance of different attack methods has never been accurately answered. In this work, we present the design, implementation, and evaluation of Canary, a platform that aims to answer this question. Canary uses a common scoring framework that includes 4 dimensions with 26 (sub)metrics for evaluation. First, Canary generates and selects valid adversarial examples and collects metrics data through a series of tests. Then it uses a two-way evaluation strategy to guide the data organization and finally integrates all the data to give the scores for model robustness and attack effectiveness. In this process, we use Item Response Theory (IRT) for the first time to ensure that all the metrics can be fairly calculated into a score that can visually measure the capability. In order to fully demonstrate the effectiveness of Canary, we conducted large-scale testing of 15 representative models trained on the ImageNet dataset using 12 white-box attacks and 12 black-box attacks and came up with a series of in-depth and interesting findings. This further illustrates the capabilities and strengths of Canary as a benchmarking platform. Our paper provides an open-source framework for model robustness evaluation, allowing researchers to perform comprehensive and rapid evaluations of models or attack/defense algorithms, thus inspiring further improvements and greatly benefiting future work. [ABSTRACT FROM AUTHOR]

Intrusion detection systems are increasingly using machine learning. While machine learning has shown excellent performance in identifying malicious traffic, it may increase the risk of privacy leakage. This paper focuses on implementing a model stealing attack on intrusion detection systems. Existing model stealing attacks are hard to implement in practical network environments, as they either need private data of the victim dataset or frequent access to the victim model. In this paper, we propose a novel solution called Fast Model Stealing Attack (FMSA) to address the problem in the field of model stealing attacks. We also highlight the risks of using ML-NIDS in network security. First, meta-learning frameworks are introduced into the model stealing algorithm to clone the victim model in a black-box state. Then, the number of accesses to the target model is used as an optimization term, resulting in minimal queries to achieve model stealing. Finally, adversarial training is used to simulate the data distribution of the target model and achieve the recovery of privacy data. Through experiments on multiple public datasets, compared to existing state-of-the-art algorithms, FMSA reduces the number of accesses to the target model and improves the accuracy of the clone model on the test dataset to 88.9% and the similarity with the target model to 90.1%. We can demonstrate the successful execution of model stealing attacks on the ML-NIDS system even with protective measures in place to limit the number of anomalous queries. [ABSTRACT FROM AUTHOR]

Machine learning (ML) has been universally adopted for automated decisions in a variety of fields, including recognition and classification applications, recommendation systems, natural language processing, and so on. However, in light of high expenses on training data and computing resources, recent years have witnessed a rapid increase in outsourced ML training, either partially or completely, which provides vulnerabilities for adversaries to exploit. A prime threat in training phase is called poisoning attack, where adversaries strive to subvert the behavior of machine learning systems by poisoning training data or other means of interference. Although a growing number of relevant studies have been proposed, the research among poisoning attack is still overly scattered, with each paper focusing on a particular task in a specific domain. In this survey, we summarize and categorize existing attack methods and corresponding defenses, as well as demonstrate compelling application scenarios, thus providing a unified framework to analyze poisoning attacks. Besides, we also discuss the main limitations of current works, along with the corresponding future directions to facilitate further researches. Our ultimate motivation is to provide a comprehensive and self-contained survey of this growing field of research and lay the foundation for a more standardized approach to reproducible studies. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) are easily exposed to backdoor threats when training with poisoned training samples. Models using backdoor attack have normal performance for benign samples, and possess poor performance for poisoned samples manipulated with pre-defined trigger patterns. Currently, research on backdoor attacks focuses on image classification and object detection. In this article, we investigated backdoor attacks in facial sketch synthesis, which can be beneficial for many applications, such as animation production and assisting police in searching for suspects. Specifically, we propose a simple yet effective poison-only backdoor attack suitable for generation tasks. We demonstrate that when the backdoor is integrated into the target model via our attack, it can mislead the model to synthesize unacceptable sketches of any photos stamped with the trigger patterns. Extensive experiments are executed on the benchmark datasets. Specifically, the light strokes devised by our backdoor attack strategy can significantly decrease the perceptual quality. However, the FSIM score of light strokes is 68.21% on the CUFS dataset and the FSIM scores of pseudo-sketches generated by FCN, cGAN, and MDAL are 69.35%, 71.53%, and 72.75%, respectively. There is no big difference, which proves the effectiveness of the proposed backdoor attack method. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) technology is crucial for developing autonomous ships in the maritime industry. Autonomous ships, based on the collected information, recognize the environment without any human intervention and operate themselves using their own judgment. However, ship-to-land connectivity increased, owing to the real-time monitoring and remote control (for unexpected circumstances) from land; this poses a potential cyberthreat to various data collected inside and outside the ships and to the applied AI technology. For the safety of autonomous ships, cybersecurity around AI technology needs to be considered, in addition to the cybersecurity of the ship systems. By identifying various vulnerabilities and via research cases of the ship systems and AI technologies, this study presents possible cyberattack scenarios on the AI technologies applied to autonomous ships. Based on these attack scenarios, cyberthreats and cybersecurity requirements are formulated for autonomous ships by employing the security quality requirements engineering (SQUARE) methodology. [ABSTRACT FROM AUTHOR]

Recent efforts have shown that training data is not secured through the generalization and abstraction of algorithms. This vulnerability to the training data has been expressed through membership inference attacks that seek to discover the use of specific records within the training dataset of a model. Additionally, disparate membership inference attacks have been shown to achieve better accuracy compared with their macro attack counterparts. These disparate membership inference attacks use a pragmatic approach to attack individual, more vulnerable sub-sets of the data, such as underrepresented classes. While previous work in this field has explored model vulnerability to these attacks, this effort explores the vulnerability of datasets themselves to disparate membership inference attacks. This is accomplished through the development of a vulnerability-classification model that classifies datasets as vulnerable or secure to these attacks. To develop this model, a vulnerability-classification dataset is developed from over 100 datasets—including frequently cited datasets within the field. These datasets are described using a feature set of over 100 features and assigned labels developed from a combination of various modeling and attack strategies. By averaging the attack accuracy over 13 different modeling and attack strategies, the authors explore the vulnerabilities of the datasets themselves as opposed to a particular modeling or attack effort. The in-class observational distance, width ratio, and the proportion of discrete features are found to dominate the attributes defining dataset vulnerability to disparate membership inference attacks. These features are explored in deeper detail and used to develop exploratory methods for hardening these class-based sub-datasets against attacks showing preliminary mitigation success with combinations of feature reduction and class-balancing strategies.

Federated learning (FL), a variant of distributed learning (DL), supports the training of a shared model without accessing private data from different sources. Despite its benefits with regard to privacy preservation, FL's distributed nature and privacy constraints make it vulnerable to data poisoning attacks. Existing defenses, primarily designed for DL, are typically not well adapted to FL. In this paper, we study such attacks and defenses. In doing so, we start from the perspective of DL and then give consideration to a real-world FL scenario, with the aim being to explore the requisites of a desirable defense in FL. Our study shows that (i) the batch size used in each training round affects the effectiveness of defenses in DL, (ii) the defenses investigated are somewhat effective and moderately influenced by batch size in FL settings and (iii) the non-IID data makes it more difficult to defend against data poisoning attacks in FL. Based on the findings, we discuss the key challenges and possible directions in defending against such attacks in FL. In addition, we propose detect and suppress the potential outliers(DSPO), a defense against data poisoning attacks in FL scenarios. Our results show that DSPO outperforms other defenses in several cases. [ABSTRACT FROM AUTHOR]

As deep learning models have made remarkable strides in numerous fields, a variety of adversarial attack methods have emerged to interfere with deep learning models. Adversarial examples apply a minute perturbation to the original image, which is inconceivable to the human but produces a massive error in the deep learning model. Existing attack methods have achieved good results when the network structure is known. However, in the case of unknown network structures, the effectiveness of the attacks still needs to be improved. Therefore, transfer-based attacks are now very popular because of their convenience and practicality, allowing adversarial samples generated on known models to be used in attacks on unknown models. In this paper, we extract sensitive features by Grad-CAM and propose two single-step attacks methods and a multi-step attack method to corrupt sensitive features. In two single-step attacks, one corrupts the features extracted from a single model and the other corrupts the features extracted from multiple models. In multi-step attack, our method improves the existing attack method, thus enhancing the adversarial sample transferability to achieve better results on unknown models. Our method is also validated on CIFAR-10 and MINST, and achieves a 1%-3% improvement in transferability. [ABSTRACT FROM AUTHOR]

Recent efforts have shown that training data is not secured through the generalization and abstraction of algorithms. This vulnerability to the training data has been expressed through membership inference attacks that seek to discover the use of specific records within the training dataset of a model. Additionally, disparate membership inference attacks have been shown to achieve better accuracy compared with their macro attack counterparts. These disparate membership inference attacks use a pragmatic approach to attack individual, more vulnerable sub-sets of the data, such as underrepresented classes. While previous work in this field has explored model vulnerability to these attacks, this effort explores the vulnerability of datasets themselves to disparate membership inference attacks. This is accomplished through the development of a vulnerability-classification model that classifies datasets as vulnerable or secure to these attacks. To develop this model, a vulnerability-classification dataset is developed from over 100 datasets—including frequently cited datasets within the field. These datasets are described using a feature set of over 100 features and assigned labels developed from a combination of various modeling and attack strategies. By averaging the attack accuracy over 13 different modeling and attack strategies, the authors explore the vulnerabilities of the datasets themselves as opposed to a particular modeling or attack effort. The in-class observational distance, width ratio, and the proportion of discrete features are found to dominate the attributes defining dataset vulnerability to disparate membership inference attacks. These features are explored in deeper detail and used to develop exploratory methods for hardening these class-based sub-datasets against attacks showing preliminary mitigation success with combinations of feature reduction and class-balancing strategies. [ABSTRACT FROM AUTHOR]

Together with impressive advances touching every aspect of our society, AI technology based on Deep Neural Networks (DNN) is bringing increasing security concerns. While attacks operating at test time have monopolised the initial attention of researchers, backdoor attacks, exploiting the possibility of corrupting DNN models by interfering with the training process, represent a further serious threat undermining the dependability of AI techniques. In backdoor attacks, the attacker corrupts the training data to induce an erroneous behaviour at test time. Test-time errors, however, are activated only in the presence of a triggering event. In this way, the corrupted network continues to work as expected for regular inputs, and the malicious behaviour occurs only when the attacker decides to activate the backdoor hidden within the network. Recently, backdoor attacks have been an intense research domain focusing on both the development of new classes of attacks, and the proposal of possible countermeasures. The goal of this overview is to review the works published until now, classifying the different types of attacks and defences proposed so far. The classification guiding the analysis is based on the amount of control that the attacker has on the training process, and the capability of the defender to verify the integrity of the data used for training, and to monitor the operations of the DNN at training and test time. Hence, the proposed analysis is suited to highlight the strengths and weaknesses of both attacks and defences with reference to the application scenarios they are operating in.

Adversarial machine learning (AML) is a class of data manipulation techniques that cause alterations in the behavior of artificial intelligence (AI) systems while going unnoticed by humans. These alterations can cause serious vulnerabilities to mission-critical AI-enabled applications. This work introduces an AI architecture augmented with adversarial examples and defense algorithms to safeguard, secure, and make more reliable AI systems. This can be conducted by robustifying deep neural network (DNN) classifiers and explicitly focusing on the specific case of convolutional neural networks (CNNs) used in non-trivial manufacturing environments prone to noise, vibrations, and errors when capturing and transferring data. The proposed architecture enables the imitation of the interplay between the attacker and a defender based on the deployment and cross-evaluation of adversarial and defense strategies. The AI architecture enables (i) the creation and usage of adversarial examples in the training process, which robustify the accuracy of CNNs, (ii) the evaluation of defense algorithms to recover the classifiers' accuracy, and (iii) the provision of a multiclass discriminator to distinguish and report on non-attacked and attacked data. The experimental results show promising results in a hybrid solution combining the defense algorithms and the multiclass discriminator in an effort to revitalize the attacked base models and robustify the DNN classifiers. The proposed architecture is ratified in the context of a real manufacturing environment utilizing datasets stemming from the actual production lines. [ABSTRACT FROM AUTHOR]

Deep neural networks have been widely used in detection tasks based on optical remote sensing images. However, in recent studies, deep neural networks have been shown to be vulnerable to adversarial examples. Adversarial examples are threatening in both the digital and physical domains. Specifically, they make it possible for adversarial examples to attack aerial remote sensing detection. To defend against adversarial attacks on aerial remote sensing detection, we propose a cascaded adversarial defense framework, which locates the adversarial patch according to its high frequency and saliency information in the gradient domain and removes it directly. The original image semantic and texture information is then restored by the image inpainting method. When combined with the random erasing algorithm, the robustness of detection is further improved. Our method is the first attempt to defend against adversarial examples in remote sensing detection. The experimental results show that our method is very effective in defending against real-world adversarial attacks. In particular, when using the YOLOv3 and YOLOv4 algorithms for robust detection of single-class targets, the AP60 of YOLOv3 and YOLOv4 only drop by 2.11% and 2.17%, respectively, under the adversarial example. [ABSTRACT FROM AUTHOR]

Model extraction attack focuses on reverse engineering architecture and weights of DNN model deployed in edge. Model extraction attack is a basic security problem in AI security, it underlies advanced attacks as data provider, such as adversarial sample and data poisoning. A novel method named Cluster-based SCA was proposed, this method did not need leakage model. Cluster-based SCA was based on vulnerability of quantized inference. There exist a phenomenon in multiplication operation in quantized inference, which the output of different weights were not equivalent in respect of classification. It can be used to distinguish different weights. The proposed method computed output activations of each DNN layer with guessing weight. Then acquired side channel signal were classified into different class, the taxonomy was corresponding output activations' value. Average dispersion of all classes was used to decide whether guess was right. The effectiveness of Cluster-based SCA method was verified by simulation experiment and HW model was used as target leakage model. For all weights from first convolution layer of target CNN model, TOP2 recovery rate was 52.66%. And for large weights in significant interval,TOP2 recover rate was 100%.

Deep neural networks (DNNs) are easily exposed to backdoor threats when training with poisoned training samples. Models using backdoor attack have normal performance for benign samples, and possess poor performance for poisoned samples manipulated with pre-defined trigger patterns. Currently, research on backdoor attacks focuses on image classification and object detection. In this article, we investigated backdoor attacks in facial sketch synthesis, which can be beneficial for many applications, such as animation production and assisting police in searching for suspects. Specifically, we propose a simple yet effective poison-only backdoor attack suitable for generation tasks. We demonstrate that when the backdoor is integrated into the target model via our attack, it can mislead the model to synthesize unacceptable sketches of any photos stamped with the trigger patterns. Extensive experiments are executed on the benchmark datasets. Specifically, the light strokes devised by our backdoor attack strategy can significantly decrease the perceptual quality. However, the FSIM score of light strokes is 68.21% on the CUFS dataset and the FSIM scores of pseudo-sketches generated by FCN, cGAN, and MDAL are 69.35%, 71.53%, and 72.75%, respectively. There is no big difference, which proves the effectiveness of the proposed backdoor attack method.

Artificial intelligence (AI) technology is crucial for developing autonomous ships in the maritime industry. Autonomous ships, based on the collected information, recognize the environment without any human intervention and operate themselves using their own judgment. However, ship-to-land connectivity increased, owing to the real-time monitoring and remote control (for unexpected circumstances) from land; this poses a potential cyberthreat to various data collected inside and outside the ships and to the applied AI technology. For the safety of autonomous ships, cybersecurity around AI technology needs to be considered, in addition to the cybersecurity of the ship systems. By identifying various vulnerabilities and via research cases of the ship systems and AI technologies, this study presents possible cyberattack scenarios on the AI technologies applied to autonomous ships. Based on these attack scenarios, cyberthreats and cybersecurity requirements are formulated for autonomous ships by employing the security quality requirements engineering (SQUARE) methodology.

Backdoor attacks cause model misbehaving by first implanting backdoors in deep neural networks (DNNs) during training and then activating the backdoor via samples with triggers during inference. The compromised models could pose serious security risks to artificial intelligence systems, such as misidentifying ‘stop’ traffic sign into ‘80km/h’. In this paper, we investigate the connection characteristic between the backdoor and the trigger in DNNs and observe the fact that the backdoor is implanted via establishing a link between a cluster of neurons, representing the backdoor, and the triggers. Based on this observation, we design LinkBreaker, a new generic scheme for defending against backdoor attacks. In particular, LinkBreaker deploys a neuron consistency check mechanism for identifying compromised neuron set related to the trigger. Then, the LinkBreaker regulates the model to make predictions based on benign neuron set only and thus breaks the link between the backdoor and the trigger. Compared to previous defenses, LinkBreaker offers a more general backdoor countermeasure that is not only effective against input-agnostic backdoors but also source-specific backdoors, which the later can not be defeated by majority of state-of-the-arts. Besides, LinkBreaker is robust against adversarial examples, which, to a large extent, provides a holistic defense against adversarial example attacks on DNNs, while almost all current backdoor defenses do not have such consideration and capability. Extensive experimental evaluations on real datasets demonstrate that LinkBreaker is with high efficacy of suppressing trigger inputs while incurring no noticeable accuracy deterioration on benign inputs. [ABSTRACT FROM AUTHOR]

Machine learning-based network intrusion detection systems (ML-NIDS) are extensively used for network security against unknown attacks. Existing intrusion detection systems can effectively defend traditional network attacks, however, they face AI based threats. The current known AI attacks cannot balance the escape rate and attack effectiveness. In addition, the time cost of existing AI attacks is very high. In this paper, we propose a backdoor attack called VulnerGAN, which features high concealment, high aggressiveness, and high timeliness. The backdoor can make the specific attack traffic bypass the detection of ML-NIDS without affecting the performance of ML-NIDS in identifying other attack traffic. VulnerGAN uses generative adversarial networks (GAN) to calculate poisoning and adversarial samples based on machine learning model vulnerabilities. It can make traditional network attack traffic escape black-box online ML-NIDS. At the same time, model extraction and fuzzing test are used to enhance the convergence of VulnerGAN. Compared with the state-of-the-art algorithms, the VulnerGAN backdoor attack increases 33.28% in concealment, 18.48% in aggressiveness, and 46.32% in timeliness. [ABSTRACT FROM AUTHOR]

Fifth generation mobile networks (5G) private networks are being actively promoted not only by the government but in the industry. The open radio access network (O-RAN) architecture is adopted to meet the demand of providing high-quality and low-cost services, and thus, has become a popular solution for customized application scenarios, such as smart factories, smart hospitals, public utilities, etc. Although the 5G private networks can develop a new wireless scenario providing high-quality applications, the open RAN architecture unavoidably raises new technical challenges on communications security. This article addresses the differences of security requirements between 5G public networks and 5G private networks, meanwhile it discusses the interrelation and trends of the 5G O-RAN architecture, security standards and the risk of communication security. The integration and management issues with corresponding security challenges over 5G/Wi-Fi private networks are elaborated and followed by a discussion on how artificial intelligence (AI) can play a role in the trends of 5G and beyond 5G (B5G) security. This article shares our viewpoints on 5G information security and hopefully can motivate more opportunities for industry-academia cooperation and new research topics. [ABSTRACT FROM AUTHOR]

Adversarial examples, which add carefully planned perturbations to images, pose a serious threat to neural network applications. Transferable adversarial attacks, in which adversarial examples generated on the source model can successfully attack the target model, provide a realistic and undetectable method. Existing transfer-based attacks tend to improve the transferability of adversarial examples by destroying their intrinsic features. They destabilized features differentially by assessing their importance, thus rendering the model incapable of inference. However, the existing methods generate feature-importance assessments that are overly dependent on the source model, leading to inaccurate importance guidance and insufficient feature destruction. In this paper, we propose neighborhood expectancy attribution attacks (NEAA) that accurately guide the destruction of deep features, leading to highly transferable adversarial examples. First, we design a highly versatile attribution tool called neighborhood attribution to represent the importance of features that attribute highly similar results to various source models. Specifically, we discard the imputation of a single baseline and adopt the imputed expectation of a baseline within the neighborhood of the image. Subsequently, we generalize the neighborhood attribution to the middle layer of the model and simplify the computation by assuming linear independence. Finally, the attribution result guides the attack to destroy the intrinsic features of the image and obtain highly transferable adversarial examples. Numerous experiments demonstrate the effectiveness of the proposed method. Code is available at Github: https://github.com/KWPCCC/NEAA. • Adversarial attacks focus on "How should images be understood?". • Light-modeled and heavy-featured importance assessments are key to reducing model overfitting and improving the transferability of adversarial examples. • The multipath neighborhood baseline focuses the attribution result on the image itself. • Adversarial examples crafted through broken intermediate features have good offense against defensive models. [ABSTRACT FROM AUTHOR]

Backdoor attacks against Deep Neural Networks (DNNs) have surfaced as a substantial and concerning security challenge. These backdoor vulnerabilities in DNNs can be introduced by third-party sources through maliciously manipulated training data. Existing backdoor attacks are primarily built on perturbation trigger patterns in the spatial domain, which makes practical deployment arduous due to the ease of detection by inspectors. Moreover, shortcut learning renders the backdoor network less robust against defense methods. This work advances an effective and adaptable approach to backdoor attacks, situated in the frequency domain. This methodology involves the incorporation of specific natural perturbations within the frequency domain of images. Remarkably, these introduced triggers yield minimal alterations in the image's semantic content, rendering them nearly imperceptible to human observers. To evade detection by machine-based defenders, we introduce a new training paradigm that incorporates negative sampling techniques. This approach compels the neural network to learn richer differences as trigger patterns. We evaluate our attacks on popular convolutional neural networks, visual transformers, and MLP-Mixer models as well as four standard datasets including MNIST, CIFAR-10, GTSRB, and ImageNet. Experimental results demonstrate that the trained networks can be successfully injected with backdoors. Our attack methods exhibit remarkable efficacy, achieving high attack success rates in both All-to-one (near 100% on all datasets) and All-to-all (over 90% except on ImageNet) scenarios and also demonstrate robustness against contemporary state-of-the-art defense mechanisms. Furthermore, our work reveals that DNNs can capture discrepancies in the frequency components of images that are barely perceptible to humans. [ABSTRACT FROM AUTHOR]

To realize Society 5.0, edge AI techniques have attracted attention. On the other hand, security issues of edge AI have been reported. In addition, in the field of hardware security, the threat of hardware Trojan (HT) is emphasized. To defend the AI device from malicious attacks, it is important to check the vulnerability against various attacks. Therefore, this study proposes a new HT for AI inference devices. The proposed HT falsifies the inference result with respect to an arbitrary trigger input. The proposed HT concentrates on the Lookup Table (LUT) structure, and can be achieved by rewriting the LUT table information. As a result, the proposed HT does not need additional trojan trigger and payload circuits, that is, it can be implemented without the circuit overhead. Experiments by field programable gate array show the validity of the proposed HT. [ABSTRACT FROM AUTHOR]

In recent years, adversarial attack methods have been deceived rather easily on deep neural networks (DNNs). In practice, adversarial patches cause misclassification that can be extremely effective. However, many existing adversarial patches are used for attacking DNNs, and only a few of them apply to both the DNN and its explanation model. In this paper, we present different adversarial patches that misguide the prediction of DNN models and change the cause of prediction results of interpretation models, such as gradient-weighted class activation mapping. The proposed adversarial patches have appropriate location and perturbation ratios, which comprise visible or less visible adversarial patches. In addition, image patches within small arrays are localized without covering or overlapping with any of the main objects in a natural image. In particular, we generate two adversarial patches that cover only 3% and 1.5% of the pixels in the original image, while they do not cover the main objects in the natural image. Our experiments are performed using four pre-trained DNN models and the ImageNet dataset. We also examine the inaccurate results of the interpretation models through mask and heatmap visualization. The proposed adversarial attack method could be a reference for developing robust network interpretation models that are more reliable for the decision-making process of pre-trained DNN models.