Search

In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.

In the Standard Model (SM), electroweak (EW) corrections become significant at high energies, particularly at the tera-electronvolt scale and beyond, due to the presence of Sudakov logarithms. At these energy scales, the Standard Model Effective Field Theory (SMEFT) framework provides an enhanced sensitivity to potential new physics effects. This motivates the inclusion of EW corrections not only for SM predictions but also for analyses within SMEFT. In this work, we compute EW corrections in the high-energy limit for a selected set of dimension-six operators, specifically the class of four-fermion contact interactions, in key hard-scattering processes relevant to both current and future colliders: top-quark pair production at the Large Hadron Collider (LHC) and in a muon collider scenario, as well as the Drell-Yan process at the LHC. We first discuss the technical details and challenges associated with evaluating EW Sudakov logarithms in SMEFT, contrasting them with the SM case. We then present phenomenological results for the aforementioned processes, highlighting the non-trivial effects introduced by EW corrections arising from the insertion of dimension-six, four-fermion operators. Importantly, the resulting $K$-factors exhibit significant deviations from their SM counterparts, with dependencies not only on the process but also on the specific operators considered. Finally, we explore the potential to lift flat directions in the SMEFT parameter space by incorporating higher-order corrections, using Fisher information techniques., Comment: The article is 60 pages, including figures, tables, appendices, and references

Expansion Microscopy is a super-resolution technique in which physically enlarging samples in an isotropic manner increases inter-molecular distances such that nano-scale structures can be resolved using light microscopy. This is particularly useful in neuroscience as many important structures are smaller than the diffraction limit. Since its invention in 2015, a variety of Expansion Microscopy protocols have been generated and applied to advance knowledge in many prominent organisms in neuroscience, including zebrafish, mice, Drosophila, and C. elegans. Here we review the last decade of Expansion Microscopy-enabled advances with a focus on neuroscience.

The covariance for clean data given a noisy observation is an important quantity in many conditional generation methods for diffusion models. Current methods require heavy test-time computation, altering the standard diffusion training process or denoiser architecture, or making heavy approximations. We propose a new framework that sidesteps these issues by using covariance information that is available for free from training data and the curvature of the generative trajectory, which is linked to the covariance through the second-order Tweedie's formula. We integrate these sources of information using {\em (i)} a novel method to transfer covariance estimates across noise levels and (ii) low-rank updates in a given noise level. We validate the method on linear inverse problems, where it outperforms recent baselines, especially with fewer diffusion steps., Comment: 24 pages, 11 figures

The training phase of machine learning models is a delicate step, especially in cybersecurity contexts. Recent research has surfaced a series of insidious training-time attacks that inject backdoors in models designed for security classification tasks without altering the training labels. With this work, we propose new techniques that leverage insights in cybersecurity threat models to effectively mitigate these clean-label poisoning attacks, while preserving the model utility. By performing density-based clustering on a carefully chosen feature subspace, and progressively isolating the suspicious clusters through a novel iterative scoring procedure, our defensive mechanism can mitigate the attacks without requiring many of the common assumptions in the existing backdoor defense literature. To show the generality of our proposed mitigation, we evaluate it on two clean-label model-agnostic attacks on two different classic cybersecurity data modalities: network flows classification and malware classification, using gradient boosting and neural network models., Comment: 14 pages, 8 figures

Data derived from the realm of the social sciences is often produced in digital text form, which motivates its use as a source for natural language processing methods. Researchers and practitioners have developed and relied on artificial intelligence techniques to collect, process, and analyze documents in the legal field, especially for tasks such as text summarization and classification. While increasing procedural efficiency is often the primary motivation behind natural language processing in the field, several works have proposed solutions for human rights-related issues, such as assessment of public policy and institutional social settings. One such issue is the presence of gender biases in court decisions, which has been largely studied in social sciences fields; biased institutional responses to gender-based violence are a violation of international human rights dispositions since they prevent gender minorities from accessing rights and hamper their dignity. Natural language processing-based approaches can help detect these biases on a larger scale. Still, the development and use of such tools require researchers and practitioners to be mindful of legal and ethical aspects concerning data sharing and use, reproducibility, domain expertise, and value-charged choices. In this work, we (a) present an experimental framework developed to automatically detect gender biases in court decisions issued in Brazilian Portuguese and (b) describe and elaborate on features we identify to be critical in such a technology, given its proposed use as a support tool for research and assessment of court~activity., Comment: 27 pages; 2 figures; 6 tables. To appear in the proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 24), June 3 to 6, 2024, Rio de Janeiro, Brazil

Retrieval Augmented Generation (RAG) expands the capabilities of modern large language models (LLMs), by anchoring, adapting, and personalizing their responses to the most relevant knowledge sources. It is particularly useful in chatbot applications, allowing developers to customize LLM output without expensive retraining. Despite their significant utility in various applications, RAG systems present new security risks. In this work, we propose new attack vectors that allow an adversary to inject a single malicious document into a RAG system's knowledge base, and mount a backdoor poisoning attack. We design Phantom, a general two-stage optimization framework against RAG systems, that crafts a malicious poisoned document leading to an integrity violation in the model's output. First, the document is constructed to be retrieved only when a specific trigger sequence of tokens appears in the victim's queries. Second, the document is further optimized with crafted adversarial text that induces various adversarial objectives on the LLM output, including refusal to answer, reputation damage, privacy violations, and harmful behaviors. We demonstrate our attacks on multiple LLM architectures, including Gemma, Vicuna, and Llama, and show that they transfer to GPT-3.5 Turbo and GPT-4. Finally, we successfully conducted a Phantom attack on NVIDIA's black-box production RAG system, "Chat with RTX".

In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance., Comment: 10 pages, 7 figures

Retrosynthesis, the task of identifying precursors for a given molecule, can be naturally framed as a conditional graph generation task. Diffusion models are a particularly promising modelling approach, enabling post-hoc conditioning and trading off quality for speed during generation. We show mathematically that permutation equivariant denoisers severely limit the expressiveness of graph diffusion models and thus their adaptation to retrosynthesis. To address this limitation, we relax the equivariance requirement such that it only applies to aligned permutations of the conditioning and the generated graphs obtained through atom mapping. Our new denoiser achieves the highest top-$1$ accuracy ($54.7$\%) across template-free and template-based methods on USPTO-50k. We also demonstrate the ability for flexible post-training conditioning and good sample quality with small diffusion step counts, highlighting the potential for interactive applications and additional controls for multi-step planning., Comment: 28 pages, 9 figures

In this paper we present a Networked Music Performance system currently under development at Politecnico di Torino. We demonstrate its use in a distributed concert held in June 2023, which featured three musicians in Turin (Italy) and three musicians in Wroc{\l}aw (Poland). Although in its early stages, the system proved to be already stable enough to appear transparent to the remote audience.

We study the quantum properties of top quark pairs in lepton colliders with unpolarised beams, including spin correlations, entanglement, and violation of Bell inequalities. We present analytical results in the SM and in the SMEFT and discuss several practical aspects, like the choice of quantization axes and $t \bar t$ threshold effects. We also note a correspondence between parity symmetry and entanglement. We find that quantum observables exhibit a rich phenomenology in the SM, and can also provide additional leverage in detecting new physics residing at higher scales., Comment: 33 pages plus Appendix, 16 figures

The recent observation of entanglement between top and anti-top quarks at the LHC opens the way to interpreting collider data with quantum information tools. In this work we investigate the relevance of quantum observables in searches of new physics. To this aim, we study spin correlations of top/anti-top pairs originating from various intermediate resonances, and compare the discovery reach of quantum observables compared to classical ones. We find that they provide complementary information and, in several notable cases, also the additional leverage necessary to detect new effects., Comment: 23 pages plus appendix, 12 figures. Version accepted for publication

The integration of machine learning (ML) in numerous critical applications introduces a range of privacy concerns for individuals who provide their datasets for model training. One such privacy risk is Membership Inference (MI), in which an attacker seeks to determine whether a particular data sample was included in the training dataset of a model. Current state-of-the-art MI attacks capitalize on access to the model's predicted confidence scores to successfully perform membership inference, and employ data poisoning to further enhance their effectiveness. In this work, we focus on the less explored and more realistic label-only setting, where the model provides only the predicted label on a queried sample. We show that existing label-only MI attacks are ineffective at inferring membership in the low False Positive Rate (FPR) regime. To address this challenge, we propose a new attack Chameleon that leverages a novel adaptive data poisoning strategy and an efficient query selection method to achieve significantly more accurate membership inference than existing label-only attacks, especially at low FPRs., Comment: To appear at International Conference on Learning Representations (ICLR) 2024

Most current approaches for protecting privacy in machine learning (ML) assume that models exist in a vacuum. Yet, in reality, these models are part of larger systems that include components for training data filtering, output monitoring, and more. In this work, we introduce privacy side channels: attacks that exploit these system-level components to extract private information at far higher rates than is otherwise possible for standalone models. We propose four categories of side channels that span the entire ML lifecycle (training data filtering, input preprocessing, output post-processing, and query filtering) and allow for enhanced membership inference, data extraction, and even novel threats such as extraction of users' test queries. For example, we show that deduplicating training data before applying differentially-private training creates a side-channel that completely invalidates any provable privacy guarantees. We further show that systems which block language models from regenerating training data can be exploited to exfiltrate private keys contained in the training set--even if the model did not memorize these keys. Taken together, our results demonstrate the need for a holistic, end-to-end privacy analysis of machine learning systems., Comment: USENIX Security 2024

As machine learning (ML) classifiers increasingly oversee the automated monitoring of network traffic, studying their resilience against adversarial attacks becomes critical. This paper focuses on poisoning attacks, specifically backdoor attacks, against network traffic flow classifiers. We investigate the challenging scenario of clean-label poisoning where the adversary's capabilities are constrained to tampering only with the training data - without the ability to arbitrarily modify the training labels or any other component of the training process. We describe a trigger crafting strategy that leverages model interpretability techniques to generate trigger patterns that are effective even at very low poisoning rates. Finally, we design novel strategies to generate stealthy triggers, including an approach based on generative Bayesian network models, with the goal of minimizing the conspicuousness of the trigger, and thus making detection of an ongoing poisoning campaign more challenging. Our findings provide significant insights into the feasibility of poisoning attacks on network traffic classifiers used in multiple scenarios, including detecting malicious communication and application classification., Comment: 14 pages, 8 figures

Although promising for biomedicine, the clinical translation of inorganic nanoparticles (NPs) is limited by low biocompatibility and stability in biological fluids. A common strategy to circumvent this drawback consists in disguising the active inorganic core with a lipid bilayer coating, reminiscent of the structure of the cell membrane to redefine the chemical and biological identity of NPs. While recent reports introduced membrane coating procedures for NPs, a robust and accessible method to quantify the integrity of the bilayer coverage is not yet available. To fill this gap, we prepared SiO2 nanoparticles (SiO2NPs) with different membrane coverage degrees and monitored their interaction with AuNPs by combining microscopic, scattering, and optical techniques. The membrane-coating on SiO2NPs induces spontaneous clustering of AuNPs, whose extent depends on the coating integrity. Remarkably, we discovered a linear correlation between the membrane coverage and a spectral descriptor for the AuNPs plasmonic resonance, spanning a wide range of coating yields. These results provide a fast and cost-effective assay to monitor the compatibilization of NPs with biological environments, essential for bench tests and scale-up. In addition, we introduce a robust and scalable method to prepare SiO2NPsAuNPs hybrids through spontaneous self assembly, with a high fidelity structural control mediated by a lipid bilayer.

Large language models (LLMs) have recently taken the world by storm. They can generate coherent text, hold meaningful conversations, and be taught concepts and basic sets of instructions - such as the steps of an algorithm. In this context, we are interested in exploring the application of LLMs to graph drawing algorithms by performing experiments on ChatGPT. These algorithms are used to improve the readability of graph visualizations. The probabilistic nature of LLMs presents challenges to implementing algorithms correctly, but we believe that LLMs' ability to learn from vast amounts of data and apply complex operations may lead to interesting graph drawing results. For example, we could enable users with limited coding backgrounds to use simple natural language to create effective graph visualizations. Natural language specification would make data visualization more accessible and user-friendly for a wider range of users. Exploring LLMs' capabilities for graph drawing can also help us better understand how to formulate complex algorithms for LLMs; a type of knowledge that could transfer to other areas of computer science. Overall, our goal is to shed light on the exciting possibilities of using LLMs for graph drawing while providing a balanced assessment of the challenges and opportunities they present. A free copy of this paper with all supplemental materials required to reproduce our results is available on https://osf.io/n5rxd/?view_only=f09cbc2621f44074810b7d843f1e12f9

We explore the impact of Renormalisation Group (RG) effects in the Standard Model Effective Field Theory (SMEFT) interpretations of LHC measurements. We implement the RG running and mixing for the Wilson coefficients as obtained from the one-loop anomalous dimension matrix in the SMEFT into a Monte Carlo generator. This allows to consistently predict and combine in global fits collider observables characterised by different scales. As a showcase, we examine the impact of RG running in the strong coupling on the SMEFT predictions for $t \bar t$ production cross sections and differential distributions as well as on the bounds on the Wilson coefficients that can be obtained from current LHC data., Comment: 26 pages, 9 figures; 3 appendices

Natural language processing techniques have helped domain experts solve legal problems. Digital availability of court documents increases possibilities for researchers, who can access them as a source for building datasets -- whose disclosure is aligned with good reproducibility practices in computational research. Large and digitized court systems, such as the Brazilian one, are prone to be explored in that sense. However, personal data protection laws impose restrictions on data exposure and state principles about which researchers should be mindful. Special caution must be taken in cases with human rights violations, such as gender discrimination, over which we elaborate as an example of interest. We present legal and ethical considerations on the issue, as well as guidelines for researchers dealing with this kind of data and deciding whether to disclose it., Comment: 10 pages, 2 figures. To be published in the 4th Workshop on Natural Legal Language Processing (NLLP 2022), co-located with the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)

We present the first analysis of top spin polarizations, $t \bar t$ spin correlations, and $t \bar t$ spin entanglement at the LHC in the context of the Standard Model Effective Field Theory, that goes beyond Leading Order QCD accuracy. The complete set of independent dimension-6 operators entering $t \bar t$ production is identified, and their effects on all top spin observables are extracted at linear and quadratic order in $c/\Lambda^2$. By comparing results at LO and NLO, we note that the inclusion of higher orders, while not dramatically changing the picture, often amounts to notable numerical differences, that are not fully captured by LO scale variation. We also find that the expected deviations from the SM have an intricate phase space structure, and show up predominantly at large top $p_T$. For this reason, we advocate for the measurement of spin observables differentially or doubly-differentially in the $t \bar t$ phase-space. We show how the inclusion of present and future top spin measurements will improve global fits to top LHC data, by also addressing the issue of flat directions., Comment: Accepted for publication on JHEP

Recent work by Jia et al., showed the possibility of effectively computing pairwise model distances in weight space, using a model explanation technique known as LIME. This method requires query-only access to the two models under examination. We argue this insight can be leveraged by an adversary to reduce the net cost (number of queries) of launching an evasion campaign against a deployed model. We show that there is a strong negative correlation between the success rate of adversarial transfer and the distance between the victim model and the surrogate used to generate the evasive samples. Thus, we propose and evaluate a method to reduce adversarial costs by finding the closest surrogate model for adversarial transfer.

Federated learning is a popular strategy for training models on distributed, sensitive data, while preserving data privacy. Prior work identified a range of security threats on federated learning protocols that poison the data or the model. However, federated learning is a networked system where the communication between clients and server plays a critical role for the learning task performance. We highlight how communication introduces another vulnerability surface in federated learning and study the impact of network-level adversaries on training federated learning models. We show that attackers dropping the network traffic from carefully selected clients can significantly decrease model accuracy on a target population. Moreover, we show that a coordinated poisoning campaign from a few clients can amplify the dropping attacks. Finally, we develop a server-side defense which mitigates the impact of our attacks by identifying and up-sampling clients likely to positively contribute towards target accuracy. We comprehensively evaluate our attacks and defenses on three datasets, assuming encrypted communication channels and attackers with partial visibility of the network., Comment: 12 pages. Appearing at IEEE CNS 2022

While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.

We present the prospects of detecting quantum entanglement and the violation of Bell inequalities in $t\bar{t}$ events at the LHC. We introduce a unique set of observables suitable for both measurements, and then perform the corresponding analyses using simulated events in the dilepton final state, reconstructing up to the unfolded level. We find that entanglement can be established at better than 5$\sigma$ both at threshold as well as at high $p_T$ already in the LHC Run 2 dataset. On the other hand, only very high-$p_T$ events are sensitive to a violation of Bell inequalities, making it significantly harder to observe experimentally. By employing a sensitive and robust observable, two different unfolding methods and independent statistical approaches, we conclude that, at variance with previous estimates, testing Bell inequalities will be challenging even in the high luminosity LHC run., Comment: Accepted for publication in EPJC