Search

The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of parameters and trillions of tokens require sophisticated distributed systems that enable composing and comparing several state-of-the-art techniques in order to efficiently scale across thousands of accelerators. However, existing solutions are complex, scattered across multiple libraries/repositories, lack interoperability, and are cumbersome to maintain. Thus, curating and empirically comparing training recipes require non-trivial engineering effort. This paper introduces TorchTitan, an open-source, PyTorch-native distributed training system that unifies state-of-the-art techniques, streamlining integration and reducing overhead. TorchTitan enables 3D parallelism in a modular manner with elastic scaling, providing comprehensive logging, checkpointing, and debugging tools for production-ready training. It also incorporates hardware-software co-designed solutions, leveraging features like Float8 training and SymmetricMemory. As a flexible test bed, TorchTitan facilitates custom recipe curation and comparison, allowing us to develop optimized training recipes for Llama 3.1 and provide guidance on selecting techniques for maximum efficiency based on our experiences. We thoroughly assess TorchTitan on the Llama 3.1 family of LLMs, spanning 8 billion to 405 billion parameters, and showcase its exceptional performance, modular composability, and elastic scalability. By stacking training optimizations, we demonstrate accelerations of 65.08% with 1D parallelism at the 128-GPU scale (Llama 3.1 8B), an additional 12.59% with 2D parallelism at the 256-GPU scale (Llama 3.1 70B), and an additional 30% with 3D parallelism at the 512-GPU scale (Llama 3.1 405B) on NVIDIA H100 GPUs over optimized baselines.

Foundation Models (FMs) display exceptional performance in tasks such as natural language processing and are being applied across a growing range of disciplines. Although typically trained on large public datasets, FMs are often fine-tuned or integrated into Retrieval-Augmented Generation (RAG) systems, which rely on private data. This access, along with their size and costly training, heightens the risk of intellectual property theft. Moreover, multimodal FMs may expose sensitive information. In this work, we examine the FM threat model and discuss the practicality and comprehensiveness of various approaches for securing against them, such as ML-based methods and trusted execution environments (TEEs). We demonstrate that TEEs offer an effective balance between strong security properties, usability, and performance. Specifically, we present a solution achieving less than 10\% overhead versus bare metal for the full Llama2 7B and 13B inference pipelines running inside \intel\ SGX and \intel\ TDX. We also share our configuration files and insights from our implementation. To our knowledge, our work is the first to show the practicality of TEEs for securing FMs.

Magnetic field-induced electric polarization has been observed in trigonal non-centrosymmetric paramagnetic Pr3Ga5SiO14 langasite. We detected quadratic electric polarization along the a-axis in the basal ab plane for various magnetic-field orientations. Electric polarization along the c-axis is only evident starting from the fourth power of magnetic field, in accordance with the trigonal symmetry. The magnetic properties of Pr3Ga5SiO14 primarily stem from the local anisotropic magnetic moment of the two lowest Pr3+ singlets (quasi-doublet) in the crystal electric field. The random distribution of Ga/Si in the 2d positions leads to a local distortion of C2 symmetry and to a splitting distribution of the quasi-doublet. By considering the interactions of local moments among different Pr3+ positions within a phenomenological approach for the allowed magnetoelectric coupling, we derive the electric polarization in terms of symmetry-allowed combinations of local magnetic susceptibilities and field components. The magnetic field dependence of electric polarization in the basal plane, P_{a,b*}, is mainly determined by the accumulation of effective local susceptibilities, exhibiting similar behavior in low fields, while polarization along the c-axis, P_c, arises from the non-equivalence of local effective magnetic susceptibilities in different Pr3+ positions. Our findings suggest that the temperature dependencies of magnetic and magnetoelectric susceptibilities are highly sensitive to the distribution of the quasi-doublet splitting, which reflects the local symmetry breaking.

The generalised-geometric formulation of 10-dimensional supergravity suggests a particular simple "limit", which results in a theory whose only dynamical degrees of freedom are the dilaton and the dilatino. The theory is still invariant both under generalised diffeomorphisms and a local supersymmetry and in many aspects is structurally similar to the original supergravity, which makes it a convenient playground for understanding more subtle aspects of the full physical setup. In particular, the simplicity and the geometric nature of the dilatonic theory allow us to build a full BV extension to all orders in the fermionic variables., Comment: 16 pages, 1 figure

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Recent advancements in understanding the brain's functional organization related to behavior have been pivotal, particularly in the development of predictive models based on brain connectivity. Traditional methods in this domain often involve a two-step process by first constructing a connectivity matrix from predefined brain regions, and then linking these connections to behaviors or clinical outcomes. However, these approaches with unsupervised node partitions predict outcomes inefficiently with independently established connectivity. In this paper, we introduce the Supervised Brain Parcellation (SBP), a brain node parcellation scheme informed by the downstream predictive task. With voxel-level functional time courses generated under resting-state or cognitive tasks as input, our approach clusters voxels into nodes in a manner that maximizes the correlation between inter-node connections and the behavioral outcome, while also accommodating intra-node homogeneity. We rigorously evaluate the SBP approach using resting-state and task-based fMRI data from both the Adolescent Brain Cognitive Development (ABCD) study and the Human Connectome Project (HCP). Our analyses show that SBP significantly improves out-of-sample connectome-based predictive performance compared to conventional step-wise methods under various brain atlases. This advancement holds promise for enhancing our understanding of brain functional architectures with behavior and establishing more informative network neuromarkers for clinical applications.

The electroretinogram (ERG) is a clinical test that records the retina's electrical response to light. The ERG is a promising way to study different neurodevelopmental and neurodegenerative disorders, including autism spectrum disorder (ASD) - a neurodevelopmental condition that impacts language, communication, and reciprocal social interactions. However, in heterogeneous populations, such as ASD, where the ability to collect large datasets is limited, the application of artificial intelligence (AI) is complicated. Synthetic ERG signals generated from real ERG recordings carry similar information as natural ERGs and, therefore, could be used as an extension for natural data to increase datasets so that AI applications can be fully utilized. As proof of principle, this study presents a Generative Adversarial Network capable of generating synthetic ERG signals of children with ASD and typically developing control individuals. We applied a Time Series Transformer and Visual Transformer with Continuous Wavelet Transform to enhance classification results on the extended synthetic signals dataset. This approach may support classification models in related psychiatric conditions where the ERG may help classify disorders.

The magnetic properties of Ho-doped langasites (La:Ho)$_3$Ga$_5$SiO$_{14}$ are dominated by the Ising-like magnetic moments of the Ho$^{3+}$ ions. In their saturated regime, the induced magnetic state breaks both time and space inversion symmetries, leading to a novel linear magnetoelectric effect. However, due to distortions induced by a shared Ga/Si occupancy of the 2d sites, resolving the microscopic nature of the magnetic configuration remains a difficult task. Here we combine polarized neutron diffraction and angular dependent magnetization experiments to determine the local distortions of the Ho$^{3+}$ magnetic moments in doped langasites (La$_{1-x}$Ho$_x$)$_3$Ga$_5$SiO$_{14}$ with $x \approx 0.015$ and $x \approx 0.045$. We propose a model for a field-induced magnetic configuration with arbitrary orientations of the local Ising axis of Ho$^{3+}$ in distorted positions. The operations of broken local $C_2$ symmetry and rotations around the trigonal $C_3$ axis connect different sites, restoring the global P321 symmetry of the crystal and simplifying the description of the magnetic properties. The superposition of two distorted Ho$^{3+}$ positions connected by $C_2$ symmetry determines the local magnetic susceptibility tensor, which no longer appears Ising-like at low fields.

We consider a variety of physical systems in which one has states that can be thought of as generalised instantons. These include Yang--Mills theories on manifolds with a torsion-free $G$-structure, analogous gravitational instantons and certain supersymmetric solutions of ten-dimensional supergravity, using their formulation as generalised $G$-structures on Courant algebroids. We provide a universal algebraic construction of a complex, which we call the BPS complex, that computes the infinitesimal moduli space of the instanton as one of its cohomologies. We call a class of these spinor type complexes, which are closely connected to supersymmetric systems, and show how their Laplacians have nice properties. In the supergravity context, the BPS complex becomes a double complex, in a way that corresponds to the left- and right-moving sectors of the string, and becomes much like the double complex of $(p,q)$-forms on a K\"ahler manifold. If the BPS complex has a symplectic inner product, one can write down an associated linearised BV Chern--Simons theory, which reproduces several classic examples in gauge theory. We discuss applications to (quasi-)topological string theories and heterotic superpotential functionals, whose quadratic parts can also be constructed naturally from the BPS complex., Comment: 56 + 1 pages

Visual electrophysiology is often used clinically to determine functional changes associated with retinal or neurological conditions. The full-field flash electroretinogram (ERG) assesses the global contribution of the outer and inner retinal layers initiated by the rods and cone pathways depending on the state of retinal adaptation. Within clinical centers reference normative data are used to compare with clinical cases that may be rare or underpowered within a specific demographic. To bolster either reference or case datasets the application of synthetic ERG waveforms may offer benefits to disease classification and case-control studies. In this study and as a proof of concept, artificial intelligence (AI) to generate synthetic signals using Generative Adversarial Networks is deployed to up-scale male participants within an ISCEV reference dataset containing 68 participants, with waveforms from the right and left eye. Random Forest Classifiers further improved classification for sex within the group from a balanced accuracy of 0.72 to 0.83 with the added synthetic male waveforms. This is the first study to demonstrate the generation of synthetic ERG waveforms to improve machine learning classification modelling with electroretinogram waveforms.

When technical requirements are high, and patient outcomes are critical, opportunities for monitoring and improving surgical skills via objective motion analysis feedback may be particularly beneficial. This narrative review synthesises work on technical and non-technical surgical skills, collaborative task performance, and pose estimation to illustrate new opportunities to advance cardiothoracic surgical performance with innovations from computer vision and artificial intelligence. These technological innovations are critically evaluated in terms of the benefits they could offer the cardiothoracic surgical community, and any barriers to the uptake of the technology are elaborated upon. Like some other specialities, cardiothoracic surgery has relatively few opportunities to benefit from tools with data capture technology embedded within them (as with robotic-assisted laparoscopic surgery, for example). In such cases, pose estimation techniques that allow for movement tracking across a conventional operating field without using specialist equipment or markers offer considerable potential. With video data from either simulated or real surgical procedures, these tools can (1) provide insight into the development of expertise and surgical performance over a surgeon's career, (2) provide feedback to trainee surgeons regarding areas for improvement, (3) provide the opportunity to investigate what aspects of skill may be linked to patient outcomes which can (4) inform the aspects of surgical skill which should be focused on within training or mentoring programmes. Classifier or assessment algorithms that use artificial intelligence to 'learn' what expertise is from expert surgical evaluators could further assist educators in determining if trainees meet competency thresholds.

We develop a theory of Ricci flow for metrics on Courant algebroids which unifies and extends the analytic theory of various geometric flows, yielding a general tool for constructing solutions to supergravity equations. We prove short time existence and uniqueness of solutions on compact manifolds, in turn showing that the Courant isometry group is preserved by the flow. We show a scalar curvature monotonicity formula and prove that generalized Ricci flow is a gradient flow, extending fundamental works of Hamilton and Perelman. Using these we show a convergence result for certain nonsingular solutions to generalized Ricci flow., Comment: 38 pages, 1 figure

Optical coherence tomography (OCT) is a non-invasive imaging technique with extensive clinical applications in ophthalmology. OCT enables the visualization of the retinal layers, playing a vital role in the early detection and monitoring of retinal diseases. OCT uses the principle of light wave interference to create detailed images of the retinal microstructures, making it a valuable tool for diagnosing ocular conditions. This work presents an open-access OCT dataset (OCTDL) comprising over 2000 OCT images labeled according to disease group and retinal pathology. The dataset consists of OCT records of patients with Age-related Macular Degeneration (AMD), Diabetic Macular Edema (DME), Epiretinal Membrane (ERM), Retinal Artery Occlusion (RAO), Retinal Vein Occlusion (RVO), and Vitreomacular Interface Disease (VID). The images were acquired with an Optovue Avanti RTVue XR using raster scanning protocols with dynamic scan length and image resolution. Each retinal b-scan was acquired by centering on the fovea and interpreted and cataloged by an experienced retinal specialist. In this work, we applied Deep Learning classification techniques to this new open-access dataset.

We consider a heterotic version of six-dimensional Kodaira-Spencer gravity derived from the heterotic superpotential. We compute the one-loop partition function and find it can be expressed as a product of holomorphic Ray-Singer torsions. We discuss its topological properties and potential gauge and gravitational anomalies. We show these anomalies can be cancelled using Green-Schwarz-like counter-terms. We also discuss the dependence on the background geometry, and in particular the choice of hermitian metric needed for quantisation. Given suitable topological constraints, this dependence may again be cancelled by the addition of purely background-dependent counter-terms. We also explain how our methods provide the one-loop partition functions of a large class of more general holomorphic field theories in terms of holomorphic Ray-Singer torsions., Comment: 63 pages, 2 figures; v2: updated references and fixed typos

Handing objects to humans is an essential capability for collaborative robots. Previous research works on human-robot handovers focus on facilitating the performance of the human partner and possibly minimising the physical effort needed to grasp the object. However, altruistic robot behaviours may result in protracted and awkward robot motions, contributing to unpleasant sensations by the human partner and affecting perceived safety and social acceptance. This paper investigates whether transferring the cognitive science principle that "humans act coefficiently as a group" (i.e. simultaneously maximising the benefits of all agents involved) to human-robot cooperative tasks promotes a more seamless and natural interaction. Human-robot coefficiency is first modelled by identifying implicit indicators of human comfort and discomfort as well as calculating the robot energy consumption in performing the desired trajectory. We then present a reinforcement learning approach that uses the human-robot coefficiency score as reward to adapt and learn online the combination of robot interaction parameters that maximises such coefficiency. Results proved that by acting coefficiently the robot could meet the individual preferences of most subjects involved in the experiments, improve the human perceived comfort, and foster trust in the robotic partner., Comment: 8 pages, 6 figures, IEEE Robotics and Automation Letters

Deep learning (DL) has been a revolutionary technique in various domains. To facilitate the model development and deployment, many deep learning frameworks are proposed, among which PyTorch is one of the most popular solutions. The performance of ecosystem around PyTorch is critically important, which saves the costs of training models and reduces the response time of model inferences. In this paper, we propose TorchBench, a novel benchmark suite to study the performance of PyTorch software stack. Unlike existing benchmark suites, TorchBench encloses many representative models, covering a large PyTorch API surface. TorchBench is able to comprehensively characterize the performance of the PyTorch software stack, guiding the performance optimization across models, PyTorch framework, and GPU libraries. We show two practical use cases of TorchBench. (1) We profile TorchBench to identify GPU performance inefficiencies in PyTorch. We are able to optimize many performance bugs and upstream patches to the official PyTorch repository. (2) We integrate TorchBench into PyTorch continuous integration system. We are able to identify performance regression in multiple daily code checkins to prevent PyTorch repository from introducing performance bugs. TorchBench is open source and keeps evolving.

Converging evidence indicates that the heterogeneity of cognitive profiles may arise through detectable alternations in brain functions. Particularly, brain functional connectivity, measured under resting and cognitive states, characterizes the unique neuronal interconnections across large-scale brain networks. Despite an unprecedented opportunity to uncover neurobiological subtypes through clustering or subtyping analyses on multi-state functional connectivity, few existing approaches are applicable here to accommodate the network topology and unique biological architecture of functional connectivity. To address this issue, we propose an innovative Bayesian nonparametric network-variate clustering analysis to uncover subgroups with homogeneous brain functional network patterns integrating different cognitive states. In light of the existing neuroscience literature, we assume there are unknown state-specific modular structures within functional connectivity and simultaneously impose selection to identify informative network features for defining subtypes within unsupervised learning. To further facilitate practical use, we develop a computationally efficient variational inference algorithm to perform posterior inference with satisfactory estimation accuracy. Extensive simulations show the superior clustering accuracy and plausible result of our method. Applying the method to the landmark Adolescent Brain Cognitive Development (ABCD) study, we successfully establish neurodevelopmental subtypes linked with impulsivity related behavior trait, and identify brain sub-network phenotypes under each state to signal neurobiological heterogeneity.

Complex system stability can be studied via linear stability analysis using Random Matrix Theory (RMT) or via feasibility (requiring positive equilibrium abundances). Both approaches highlight the importance of interaction structure. Here we show, analytically and numerically, how RMT and feasibility approaches can be complementary. In generalised Lotka-Volterra (GLV) models with random interaction matrices, feasibility increases when predator-prey interactions increase; increasing competition/mutualism has the opposite effect. These changes have crucial impact on the stability of the GLV model., Comment: Manuscript is 8 pages long, containing 4 figures. Pages 9 to 25 is the Supplemental Material

Randomized, skewed caches (RSCs) such as CEASER-S have recently received much attention to defend against contention-based cache side channels. By randomizing and regularly changing the mapping(s) of addresses to cache sets, these techniques are designed to obfuscate the leakage of memory access patterns. However, new attack techniques, e.g., Prime+Prune+Probe, soon demonstrated the limits of RSCs as they allow attackers to more quickly learn which addresses contend in the cache and use this information to circumvent the randomization. To yet maintain side-channel resilience, RSCs must change the random mapping(s) more frequently with adverse effects on performance and implementation complexity. This work aims to make randomization-based approaches more robust to allow for reduced re-keying rates and presents Chameleon Cache. Chameleon Cache extends RSCs with a victim cache (VC) to decouple contention in the RSC from evictions observed by the user. The VC allows Chameleon Cache to make additional use of the multiple mappings RSCs provide to translate addresses to cache set indices: when a cache line is evicted from the RSC to the VC under one of its mappings, the VC automatically reinserts this evicted line back into the RSC by using a different mapping. As a result, the effects of previous RSC set contention are hidden and Chameleon Cache exhibits side-channel resistance and eviction patterns similar to fully associative caches with random replacement. We show that Chameleon Cache has performance overheads of < 1% and stress that VCs are more generically helpful to increase side-channel resistance and re-keying intervals of randomized caches., Comment: 12 pages, 9 figures, 6 algorithms, 1 table

While the spin liquid state in the frustrated pyrochlore Tb$_{2+x}$Ti$_{2-x}$O$_{7+y}$ has been studied both experimentally and theoretically for more than two decades, no definite description of this unconventional state has been achieved. Using synchrotron based THz spectroscopy in combination with quantum numerical simulations, we highlight a significant link between two previously unrelated features: the existence of a quadrupolar order following an ice rule and the presence of strong magneto-elastic coupling in the form of hybridized Tb$^{3+}$ crystal-field and phonon modes. The magnitude of this so-called vibronic process, which involves quadrupolar degrees of freedom, is significantly dependent on small off-stoichiometry $x$ and favors all-in all-out like correlations between quadrupoles. This mechanism competes with the long range ordered quadrupolar ice, and for slightly different stoichiometry, is able to destabilize it., Comment: Main text: 7 pages, 3 figures ; Supplemental Material: 6 pages, 2 figures