Search

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Empirical Risk Minimization (ERM) is fragile in scenarios with insufficient labeled samples. A vanilla extension of ERM to unlabeled samples is Entropy Minimization (EntMin), which employs the soft-labels of unlabeled samples to guide their learning. However, EntMin emphasizes prediction discriminability while neglecting prediction diversity. To alleviate this issue, in this paper, we rethink the guidance information to utilize unlabeled samples. By analyzing the learning objective of ERM, we find that the guidance information for labeled samples in a specific category is the corresponding label encoding. Inspired by this finding, we propose a Label-Encoding Risk Minimization (LERM). It first estimates the label encodings through prediction means of unlabeled samples and then aligns them with their corresponding ground-truth label encodings. As a result, the LERM ensures both prediction discriminability and diversity, and it can be integrated into existing methods as a plugin. Theoretically, we analyze the relationships between LERM and ERM as well as EntMin. Empirically, we verify the superiority of the LERM under several label insufficient scenarios. The codes are available at https://github.com/zhangyl660/LERM., Comment: Accepted to ICML 2024

The spectral and transport properties of strongly correlated metals, such as SrVO$_3$ (SVO), are widely attributed to electron-electron ($e$-$e$) interactions, with lattice vibrations (phonons) playing a secondary role. Here, using first-principles electron-phonon ($e$-ph) and dynamical mean field theory calculations, we show that $e$-ph interactions play an essential role in SVO: they govern the electron scattering and resistivity in a wide temperature range down to 30 K, and induce an experimentally observed kink in the spectral function. In contrast, the $e$-$e$ interactions control quasiparticle renormalizations and low temperature transport, and enhance the $e$-ph coupling. We clarify the origin of the near $T^2$ temperature dependence of the resistivity by analyzing the $e$-$e$ and $e$-ph limited transport regimes. Our work disentangles the electronic and lattice degrees of freedom in a prototypical correlated metal, revealing the dominant role of $e$-ph interactions in SVO., Comment: 7 pages, 4 figures

Document-level Event Causality Identification (DECI) aims to identify causal relations between two events in documents. Recent research tends to use pre-trained language models to generate the event causal relations. Whereas, these methods are prone to the errors of sequential generation due to multiple events in a document. Moreover, the potential structures such as event coreference and related causal chain are neglected. In this paper, we propose a multi-task learning framework to enhance event causality identification with rationale and structure-aware causal question answering. Specifically, the DECI task is transformed into multiple-choice question answering, and the causes and effects of the questioned event are generated with large language models. In addition, we generate the rationales to explain why these events have causal relations. Moreover, we construct an event structure graph, which models the multi-hop potential relations for causal reasoning of the current event. Experiments on two benchmark datasets show the great advantages of our proposed approach compared to the state-of-the-art methods. Moreover, we conduct both quantitative and qualitative analyses, which shed light on why each component of our approach can lead to great improvements.

Twisted bilayer graphene (tBLG) has emerged as an exciting platform for novel condensed matter physics. However, electron-phonon ($e$-ph) interactions in tBLG and their effects on electronic transport are not completely understood. Here we show first-principles calculations of $e$-ph interactions and resistivity in commensurate tBLG with large twist angles of 13.2 and 21.8 degrees. These calculations overcome key technical barriers, including large unit cells of up to 76 atoms, Brillouin-zone folding of the $e$-ph interactions, and unstable lattice vibrations due to the AA-stacked domains. We show that $e$-ph interactions due to layer-breathing (LB) phonons enhance intervalley scattering in large-angle tBLG. This interaction effectively couples the two layers, which are otherwise electronically decoupled at such large twist angles. As a result, the phonon-limited resistivity in tBLG deviates from the temperature-linear trend characteristic of monolayer graphene and tBLG near the magic angle. Taken together, our work quantifies $e$-ph interactions and scattering mechanisms in tBLG, revealing subtle interlayer coupling effects at large twist angles.

Understanding electronic interactions in high-temperature superconductors is an outstanding challenge. In the widely studied cuprate materials, experimental evidence points to strong electron-phonon ($e$-ph) coupling and broad photoemission spectra. Yet, the microscopic origin of this behavior is not fully understood. Here we study $e$-ph interactions and polarons in a prototypical parent (undoped) cuprate, La$_2$CuO$_4$ (LCO), by means of first-principles calculations. Leveraging parameter-free Hubbard-corrected density functional theory, we obtain a ground state with band gap and Cu magnetic moment in nearly exact agreement with experiments. This enables a quantitative characterization of $e$-ph interactions. Our calculations reveal two classes of longitudinal optical (LO) phonons with strong $e$-ph coupling to hole states. These modes consist of Cu-O plane bond-stretching and bond-bending as well as vibrations of apical O atoms. The hole spectral functions, obtained with a cumulant method that can capture strong $e$-ph coupling, exhibit broad quasiparticle peaks with a small spectral weight ($Z\approx0.25$) and pronounced LO-phonon sidebands characteristic of polaron effects. Our calculations predict features observed in photoemission spectra, including a 40-meV peak in the $e$-ph coupling distribution function not explained by existing models. These results show that the universal strong $e$-ph coupling found experimentally in lanthanum cuprates is an intrinsic feature of the parent compound, and elucidates its microscopic origin., Comment: 6 pages, 4 figures

By systematic theoretical calculations, we have revealed an excitonic insulator (EI) in the Ta2Pd3Te5 monolayer. The bulk Ta2Pd3Te5 is a van der Waals (vdW) layered compound, whereas the vdW layer can be obtained through exfoliation or molecular-beam epitaxy. First-principles calculations show that the monolayer is a nearly zero-gap semiconductor with the modified Becke-Johnson functional. Due to the same symmetry of the band-edge states, the two-dimensional polarization $\alpha_{2D}$ would be finite as the band gap goes to zero, allowing for an EI state in the compound. Using the first-principles many-body perturbation theory, the GW plus Bethe-Salpeter equation calculation reveals that the exciton binding energy is larger than the single-particle band gap, indicating the excitonic instability. The computed phonon spectrum suggests that the monolayer is dynamically stable without lattice distortion. Our findings suggest that the Ta2Pd3Te5 monolayer is an excitonic insulator without structural distortion., Comment: 6 pages, 4 figures

As a critical clue of video super-resolution (VSR), inter-frame alignment significantly impacts overall performance. However, accurate pixel-level alignment is a challenging task due to the intricate motion interweaving in the video. In response to this issue, we introduce a novel paradigm for VSR named Semantic Lens, predicated on semantic priors drawn from degraded videos. Specifically, video is modeled as instances, events, and scenes via a Semantic Extractor. Those semantics assist the Pixel Enhancer in understanding the recovered contents and generating more realistic visual results. The distilled global semantics embody the scene information of each frame, while the instance-specific semantics assemble the spatial-temporal contexts related to each instance. Furthermore, we devise a Semantics-Powered Attention Cross-Embedding (SPACE) block to bridge the pixel-level features with semantic knowledge, composed of a Global Perspective Shifter (GPS) and an Instance-Specific Semantic Embedding Encoder (ISEE). Concretely, the GPS module generates pairs of affine transformation parameters for pixel-level feature modulation conditioned on global semantics. After that, the ISEE module harnesses the attention mechanism to align the adjacent frames in the instance-centric semantic space. In addition, we incorporate a simple yet effective pre-alignment module to alleviate the difficulty of model training. Extensive experiments demonstrate the superiority of our model over existing state-of-the-art VSR methods., Comment: Accepted to AAAI 2024

Cloud-aided mobile edge networks (CAMENs) allow edge servers (ESs) to purchase resources from remote cloud servers (CSs), while overcoming resource shortage when handling computation-intensive tasks of mobile users (MUs). Conventional trading mechanisms (e.g., onsite trading) confront many challenges, including decision-making overhead (e.g., latency) and potential trading failures. This paper investigates a series of cross-layer matching mechanisms to achieve stable and cost-effective resource provisioning across different layers (i.e., MUs, ESs, CSs), seamlessly integrated into a novel hybrid paradigm that incorporates futures and spot trading. In futures trading, we explore an overbooking-driven aforehand cross-layer matching (OA-CLM) mechanism, facilitating two future contract types: contract between MUs and ESs, and contract between ESs and CSs, while assessing potential risks under historical statistical analysis. In spot trading, we design two backup plans respond to current network/market conditions: determination on contractual MUs that should switch to local processing from edge/cloud services; and an onsite cross-layer matching (OS-CLM) mechanism that engages participants in real-time practical transactions. We next show that our matching mechanisms theoretically satisfy stability, individual rationality, competitive equilibrium, and weak Pareto optimality. Comprehensive simulations in real-world and numerical network settings confirm the corresponding efficacy, while revealing remarkable improvements in time/energy efficiency and social welfare.

As Earth science enters the era of big data, artificial intelligence (AI) not only offers great potential for solving geoscience problems, but also plays a critical role in accelerating the understanding of the complex, interactive, and multiscale processes of Earth's behavior. As geoscience AI models are progressively utilized for significant predictions in crucial situations, geoscience researchers are increasingly demanding their interpretability and versatility. This study proposes an interpretable geoscience artificial intelligence (XGeoS-AI) framework to unravel the mystery of image recognition in the Earth sciences, and its effectiveness and versatility is demonstrated by taking computed tomography (CT) image recognition as an example. Inspired by the mechanism of human vision, the proposed XGeoS-AI framework generates a threshold value from a local region within the whole image to complete the recognition. Different kinds of artificial intelligence (AI) methods, such as Support Vector Regression (SVR), Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), can be adopted as the AI engines of the proposed XGeoS-AI framework to efficiently complete geoscience image recognition tasks. Experimental results demonstrate that the effectiveness, versatility, and heuristics of the proposed framework have great potential in solving geoscience image recognition problems. Interpretable AI should receive more and more attention in the field of the Earth sciences, which is the key to promoting more rational and wider applications of AI in the field of Earth sciences. In addition, the proposed interpretable framework may be the forerunner of technological innovation in the Earth sciences., Comment: there are some erros in the results, and a newer revision is still preparing

Satellite-based greenhouse gases (GHG) sensing technologies play a critical role in the study of global carbon emissions and climate change. However, none of the existing satellite-based GHG sensing technologies can achieve the measurement of broad bandwidth, high temporal-spatial resolution, and high sensitivity at the same time. Recently, dual-comb spectroscopy (DCS) has been proposed as a superior candidate technology for GHG sensing because it can measure broadband spectra with high temporal-spatial resolution and high sensitivity. The main barrier to DCS's display on satellites is its short measurement distance in open air achieved thus far. Prior research has not been able to implement DCS over 20 km of open-air path. Here, by developing a bistatic setup using time-frequency dissemination and high-power optical frequency combs, we have implemented DCS over a 113 km turbulent horizontal open-air path. Our experiment successfully measured GHG with 7 nm spectral bandwidth and a 10 kHz frequency and achieved a CO2 sensing precision of <2 ppm in 5 minutes and <0.6 ppm in 36 minutes. Our results represent a significant step towards advancing the implementation of DCS as a satellite-based technology and improving technologies for GHG monitoring, Comment: 24 pages, 6 figures

Surveillance videos are an essential component of daily life with various critical applications, particularly in public security. However, current surveillance video tasks mainly focus on classifying and localizing anomalous events. Existing methods are limited to detecting and classifying the predefined events with unsatisfactory semantic understanding, although they have obtained considerable performance. To address this issue, we propose a new research direction of surveillance video-and-language understanding, and construct the first multimodal surveillance video dataset. We manually annotate the real-world surveillance dataset UCF-Crime with fine-grained event content and timing. Our newly annotated dataset, UCA (UCF-Crime Annotation), contains 23,542 sentences, with an average length of 20 words, and its annotated videos are as long as 110.7 hours. Furthermore, we benchmark SOTA models for four multimodal tasks on this newly created dataset, which serve as new baselines for surveillance video-and-language understanding. Through our experiments, we find that mainstream models used in previously publicly available datasets perform poorly on surveillance video, which demonstrates the new challenges in surveillance video-and-language understanding. To validate the effectiveness of our UCA, we conducted experiments on multimodal anomaly detection. The results demonstrate that our multimodal surveillance learning can improve the performance of conventional anomaly detection tasks. All the experiments highlight the necessity of constructing this dataset to advance surveillance AI. The link to our dataset is provided at: https://xuange923.github.io/Surveillance-Video-Understanding.

Artificial intelligence (AI) has significantly advanced Earth sciences, yet its full potential in to comprehensively modeling Earth's complex dynamics remains unrealized. Geoscience foundation models (GFMs) emerge as a paradigm-shifting solution, integrating extensive cross-disciplinary data to enhance the simulation and understanding of Earth system dynamics. These data-centric AI models extract insights from petabytes of structured and unstructured data, effectively addressing the complexities of Earth systems that traditional models struggle to capture. The unique strengths of GFMs include flexible task specification, diverse input-output capabilities, and multi-modal knowledge representation, enabling analyses that surpass those of individual data sources or traditional AI methods. This review not only highlights the key advantages of GFMs, but also presents essential techniques for their construction, with a focus on transformers, pre-training, and adaptation strategies. Subsequently, we examine recent advancements in GFMs, including large language models, vision models, and vision-language models, particularly emphasizing the potential applications in remote sensing. Additionally, the review concludes with a comprehensive analysis of the challenges and future trends in GFMs, addressing five critical aspects: data integration, model complexity, uncertainty quantification, interdisciplinary collaboration, and concerns related to privacy, trust, and security. This review offers a comprehensive overview of emerging geoscientific research paradigms, emphasizing the untapped opportunities at the intersection of advanced AI techniques and geoscience. It examines major methodologies, showcases advances in large-scale models, and discusses the challenges and prospects that will shape the future landscape of GFMs., Comment: the manuscript is under review

To effectively process high volume of data across a fleet of dynamic and distributed vehicles, it is crucial to implement resource provisioning techniques that can provide reliable, cost-effective, and timely computing services. This article explores computation-intensive task scheduling over mobile vehicular clouds (MVCs). We use undirected weighted graphs (UWGs) to model both the execution of tasks and communication patterns among vehicles in an MVC. We then study reliable and timely scheduling of UWG tasks through a novel mechanism, operating on two complementary decision-making stages: Plan A and Plan B. Plan A entails a proactive decision-making approach, leveraging historical statistical data for the preemptive creation of an optimal mapping ($\alpha$) between tasks and the MVC prior to practical task scheduling. In contrast, Plan B explores a real-time decision-making paradigm, functioning as a reliable contingency plan. It seeks a viable mapping ($\beta$) if $\alpha$ encounters failures during task scheduling due to the unpredictable nature of the network. Furthermore, we provide an in-depth exploration of the procedural intricacies and key contributing factors that underpin the success of our mechanism. Additionally, we present a case study showcasing the superior performance on time efficiency and computation overhead. We further discuss a series of open directions for future research.

GAN-based image compression schemes have shown remarkable progress lately due to their high perceptual quality at low bit rates. However, there are two main issues, including 1) the reconstructed image perceptual degeneration in color, texture, and structure as well as 2) the inaccurate entropy model. In this paper, we present a novel GAN-based image compression approach with improved rate-distortion optimization (RDO) process. To achieve this, we utilize the DISTS and MS-SSIM metrics to measure perceptual degeneration in color, texture, and structure. Besides, we absorb the discretized gaussian-laplacian-logistic mixture model (GLLMM) for entropy modeling to improve the accuracy in estimating the probability distributions of the latent representation. During the evaluation process, instead of evaluating the perceptual quality of the reconstructed image via IQA metrics, we directly conduct the Mean Opinion Score (MOS) experiment among different codecs, which fully reflects the actual perceptual results of humans. Experimental results demonstrate that the proposed method outperforms the existing GAN-based methods and the state-of-the-art hybrid codec (i.e., VVC).

Electron-electron ($e$-$e$) and electron-phonon ($e$-ph) interactions are challenging to describe in correlated materials, where their joint effects govern unconventional transport, phase transitions, and superconductivity. Here we combine first-principles $e$-ph calculations with dynamical mean field theory (DMFT) as a step toward a unified description of $e$-$e$ and $e$-ph interactions in correlated materials. We compute the $e$-ph self-energy using the DMFT electron Green's function, and combine it with the $e$-$e$ self-energy from DMFT to obtain a Green's function including both interactions. This approach captures the renormalization of quasiparticle dispersion and spectral weight on equal footing. Using our method, we study the $e$-ph and $e$-$e$ contributions to the resistivity and spectral functions in the correlated metal Sr$_2$RuO$_4$. In this material, our results show that $e$-$e$ interactions dominate transport and spectral broadening in the temperature range we study (50$-$310~K), while $e$-ph interactions are relatively weak and account for only $\sim$10\% of the experimental resistivity. We also compute effective scattering rates, and find that the $e$-$e$ interactions result in scattering several times greater than the Planckian value $k_BT$, whereas $e$-ph interactions are associated with scattering rates lower than $k_BT$. Our work demonstrates a first-principles approach to combine electron dynamical correlations from DMFT with $e$-ph interactions in a consistent way, advancing quantitative studies of correlated materials., Comment: 10 pages, 5 figures. This revision has been accepted in PR Materials

Bulk Dirac semimetals (DSMs) exhibit unconventional transport properties and phase transitions due to their peculiar low-energy band structure. Yet the electronic interactions governing nonequilibrium phenomena in DSMs are not fully understood. Here we show that electron-phonon (e-ph) interactions in a prototypical bulk DSM, Na3Bi, are predominantly two-dimensional (2D). Our first-principles calculations discover a 2D optical phonon with strong e-ph interactions associated with in-plane vibrations of Na atoms. We show that this 2D mode governs e-ph scattering and charge transport in Na3Bi, and induces a dynamical phase transition to a Weyl semimetal. Our work advances quantitative analysis of electron interactions in topological semimetals and reveals dominant low-dimensional interactions in bulk quantum materials., Comment: Single column, 22 pages, 5 figures

Recently, with the development of deep learning, a number of Just Noticeable Difference (JND) datasets have been built for JND modeling. However, all the existing JND datasets only label the JND points based on the level of compression distortion. Hence, JND models learned from such datasets can only be used for image/video compression. As known, JND is a major characteristic of the human visual system (HVS), which reflects the maximum visual distortion that the HVS can tolerate. Hence, a generalized JND modeling should take more kinds of distortion types into account. To benefit JND modeling, this work establishes a generalized JND dataset with a coarse-to-fine JND selection, which contains 106 source images and 1,642 JND maps, covering 25 distortion types. To this end, we proposed a coarse JND candidate selection scheme to select the distorted images from the existing Image Quality Assessment (IQA) datasets as JND candidates instead of generating JND maps ourselves. Then, a fine JND selection is carried out on the JND candidates with a crowdsourced subjective assessment.

Recently, learned image compression schemes have achieved remarkable improvements in image fidelity (e.g., PSNR and MS-SSIM) compared to conventional hybrid image coding ones due to their high-efficiency non-linear transform, end-to-end optimization frameworks, etc. However, few of them take the Just Noticeable Difference (JND) characteristic of the Human Visual System (HVS) into account and optimize learned image compression towards perceptual quality. To address this issue, a JND-based perceptual quality loss is proposed. Considering that the amounts of distortion in the compressed image at different training epochs under different Quantization Parameters (QPs) are different, we develop a distortion-aware adjustor. After combining them together, we can better assign the distortion in the compressed image with the guidance of JND to preserve the high perceptual quality. All these designs enable the proposed method to be flexibly applied to various learned image compression schemes with high scalability and plug-and-play advantages. Experimental results on the Kodak dataset demonstrate that the proposed method has led to better perceptual quality than the baseline model under the same bit rate., Comment: 5 pages, 5 figures, conference

Application of the nonadiabatic molecular dynamics (NAMD) approach is severely limited to studying carrier dynamics in the momentum space, since a supercell is required to sample the phonon excitation and electron-phonon (e-ph) interaction at different momenta in a molecular dynamics simulation. Here, we develop an ab initio approach for the real-time quantum dynamics for charge carriers in the momentum space (NAMD_k) by directly introducing the e-ph coupling into the Hamiltonian based on the harmonic approximation. The NAMD_k approach maintains the quantum zero-point energy and proper phonon dispersion, and includes memory effects of phonon excitation. The application of NAMD_k to the hot carrier dynamics in graphene reveals the phonon-specific relaxation mechanism. An energy threshold of 0.2eV, defined by two optical phonon modes strongly coupled to the electrons, separates the hot electron relaxation into fast and slow regions with the lifetimes of pico- and nano-seconds, respectively. The NAMD_k approach provides a powerful tool to understand real-time carrier dynamics in the momentum space for different materials.

First-principles calculations enable accurate predictions of electronic interactions and dynamics. However, computing the electron spin dynamics remains challenging. The spin-orbit interaction causes various dynamical phenomena that couple with phonons, such as spin precession and spin-flip e-ph scattering, which are difficult to describe with current first-principles calculations. In this work, we show a rigorous framework to study phonon-induced spin relaxation and decoherence, by computing the spin-spin correlation function and its vertex corrections due to e-ph interactions. We apply this approach to a model system and develop corresponding first-principles calculations of spin relaxation in GaAs. Our vertex-correction formalism is shown to capture the Elliott-Yafet, Dyakonov-Perel, and strong-precession mechanisms - three independent spin decoherence regimes with distinct physical origins - thereby unifying their theoretical treatment and calculation. Our method is general and enables quantitative studies of spin relaxation, decoherence, and transport in a wide range of materials and devices., Comment: 13 pages, 4 figures