Search

Three sets of complete multi-color light curves of V694 Peg observed in 2013, 2015 and 2019 were presented and analyzed. Our photometric solutions show that this system is an A-type shallow contact binary in 2013 and 2015, while it converted to a W-type one in 2019. A large cool spot on the component of this binary could explain the conversion, implying the W-type phenomena may be caused by magnetic activity of the components. We have collected available data of this binary and calculated 505 times of light minimum, which span 17 years. The orbital period investigation based on these timings shows there is a long-term period increase at a rate of $dP/dt$ = 4.3($\pm$ 0.3)$\times$ 10$^{-9}$ d yr$^{-1}$ superposed on a periodic variation with a period of 11.81($\pm$ 0.06) years. The cyclic orbital variation may be the result of magnetic activity cycles or the existence of a third body. Till now, only 8 transformed systems including V694 Peg have been reported. Compared with other converting contact systems between A-type and W-type, V694 Peg is recorded as the shortest-period one. All of these converting systems are late-type (later than F7) contact binaries with O'Connell effect and show cyclic period variation, which indicates that magnetic activity may be the reason for the conversion between the two types of contact binaries. For investigating the nature of A-type and W-type phenomena, the discovery of more converting contact binaries is essential.

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Diffusion models (DMs) have gained attention in Missing Data Imputation (MDI), but there remain two long-neglected issues to be addressed: (1). Inaccurate Imputation, which arises from inherently sample-diversification-pursuing generative process of DMs. (2). Difficult Training, which stems from intricate design required for the mask matrix in model training stage. To address these concerns within the realm of numerical tabular datasets, we introduce a novel principled approach termed Kernelized Negative Entropy-regularized Wasserstein gradient flow Imputation (KnewImp). Specifically, based on Wasserstein gradient flow (WGF) framework, we first prove that issue (1) stems from the cost functionals implicitly maximized in DM-based MDI are equivalent to the MDI's objective plus diversification-promoting non-negative terms. Based on this, we then design a novel cost functional with diversification-discouraging negative entropy and derive our KnewImp approach within WGF framework and reproducing kernel Hilbert space. After that, we prove that the imputation procedure of KnewImp can be derived from another cost functional related to the joint distribution, eliminating the need for the mask matrix and hence naturally addressing issue (2). Extensive experiments demonstrate that our proposed KnewImp approach significantly outperforms existing state-of-the-art methods.

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score". By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved training efficiency for LM models. Experimental results over multiple models and datasets demonstrate the efficacy of this approach, showcasing substantial gains in training effectiveness and highlighting the potential for resource-efficient LM training. For example, we observe an absolute accuracy improvement of 0.9% averaged over 14 downstream evaluation tasks for multiple LM models while using 40% lesser data and training 42% faster when training on the OpenWebText dataset and 0.8% average absolute accuracy improvement while using 20% lesser data and training 21% faster on the Wikipedia dataset.

The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn a system of CLIP data experts via clustering. Each data expert is trained on one data cluster, being less sensitive to false negative noises in other clusters. At inference time, we ensemble their outputs by applying weights determined through the correlation between task metadata and cluster conditions. To estimate the correlation precisely, the samples in one cluster should be semantically similar, but the number of data experts should still be reasonable for training and inference. As such, we consider the ontology in human language and propose to use fine-grained cluster centers to represent each data expert at a coarse-grained level. Experimental studies show that four CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and OpenCLIP on zero-shot image classification but with less ($<$35\%) training cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly include new data experts. The code is available at https://github.com/facebookresearch/MetaCLIP/tree/main/mode., Comment: IEEE CVPR 2024 Camera Ready. Code Link: https://github.com/facebookresearch/MetaCLIP/tree/main/mode

Based on first-principles calculations, we propose a potential access to the yet unrealized freestanding blue phosphorus (blueP) through transformation of black phosphorus (blackP) induced by surface bromine (Br) adsorption. Formation of the Br-P bonds disrupts the original sp3 configurations in blackP, generates unpaired pz electrons and induces a structural transformation that results in blueP formation by re-pairing the pz orbitals. Ab initio molecular dynamics simulations confirm that randomly adsorbed Br adatoms on bilayer blackP spontaneously diffuse into specific patterns to render the emergence of the blueP phase. The expected obtainment Br-passivated blueP nanoribbons exhibit tunable band gaps in a wide range and high carrier mobilities of the order of 1000 cm2V-1s-1. This study provides an opportunity to fabricate blueP through the conversion from blackP by tuning its surface chemistry.

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts from a seed model, which is branched to train experts in embarrassingly parallel fashion with high throughput and reduced communication cost. After individual experts are asynchronously trained, BTX brings together their feedforward parameters as experts in Mixture-of-Expert (MoE) layers and averages the remaining parameters, followed by an MoE-finetuning stage to learn token-level routing. BTX generalizes two special cases, the Branch-Train-Merge method, which does not have the MoE finetuning stage to learn routing, and sparse upcycling, which omits the stage of training experts asynchronously. Compared to alternative approaches, BTX achieves the best accuracy-efficiency tradeoff.

Periodic origami patterns made with repeating unit cells of creases and panels bend and twist in complex ways. In principle, such soft modes of deformation admit a simplified asymptotic description in the limit of a large number of cells. Starting from a bar and hinge model for the elastic energy of a generic four parallelogram panel origami pattern, we derive a complete set of geometric compatibility conditions identifying the pattern's soft modes in this limit. The compatibility equations form a system of partial differential equations constraining the actuation of the origami's creases (a scalar angle field) and the relative rotations of its unit cells (a pair of skew tensor fields). We show that every solution of the compatibility equations is the limit of a sequence of soft modes -- origami deformations with finite bending energy and negligible stretching. Using these sequences, we derive a plate-like theory for parallelogram origami patterns with an explicit coarse-grained quadratic energy depending on the gradient of the crease-actuation and the relative rotations of the cells. Finally, we illustrate our theory in the context of two well-known origami designs: the Miura and Eggbox patterns. Though these patterns are distinguished in their anticlastic and synclastic bending responses, they show a universal twisting response. General soft modes captured by our theory involve a rich nonlinear interplay between actuation, bending and twisting, determined by the underlying crease geometry., Comment: minor improvements to exposition

The use of Autonomous Surface Vessels (ASVs) is growing rapidly. For safe and efficient surface auto-driving, a reliable perception system is crucial. Such systems allow the vessels to sense their surroundings and make decisions based on the information gathered. During the perception process, free space segmentation is essential to distinguish the safe mission zone and segment the operational waterways. However, ASVs face particular challenges in free space segmentation due to nearshore reflection interference, complex water textures, and random motion vibrations caused by the water surface conditions. To deal with these challenges, we propose a visual temporal fusion based free space segmentation model to utilize the previous vision information. In addition, we also introduce a new evaluation procedure and a contour position based loss calculation function, which are more suitable for surface free space segmentation tasks. The proposed model and process are tested on a continuous video segmentation dataset and achieve both high-accuracy and robust results. The dataset is also made available along with this paper.

Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP., Comment: 17 pages. arXiv admin note: text overlap with arXiv:2103.00020 by other authors

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

There has been a longstanding belief that generation can facilitate a true understanding of visual data. In line with this, we revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE). Our approach is capable of (i) serving as a strong initialization for downstream recognition tasks, (ii) conducting high-quality image inpainting, and (iii) being effortlessly extended to video where it produces state-of-the-art classification accuracy. We further perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders., Comment: Tech report. Project page: https://weichen582.github.io/diffmae.html

Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach., Comment: EMNLP 2022

Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford. This paper trades generality for efficiency and presents Curation in Training (CiT), a simple and efficient vision-text learning algorithm that couples a data objective into training. CiT automatically yields quality data to speed-up contrastive image-text training and alleviates the need for an offline data filtering pipeline, allowing broad data sources (including raw image-text pairs from the web). CiT contains two loops: an outer loop curating the training data and an inner loop consuming the curated training data. The text encoder connects the two loops. Given metadata for tasks of interest, e.g., class names, and a large pool of image-text pairs, CiT alternatively selects relevant training data from the pool by measuring the similarity of their text embeddings and embeddings of the metadata. In our experiments, we observe that CiT can speed up training by over an order of magnitude, especially if the raw data size is large., Comment: Technical Report

We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks., Comment: Technical report

Unconventional Weyl phonons with higher topological charges in crystalline solids have attracted increasing attention. By symmetry analysis and low-energy $k.p$ effective Hamiltonian, we propose the symmetry enforced triple hourglass Weyl phonons (THWPs) with Chern number $\mathcal C$ = $\pm$ 3 protected by $6_3$ screw rotation symmetry in chiral space groups 173 ($P6_3$) and 182 ($P6_322$). We take LiIO$_3$ with space group 173 as a candidate and confirm that it possesses THWP with linear and quadratic dispersions along the $k_z$ direction and in the $k_x$-$k_y$ plane, respectively. Due to the constraints of crystal symmetry and topological charge conservation, six equivalent single Weyl phonons (SWPs) emerge and lie in the $k_z$ = 0 plane. Therefore, the unique phonon surface arcs connect the projections of two THWPs and six SWPs, leading to nontrivial sextuple-helicoid surface arcs on the (001) surface Brillouin zone. Our work proposes a class of topological phonons and realizes it in realistic materials, providing a perfect platform for experimental observation of THWPs. We expect our work to provide a new idea for detection of unconventional quasiparticles., Comment: 11 pages, 7 figures

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.

We report successful growth of millimeter-sized high quality single crystals of V$_3$S$_4$, a candidate topological semimetal belonging to a low-symmetry space group and consisting of only low atomic number elements. Using density functional theory calculations and angle-resolved photoemission spectroscopy, we show that the nonmagnetic phase of monoclinic V$_3$S$_4$ hosts type-II Dirac-like quasiparticles which opens a sizable gap due to spin orbit coupling, as well as theoretical multiple nodal lines that are eliminated also by spin orbit coupling. These results suggest that relativistic effects give rise to profound modifications of the topological properties even in compounds with low-weight elements., Comment: 19 Pages, 3 Figures. To be published in Journal of Alloys and Compounds

Space groups describing the symmetry of lattice structure allow the emergence of fermionic quasiparticles with various degeneracy in the band structure. Theoretical efforts have predicted many materials hosting fermions with the highest degeneracy, i.e., eightfold fermions, yet lacking experimental realization. Here, we explore the band degeneracies in TaCo$_2$Te$_2$ crystals. Through systematic experimental and theoretical analyses, we establish TaCo$_2$Te$_2$ as a nonsymmorphic crystal with negligible spin-orbit coupling (SOC) and long-range magnetic order. These critical properties guarantee the first realization of practical eightfold fermions and fourfold van Hove singularity, as directly observed by photoemission spectroscopy. TaCo$_2$Te$_2$ serves as a topological quantum critical platform, which can be tuned into various magnetic, topologically trivial, and nontrivial phases by adding strain, magnetic field, or SOC. The latter is demonstrated by our first-principles calculations, which show that enhancing SOC in TaCo$_2$Te$_2$ will promote the experimental observation of bulk hourglass fermions. Our results establish TaCo$_2$Te$_2$ as a unique platform to explore states of matter intertwining magnetism, correlation, symmetry, and band topology.

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE., Comment: Accepted at NeurIPS 2022

While the growth of germanene has been claimed on many substrates, the exact crystal structures remain controversial. Here, we systematically explore the possible structures formed by Ge deposition onto Al(111) surface by combining density-functional theory (DFT) and global optimization algorithm. We show that, by high-level random-phase approximation (RPA) calculations, the formation of germanene on Al(111) is energetically unfavorable with positive formation energy. The two experimental phases are identified as honeycomb alloys Al3Ge3/Al(111)(r7xr7) and Al3Ge4/Al(111)(3x3), by combining ab initio evolutionary simulations, RPA calculations, and available experimental data from scanning tunneling microscopy (STM) and low-energy electron diffraction (LEED). Al3Ge4/Al(111)(3x3) is an interesting structure with a vacancy in the substrate, which accounts for the dark clover pattern in the experimental STM image. Our results clarify the structural controversy of the Ge/Al(111) system and indicate the fabrication of germanene may remain challenging., Comment: 16 pages, 4 figures

The first metallic glass of Au-Si alloy has been discovered for over half a century, but its atomic structure is still puzzling. Herein, Au 8 Si dodecahedrons with local five-fold symmetry are revealed as building blocks in Au-Si metallic glass, and the interconnection modes of Au 8 Si dodecahedrons determine the medium-range order. With dimensionality reduction, the surface ordering is attributed to the motif transformation of Au 8 Si dodecahedrons into planar Au 5 Si pyramids with five-fold symmetry, and thus the self-assembly of Au 5 Si pyramids leads to the formation of the ordered Au 2 Si monolayer with the lowest energy. Furthermore, the structural similarity analysis is performed to unveil the physical origin of structural characteristics in different dimensions. The amorphism of Au-Si is due to the smooth energy landscape around the global minimum, while the ordered surface structure occurs due to the steep energy landscape.

Aspect-based sentiment analysis (ABSA) typically requires in-domain annotated data for supervised training/fine-tuning. It is a big challenge to scale ABSA to a large number of new domains. This paper aims to train a unified model that can perform zero-shot ABSA without using any annotated data for a new domain. We propose a method called contrastive post-training on review Natural Language Inference (CORN). Later ABSA tasks can be cast into NLI for zero-shot transfer. We evaluate CORN on ABSA tasks, ranging from aspect extraction (AE), aspect sentiment classification (ASC), to end-to-end aspect-based sentiment analysis (E2E ABSA), which show ABSA can be conducted without any human annotated ABSA data.