Search

Masked image modeling (MIM) is a promising option for training Vision Transformers among various self-supervised learning (SSL) methods. The essence of MIM lies in token-wise masked token predictions, with targets tokenized from images or generated by pre-trained models such as vision-language models. While tokenizers or pre-trained models are plausible MIM targets, they often offer spatially inconsistent targets even for neighboring tokens, complicating models to learn unified discriminative representations. Our pilot study confirms that addressing spatial inconsistencies has the potential to enhance representation quality. Motivated by the findings, we introduce a novel self-supervision signal called Dynamic Token Morphing (DTM), which dynamically aggregates contextually related tokens to yield contextualized targets. DTM is compatible with various SSL frameworks; we showcase an improved MIM by employing DTM, barely introducing extra training costs. Our experiments on ImageNet-1K and ADE20K demonstrate the superiority of our methods compared with state-of-the-art, complex MIM methods. Furthermore, the comparative evaluation of the iNaturalists and fine-grained visual classification datasets further validates the transferability of our method on various downstream tasks. Code is available at https://github.com/naver-ai/dtm, Comment: 27 pages, 17 tables, 6 figures

Recent deep-learning-based single image super-resolution (SISR) methods have shown impressive performance whereas typical methods train their networks by minimizing the pixel-wise distance with respect to a given high-resolution (HR) image. However, despite the basic training scheme being the predominant choice, its use in the context of ill-posed inverse problems has not been thoroughly investigated. In this work, we aim to provide a better comprehension of the underlying constituent by decomposing target HR images into two subcomponents: (1) the optimal centroid which is the expectation over multiple potential HR images, and (2) the inherent noise defined as the residual between the HR image and the centroid. Our findings show that the current training scheme cannot capture the ill-posed nature of SISR and becomes vulnerable to the inherent noise term, especially during early training steps. To tackle this issue, we propose a novel optimization method that can effectively remove the inherent noise term in the early steps of vanilla training by estimating the optimal centroid and directly optimizing toward the estimation. Experimental results show that the proposed method can effectively enhance the stability of vanilla training, leading to overall performance gain. Codes are available at github.com/2minkyulee/ECO., Comment: Accepted to AAAI 2024. Codes are available at github.com/2minkyulee/ECO

Zero-Shot Object Counting (ZSOC) aims to count referred instances of arbitrary classes in a query image without human-annotated exemplars. To deal with ZSOC, preceding studies proposed a two-stage pipeline: discovering exemplars and counting. However, there remains a challenge of vulnerability to error propagation of the sequentially designed two-stage process. In this work, an one-stage baseline, Visual-Language Baseline (VLBase), exploring the implicit association of the semantic-patch embeddings of CLIP is proposed. Subsequently, the extension of VLBase to Visual-language Counter (VLCounter) is achieved by incorporating three modules devised to tailor VLBase for object counting. First, Semantic-conditioned Prompt Tuning (SPT) is introduced within the image encoder to acquire target-highlighted representations. Second, Learnable Affine Transformation (LAT) is employed to translate the semantic-patch similarity map to be appropriate for the counting task. Lastly, the layer-wisely encoded features are transferred to the decoder through Segment-aware Skip Connection (SaSC) to keep the generalization capability for unseen classes. Through extensive experiments on FSC147, CARPK, and PUCPR+, the benefits of the end-to-end framework, VLCounter, are demonstrated., Comment: Accepted to AAAI 2024. Code is available at https://github.com/Seunggu0305/VLCounter

Few-shot segmentation aims to accurately segment novel target objects within query images using only a limited number of annotated support images. The recent works exploit support background as well as its foreground to precisely compute the dense correlations between query and support. However, they overlook the characteristics of the background that generally contains various types of objects. In this paper, we highlight this characteristic of background which can bring problematic cases as follows: (1) when the query and support backgrounds are dissimilar and (2) when objects in the support background are similar to the target object in the query. Without any consideration of the above cases, adopting the entire support background leads to a misprediction of the query foreground as background. To address this issue, we propose Task-disruptive Background Suppression (TBS), a module to suppress those disruptive support background features based on two spatial-wise scores: query-relevant and target-relevant scores. The former aims to mitigate the impact of unshared features solely existing in the support background, while the latter aims to reduce the influence of target-similar support background features. Based on these two scores, we define a query background relevant score that captures the similarity between the backgrounds of the query and the support, and utilize it to scale support background features to adaptively restrict the impact of disruptive support backgrounds. Our proposed method achieves state-of-the-art performance on PASCAL-5 and COCO-20 datasets on 1-shot segmentation. Our official code is available at github.com/SuhoPark0706/TBSNet.

In this paper, we first investigate a visual quality degradation problem observed in recent high-resolution virtual try-on approach. The tendency is empirically found that the textures of clothes are squeezed at the sleeve, as visualized in the upper row of Fig.1(a). A main reason for the issue arises from a gradient conflict between two popular losses, the Total Variation (TV) and adversarial losses. Specifically, the TV loss aims to disconnect boundaries between the sleeve and torso in a warped clothing mask, whereas the adversarial loss aims to combine between them. Such contrary objectives feedback the misaligned gradients to a cascaded appearance flow estimation, resulting in undesirable squeezing artifacts. To reduce this, we propose a Sequential Deformation (SD-VITON) that disentangles the appearance flow prediction layers into TV objective-dominant (TVOB) layers and a task-coexistence (TACO) layer. Specifically, we coarsely fit the clothes onto a human body via the TVOB layers, and then keep on refining via the TACO layer. In addition, the bottom row of Fig.1(a) shows a different type of squeezing artifacts around the waist. To address it, we further propose that we first warp the clothes into a tucked-out shirts style, and then partially erase the texture from the warped clothes without hurting the smoothness of the appearance flows. Experimental results show that our SD-VITON successfully resolves both types of artifacts and outperforms the baseline methods. Source code will be available at https://github.com/SHShim0513/SD-VITON., Comment: Accepted to AAAI 2024

We address the medication recommendation problem, which aims to recommend effective medications for a patient's current visit by utilizing information (e.g., diagnoses and procedures) given at the patient's current and past visits. While there exist a number of recommender systems designed for this problem, we point out that they are challenged in accurately capturing the relation (spec., the degree of relevance) between the current and each of the past visits for the patient when obtaining her current health status, which is the basis for recommending medications. To address this limitation, we propose a novel medication recommendation framework, named VITA, based on the following two novel ideas: (1) relevant-Visit selectIon; (2) Target-aware Attention. Through extensive experiments using real-world datasets, we demonstrate the superiority of VITA (spec., up to 5.56% higher accuracy, in terms of Jaccard, than the best competitor) and the effectiveness of its two core ideas. The code is available at https://github.com/jhheo0123/VITA., Comment: Accepted by AAAI 2024

Recent advancements in Deep Neural Network (DNN) models have significantly improved performance across computer vision tasks. However, achieving highly generalizable and high-performing vision models requires expansive datasets, resulting in significant storage requirements. This storage challenge is a critical bottleneck for scaling up models. A recent breakthrough by SeiT proposed the use of Vector-Quantized (VQ) feature vectors (i.e., tokens) as network inputs for vision classification. This approach achieved 90% of the performance of a model trained on full-pixel images with only 1% of the storage. While SeiT needs labeled data, its potential in scenarios beyond fully supervised learning remains largely untapped. In this paper, we extend SeiT by integrating Masked Token Modeling (MTM) for self-supervised pre-training. Recognizing that self-supervised approaches often demand more data due to the lack of labels, we introduce TokenAdapt and ColorAdapt. These methods facilitate comprehensive token-friendly data augmentation, effectively addressing the increased data requirements of self-supervised learning. We evaluate our approach across various scenarios, including storage-efficient ImageNet-1k classification, fine-grained classification, ADE-20k semantic segmentation, and robustness benchmarks. Experimental results demonstrate consistent performance improvement in diverse experiments, validating the effectiveness of our method. Code is available at https://github.com/naver-ai/seit., Comment: Accepted to ECCV 2024. First two authors contributed equally

In speaker verification, ECAPA-TDNN has shown remarkable improvement by utilizing one-dimensional(1D) Res2Net block and squeeze-and-excitation(SE) module, along with multi-layer feature aggregation (MFA). Meanwhile, in vision tasks, ConvNet structures have been modernized by referring to Transformer, resulting in improved performance. In this paper, we present an improved block design for TDNN in speaker verification. Inspired by recent ConvNet structures, we replace the SE-Res2Net block in ECAPA-TDNN with a novel 1D two-step multi-scale ConvNeXt block, which we call TS-ConvNeXt. The TS-ConvNeXt block is constructed using two separated sub-modules: a temporal multi-scale convolution (MSC) and a frame-wise feed-forward network (FFN). This two-step design allows for flexible capturing of inter-frame and intra-frame contexts. Additionally, we introduce global response normalization (GRN) for the FFN modules to enable more selective feature propagation, similar to the SE module in ECAPA-TDNN. Experimental results demonstrate that NeXt-TDNN, with a modernized backbone block, significantly improved performance in speaker verification tasks while reducing parameter size and inference time. We have released our code for future studies., Comment: Accepted by ICASSP 2024

Model explanations can be valuable for interpreting and debugging predictive models. We study a specific kind called Concept Explanations, where the goal is to interpret a model using human-understandable concepts. Although popular for their easy interpretation, concept explanations are known to be noisy. We begin our work by identifying various sources of uncertainty in the estimation pipeline that lead to such noise. We then propose an uncertainty-aware Bayesian estimation method to address these issues, which readily improved the quality of explanations. We demonstrate with theoretical analysis and empirical evaluation that explanations computed by our method are robust to train-time choices while also being label-efficient. Further, our method proved capable of recovering relevant concepts amongst a bank of thousands, in an evaluation with real-datasets and off-the-shelf models, demonstrating its scalability. We believe the improved quality of uncertainty-aware concept explanations make them a strong candidate for more reliable model interpretation. We release our code at https://github.com/vps-anonconfs/uace.

Despite the impressive generative capabilities of diffusion models, existing diffusion model-based style transfer methods require inference-stage optimization (e.g. fine-tuning or textual inversion of style) which is time-consuming, or fails to leverage the generative ability of large-scale diffusion models. To address these issues, we introduce a novel artistic style transfer method based on a pre-trained large-scale diffusion model without any optimization. Specifically, we manipulate the features of self-attention layers as the way the cross-attention mechanism works; in the generation process, substituting the key and value of content with those of style image. This approach provides several desirable characteristics for style transfer including 1) preservation of content by transferring similar styles into similar image patches and 2) transfer of style based on similarity of local texture (e.g. edge) between content and style images. Furthermore, we introduce query preservation and attention temperature scaling to mitigate the issue of disruption of original content, and initial latent Adaptive Instance Normalization (AdaIN) to deal with the disharmonious color (failure to transfer the colors of style). Our experimental results demonstrate that our proposed method surpasses state-of-the-art methods in both conventional and diffusion-based style transfer baselines., Comment: Accepted to CVPR 2024. Project page: https://jiwoogit.github.io/StyleID_site

In recent years, online Video Instance Segmentation (VIS) methods have shown remarkable advancement with their powerful query-based detectors. Utilizing the output queries of the detector at the frame-level, these methods achieve high accuracy on challenging benchmarks. However, our observations demonstrate that these methods heavily rely on location information, which often causes incorrect associations between objects. This paper presents that a key axis of object matching in trackers is appearance information, which becomes greatly instructive under conditions where positional cues are insufficient for distinguishing their identities. Therefore, we suggest a simple yet powerful extension to object decoders that explicitly extract embeddings from backbone features and drive queries to capture the appearances of objects, which greatly enhances instance association accuracy. Furthermore, recognizing the limitations of existing benchmarks in fully evaluating appearance awareness, we have constructed a synthetic dataset to rigorously validate our method. By effectively resolving the over-reliance on location information, we achieve state-of-the-art results on YouTube-VIS 2019/2021 and Occluded VIS (OVIS). Code is available at https://github.com/KimHanjung/VISAGE., Comment: Technical report

In this paper, we introduce a quantum-secured single-pixel imaging (QS-SPI) technique designed to withstand spoofing attacks, wherein adversaries attempt to deceive imaging systems with fake signals. Unlike previous quantum-secured protocols that impose a threshold error rate limiting their operation, even with the existence of true signals, our approach not only identifies spoofing attacks but also facilitates the reconstruction of a true image. Our method involves the analysis of a specific mode correlation of a photon-pair, which is independent of the mode used for image construction, to check security. Through this analysis, we can identify both the targeted image region by the attack and the type of spoofing attack, enabling reconstruction of the true image. A proof-of-principle demonstration employing polarization-correlation of a photon-pair is provided, showcasing successful image reconstruction even under the condition of spoofing signals 2000 times stronger than the true signals. We expect our approach to be applied to quantum-secured signal processing such as quantum target detection or ranging., Comment: 10 pages, 6 figures

We introduce a hypergoemetirc series with two complex variables, which generalizes Appell's, Lauricella's and Kemp\'e de F\'eriet's hypergeometric series, and study the system of differential equations that it satisfies. We determine the singularities, the rank and the condition for the reducibility of the system. We give complete local solutions of the system at many singular points of the system and solve the connection problem among these local solutions. Under some assumptions, the system is written as a KZ equation. We determine its spectral type in the direction of coordinates as well as simultaneous eigenspace decompositions of residue matrices. The system may or may not be rigid in the sense of N.~Katz viewed as an ordinary differential equation in some direction. We also show that the system is a special case of Gel'fand-Kapranov-Zelevinsky system. From this point of view, we discuss multivariate generalizations., Comment: 69 pages, revised version

Lane detection is a vital task for vehicles to navigate and localize their position on the road. To ensure reliable driving, lane detection models must have robust generalization performance in various road environments. However, despite the advanced performance in the trained domain, their generalization performance still falls short of expectations due to the domain discrepancy. To bridge this gap, we propose a novel generative framework using HD Maps for Single-Source Domain Generalization (SSDG) in lane detection. We first generate numerous front-view images from lane markings of HD Maps. Next, we strategically select a core subset among the generated images using (i) lane structure and (ii) road surrounding criteria to maximize their diversity. In the end, utilizing this core set, we train lane detection models to boost their generalization performance. We validate that our generative framework from HD Maps outperforms the Domain Adaptation model MLDA with +3.01%p accuracy improvement, even though we do not access the target domain images., Comment: Accepted by CVPR Data-Driven Autonomous Driving Simulation Workshop, 2024

This study is concerned with the estimation of long-term fatigue damage for a floating offshore wind turbine. With the ultimate goal of efficient evaluation of fatigue limit states for floating offshore wind turbine systems, a detailed computational framework is introduced and used to develop a surrogate model using Gaussian process regression. The surrogate model, at first, relies only on a small subset of representative sea states and, then, is supplemented by the evaluation of additional sea states that leads to efficient convergence and accurate prediction of fatigue damage. A 5-MW offshore wind turbine supported by a semi-submersible floating platform is selected to demonstrate the proposed framework. The fore-aft bending moment at the turbine tower base and the fairlead tension in the windward mooring line are used for evaluation. Metocean data provide information on joint statistics of the wind and wave along with their relative likelihoods for the installation site in the Mediterranean Sea, near the coast of Sicily. \textcolor{black}{A coupled frequency-domain model} provides needed power spectra for the desired response processes. The proposed approach offers an efficient and accurate alternative to the exhaustive evaluation of a larger number of sea states and, as such, avoids excessive response simulations.

This study utilizes data from the Baccalaureate and Beyond Longitudinal Study to explore factors associated with the likelihood of students' employment in STEM fields one year after graduation. We examined various factors related to students' individual characteristics (e.g., gender, race, and financial situation), institutional experiences (e.g., major, academic standing, research involvement, internships, extracurricular activities, and undergraduate practicum), and institutional and national trends. The results indicate lower STEM employment likelihood for minority groups and students with academic probation. The findings also highlight the positive impact of undergraduate practicum and job relevance to major on STEM employment likelihood. On the contrary, career services were negatively associated with the likelihood of students' STEM occupation choice, suggesting potential shortcomings in STEM job preparation within these services. The study provides valuable insights and actionable recommendations for policymakers and educators seeking to increase diversity and inclusion in STEM fields, suggesting the need for more efficient and tailored educational interventions and curriculum development.

Accurately predicting the lifespan of lithium-ion batteries is crucial for optimizing operational strategies and mitigating risks. While numerous studies have aimed at predicting battery lifespan, few have examined the interpretability of their models or how such insights could improve predictions. Addressing this gap, we introduce three innovative models that integrate shallow attention layers into a foundational model from our previous work, which combined elements of recurrent and convolutional neural networks. Utilizing a well-known public dataset, we showcase our methodology's effectiveness. Temporal attention is applied to identify critical timesteps and highlight differences among test cell batches, particularly underscoring the significance of the "rest" phase. Furthermore, by applying cyclic attention via self-attention to context vectors, our approach effectively identifies key cycles, enabling us to strategically decrease the input size for quicker predictions. Employing both single- and multi-head attention mechanisms, we have systematically minimized the required input from 100 to 50 and then to 30 cycles, refining this process based on cyclic attention scores. Our refined model exhibits strong regression capabilities, accurately forecasting the initiation of rapid capacity fade with an average deviation of only 58 cycles by analyzing just the initial 30 cycles of easily accessible input data.

Temporal Grounding is to identify specific moments or highlights from a video corresponding to textual descriptions. Typical approaches in temporal grounding treat all video clips equally during the encoding process regardless of their semantic relevance with the text query. Therefore, we propose Correlation-Guided DEtection TRansformer (CG-DETR), exploring to provide clues for query-associated video clips within the cross-modal attention. First, we design an adaptive cross-attention with dummy tokens. Dummy tokens conditioned by text query take portions of the attention weights, preventing irrelevant video clips from being represented by the text query. Yet, not all words equally inherit the text query's correlation to video clips. Thus, we further guide the cross-attention map by inferring the fine-grained correlation between video clips and words. We enable this by learning a joint embedding space for high-level concepts, i.e., moment and sentence level, and inferring the clip-word correlation. Lastly, we exploit the moment-specific characteristics and combine them with the context of each video to form a moment-adaptive saliency detector. By exploiting the degrees of text engagement in each video clip, it precisely measures the highlightness of each clip. CG-DETR achieves state-of-the-art results on various benchmarks for temporal grounding. Codes are available at https://github.com/wjun0830/CGDETR., Comment: 29 pages, 15 figures, 14 tables, Code is available at https://github.com/wjun0830/CGDETR

The acoustic response of defect-based acoustic interferometer-like designs, known as Coupled Resonator Acoustic Waveguides (CRAWs), in two-dimensional phononic crystals (PnCs) is reported. The PnC is composed of steel cylinders arranged in a square lattice within a water matrix with defects induced by selectively removing cylinders to create Mach-Zehnder-like (MZ) defect-based interferometers. Two defect-based acoustic interferometers of MZ-type are fabricated, one with arms oriented horizontally and another one with arms oriented diagonally, and their transmission features are experimentally characterized using ultrasonic spectroscopy. The experimental data are compared with finite element method (FEM) simulations and with tight-binding (TB) calculations in which each defect is treated as a resonator coupled to its neighboring ones. Significantly, the results exhibit excellent agreement indicating the reliability of the proposed approach. This comprehensive match is of paramount importance for accurately predicting and optimizing resonant modes supported by defect arrays, thus enabling the tailoring of phononic structures and defect-based waveguides to meet specific requirements. This successful implementation of FEM and TB calculations in investigating CRAWs systems within phononic crystals paves the way for designing advanced acoustic devices with desired functionalities for various practical applications, demonstrating the application of solid-state electronics principles to underwater acoustic devices description.

Advances in lightweight neural networks have revolutionized computer vision in a broad range of IoT applications, encompassing remote monitoring and process automation. However, the detection of small objects, which is crucial for many of these applications, remains an underexplored area in current computer vision research, particularly for embedded devices. To address this gap, the paper proposes a novel adaptive tiling method that can be used on top of any existing object detector including the popular FOMO network for object detection on microcontrollers. Our experimental results show that the proposed tiling method can boost the F1-score by up to 225% while reducing the average object count error by up to 76%. Furthermore, the findings of this work suggest that using a soft F1 loss over the popular binary cross-entropy loss can significantly reduce the negative impact of imbalanced data. Finally, we validate our approach by conducting experiments on the Sony Spresense microcontroller, showcasing the proposed method's ability to strike a balance between detection performance, low latency, and minimal memory consumption.

Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language., Comment: ICLR 2024 (v2: 34 pages, 19 figures)

We report on the chemo-dynamical analysis of SPLUS J142445.34-254247.1, an extremely metal-poor halo star enhanced in elements formed by the rapid neutron-capture process. This star was first selected as a metal-poor candidate from its narrow-band S-PLUS photometry and followed up spectroscopically in medium-resolution with Gemini South/GMOS, which confirmed its low-metallicity status. High-resolution spectroscopy was gathered with GHOST at Gemini South, allowing for the determination of chemical abundances for 36 elements, from carbon to thorium. At [Fe/H]=-3.39, SPLUS J1424-2542 is one of the lowest metallicity stars with measured Th and has the highest logeps(Th/Eu) observed to date, making it part of the "actinide-boost" category of r-process enhanced stars. The analysis presented here suggests that the gas cloud from which SPLUS J1424-2542 was formed must have been enriched by at least two progenitor populations. The light-element (Z<=30) abundance pattern is consistent with the yields from a supernova explosion of metal-free stars with 11.3-13.4 Msun, and the heavy-element (Z>=38) abundance pattern can be reproduced by the yields from a neutron star merger (1.66Msun and 1.27Msun) event. A kinematical analysis also reveals that SPLUS J1424-2542 is a low-mass, old halo star with a likely in-situ origin, not associated with any known early merger events in the Milky Way., Comment: 26 pages, 11 figures, accepted for publication on ApJ

Coyote C++ is an automated testing tool that uses a sophisticated concolic-execution-based approach to realize fully automated unit testing for C and C++. While concolic testing has proven effective for languages such as C and Java, tools have struggled to achieve a practical level of automation for C++ due to its many syntactical intricacies and overall complexity. Coyote C++ is the first automated testing tool to breach the barrier and bring automated unit testing for C++ to a practical level suitable for industrial adoption, consistently reaching around 90% code coverage. Notably, this testing process requires no user involvement and performs test harness generation, test case generation and test execution with "one-click" automation. In this paper, we introduce Coyote C++ by outlining its high-level structure and discussing the core design decisions that shaped the implementation of its concolic execution engine. Finally, we demonstrate that Coyote C++ is capable of achieving high coverage results within a reasonable timespan by presenting the results from experiments on both open-source and industrial software.

Titanium nitride (TiN) presents superior electrical conductivity with mechanical and chemical stability and compatibility with the semiconductor fabrication process. Here, we fabricated epitaxial and polycrystalline TiN (111) thin films on MgO (111), sapphire (001), and mica substrates at 640oC and room temperature by using a DC sputtering, respectively. The epitaxial films show less amount of surface oxidation than the polycrystalline ones grown at room temperature. The epitaxial films show drastically reduced resistivity (~30 micro-ohm-cm), much smaller than the polycrystalline films. Temperature-dependent resistivity measurements show a nearly monotonic temperature slope down to low temperature. These results demonstrate that high temperature growth of TiN thin films leads to significant enhancement of electrical conductivity, promising for durable and scalable electrode applications., Comment: 14 pages, 3 figures

Masked image modeling (MIM) has become a leading self-supervised learning strategy. MIMs such as Masked Autoencoder (MAE) learn strong representations by randomly masking input tokens for the encoder to process, with the decoder reconstructing the masked tokens to the input. However, MIM pre-trained encoders often exhibit a limited attention span, attributed to MIM's sole focus on regressing masked tokens only, which may impede the encoder's broader context learning. To tackle the limitation, we improve MIM by explicitly incorporating unmasked tokens into the training process. Specifically, our method enables the encoder to learn from broader context supervision, allowing unmasked tokens to experience broader contexts while the decoder reconstructs masked tokens. Thus, the encoded unmasked tokens are equipped with extensive contextual information, empowering masked tokens to leverage the enhanced unmasked tokens for MIM. As a result, our simple remedy trains more discriminative representations revealed by achieving 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain. We attribute the success to the enhanced pre-training method, as evidenced by the singular value spectrum and attention analyses. Finally, our models achieve significant performance gains at the downstream semantic segmentation and fine-grained visual classification tasks; and on diverse robust evaluation metrics. Code is available at https://github.com/naver-ai/lut, Comment: 24 pages, 6 figures, 10 tables. To be presented at ECCV'24

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io., Comment: Project website: https://robotics-transformer-x.github.io

To understand the biological characteristics of neurological disorders with functional connectivity (FC), recent studies have widely utilized deep learning-based models to identify the disease and conducted post-hoc analyses via explainable models to discover disease-related biomarkers. Most existing frameworks consist of three stages, namely, feature selection, feature extraction for classification, and analysis, where each stage is implemented separately. However, if the results at each stage lack reliability, it can cause misdiagnosis and incorrect analysis in afterward stages. In this study, we propose a novel unified framework that systemically integrates diagnoses (i.e., feature selection and feature extraction) and explanations. Notably, we devised an adaptive attention network as a feature selection approach to identify individual-specific disease-related connections. We also propose a functional network relational encoder that summarizes the global topological properties of FC by learning the inter-network relations without pre-defined edges between functional networks. Last but not least, our framework provides a novel explanatory power for neuroscientific interpretation, also termed counter-condition analysis. We simulated the FC that reverses the diagnostic information (i.e., counter-condition FC): converting a normal brain to be abnormal and vice versa. We validated the effectiveness of our framework by using two large resting-state functional magnetic resonance imaging (fMRI) datasets, Autism Brain Imaging Data Exchange (ABIDE) and REST-meta-MDD, and demonstrated that our framework outperforms other competing methods for disease identification. Furthermore, we analyzed the disease-related neurological patterns based on counter-condition analysis.

Deep learning (DL) for predicting Alzheimer's disease (AD) has provided timely intervention in disease progression yet still demands attentive interpretability to explain how their DL models make definitive decisions. Recently, counterfactual reasoning has gained increasing attention in medical research because of its ability to provide a refined visual explanatory map. However, such visual explanatory maps based on visual inspection alone are insufficient unless we intuitively demonstrate their medical or neuroscientific validity via quantitative features. In this study, we synthesize the counterfactual-labeled structural MRIs using our proposed framework and transform it into a gray matter density map to measure its volumetric changes over the parcellated region of interest (ROI). We also devised a lightweight linear classifier to boost the effectiveness of constructed ROIs, promoted quantitative interpretation, and achieved comparable predictive performance to DL methods. Throughout this, our framework produces an ``AD-relatedness index'' for each ROI and offers an intuitive understanding of brain status for an individual patient and across patient groups with respect to AD progression., Comment: 15 pages, 5 figures, 4 tables

Pristine_183.6849+04.8619 (P1836849) is an extremely metal-poor ([Fe/H]$=-3.3\pm0.1$) star on a prograde orbit confined to the Galactic disk. Such stars are rare and may have their origins in protogalactic fragments that formed the early Milky Way, in low mass satellites accreted later, or forming in situ in the Galactic plane. Here we present a chemo-dynamical analysis of the spectral features between $3700-11000$\r{A} from a high-resolution spectrum taken during Science Verification of the new Gemini High-resolution Optical SpecTrograph (GHOST). Spectral features for many chemical elements are analysed (Mg, Al, Si, Ca, Sc, Ti, Cr, Mn, Fe, Ni), and valuable upper limits are determined for others (C, Na, Sr, Ba). This main sequence star exhibits several rare chemical signatures, including (i) extremely low metallicity for a star in the Galactic disk, (ii) very low abundances of the light $\alpha$-elements (Na, Mg, Si) compared to other metal-poor stars, and (iii) unusually large abundances of Cr and Mn, where [Cr, Mn/Fe]$_{\rm NLTE}>+0.5$. A comparison to theoretical yields from supernova models suggests that two low mass Population III objects (one 10 M$_\odot$ supernova and one 17 M$_\odot$ hypernova) can reproduce the abundance pattern well (reduced $\chi^2<1$). When this star is compared to other extremely metal-poor stars on quasi-circular, prograde planar orbits, differences in both chemistry and kinematics imply there is little evidence for a common origin. The unique chemistry of P1836849 is discussed in terms of the earliest stages in the formation of the Milky Way., Comment: 16 pages, 10 figures, 6 tables. Accepted by MNRAS November 22; Revisions include comparisons to more EMP stars, results unchanged

Spin nematic (SN) is a magnetic analog of classical liquid crystals, a fourth state of matter exhibiting characteristics of both liquid and solid. Particularly intriguing is a valence-bond SN, in which spins are quantum entangled to form a multi-polar order without breaking time-reversal symmetry, but its unambiguous experimental realization remains elusive. Here, we establish a SN phase in the square-lattice iridate Sr$_2$IrO$_4$, which approximately realizes a pseudospin one-half Heisenberg antiferromagnet (AF) in the strong spin-orbit coupling limit. Upon cooling, the transition into the SN phase at T$_C$ $\approx$ 263 K is marked by a divergence in the static spin quadrupole susceptibility extracted from our Raman spectra, and concomitant emergence of a collective mode associated with the spontaneous breaking of rotational symmetries. The quadrupolar order persists in the antiferromagnetic (AF) phase below T$_N$ $\approx$ 230 K, and becomes directly observable through its interference with the AF order in resonant x-ray diffraction, which allows us to uniquely determine its spatial structure. Further, we find using resonant inelastic x-ray scattering a complete breakdown of coherent magnon excitations at short-wavelength scales, suggesting a resonating-valence-bond-like quantum entanglement in the AF state. Taken together, our results reveal a quantum order underlying the N\'eel AF that is widely believed to be intimately connected to the mechanism of high temperature superconductivity (HTSC)., Comment: Published in https://www.nature.com/articles/s41586-023-06829-4