Search

Continuous normalizing flows (CNFs) can model data distributions with expressive infinite-length architectures. But this modeling involves computationally expensive process of solving an ordinary differential equation (ODE) during maximum likelihood training. Recently proposed flow matching (FM) framework allows to substantially simplify the training phase using a regression objective with the interpolated forward vector field. In this paper, we propose an interpolant-free dual flow matching (DFM) approach without explicit assumptions about the modeled vector field. DFM optimizes the forward and, additionally, a reverse vector field model using a novel objective that facilitates bijectivity of the forward and reverse transformations. Our experiments with the SMAP unsupervised anomaly detection show advantages of DFM when compared to the CNF trained with either maximum likelihood or FM objectives with the state-of-the-art performance metrics., Comment: Extended Abstract Track at the Unifying Representations in Neural Models Workshop (NeurIPS 2024)

In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens and require additional training data. Differently, we propose an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs. Concretely, given that visual tokens complement text tokens in VLMs for linguistic reasoning, we select visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix extracted from the VLMs. Then we progressively prune irrelevant tokens. To maximize sparsity while retaining essential information, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that our SparseVLM improves the efficiency of various VLMs across a range of image and video understanding tasks. In particular, LLaVA equipped with SparseVLM reduces 61% to 67% FLOPs with a compression ratio of 78% while maintaining 93% of the accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs., Comment: 17 pages

The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset., Comment: Poster presentation at the 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024)

Normalizing flow-based generative models have been widely used in applications where the exact density estimation is of major importance. Recent research proposes numerous methods to improve their expressivity. However, conditioning on a context is largely overlooked area in the bijective flow research. Conventional conditioning with the vector concatenation is limited to only a few flow types. More importantly, this approach cannot support a practical setup where a set of context-conditioned (specialist) models are trained with the fixed pretrained general-knowledge (generalist) model. We propose ContextFlow++ approach to overcome these limitations using an additive conditioning with explicit generalist-specialist knowledge decoupling. Furthermore, we support discrete contexts by the proposed mixed-variable architecture with context encoders. Particularly, our context encoder for discrete variables is a surjective flow from which the context-conditioned continuous variables are sampled. Our experiments on rotated MNIST-R, corrupted CIFAR-10C, real-world ATM predictive maintenance and SMAP unsupervised anomaly detection benchmarks show that the proposed ContextFlow++ offers faster stable training and achieves higher performance metrics. Our code is publicly available at https://github.com/gudovskiy/contextflow., Comment: Accepted to UAI 2024. Preprint

Finetuning a pretrained vision model (PVM) is a common technique for learning downstream vision tasks. However, the conventional finetuning process with randomly sampled data points results in diminished training efficiency. To address this drawback, we propose a novel approach, Vision-language Collaborative Active Finetuning (VeCAF). With the emerging availability of labels and natural language annotations of images through web-scale crawling or controlled generation, VeCAF makes use of these information to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence to meet the performance goal. This process is assisted by the inherent semantic richness of the text embedding space which we use to augment image features. Furthermore, the flexibility of text-domain augmentation allows VeCAF to handle out-of-distribution scenarios without external data. Extensive experiments show the leading performance and high computational efficiency of VeCAF that is superior to baselines in both in-distribution and out-of-distribution image classification tasks. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.7% over the state-of-the-art active finetuning method with the same number of batches., Comment: 13 pages

The Mixture-of-Experts (MoE) approach has demonstrated outstanding scalability in multi-task learning including low-level upstream tasks such as concurrent removal of multiple adverse weather effects. However, the conventional MoE architecture with parallel Feed Forward Network (FFN) experts leads to significant parameter and computational overheads that hinder its efficient deployment. In addition, the naive MoE linear router is suboptimal in assigning task-specific features to multiple experts which limits its further scalability. In this work, we propose an efficient MoE architecture with weight sharing across the experts. Inspired by the idea of linear feature modulation (FM), our architecture implicitly instantiates multiple experts via learnable activation modulations on a single shared expert block. The proposed Feature Modulated Expert (FME) serves as a building block for the novel Mixture-of-Feature-Modulation-Experts (MoFME) architecture, which can scale up the number of experts with low overhead. We further propose an Uncertainty-aware Router (UaR) to assign task-specific features to different FM modules with well-calibrated weights. This enables MoFME to effectively learn diverse expert functions for multiple tasks. The conducted experiments on the multi-deweather task show that our MoFME outperforms the baselines in the image restoration quality by 0.1-0.2 dB and achieves SOTA-compatible performance while saving more than 72% of parameters and 39% inference time over the conventional MoE counterpart. Experiments on the downstream segmentation and classification tasks further demonstrate the generalizability of MoFME to real open-world applications., Comment: aaai2024

Uncertainty estimation is crucial for machine learning models to detect out-of-distribution (OOD) inputs. However, the conventional discriminative deep learning classifiers produce uncalibrated closed-set predictions for OOD data. A more robust classifiers with the uncertainty estimation typically require a potentially unavailable OOD dataset for outlier exposure training, or a considerable amount of additional memory and compute to build ensemble models. In this work, we improve on uncertainty estimation without extra OOD data or additional inference costs using an alternative Split-Ensemble method. Specifically, we propose a novel subtask-splitting ensemble training objective, where a common multiclass classification task is split into several complementary subtasks. Then, each subtask's training data can be considered as OOD to the other subtasks. Diverse submodels can therefore be trained on each subtask with OOD-aware objectives. The subtask-splitting objective enables us to share low-level features across submodels to avoid parameter and computational overheads. In particular, we build a tree-like Split-Ensemble architecture by performing iterative splitting and pruning from a shared backbone model, where each branch serves as a submodel corresponding to a subtask. This leads to improved accuracy and uncertainty estimation across submodels under a fixed ensemble computation budget. Empirical study with ResNet-18 backbone shows Split-Ensemble, without additional computation cost, improves accuracy over a single model by 0.8%, 1.8%, and 25.5% on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively. OOD detection for the same backbone and in-distribution datasets surpasses a single model baseline by, correspondingly, 2.2%, 8.1%, and 29.6% mean AUROC., Comment: ICML2024. Project website is available at https://antonioo-c.github.io/projects/split-ensemble

Recent semantic segmentation models accurately classify test-time examples that are similar to a training dataset distribution. However, their discriminative closed-set approach is not robust in practical data setups with distributional shifts and out-of-distribution (OOD) classes. As a result, the predicted probabilities can be very imprecise when used as confidence scores at test time. To address this, we propose a generative model for concurrent in-distribution misclassification (IDM) and OOD detection that relies on a normalizing flow framework. The proposed flow-based detector with an energy-based inputs (FlowEneDet) can extend previously deployed segmentation models without their time-consuming retraining. Our FlowEneDet results in a low-complexity architecture with marginal increase in the memory footprint. FlowEneDet achieves promising results on Cityscapes, Cityscapes-C, FishyScapes and SegmentMeIfYouCan benchmarks in IDM/OOD detection when applied to pretrained DeepLabV3+ and SegFormer semantic segmentation models., Comment: Accepted to UAI 2023. Preprint

Recently, DEtection TRansformer (DETR), an end-to-end object detection pipeline, has achieved promising performance. However, it requires large-scale labeled data and suffers from domain shift, especially when no labeled data is available in the target domain. To solve this problem, we propose an end-to-end cross-domain detection Transformer based on the mean teacher framework, MTTrans, which can fully exploit unlabeled target domain data in object detection training and transfer knowledge between domains via pseudo labels. We further propose the comprehensive multi-level feature alignment to improve the pseudo labels generated by the mean teacher framework taking advantage of the cross-scale self-attention mechanism in Deformable DETR. Image and object features are aligned at the local, global, and instance levels with domain query-based feature alignment (DQFA), bi-level graph-based prototype alignment (BGPA), and token-wise image feature alignment (TIFA). On the other hand, the unlabeled target domain data pseudo-labeled and available for the object detection training by the mean teacher framework can lead to better feature extraction and alignment. Thus, the mean teacher framework and the comprehensive multi-level feature alignment can be optimized iteratively and mutually based on the architecture of Transformers. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in three domain adaptation scenarios, especially the result of Sim10k to Cityscapes scenario is remarkably improved from 52.6 mAP to 57.9 mAP. Code will be released., Comment: Accepted by ECCV 2022

In spite of their importance as standard candles in cosmology and as major major sites of nucleosynthesis in the Universe, what kinds of progenitor systems lead to type Ia supernovae (SN) remains a subject of considerable debate in the literature. This is true even for the case of Tycho's SN exploded in 1572 although it has been deeply studied both observationally and theoretically. Analyzing X-ray data of Tycho's supernova remnant (SNR) obtained with Chandra in 2003, 2007, 2009, and 2015, we discover that the expansion before 2007 was substantially faster than radio measurements reported in the past decades and then rapidly decelerated during the last ~ 15 years. The result is well explained if the shock waves recently hit a wall of dense gas surrounding the SNR. Such a gas structure is in fact expected in the so-called single-degenerate scenario, in which the progenitor is a binary system consisting of a white dwarf and a stellar companion, whereas it is not generally predicted by a competing scenario, the double-degenerate scenario, which has a binary of two white dwarfs as the progenitor. Our result thus favors the former scenario. This work also demonstrates a novel technique to probe gas environments surrounding SNRs and thus disentangle the two progenitor scenarios for Type Ia SNe., Comment: 8 pages, 3 figures, 2 tables, accepted for publication in ApJL

Analyzing Chandra data of Tycho's supernova remnant (SNR) taken in 2000, 2003, 2007, 2009, and 2015, we search for time variable features of synchrotron X-rays in the southwestern part of the SNR, where stripe structures of hard X-ray emission were previous found. By comparing X-ray images obtained at each epoch, we discover a knot-like structure in the northernmost part of the stripe region became brighter particularly in 2015. We also find a bright filamentary structure gradually became fainter and narrower as it moved outward. Our spectral analysis reveal that not only the nonthermal X-ray flux but also the photon indices of the knot-like structure change from year to year. During the period from 2000 to 2015, the small knot shows brightening of $\sim 70\%$ and hardening of $\Delta \Gamma \sim 0.45$. The time variability can be explained if the magnetic field is amplified to $\sim 100~\mathrm{\mu G}$ and/or if magnetic turbulence significantly changes with time., Comment: 8 pages, 3 figures, 2 tables, accepted for publication in ApJ

We report on a measurement of the size of charge clouds produced by X-ray photons in X-ray SOI (Silicon-On-Insulator) pixel sensor named XRPIX. We carry out a beam scanning experiment of XRPIX using a monochromatic X-ray beam at 5.0 keV collimated to $\sim 10$ $\mu$m with a 4-$\mu$m$\phi$ pinhole, and obtain the spatial distribution of single-pixel events at a sub-pixel scale. The standard deviation of charge clouds of 5.0 keV X-ray is estimated to be $\sigma_{\rm cloud} = 4.30 \pm 0.07$ $\mu$m. Compared to the detector response simulation, the estimated charge cloud size is well explained by a combination of photoelectron range, thermal diffusion, and Coulomb repulsion. Moreover, by analyzing the fraction of multi-pixel events in various energies, we find that the energy dependence of the charge cloud size is also consistent with the simulation., Comment: 9 pages, 12 figures, accepted for publication in IEEE Transactions on Nuclear Science

We have been developing a monolithic active pixel sensor, ``XRPIX``, for the Japan led future X-ray astronomy mission ``FORCE`` observing the X-ray sky in the energy band of 1-80 keV with angular resolution of better than 15``. XRPIX is an upper part of a stack of two sensors of an imager system onboard FORCE, and covers the X-ray energy band lower than 20 keV. The XRPIX device consists of a fully depleted high-resistivity silicon sensor layer for X-ray detection, a low resistivity silicon layer for CMOS readout circuit, and a buried oxide layer in between, which is fabricated with 0.2 $\mu$ m CMOS silicon-on-insulator (SOI) technology. Each pixel has a trigger circuit with which we can achieve a 10 $\mu$ s time resolution, a few orders of magnitude higher than that with X-ray astronomy CCDs. We recently introduced a new type of a device structure, a pinned depleted diode (PDD), in the XRPIX device, and succeeded in improving the spectral performance, especially in a readout mode using the trigger function. In this paper, we apply a mesh experiment to the XRPIX devices for the first time in order to study the spectral response of the PDD device at the subpixel resolution. We confirmed that the PDD structure solves the significant degradation of the charge collection efficiency at the pixel boundaries and in the region under the pixel circuits, which is found in the single SOI structure, the conventional type of the device structure. On the other hand, the spectral line profiles are skewed with low energy tails and the line peaks slightly shift near the pixel boundaries, which contribute to a degradation of the energy resolution., Comment: 10 pages, 7 figures, , Conference proceedings for PIXEL2018

We have been developing monolithic active pixel sensors for X-rays based on the silicon-on-insulator technology. Our device consists of a low-resistivity Si layer for readout CMOS electronics, a high-resistivity Si sensor layer, and a SiO$_2$ layer between them. This configuration allows us both high-speed readout circuits and a thick (on the order of $100~\mu{\rm m}$) depletion layer in a monolithic device. Each pixel circuit contains a trigger output function, with which we can achieve a time resolution of $\lesssim 10~\mu{\rm s}$. One of our key development items is improvement of the energy resolution. We recently fabricated a device named XRPIX6E, to which we introduced a pinned depleted diode (PDD) structure. The structure reduces the capacitance coupling between the sensing area in the sensor layer and the pixel circuit, which degrades the spectral performance. With XRPIX6E, we achieve an energy resolution of $\sim 150$~eV in full width at half maximum for 6.4-keV X-rays. In addition to the good energy resolution, a large imaging area is required for practical use. We developed and tested XRPIX5b, which has an imaging area size of $21.9~{\rm mm} \times 13.8~{\rm mm}$ and is the largest device that we ever fabricated. We successfully obtain X-ray data from almost all the $608 \times 384$ pixels with high uniformity., Comment: 5 pages, 9 figures, submitted to Conference Record of IEEE NSS-MIC 2018

The young shell-type supernova remnant (SNR) RX J1713.7-3946 has been studied as a suitable target to test the SNR paradigm for the origin of Galactic cosmic rays. We present a spatially resolved spectroscopy of the nonthermal X-ray emission in RX J1713.7-3946 with Chandra. In order to obtain X-ray properties of the filamentary structures and their surrounding regions, we divide the southeastern (SE), southwestern (SW), and northwestern (NW) parts of the SNR into subregions on the typical order of several 10" and extract spectra from each subregion. Their photon indices are significantly different among the subregions with a range of 1.8 < {\Gamma} < 3. In the SE part, the clear filaments are harder ({\Gamma} ~ 2.0) than the surrounding regions. This is a common feature often observed in young SNRs and naturally interpreted as a consequence of synchrotron cooling. On the other hand, the bright filamentary regions do not necessarily coincide with the hardest regions in the SW and NW parts. We also find the SW filamentary region is rather relatively soft ({\Gamma} ~ 2.7). In addition, we find that hard regions with photon indices of 2.0-2.2 exist around the bright emission although they lie in the downstream region and does not appear to be the blast wave shock front. Both two aforementioned characteristic regions in SW are located close to peaks of the interstellar gas. We discuss possible origins of the spatial variation of the photon indices, paying particular attention to the shock-cloud interactions., Comment: 8 pages, 7 figures, 1 table, accepted for publication in PASJ on May 17, 2018