Search

Using multiple spatial modalities has been proven helpful in improving semantic segmentation performance. However, there are several real-world challenges that have yet to be addressed: (a) improving label efficiency and (b) enhancing robustness in realistic scenarios where modalities are missing at the test time. To address these challenges, we first propose a simple yet efficient multi-modal fusion mechanism Linear Fusion, that performs better than the state-of-the-art multi-modal models even with limited supervision. Second, we propose M3L: Multi-modal Teacher for Masked Modality Learning, a semi-supervised framework that not only improves the multi-modal performance but also makes the model robust to the realistic missing modality scenario using unlabeled data. We create the first benchmark for semi-supervised multi-modal semantic segmentation and also report the robustness to missing modalities. Our proposal shows an absolute improvement of up to 10% on robust mIoU above the most competitive baselines. Our code is available at https://github.com/harshm121/M3L

Recent developments for Semi-Supervised Object Detection (SSOD) have shown the promise of leveraging unlabeled data to improve an object detector. However, thus far these methods have assumed that the unlabeled data does not contain out-of-distribution (OOD) classes, which is unrealistic with larger-scale unlabeled datasets. In this paper, we consider a more practical yet challenging problem, Open-Set Semi-Supervised Object Detection (OSSOD). We first find the existing SSOD method obtains a lower performance gain in open-set conditions, and this is caused by the semantic expansion, where the distracting OOD objects are mispredicted as in-distribution pseudo-labels for the semi-supervised training. To address this problem, we consider online and offline OOD detection modules, which are integrated with SSOD methods. With the extensive studies, we found that leveraging an offline OOD detector based on a self-supervised vision transformer performs favorably against online OOD detectors due to its robustness to the interference of pseudo-labeling. In the experiment, our proposed framework effectively addresses the semantic expansion issue and shows consistent improvements on many OSSOD benchmarks, including large-scale COCO-OpenImages. We also verify the effectiveness of our framework under different OSSOD conditions, including varying numbers of in-distribution classes, different degrees of supervision, and different combinations of unlabeled sets., Comment: Project Page is at https://ycliu93.github.io/projects/ossod.html

Adapting large-scale pretrained models to various downstream tasks via fine-tuning is a standard method in machine learning. Recently, parameter-efficient fine-tuning methods show promise in adapting a pretrained model to different tasks while training only a few parameters. Despite their success, most existing methods are proposed in Natural Language Processing tasks with language Transformers, and adaptation to Computer Vision tasks with Vision Transformers remains under-explored, especially for dense vision tasks. Further, in multi-task settings, individually fine-tuning and storing separate models for different tasks is inefficient. In this work, we provide an extensive multi-task parameter-efficient benchmark and examine existing parameter-efficient fine-tuning NLP methods for vision tasks. Our results on four different dense vision tasks showed that existing methods cannot be efficiently integrated due to the hierarchical nature of the Hierarchical Vision Transformers. To overcome this issue, we propose Polyhistor and Polyhistor-Lite, consisting of Decomposed HyperNetworks and Layer-wise Scaling Kernels, to share information across different tasks with a few trainable parameters. This leads to favorable performance improvements against existing parameter-efficient methods while using fewer trainable parameters. Specifically, Polyhistor achieves competitive accuracy compared to the state-of-the-art while only using ~10% of their trainable parameters. Furthermore, our methods show larger performance gains when large networks and more pretraining data are used., Comment: Accepted to NeurIPS 2022; Project Page is at https://ycliu93.github.io/projects/polyhistor.html

With the recent development of Semi-Supervised Object Detection (SS-OD) techniques, object detectors can be improved by using a limited amount of labeled data and abundant unlabeled data. However, there are still two challenges that are not addressed: (1) there is no prior SS-OD work on anchor-free detectors, and (2) prior works are ineffective when pseudo-labeling bounding box regression. In this paper, we present Unbiased Teacher v2, which shows the generalization of SS-OD method to anchor-free detectors and also introduces Listen2Student mechanism for the unsupervised regression loss. Specifically, we first present a study examining the effectiveness of existing SS-OD methods on anchor-free detectors and find that they achieve much lower performance improvements under the semi-supervised setting. We also observe that box selection with centerness and the localization-based labeling used in anchor-free detectors cannot work well under the semi-supervised setting. On the other hand, our Listen2Student mechanism explicitly prevents misleading pseudo-labels in the training of bounding box regression; we specifically develop a novel pseudo-labeling selection mechanism based on the Teacher and Student's relative uncertainties. This idea contributes to favorable improvement in the regression branch in the semi-supervised setting. Our method, which works for both anchor-free and anchor-based methods, consistently performs favorably against the state-of-the-art methods in VOC, COCO-standard, and COCO-additional., Comment: Project Page is at http://ycliu93.github.io/projects/unbiasedteacher2.html

Neural Networks can perform poorly when the training label distribution is heavily imbalanced, as well as when the testing data differs from the training distribution. In order to deal with shift in the testing label distribution, which imbalance causes, we motivate the problem from the perspective of an optimal Bayes classifier and derive a post-training prior rebalancing technique that can be solved through a KL-divergence based optimization. This method allows a flexible post-training hyper-parameter to be efficiently tuned on a validation set and effectively modify the classifier margin to deal with this imbalance. We further combine this method with existing likelihood shift methods, re-interpreting them from the same Bayesian perspective, and demonstrating that our method can deal with both problems in a unified way. The resulting algorithm can be conveniently used on probabilistic classification problems agnostic to underlying architectures. Our results on six different datasets and five different architectures show state of art accuracy, including on large-scale imbalanced datasets such as iNaturalist for classification and Synthia for semantic segmentation. Please see https://github.com/GT-RIPL/UNO-IC.git for implementation., Comment: Accepted to NeurIPS 2020

Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mainstream methods. We conclude with several suggestions for creating harder evaluation scenarios and future research directions. The code is available at https://github.com/GT-RIPL/Continual-Learning-Benchmark, Comment: Continual Learning Workshop, 32nd Conference on Neural Information Processing Systems (NIPS 2018)

We present a novel and unified deep learning framework which is capable of learning domain-invariant representation from data across multiple domains. Realized by adversarial training with additional ability to exploit domain-specific information, the proposed network is able to perform continuous cross-domain image translation and manipulation, and produces desirable output images accordingly. In addition, the resulting feature representation exhibits superior performance of unsupervised domain adaptation, which also verifies the effectiveness of the proposed model in learning disentangled features for describing cross-domain data., Comment: NIPS 2018

To detect spatially-modulated signals, a receiver needs the channel state information (CSI) of each transmit- receive antenna pair. Although the CSI is never perfect and varies in time, most studies on spatial modulation (SM) systems assume perfectly known CSI and time-invariant channel. The spatial correlations among multiple spatial subchannels, which have to be considered when CSI is imperfect, are also often neglected. In this paper, we release the above assumptions and take the CSI uncertainty along with the spatial-temporal selectivities into account. We derive the channel estimation error aware maximum likelihood (CEEA-ML) detectors as well as several low- complexity alternatives for PSK and QAM signals. As the CSI uncertainty depends on the channel estimator used, we consider both decision feedback and model based estimators in our study. The error rate performance of the ML and some suboptimal detectors is analyzed. Numerical results obtained by simulations and analysis show that the CEEA-ML detectors offer clear performance gain against conventional mismatched SM detectors and, in many cases, the proposed suboptimal detectors incur only minor performance loss., Comment: 15 pages, 9 figures, 4 tables, submitted to IEEE Trans. Veh. Techol. in May 2017

We present two reduced-rank channel estimators for large-scale multiple-input, multiple-output (MIMO) systems and analyze their mean square error (MSE) performance. Taking advantage of the channel's transform domain sparseness, the estimators yield outstanding performance and may offer additional mean angle-of-arrival (AoA) information. It is shown that, for the estimators to be effective, one has to select a proper predetermined unitary basis (transform) and be able to determine the dominant channel rank and the associated subspace. Further MSE analysis reveals the relations among the array size, channel rank, signal-to-noise ratio (SNR), and the estimators' performance. It provides rationales for the proposed rank determination and mean AoA estimation algorithms as well. An angle alignment operation included in one of our channel models is proved to be effective in further reducing the required rank, shifting the dominant basis vectors' range (channel spectrum) and improving the estimator's performance when a suitable basis is used. We also draw insightful analogies among MIMO channel modeling, transform coding, parallel spatial search, and receive beamforming. Computer experiment results are provided to examine the numerical effects of various estimator parameters and the advantages of the proposed channel estimators and rank determination method., Comment: 12 pages, 10 figures, submitted to IEEE Trans. Wireless Commun

We consider a multiuser (MU) multiple-input multiple-output (MIMO) time-division duplexing (TDD) system in which the base station (BS) is equipped with a large number of antennas for communicating with single-antenna mobile users. In such a system the BS has to estimate the channel state information (CSI) that includes large-scale fading coefficients (LSFCs) and small-scale fading coefficients (SSFCs) by uplink pilots. Although information about the former FCs are indispensable in a MU-MIMO or distributed MIMO system, they are usually ignored or assumed perfectly known when treating the MIMO CSI estimation problem. We take advantage of the large spatial samples of a massive MIMO BS to derive accurate LSFC estimates in the absence of SSFC information. With estimated LSFCs, SSFCs are then obtained using a rank-reduced (RR) channel model which in essence transforms the channel vector into a lower dimension representation. We analyze the mean squared error (MSE) performance of the proposed composite channel estimator and prove that the separable angle of arrival (AoA) information provided by the RR model is beneficial for enhancing the estimator's performance, especially when the angle spread of the uplink signal is not too large., Comment: 11 pages, 9 figures, submitted to IEEE Journal of Selected Topics in Signal Processing