Search

As deep neural networks evolve from convolutional neural networks (ConvNets) to advanced vision transformers (ViTs), there is an increased need to eliminate redundant data for faster processing without compromising accuracy. Previous methods are often architecture-specific or necessitate re-training, restricting their applicability with frequent model updates. To solve this, we first introduce a novel property of lightweight ConvNets: their ability to identify key discriminative patch regions in images, irrespective of model's final accuracy or size. We demonstrate that fully-connected layers are the primary bottleneck for ConvNets performance, and their suppression with simple weight recalibration markedly enhances discriminative patch localization performance. Using this insight, we introduce PaPr, a method for substantially pruning redundant patches with minimal accuracy loss using lightweight ConvNets across a variety of deep learning architectures, including ViTs, ConvNets, and hybrid transformers, without any re-training. Moreover, the simple early-stage one-step patch pruning with PaPr enhances existing patch reduction methods. Through extensive testing on diverse architectures, PaPr achieves significantly higher accuracy over state-of-the-art patch reduction methods with similar FLOP count reduction. More specifically, PaPr reduces about 70% of redundant patches in videos with less than 0.8% drop in accuracy, and up to 3.7x FLOPs reduction, which is a 15% more reduction with 2.5% higher accuracy. Code is released at https://github.com/tanvir-utexas/PaPr., Comment: Accepted in ECCV 2024. Code: https://github.com/tanvir-utexas/PaPr

The rapid advancement of Large AI Models (LAIMs), particularly diffusion models and large language models, has marked a new era where AI-generated multimedia is increasingly integrated into various aspects of daily life. Although beneficial in numerous fields, this content presents significant risks, including potential misuse, societal disruptions, and ethical concerns. Consequently, detecting multimedia generated by LAIMs has become crucial, with a marked rise in related research. Despite this, there remains a notable gap in systematic surveys that focus specifically on detecting LAIM-generated multimedia. Addressing this, we provide the first survey to comprehensively cover existing research on detecting multimedia (such as text, images, videos, audio, and multimodal content) created by LAIMs. Specifically, we introduce a novel taxonomy for detection methods, categorized by media modality, and aligned with two perspectives: pure detection (aiming to enhance detection performance) and beyond detection (adding attributes like generalizability, robustness, and interpretability to detectors). Additionally, we have presented a brief overview of generation mechanisms, public datasets, and online detection tools to provide a valuable resource for researchers and practitioners in this field. Furthermore, we identify current challenges in detection and propose directions for future research that address unexplored, ongoing, and emerging issues in detecting multimedia generated by LAIMs. Our aim for this survey is to fill an academic gap and contribute to global AI security efforts, helping to ensure the integrity of information in the digital realm. The project link is https://github.com/Purdue-M2/Detect-LAIM-generated-Multimedia-Survey.

Despite significant progress in semi-supervised learning for image object detection, several key issues are yet to be addressed for video object detection: (1) Achieving good performance for supervised video object detection greatly depends on the availability of annotated frames. (2) Despite having large inter-frame correlations in a video, collecting annotations for a large number of frames per video is expensive, time-consuming, and often redundant. (3) Existing semi-supervised techniques on static images can hardly exploit the temporal motion dynamics inherently present in videos. In this paper, we introduce SSVOD, an end-to-end semi-supervised video object detection framework that exploits motion dynamics of videos to utilize large-scale unlabeled frames with sparse annotations. To selectively assemble robust pseudo-labels across groups of frames, we introduce \textit{flow-warped predictions} from nearby frames for temporal-consistency estimation. In particular, we introduce cross-IoU and cross-divergence based selection methods over a set of estimated predictions to include robust pseudo-labels for bounding boxes and class labels, respectively. To strike a balance between confirmation bias and uncertainty noise in pseudo-labels, we propose confidence threshold based combination of hard and soft pseudo-labels. Our method achieves significant performance improvements over existing methods on ImageNet-VID, Epic-KITCHENS, and YouTube-VIS datasets. Code and pre-trained models will be released.

We propose an embarrassingly simple method -- instance-aware repeat factor sampling (IRFS) to address the problem of imbalanced data in long-tailed object detection. Imbalanced datasets in real-world object detection often suffer from a large disparity in the number of instances for each class. To improve the generalization performance of object detection models on rare classes, various data sampling techniques have been proposed. Repeat factor sampling (RFS) has shown promise due to its simplicity and effectiveness. Despite its efficiency, RFS completely neglects the instance counts and solely relies on the image count during re-sampling process. However, instance count may immensely vary for different classes with similar image counts. Such variation highlights the importance of both image and instance for addressing the long-tail distributions. Thus, we propose IRFS which unifies instance and image counts for the re-sampling process to be aware of different perspectives of the imbalance in long-tailed datasets. Our method shows promising results on the challenging LVIS v1.0 benchmark dataset over various architectures and backbones, demonstrating their effectiveness in improving the performance of object detection models on rare classes with a relative $+50\%$ average precision (AP) improvement over counterpart RFS. IRFS can serve as a strong baseline and be easily incorporated into existing long-tailed frameworks.

We introduce a new type of autonomous vehicle - an autonomous dozer that is expected to complete construction site tasks in an efficient, robust, and safe manner. To better handle the path planning for the dozer and ensure construction site safety, object detection plays one of the most critical components among perception tasks. In this work, we first collect the construction site data by driving around our dozers. Then we analyze the data thoroughly to understand its distribution. Finally, two well-known object detection models are trained, and their performances are benchmarked with a wide range of training strategies and hyperparameters.

Semi-supervised object detection methods are widely used in autonomous driving systems, where only a fraction of objects are labeled. To propagate information from the labeled objects to the unlabeled ones, pseudo-labels for unlabeled objects must be generated. Although pseudo-labels have proven to improve the performance of semi-supervised object detection significantly, the applications of image-based methods to video frames result in numerous miss or false detections using such generated pseudo-labels. In this paper, we propose a new approach, PseudoProp, to generate robust pseudo-labels by leveraging motion continuity in video frames. Specifically, PseudoProp uses a novel bidirectional pseudo-label propagation approach to compensate for misdetection. A feature-based fusion technique is also used to suppress inference noise. Extensive experiments on the large-scale Cityscapes dataset demonstrate that our method outperforms the state-of-the-art semi-supervised object detection methods by 7.4% on mAP75., Comment: Accepted by the Workshop on Autonomous Driving (WAD) at CVPR 2022

Understanding scenes in movies is crucial for a variety of applications such as video moderation, search, and recommendation. However, labeling individual scenes is a time-consuming process. In contrast, movie level metadata (e.g., genre, synopsis, etc.) regularly gets produced as part of the film production process, and is therefore significantly more commonly available. In this work, we propose a novel contrastive learning approach that uses movie metadata to learn a general-purpose scene representation. Specifically, we use movie metadata to define a measure of movie similarity, and use it during contrastive learning to limit our search for positive scene-pairs to only the movies that are considered similar to each other. Our learned scene representation consistently outperforms existing state-of-the-art methods on a diverse set of tasks evaluated using multiple benchmark datasets. Notably, our learned representation offers an average improvement of 7.9% on the seven classification tasks and 9.7% improvement on the two regression tasks in LVU dataset. Furthermore, using a newly collected movie dataset, we present comparative results of our scene representation on a set of video moderation tasks to demonstrate its generalizability on previously less explored tasks., Comment: Accepted to CVPR 2023

Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks., Comment: ICLR'19 Camera Ready

Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being deployed on certain computing environment (e.g., with limited power budgets). We propose MONAS, a framework for Multi-Objective Neural Architectural Search that employs reward functions considering both prediction accuracy and other important objectives (e.g., power consumption) when searching for neural network architectures. Experimental results showed that, compared to the state-ofthe-arts, models found by MONAS achieve comparable or better classification accuracy on computer vision applications, while satisfying the additional objectives such as peak power.

Eigenvalue-based detectors are considered as an important method of spectrum sensing since they do not require the information about the primary user (PU) signal. In this paper we propose a method to improve the performance of the eigenvalue-based detector. The proposed method introduces a new test statistic based on combinatorial matrix with components which are overlapping subgroups extracted from the array of received signals. As a result, its covariance matrix has a larger maximum eigenvalue and trace value than the one without overlapping. Simulation results show that our proposed method can further improve the detection performance of the optimal eigenvalue-based detector. The paper also shows the effect of different overlapping methods on the receiver operating characteristic curve., Comment: conference

Dynamic spectrum access (DSA) enables secondary networks to find and efficiently exploit spectrum opportunities. A key factor to design a DSA network is the spectrum sensing algorithms for multiple channels with multiple users. Multi-user cooperative channel sensing reduces the sensing time, and thus it increases transmission throughput. However, in a multi-channel system, the problem becomes more complex since the benefits of assigning users to sense channels in parallel must also be considered. A sensing schedule, indicating to each user the channel that it should sense at different sensing moments, must be thus created to optimize system performance. In this paper, we formulate the general sensing scheduling optimization problem and then propose several sensing strategies to schedule the users according to network parameters with homogeneous sensors. Later on we extend the results to heterogeneous sensors and propose a robust scheduling design when we have traffic and channel uncertainty. We propose three sensing strategies, and, within each one of them, several solutions, striking a balance between throughput performance and computational complexity, are proposed. In addition, we show that a sequential channel sensing strategy is the one to be preferred when the sensing time is small, the number of channels is large, and the number of users is small. For all the other cases, a parallel channel sensing strategy is recommended in terms of throughput performance. We also show that a proposed hybrid sequential-parallel channel sensing strategy achieves the best performance in all scenarios at the cost of extra memory and computation complexity.

This paper focuses on analytical studies of the primary user (PU) traffic classification problem. Observing that the gamma distribution can represent positively skewed data and exponential distribution (popular in communication networks performance analysis literature) it is considered here as the PU traffic descriptor. We investigate two PU traffic classifiers utilizing perfectly measured PU activity (busy) and inactivity (idle) periods: (i) maximum likelihood classifier (MLC) and (ii) multi-hypothesis sequential probability ratio test classifier (MSPRTC). Then, relaxing the assumption on perfect period measurement, we consider a PU traffic observation through channel sampling. For a special case of negligible probability of PU state change in between two samplings, we propose a minimum variance PU busy/idle period length estimator. Later, relaxing the assumption of the complete knowledge of the parameters of the PU period length distribution, we propose two PU traffic classification schemes: (i) estimate-then-classify (ETC), and (ii) average likelihood function (ALF) classifiers considering time domain fluctuation of the PU traffic parameters. Numerical results show that both MLC and MSPRTC are sensitive to the periods measurement errors when the distance among distribution hypotheses is small, and to the distribution parameter estimation errors when the distance among hypotheses is large. For PU traffic parameters with a partial prior knowledge of the distribution, the ETC outperforms ALF when the distance among hypotheses is small, while the opposite holds when the distance is large., Comment: Accepted to IEEE Journal on Selected Areas in Communications; Preliminary version appeared in Proc. IEEE GLOBECOM, Dec. 9-13, 2013, Atlanta, GA, USA

In this work we investigate the bounds on the estimation accuracy of Primary User (PU) traffic parameters with exponentially distributed busy and idle times. We derive closed-form expressions for the Cramer-Rao bounds on the mean squared estimation error for the blind joint estimation of the PU traffic parameters, specifically, the duty cycle, and the mean arrival and departure rates. Moreover, we present the corresponding maximum-likelihood estimators for the traffic parameters. In addition, we derive a modified likelihood function for the joint estimation of traffic parameters when spectrum sensing errors are considered, and we present the impact of spectrum sensing errors on the estimation error via simulations. Finally, we consider a duty cycle estimator, common in traffic estimation literature, that is based on averaging the traffic samples. We derive, in closed-form, the mean squared estimation error of the considered estimator under spectrum sensing errors., Comment: accepted to IEEE ICC 2013 [Extended version, with appendices]

Accurate estimation of licensed channel Primary User's (PU) temporal statistics is important for Dynamic Spectrum Access (DSA) systems. With accurate estimation of the mean duty cycle, u, and the mean off- and on-times of PUs, DSA systems can more efficiently assign PU resources to its subscribers, thus, increasing channel utilization. This paper presents a mathematical analysis of the accuracy of estimating u, as well as the PU mean off- and on-times, where the estimation accuracy is expressed as the mean squared estimation error. The analysis applies for the traffic model assuming exponentially distributed PU off- and on-times, which is a common model in traffic literature. The estimation accuracy is quantified as a function of the number of samples and observation window length, hence, this work provides guidelines on traffic parameters estimation for both energy-constrained and delay-constrained applications. For estimating u, we consider uniform, non-uniform, and weighted sample stream averaging, as well as maximum likelihood estimation. The estimation accuracy of the mean PU off- and on-times is studied when maximum likelihood estimation is employed. Furthermore, we develop algorithms for the blind estimation of the traffic parameters based on the derived theoretical estimation accuracy expressions. We show that the estimation error for all traffic parameters is lower bounded for a fixed observation window length due to the correlation between the traffic samples. Moreover, we prove that for estimating u, maximum likelihood estimation can yield the same estimation error as weighted sample averaging using only half the observation window length.