Search

Weakly-Supervised Group Activity Recognition (WSGAR) aims to understand the activity performed together by a group of individuals with the video-level label and without actor-level labels. We propose Flow-Assisted Motion Learning Network (Flaming-Net) for WSGAR, which consists of the motion-aware actor encoder to extract actor features and the two-pathways relation module to infer the interaction among actors and their activity. Flaming-Net leverages an additional optical flow modality in the training stage to enhance its motion awareness when finding locally active actors. The first pathway of the relation module, the actor-centric path, initially captures the temporal dynamics of individual actors and then constructs inter-actor relationships. In parallel, the group-centric path starts by building spatial connections between actors within the same timeframe and then captures simultaneous spatio-temporal dynamics among them. We demonstrate that Flaming-Net achieves new state-of-the-art WSGAR results on two benchmarks, including a 2.8%p higher MPCA score on the NBA dataset. Importantly, we use the optical flow modality only for training and not for inference.

Due to the distinctive characteristics of sensors, each modality exhibits unique physical properties. For this reason, in the context of multi-modal action recognition, it is important to consider not only the overall action content but also the complementary nature of different modalities. In this paper, we propose a novel network, named Modality Mixer (M-Mixer) network, which effectively leverages and incorporates the complementary information across modalities with the temporal context of actions for action recognition. A key component of our proposed M-Mixer is the Multi-modal Contextualization Unit (MCU), a simple yet effective recurrent unit. Our MCU is responsible for temporally encoding a sequence of one modality (e.g., RGB) with action content features of other modalities (e.g., depth and infrared modalities). This process encourages M-Mixer network to exploit global action content and also to supplement complementary information of other modalities. Furthermore, to extract appropriate complementary information regarding to the given modality settings, we introduce a new module, named Complementary Feature Extraction Module (CFEM). CFEM incorporates sepearte learnable query embeddings for each modality, which guide CFEM to extract complementary information and global action content from the other modalities. As a result, our proposed method outperforms state-of-the-art methods on NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA datasets. Moreover, through comprehensive ablation studies, we further validate the effectiveness of our proposed method., Comment: arXiv admin note: substantial text overlap with arXiv:2208.11314

This paper describes the design of a linear quadratic gaussian (LQG) for trajectory tracking control of UAV Bicopter. In this work, disturbance in the form of payload significantly affects the trajectory tracking control process on the UAV Bicopter when using a linear quadratic regulator (LQR) controller. The use of a LQR control will be optimal in the case of a state regulator towards an equilibrium point in a system, but for the tracking case, the LQR controller is not capable of optimally, especially in systems that have high levels of nonlinearity and system dynamic changes such as inertial disturbances. Therefore, this paper proposes the design of a LQG control that is expected to overcome system dynamic changes, in this case in the form of inertial disturbances to the UAV Bicopter when carrying a payload. The success of LQG control was tested in two scenarios, the first trajectory tracking at a circular position and the second with the position of the trajectory number "8". The simulation results show that the proposed LQG controller successfully overcame inertial disturbances when the UAV Bicopter performs trajectory tracking. When given an inertial disturbance, the trajectory tracking test results show that the LQG control has a lower root mean square error (RMSE) value than the LQR control.

This paper present a control system for the attitude and low cost design of a Bicopter. The control system uses a PID controller that receives feedback from an IMU to calculate control inputs that adjust the Bicopters attitude (roll, pitch and yaw angles) which is resistant to disturbances (wind noise) on a test bed. The control system is implemented on a hardware platform consisting of a Bicopter, an IMU sensor, and a microcontroller with low cost design. In mechanical design, the Bicopter is designed to more closely resemble the letter "V" so that the distribution of the centre of mass (CoM) of the Bicopter can be such that the servomotor torque reaction is parallel to the axis of rotation of the Bicopter during the movement of the pitch angle attitude. In electronic design, the Bicopter was developed using the ATmega328P microcontroller.

Deep learning has made significant strides in video understanding tasks, but the computation required to classify lengthy and massive videos using clip-level video classifiers remains impractical and prohibitively expensive. To address this issue, we propose Audio-Visual Glance Network (AVGN), which leverages the commonly available audio and visual modalities to efficiently process the spatio-temporally important parts of a video. AVGN firstly divides the video into snippets of image-audio clip pair and employs lightweight unimodal encoders to extract global visual features and audio features. To identify the important temporal segments, we use an Audio-Visual Temporal Saliency Transformer (AV-TeST) that estimates the saliency scores of each frame. To further increase efficiency in the spatial dimension, AVGN processes only the important patches instead of the whole images. We use an Audio-Enhanced Spatial Patch Attention (AESPA) module to produce a set of enhanced coarse visual features, which are fed to a policy network that produces the coordinates of the important patches. This approach enables us to focus only on the most important spatio-temporally parts of the video, leading to more efficient video recognition. Moreover, we incorporate various training techniques and multi-modal feature fusion to enhance the robustness and effectiveness of our AVGN. By combining these strategies, our AVGN sets new state-of-the-art performance in multiple video recognition benchmarks while achieving faster processing speed., Comment: ICCV 2023

Indonesia holds the second-highest-ranking country for the highest number of malaria cases in Southeast Asia. A different malaria parasite semantic segmentation technique based on a deep learning approach is an alternative to reduce the limitations of traditional methods. However, the main problem of the semantic segmentation technique is raised since large parasites are dominant, and the tiny parasites are suppressed. In addition, the amount and variance of data are important influences in establishing their models. In this study, we conduct two contributions. First, we collect 559 microscopic images containing 691 malaria parasites of thin blood smears. The dataset is named PlasmoID, and most data comes from rural Indonesia. PlasmoID also provides ground truth for parasite detection and segmentation purposes. Second, this study proposes a malaria parasite segmentation and detection scheme by combining Faster RCNN and a semantic segmentation technique. The proposed scheme has been evaluated on the PlasmoID dataset. It has been compared with recent studies of semantic segmentation techniques, namely UNet, ResFCN-18, DeepLabV3, DeepLabV3plus and ResUNet-18. The result shows that our proposed scheme can improve the segmentation and detection of malaria parasite performance compared to original semantic segmentation techniques.

Standard multi-modal models assume the use of the same modalities in training and inference stages. However, in practice, the environment in which multi-modal models operate may not satisfy such assumption. As such, their performances degrade drastically if any modality is missing in the inference stage. We ask: how can we train a model that is robust to missing modalities? This paper seeks a set of good practices for multi-modal action recognition, with a particular interest in circumstances where some modalities are not available at an inference time. First, we study how to effectively regularize the model during training (e.g., data augmentation). Second, we investigate on fusion methods for robustness to missing modalities: we find that transformer-based fusion shows better robustness for missing modality than summation or concatenation. Third, we propose a simple modular network, ActionMAE, which learns missing modality predictive coding by randomly dropping modality features and tries to reconstruct them with the remaining modality features. Coupling these good practices, we build a model that is not only effective in multi-modal action recognition but also robust to modality missing. Our model achieves the state-of-the-arts on multiple benchmarks and maintains competitive performances even in missing modality scenarios. Codes are available at https://github.com/sangminwoo/ActionMAE., Comment: AAAI 2023 (Oral); Code: https://github.com/sangminwoo/ActionMAE

In multi-modal action recognition, it is important to consider not only the complementary nature of different modalities but also global action content. In this paper, we propose a novel network, named Modality Mixer (M-Mixer) network, to leverage complementary information across modalities and temporal context of an action for multi-modal action recognition. We also introduce a simple yet effective recurrent unit, called Multi-modal Contextualization Unit (MCU), which is a core component of M-Mixer. Our MCU temporally encodes a sequence of one modality (e.g., RGB) with action content features of other modalities (e.g., depth, IR). This process encourages M-Mixer to exploit global action content and also to supplement complementary information of other modalities. As a result, our proposed method outperforms state-of-the-art methods on NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA datasets. Moreover, we demonstrate the effectiveness of M-Mixer by conducting comprehensive ablation studies., Comment: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023

In recent years, a ton of research has been conducted on real image denoising tasks. However, the efforts are more focused on improving real image denoising through creating a better network architecture. We explore a different direction where we propose to improve real image denoising performance through a better learning strategy that can enable test-time adaptation on the multi-task network. The learning strategy is two stages where the first stage pre-train the network using meta-auxiliary learning to get better meta-initialization. Meanwhile, we use meta-learning for fine-tuning (meta-transfer learning) the network as the second stage of our training to enable test-time adaptation on real noisy images. To exploit a better learning strategy, we also propose a network architecture with self-supervised masked reconstruction loss. Experiments on a real noisy dataset show the contribution of the proposed method and show that the proposed method can outperform other SOTA methods.

Accurate diagnosis is required before performing proper treatments for coronary heart disease. Machine learning based approaches have been proposed by many researchers to improve the accuracy of coronary heart disease diagnosis. Ensemble learning and cascade generalization are among the methods which can be used to improve the generalization ability of learning algorithm. The objective of this study is to develop heart disease diagnosis method based on ensemble learning and cascade generalization. Cascade generalization method with loose coupling strategy is proposed in this study. C4. 5 and RIPPER algorithm were used as meta-level algorithm and Naive Bayes was used as baselevel algorithm. Bagging and Random Subspace were evaluated for constructing the ensemble. The hybrid cascade ensemble methods are compared with the learning algorithms in non-ensemble mode and non-cascade mode. The methods are also compared with Rotation Forest. Based on the evaluation result, the hybrid cascade ensemble method demonstrated the best result for the given heart disease diagnosis case. Accuracy and diversity evaluation was performed to analyze the impact of the cascade strategy. Based on the result, the accuracy of the classifiers in the ensemble is increased but the diversity is decreased.

Optical Coherence Tomography allows ophthalmologist to obtain cross-section imaging of eye retina. Assisted with digital image analysis methods, effective disease detection could be performed. Various methods exist to extract feature from OCT images. The proposed study aims to compare the effectiveness of handcrafted and deep neural network features. The evaluated dataset consist of 32339 instances distributed in four classes, namely CNV, DME, DRUSEN, and NORMAL. The methods are Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), DenseNet-169, and ResNet50. As a result, the deep neural network based methods outperformed the handcrafted feature with 88% and 89% accuracy for DenseNet and ResNet compared to 50 % and 42 % for HOG and LBP respectively. The deep neural network based methods also demonstrated better result on the under represented class.