Search

Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss—first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit—second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning.

Most multi-object tracking methods have achieved good results in tracking multiple pedestrians with Kalman filter, but their tracking performance in crowded scenes is still poor due to pedestrian avoidance and frequent occlusion. In crowded scenes, the pedestrian trajectory prediction with Kalman filter alone is unreliable. In this paper, a two-dimensional field-of-view avoidance force model (AFM) is proposed to assist the Kalman filter prediction by sensing the avoidance force and then complete pedestrian tracking. In the model, each pedestrian has a two-dimensional field of view to perceive the avoidance force, which determines the next predicted trajectory. In real scenes, pedestrians tend to be more concerned about the surrounding area, so different areas are set to simulate the attention mechanisms of pedestrians in real scenes. In the FairMOT model, AFM is used to optimize the pedestrian state values of Kalman filter prediction and the optimized model is trained on the MOT16 dataset. The experimental results on the MOT20 dataset show that compared with the mainstream tracking model FairMOT, our method respectively improves MOTA by 2.7% and IDF1 by 2.2%. Our method also achieves the good performance on MOT15, MOT16, and MOT17 tracking benchmarks.

Video-level sentiment analysis is a challenging task and requires systems to obtain discriminative multimodal representations that can capture difference in sentiments across various modalities. However, due to diverse distributions of various modalities and the unified multimodal labels are not always adaptable to unimodal learning, the distance difference between unimodal representations increases, and prevents systems from learning discriminative multimodal representations. In this paper, to obtain more discriminative multimodal representations that can further improve systems’ performance, we propose a VAE-based adversarial multimodal domain transfer (VAE-AMDT) and jointly train it with a multi-attention module to reduce the distance difference between unimodal representations. We first perform variational autoencoder (VAE) to make visual, linguistic and acoustic representations follow a common distribution, and then introduce adversarial training to transfer all unimodal representations to a joint embedding space. As a result, we fuse various modalities on this joint embedding space via the multi-attention module, which consists of self-attention, cross-attention and triple-attention for highlighting important sentimental representations over time and modality. Our method improves F1-score of the state-of-the-art by 3.6% on MOSI and 2.9% on MOSEI datasets, and prove its efficacy in obtaining discriminative multimodal representations for video-level sentiment analysis.

An analytical model is proposed in this paper to predict the breakdown voltage (BV) of silicon-on-insulator (SOI) trench lateral double diffused metal oxide semiconductor (LDMOS) devices with N/P pillars. In this SOI trench LDMOS device, the U-shaped folded drift region is divided into four parts for simplifying the solution of Poisson's equation, where conformal mapping by the Schwarz-Christoffel (S-C) transformation is applied. As verified by the TCAD simulation results, the analytical model may predict the electric field distribution and the BV accurately. This analytical model can reveal the essence of the electric field peaks around the dielectric trench from the depletion effect of the N/P pillars. [ABSTRACT FROM AUTHOR]