Back to Search
Start Over
Modality Compensation Network: Cross-Modal Adaptation for Action Recognition
- Source :
- IEEE Transactions on Image Processing. 29:3957-3969
- Publication Year :
- 2020
- Publisher :
- Institute of Electrical and Electronics Engineers (IEEE), 2020.
-
Abstract
- With the prevalence of RGB-D cameras, multi-modal video data have become more available for human action recognition. One main challenge for this task lies in how to effectively leverage their complementary information. In this work, we propose a Modality Compensation Network (MCN) to explore the relationships of different modalities, and boost the representations for human action recognition. We regard RGB/optical flow videos as source modalities, skeletons as auxiliary modality. Our goal is to extract more discriminative features from source modalities, with the help of auxiliary modality. Built on deep Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks, our model bridges data from source and auxiliary modalities by a modality adaptation block to achieve adaptive representation learning, that the network learns to compensate for the loss of skeletons at test time and even at training time. We explore multiple adaptation schemes to narrow the distance between source and auxiliary modal distributions from different levels, according to the alignment of source and auxiliary data in training. In addition, skeletons are only required in the training phase. Our model is able to improve the recognition performance with source data when testing. Experimental results reveal that MCN outperforms state-of-the-art approaches on four widely-used action recognition benchmarks.<br />Accepted by IEEE Trans. on Image Processing, 2020. Project page: http://39.96.165.147/Projects/MCN_tip2020_ssj/MCN_tip_2020_ssj.html
- Subjects :
- FOS: Computer and information sciences
Modality (human–computer interaction)
Computer science
business.industry
Computer Vision and Pattern Recognition (cs.CV)
Computer Science - Computer Vision and Pattern Recognition
ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION
Optical flow
Pattern recognition
02 engineering and technology
Computer Graphics and Computer-Aided Design
Convolutional neural network
Discriminative model
0202 electrical engineering, electronic engineering, information engineering
020201 artificial intelligence & image processing
Artificial intelligence
Adaptation (computer science)
business
Software
Block (data storage)
Subjects
Details
- ISSN :
- 19410042 and 10577149
- Volume :
- 29
- Database :
- OpenAIRE
- Journal :
- IEEE Transactions on Image Processing
- Accession number :
- edsair.doi.dedup.....cfd52d26dd6fa7a42ec52b51e6c49dd0