1. RA-Net: reverse attention for generalizing residual learning
- Author
-
Zhenyuan Wang, Xuemei Xie, Jianxiu Yang, and Xiaodan Song
- Subjects
Reverse attention ,Generalized residual learning ,Identity mapping ,Modified global response normalization ,Medicine ,Science - Abstract
Abstract Since residual learning was proposed, identity mapping has been widely utilized in various neural networks. The method enables information transfer without any attenuation, which plays a significant role in training deeper networks. However, interference with unhindered transmission also affects the network’s performance. Accordingly, we propose a generalized residual learning architecture called reverse attention (RA), which applies high-level semantic features to supervise low-level information in the identity mapping branch. It means that higher semantic features selectively transmit low-level information to deeper layers. In addition, we propose a Modified Global Response Normalization(M-GRN) to implement reverse attention. RA-Net is derived by embedding M-GRN in the residual learning framework. The experiments show that the RA-Net brings significant improvements over residual networks on typical computer vision tasks. For classification on ImageNet-1K, compared with resnet101, RA-Net improves the Top-1 accuracy by 1.7% with comparable parameters and computational cost. For COCO detection, on Faster R-CNN, reverse attention improves box AP by 1.9%. Meanwhile, reverse attention improves UpperNet’s mIoU by 0.7% on ADE20K segmentation.
- Published
- 2024
- Full Text
- View/download PDF