IF-CNN: Image-Aware Inference Framework for CNN With the Collaboration of Mobile Devices and Cloud

Improving the performance of CNN-based mobile applications by offloading its computation from mobile devices to the cloud has attracted the attention of the community. Generally, there are three stages in the workflow, including local inference on the mobile device, data transmission of the intermediate result, and remote inference in the cloud. However, the time cost of local inference and data transmission are still the bottleneck in reaching the desirable inference performance. In this paper, we propose an image-aware inference framework called IF-CNN to enable fast inference based on computation offloading. In the framework, we first build a model pool consisting of CNN models with different complexities. The most efficient one from such candidate models is selected to process the corresponding image. During the selection process, we have designed an effective model to predict the confidence based on multi-task learning. After model selection, half-floating optimization and feature compression are applied to accelerate the process of distributed inference between mobile devices and cloud. Experimental results show that IF-CNN is credible to identify the most effective model for different images and the total inference performance could be significantly improved. Meanwhile, IF-CNN is complementary to other inference acceleration methods of CNN models.