Search

Deep learning methods have recently found widespread adoption for remote sensing tasks, particularly in image or pixel classification. Their flexibility and versatility has enabled researchers to propose many different designs to process remote sensing data in all spectral, spatial, and temporal dimensions. In most of the reported cases they surpass their non-deep rivals in overall classification accuracy. However, there is considerable diversity in implementation details in each case and a systematic quantitative comparison to non-deep classifiers does not exist. In this paper, we look at the major research papers that have studied deep learning image classifiers in recent years and undertake a meta-analysis on their performance compared to the most used non-deep rival, Support Vector Machine (SVM) classifiers. We focus on mono-temporal classification as the time-series image classification did not offer sufficient samples. Our work covered 103 manuscripts and included 92 cases that supported direct accuracy comparisons between deep learners and SVMs. Our general findings are the following: (i) Deep networks have better performance than non-deep spectral SVM implementations, with Convolutional Neural Networks (CNNs) performing better than other deep learners. This advantage, however, diminishes when feeding SVM with richer features extracted from data (e.g. spatial filters). (ii) Transfer learning and fine-tuning on pre-trained CNNs are offering promising results over spectral or enhanced SVM, however these pre-trained networks are currently limited to RGB input data, therefore currently lack applicability in multi/hyperspectral data. (iii) There is no strong relationship between network complexity and accuracy gains over SVM; small to medium networks perform similarly to more complex networks. (iv) Contrary to the popular belief, there are numerous cases of high deep networks performance with training proportions of 10% or less. Our study also indicates that the new generation of classifiers is often overperforming existing benchmark datasets, with accuracies surpassing 99%. There is a clear need for new benchmark dataset collections with diverse spectral, spatial and temporal resolutions and coverage that will enable us to study the design generalizations, challenge these new classifiers, and further advance remote sensing science. Our community could also benefit from a coordinated effort to create a large pre-trained network specifically designed for remote sensing images that users could later fine-tune and adjust to their study specifics.

A major issue in land cover mapping is classifier selection. Here we investigated classifier performance under different sample sizes, reference class distribution, and scene complexities. Twenty six 10 km × 10 km blocks with complete reference information across the continental US are used. Per-pixel classification took place using six spectral bands from Landsat imagery. The tested classifiers included Naive Bayes (NB), Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Bootstrap-aggregation ensemble of decision trees (BagTE), artificial neural network up to 2 hidden layers, and deep neural network (DNN) up to 3 hidden layers. For the entire block, our accuracy assessment indicated that all classifiers, with the exception of NB (a Maximum Likelihood variant), performed similarly. However, when we concentrated on edge pixels (pixels at the border of adjacent land cover classes), it was clear that the SVM and KNN offer considerable accuracy advantages, especially for larger reference datasets. Because of their relatively low execution times SVM and KNN would be recommended for classifications using Landsat's spectral inputs and Anderson's 11-level classification scheme. However, both SVM and KNN demonstrated substantial accuracy degradation during the parameter grid search. For this reason, an exhaustive parameter optimization process is suggested. While the ANN and DNN neural network variants did not perform as well, their performance may have been restricted by the lack of rich contextual information in our simple six band per-pixel input space. The effect of class distribution in the training dataset was also evident on the calculated accuracy metric. Gradual accuracy degradation as edge pixel presence increased was also observed. Future work could focus on data-rich classification problems such as change detection using Landsat stacks or expand in high spectral or spatial resolution sensors.