Search

The objectives of this paper are to investigate the trade-offs between a physically constrained neural network and a deep, convolutional neural network and to design a combined ML approach ("VarioCNN"). Our solution is provided in the framework of a cyberinfrastructure that includes a newly designed ML software, GEOCLASS-image (v1.0), modern high-resolution satellite image data sets (Maxar WorldView data), and instructions/descriptions that may facilitate solving similar spatial classification problems. Combining the advantages of the physically-driven connectionist-geostatistical classification method with those of an efficient CNN, VarioCNN provides a means for rapid and efficient extraction of complex geophysical information from submeter resolution satellite imagery. A retraining loop overcomes the difficulties of creating a labeled training data set. Computational analyses and developments are centered on a specific, but generalizable, geophysical problem: The classification of crevasse types that form during the surge of a glacier system. A surge is a glacial catastrophe, an acceleration of a glacier to typically 100–200 times its normal velocity. GEOCLASS-image is applied to study the current (2016-2024) surge in the Negribreen Glacier System, Svalbard. The geophysical result is a description of the structural evolution and expansion of the surge, based on crevasse types that capture ice deformation in six simplified classes. [ABSTRACT FROM AUTHOR]

The rapid development of neural networks has come at the cost of increased computational complexity. Neural networks are both computationally intensive and memory intensive; as such, the minimal energy and computing power of satellites pose a challenge for automatic target recognition (ATR). Knowledge distillation (KD) can distill knowledge from a cumbersome teacher network to a lightweight student network, transferring the essential information learned by the teacher network. Thus, the concept of KD can be used to improve the accuracy of student networks. Even when learning from a teacher network, there is still redundancy in the student network. Traditional networks fix the structure before training, such that training does not improve the situation. This paper proposes a distillation sparsity training (DST) algorithm based on KD and network pruning to address the above limitations. We first improve the accuracy of the student network through KD, and then through network pruning, allowing the student network to learn which connections are essential. DST allows the teacher network to teach the pruned student network directly. The proposed algorithm was tested on the CIFAR-100, MSTAR, and FUSAR-Ship data sets, with a 50% sparsity setting. First, a new loss function for the teacher-pruned student was proposed, and the pruned student network showed a performance close to that of the teacher network. Second, a new sparsity model (uniformity half-pruning UHP) was designed to solve the problem that unstructured pruning does not facilitate the implementation of general-purpose hardware acceleration and storage. Compared with traditional unstructured pruning, UHP can double the speed of neural networks. [ABSTRACT FROM AUTHOR]

The SWOT (Surface Water Ocean Topography) mission will provide high-resolution and two-dimensional measurements of sea surface height (SSH). However, despite its unprecedented precision, SWOT's Ka-band Radar Interferometer (KaRIn) still exhibits a substantial amount of random noise. In turn, the random noise limits the ability of SWOT to capture the smallest scales of the ocean's topography and its derivatives. In that context, this paper explores the feasibility, strengths and limits of a noise-reduction algorithm based on a convolutional neural network. The model is based on a U-Net architecture and is trained and tested with simulated data from the North Atlantic. Our results are compared to classical smoothing methods: a median filter, a Lanczos kernel smoother and the SWOT de-noising algorithm developed by Gomez-Navarro et al. Our U-Net model yields better results for all the evaluation metrics: 2 mm root mean square error, sub-millimetric bias, variance reduction by factor of 44 (16 dB) and an accurate power spectral density down to 10–20 km wavelengths. We also tested various scenarios to infer the robustness and the stability of the U-Net. The U-Net always exhibits good performance and can be further improved with retraining if necessary. This robustness in simulation is very encouraging: our findings show that the U-Net architecture is likely one of the best candidates to reduce the noise of flight data from KaRIn. [ABSTRACT FROM AUTHOR]

While thunderstorms can pose severe risks to property and life, forecasting remains challenging, even at short lead times, as these often arise in meta-stable atmospheric conditions. In this paper, we examine the question of how well we could perform short-term (up to 180 min) forecasts using exclusively multi-spectral satellite images and past lighting events as data. We employ representation learning based on deep convolutional neural networks in an "end-to-end" fashion. Here, a crucial problem is handling the imbalance of the positive and negative classes appropriately in order to be able to obtain predictive results (which is not addressed by many previous machine-learning-based approaches). The resulting network outperforms previous methods based on physically based features and optical flow methods (similar to operational prediction models) and generalizes across different years. A closer examination of the classifier performance over time and under masking of input data indicates that the learned model actually draws most information from structures in the visible spectrum, with infrared imaging sustaining some classification performance during the night. [ABSTRACT FROM AUTHOR]

This study investigates the potential of Sentinel-1 (S1) multi-temporal data for the early-season mapping of the rapeseed crop. Additionally, we explore the effectiveness of limiting the portion of a considered time series to map rapeseed fields. To this end, we conducted a quantitative analysis to assess several temporal windows (periods) spanning different phases of the rapeseed phenological cycle in the following two scenarios relating to the availability or constraints of providing ground samples for different years: (i) involving the same year for both training and the test, assuming the availability of ground samples for each year; and (ii) evaluating the temporal transferability of the classifier, considering the constraints of ground sampling. We employed two different classification methods that are renowned for their high performance in land cover mapping: the widely adopted random forest (RF) approach and a deep learning-based convolutional neural network, specifically the InceptionTime algorithm. To assess the classification outcomes, four evaluation metrics (recall, precision, F1 score, and Kappa) were employed. Using S1 time series data covering the entire rapeseed growth cycle, the tested algorithms achieved F1 scores close to 95% on same-year training and testing, and 92.0% when different years were used, both algorithms demonstrated robust performance. For early rapeseed detection within a two-month window post-sowing, RF and InceptionTime achieved F1 scores of 67.5% and 77.2%, respectively, and 79.8% and 88.9% when extended to six months. However, in the context of temporal transferability, both classifiers exhibited mean F1 scores below 50%. Notably, a 5-month time series, covering key growth stages such as stem elongation, inflorescence emergence, and fruit development, yielded a mean F1 score close to 95% for both algorithms when trained and tested in the same year. In the temporal transferability scenario, RF and InceptionTime achieved mean F1 scores of 92.0% and 90.0%, respectively, using a 5-month time series. Our findings underscore the importance of a concise S1 time series for effective rapeseed mapping, offering advantages in data storage and processing time. Overall, the study establishes the robustness of RF and InceptionTime in rapeseed detection scenarios, providing valuable insights for agricultural applications. [ABSTRACT FROM AUTHOR]

The objectives of this paper are to investigate the trade-offs between a physically constrained neural network and a deep, convolutional neural network and to design a combined ML approach (“VarioCNN”). Our solution is provided in the framework of a cyberinfrastructure that includes a newly designed ML software, GEOCLASS-image (v1.0), modern high-resolution satellite image data sets (Maxar WorldView data), and instructions/descriptions that may facilitate solving similar spatial classification problems. Combining the advantages of the physically-driven connectionist-geostatistical classification method with those of an efficient CNN, VarioCNN provides a means for rapid and efficient extraction of complex geophysical information from submeter resolution satellite imagery. A retraining loop overcomes the difficulties of creating a labeled training data set. Computational analyses and developments are centered on a specific, but generalizable, geophysical problem: The classification of crevasse types that form during the surge of a glacier system. A surge is a glacial catastrophe, an acceleration of a glacier to typically 100–200 times its normal velocity. GEOCLASS-image is applied to study the current (2016-2024) surge in the Negribreen Glacier System, Svalbard. The geophysical result is a description of the structural evolution and expansion of the surge, based on crevasse types that capture ice deformation in six simplified classes.

Satellite image time series (SITS) is a sequence of satellite images that record a given area at several consecutive times. The aim of such sequences is to use not only spatial information but also the temporal dimension of the data, which is used for multiple real-world applications, such as classification, segmentation, anomaly detection, and prediction. Several traditional machine learning algorithms have been developed and successfully applied to time series for predictions. However, these methods have limitations in some situations, thus deep learning (DL) techniques have been introduced to achieve the best performance. Reviews of machine learning and DL methods for time series prediction problems have been conducted in previous studies. However, to the best of our knowledge, none of these surveys have addressed the specific case of works using DL techniques and satellite images as datasets for predictions. Therefore, this paper concentrates on the DL applications for SITS prediction, giving an overview of the main elements used to design and evaluate the predictive models, namely the architectures, data, optimization functions, and evaluation metrics. The reviewed DL-based models are divided into three categories, namely recurrent neural network-based models, hybrid models, and feed-forward-based models (convolutional neural networks and multi-layer perceptron). The main characteristics of satellite images and the major existing applications in the field of SITS prediction are also presented in this article. These applications include weather forecasting, precipitation nowcasting, spatio-temporal analysis, and missing data reconstruction. Finally, current limitations and proposed workable solutions related to the use of DL for SITS prediction are also highlighted. [ABSTRACT FROM AUTHOR]

The images of the Sentinel-2 constellation can help the verification process of farmers' declarations, providing, among other things, accurate spatial explicit maps of the agricultural land cover. The aim of the study is to design, develop, and evaluate two deep learning (DL) architectures tailored for agricultural land cover and crop type mapping. The focus is on a detailed class scheme encompassing fifteen distinct classes, utilizing Sentinel-2 imagery acquired on a monthly basis throughout the year. The study's geographical scope covers a diverse rural area in North Greece, situated within southeast Europe. These architectures are a Temporal Convolutional Neural Network (CNN) and a combination of a Recurrent and a 2D Convolutional Neural Network (R-CNN), and their accuracy is compared to the well-established Random Forest (RF) machine learning algorithm. The comparative approach is not restricted to simply presenting the results given by classification metrics, but it also assesses the uncertainty of the classification results using an entropy measure and the spatial distribution of the classification errors. Furthermore, the issue of sampling strategy for the extraction of the training set is highlighted, targeting the efficient handling of both the imbalance of the dataset and the spectral variability of instances among classes. The two developed deep learning architectures performed equally well, presenting an overall accuracy of 90.13% (Temporal CNN) and 90.18% (R-CNN), higher than the 86.31% overall accuracy of the RF approach. Finally, the Temporal CNN method presented a lower entropy value (6.63%), compared both to R-CNN (7.76%) and RF (28.94%) methods, indicating that both DL approaches should be considered for developing operational EO processing workflows. [ABSTRACT FROM AUTHOR]

Over one hundred spatiotemporal fusion algorithms have been proposed, but convolutional neural networks trained with large amounts of data for spatiotemporal fusion have not shown significant advantages. In addition, no attention has been paid to whether fused images can be used for change detection. These two issues are addressed in this work. A new dataset consisting of nine pairs of images is designed to benchmark the accuracy of neural networks using one-pair spatiotemporal fusion with neural-network-based models. Notably, the size of each image is significantly larger compared to other datasets used to train neural networks. A comprehensive comparison of the radiometric, spectral, and structural losses is made using fourteen fusion algorithms and five datasets to illustrate the differences in the performance of spatiotemporal fusion algorithms with regard to various sensors and image sizes. A change detection experiment is conducted to test if it is feasible to detect changes in specific land covers using the fusion results. The experiment shows that convolutional neural networks can be used for one-pair spatiotemporal fusion if the sizes of individual images are adequately large. It also confirms that the spatiotemporally fused images can be used for change detection in certain scenes. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks have been successfully applied to lossy image compression. End-to-end optimized autoencoders, possibly variational, are able to dramatically outperform traditional transform coding schemes in terms of rate-distortion trade-off; however, this is at the cost of a higher computational complexity. An intensive training step on huge databases allows autoencoders to learn jointly the image representation and its probability distribution, possibly using a non-parametric density model or a hyperprior auxiliary autoencoder to eliminate the need for prior knowledge. However, in the context of on board satellite compression, time and memory complexities are submitted to strong constraints. The aim of this paper is to design a complexity-reduced variational autoencoder in order to meet these constraints while maintaining the performance. Apart from a network dimension reduction that systematically targets each parameter of the analysis and synthesis transforms, we propose a simplified entropy model that preserves the adaptability to the input image. Indeed, a statistical analysis performed on satellite images shows that the Laplacian distribution fits most features of their representation. A complex non parametric distribution fitting or a cumbersome hyperprior auxiliary autoencoder can thus be replaced by a simple parametric estimation. The proposed complexity-reduced autoencoder outperforms the Consultative Committee for Space Data Systems standard (CCSDS 122.0-B) while maintaining a competitive performance, in terms of rate-distortion trade-off, in comparison with the state-of-the-art learned image compression schemes. [ABSTRACT FROM AUTHOR]

At present, convolutional neural networks (CNN) have been widely used in building extraction from remote sensing imagery (RSI), but there are still some bottlenecks. On the one hand, there are so many parameters in the previous network with complex structure, which will occupy lots of memories and consume much time during training process. On the other hand, low-level features extracted by shallow layers and abstract features extracted by deep layers of artificial neural network cannot be fully fused, which leads to an inaccurate building extraction from RSI. To alleviate these disadvantages, a dense residual neural network (DR-Net) was proposed in this paper. DR-Net uses a deeplabv3+Net encoder/decoder backbone, in combination with densely connected convolution neural network (DCNN) and residual network (ResNet) structure. Compared with deeplabv3+net (containing about 41 million parameters) and BRRNet (containing about 17 million parameters), DR-Net contains about 9 million parameters; So, the number of parameters reduced a lot. The experimental results for both the WHU Building Dataset and Massachusetts Building Dataset, DR-Net show better performance in building extraction than other two state-of-the-art methods. Experiments on WHU building data set showed that Intersection over Union (IoU) increased by 2.4% and F1 score increased by 1.4%; in terms of Massachusetts Building Dataset, IoU increased by 3.8% and F1 score increased by 2.9%. [ABSTRACT FROM AUTHOR]

Radar images suffer from the impact of sidelobes. Several sidelobe-suppressing methods including the convolutional neural network (CNN)-based one has been proposed. However, the point spread function (PSF) in the radar images is sometimes spatially variant and affects the performance of the CNN. We propose the spatial-variant convolutional neural network (SV-CNN) aimed at this problem. It will also perform well in other conditions when there are spatially variant features. The convolutional kernels of the CNN can detect motifs with some distinctive features and are invariant to the local position of the motifs. This makes the convolutional neural networks widely used in image processing fields such as image recognition, handwriting recognition, image super-resolution, and semantic segmentation. They also perform well in radar image enhancement. However, the local position invariant character might not be good for radar image enhancement, when features of motifs (also known as the point spread function in the radar imaging field) vary with the positions. In this paper, we proposed an SV-CNN with spatial-variant convolution kernels (SV-CK). Its function is illustrated through a special application of enhancing the radar images. After being trained using radar images with position-codings as the samples, the SV-CNN can enhance the radar images. Because the SV-CNN reads information of the local position contained in the position-coding, it performs better than the conventional CNN. The advance of the proposed SV-CNN is tested using both simulated and real radar images. [ABSTRACT FROM AUTHOR]

The long-term spectral characteristics of the bare soil surface (BSS) in the BLUE, GREEN, RED, NIR, SWIR1, and SWIR2 Landsat spectral bands are poorly studied. Most often, the RED and NIR spectral bands are used to analyze the spatial heterogeneity of the soil cover; in our opinion, it is outmoded and seems unreasonable. The study of multi-temporal spectral characteristics requires the processing of big remote sensing data based on artificial intelligence in the form of convolutional neural networks. The analysis of BSS belongs to the direct methods of analysis of the soil cover. Soil degradation can be detected by ground methods (field reconnaissance surveys), modeling, or digital methods, and based on the remote sensing data (RSD) analysis. Ground methods are laborious, and modeling gives indirect results. RSD analysis can be based on the principles of calculation of vegetation indices (VIs) and on the BSS identification. The calculation of VIs also provides indirect information about the soil cover through the state of vegetation. BSS analysis is a direct method for analyzing soil cover heterogeneity. In this work, the informativeness of the long-term (37 years) average spectral characteristics of the BLUE, GREEN, RED, NIR, SWIR1 and SWIR2 bands of the Landsat 4–8 satellites for detecting areas of soil degradation with recognition of the BSS using deep machine learning methods was estimated. The objects of study are the spectral characteristics of kastanozems (dark chestnut soils) in the south of Russia in the territory of the Morozovsky district of the Rostov region. Soil degradation in this area is mainly caused by erosion. The following methods were used: retrospective monitoring of soil and land cover, deep machine learning using convolutional neural networks, and cartographic analysis. Six new maps of the average long-term spectral brightness of the BSS have been obtained. The information content of the BSS for six spectral bands has been verified on the basis of ground surveys. The informativeness was determined by the percentage of coincidences of degradation facts identified during the RSD analysis, and those determined in the field. It has been established that the spectral bands line up in the following descending order of information content: RED, NIR, GREEN, BLUE, SWIR1, SWIR2. The accuracy of degradation maps by band was determined as: RED—84.6%, NIR—82.9%, GREEN—78.0%, BLUE—78.0%, SWIR1—75.5%, SWIR2—62.2%. [ABSTRACT FROM AUTHOR]

This article describes the development of a machine learning (ML)-based algorithm for snowfall retrieval (Snow retrievaL ALgorithm fOr gpM–Cross Track, SLALOM-CT), exploiting ATMS radiometer measurements and using the CloudSat CPR snowfall products as references. During a preliminary analysis, different ML techniques (tree-based algorithms, shallow and convolutional neural networks—NNs) were intercompared. A large dataset (three years) of coincident observations from CPR and ATMS was used for training and testing the different techniques. The SLALOM-CT algorithm is based on four independent modules for the detection of snowfall and supercooled droplets, and for the estimation of snow water path and snowfall rate. Each module was designed by choosing the best-performing ML approach through model selection and optimization. While a convolutional NN was the most accurate for the snowfall detection module, a shallow NN was selected for all other modules. SLALOM-CT showed a high degree of consistency with CPR. Moreover, the results were almost independent of the background surface categorization and the observation angle. The reliability of the SLALOM-CT estimates was also highlighted by the good results obtained from a direct comparison with a reference algorithm (GPROF). [ABSTRACT FROM AUTHOR]

Total Cloud Cover (TCC) retrieval from ground-based optical imagery is a problem that has been tackled by several generations of researchers. The number of human-designed algorithms for the estimation of TCC grows every year. However, there has been no considerable progress in terms of quality, mostly due to the lack of systematic approach to the design of the algorithms, to the assessment of their generalization ability, and to the assessment of the TCC retrieval quality. In this study, we discuss the optimization nature of data-driven schemes for TCC retrieval. In order to compare the algorithms, we propose a framework for the assessment of the algorithms' characteristics. We present several new algorithms that are based on deep learning techniques: A model for outliers filtering, and a few models for TCC retrieval from all-sky imagery. For training and assessment of data-driven algorithms of this study, we present the Dataset of All-Sky Imagery over the Ocean (DASIO) containing over one million all-sky optical images of the visible sky dome taken in various regions of the world ocean. The research campaigns that contributed to the DASIO collection took place in the Atlantic ocean, the Indian ocean, the Red and Mediterranean seas, and the Arctic ocean. Optical imagery collected during these missions are accompanied by standard meteorological observations of cloudiness characteristics made by experienced observers. We assess the generalization ability of the presented models in several scenarios that differ in terms of the regions selected for the train and test subsets. As a result, we demonstrate that our models based on convolutional neural networks deliver a superior quality compared to all previously published approaches. As a key result, we demonstrate a considerable drop in the ability to generalize the training data in the case of a strong covariate shift between the training and test subsets of imagery which may occur in the case of region-aware subsampling. [ABSTRACT FROM AUTHOR]

In Earth observation (EO), large-scale land-surface dynamics are traditionally analyzed by investigating aggregated classes. The increase in data with a very high spatial resolution enables investigations on a fine-grained feature level which can help us to better understand the dynamics of land surfaces by taking object dynamics into account. To extract fine-grained features and objects, the most popular deep-learning model for image analysis is commonly used: the convolutional neural network (CNN). In this review, we provide a comprehensive overview of the impact of deep learning on EO applications by reviewing 429 studies on image segmentation and object detection with CNNs. We extensively examine the spatial distribution of study sites, employed sensors, used datasets and CNN architectures, and give a thorough overview of applications in EO which used CNNs. Our main finding is that CNNs are in an advanced transition phase from computer vision to EO. Upon this, we argue that in the near future, investigations which analyze object dynamics with CNNs will have a significant impact on EO research. With a focus on EO applications in this Part II, we complete the methodological review provided in Part I. [ABSTRACT FROM AUTHOR]

Deep learning (DL) has great influence on large parts of science and increasingly established itself as an adaptive method for new challenges in the field of Earth observation (EO). Nevertheless, the entry barriers for EO researchers are high due to the dense and rapidly developing field mainly driven by advances in computer vision (CV). To lower the barriers for researchers in EO, this review gives an overview of the evolution of DL with a focus on image segmentation and object detection in convolutional neural networks (CNN). The survey starts in 2012, when a CNN set new standards in image recognition, and lasts until late 2019. Thereby, we highlight the connections between the most important CNN architectures and cornerstones coming from CV in order to alleviate the evaluation of modern DL models. Furthermore, we briefly outline the evolution of the most popular DL frameworks and provide a summary of datasets in EO. By discussing well performing DL architectures on these datasets as well as reflecting on advances made in CV and their impact on future research in EO, we narrow the gap between the reviewed, theoretical concepts from CV and practical application in EO. [ABSTRACT FROM AUTHOR]