Search

Multi-image super-resolution (MISR) usually outperforms single-image super-resolution (SISR) under a proper inter-image alignment by explicitly exploiting the inter-image correlation. However, the large computational demand encumbers the deployment of MISR in practice. In this work, we propose a distributed optimization framework based on data parallelism for fast large-scale MISR using multi-GPU acceleration named FL-MISR. The scaled conjugate gradient (SCG) algorithm is applied to the distributed subfunctions and the local SCG variables are communicated to synchronize the convergence rate over multi-GPU systems towards a consistent convergence. Furthermore, an inner-outer border exchange scheme is performed to obviate the border effect between neighboring GPUs. The proposed FL-MISR is applied to the computed tomography (CT) system by super-resolving the projections acquired by subpixel detector shift. The SR reconstruction is performed on the fly during the CT acquisition such that no additional computation time is introduced. FL-MISR is extensively evaluated from different aspects and experimental results demonstrate that FL-MISR effectively improves the spatial resolution of CT systems in modulation transfer function (MTF) and visual perception. Comparing to a multi-core CPU implementation, FL-MISR achieves a more than 50× speedup on an off-the-shelf 4-GPU system., Deutsche Forschungsgemeinschaft, Projekt DEAL

Pantograph slide is a vulnerable component because of the contact and friction between the pantograph and catenary (PAC) while the train runs with a high speed. Condition monitoring of the slide is a necessary and meaningful work to improve the operation efficiency and reduce the occurrence of accident. It is a hard work to measure the slide abrasion due to the dynamic and complicated environment. In this paper, an accurate edge extraction method based on image analysis is proposed, which aims at providing an image metric property based on vision measurement to detect the wear of slide. First, a subpixel edge detection method is proposed. The gradient in horizontal direction is first taken into account, which filters out the non-horizontal edges. Then, the upper and lower edges of the slide are located using the geometric restriction. The subpixel edge feature are also extracted in the image process, which can be used to the vision measurement system for monitoring the wear condition of the slide. The experiment results shows that the proposed algorithm can precisely detect the subpixel edges of the slide. The measurement error of the slide is less than 2 mm, which demonstrates that the proposed algorithm has a high accuracy in the large field of view.

Many methods have been developed to improve the performance of image registration. In this letter, we introduce a novel method based on a local affine constraint for remote sensing image registration, which can be widely used in image processing and pattern recognition. Our algorithm has three components. First, we exploit the scale invariant feature transform (SIFT) method to extract feature points and calculate the gradient magnitude to establish feature descriptors with a circular instead of square neighborhood. Second, an initial matching is implemented by the nearest neighbor distance ratio (NNDR) and the fast sample consensus (FSC) algorithm. Finally, fine registration is established using more correct matches obtained by the local affine transformation circular region search algorithm. Experimental results show that the proposed method achieves subpixel accuracy. In addition, both the correct matching rate and registration demonstrate the effectiveness and efficiency of our method.

Uncertainty quantification is an important part of validation, because the pixel scale reference generally suffers from uncertainty caused by different factors, lowering the accuracy of validation results. In order to take a step forward to characterize the uncertainty of validation results, this study proposed a simulated shift-based pixel matching (SSPM) method with the aim of quantifying the uncertainty caused by geometric mismatch in the multiscale validation. Furthermore, its relationships with spatial heterogeneity and subpixel size were also explored. It was found that the uncertainty caused by the geometric mismatch is nonnegligible in multiscale validation, which would obscure the true accuracy of satellite products. Spatial heterogeneity makes a positive contribution to the uncertainty caused by geometric mismatch, but the magnitude depends on subpixel size, being weaker with small subpixel size and stronger with larger subpixel size. Subpixel size is generally positively related to geometric uncertainty. But in the case of very large spatial heterogeneity, their correlation is very weak. This study is an important step toward quantitatively characterizing the uncertainties of pixel scale reference in order to increase the confidence of validation results.

Thermal infrared imaging possesses the ability to monitor unmanned aerial vehicles (UAVs) in both day and night conditions. However, long-range detection of the infrared UAVs often suffers from small/dim targets, heavy clutter, and noise in the complex background. The conventional local prior-based and the nonlocal prior-based methods commonly have a high false alarm rate and low detection accuracy. In this letter, we propose a model that converts small UAV detection into a problem of predicting the residual image (i.e., background, clutter, and noise). Such novel reformulation allows us to directly learn a mapping from the input infrared image to the residual image. The constructed image-to-image network integrates the global and the local dilated residual convolution blocks into the U-Net, which can capture local and contextual structure information well and fuse the features at different scales both for image reconstruction. Additionally, subpixel convolution is utilized to upscale the image and avoid image distortion during upsampling. Finally, the small UAV target image is obtained by subtracting the residual image from the input infrared image. The comparative experiments demonstrate that the proposed method outperforms state-of-the-art ones in detecting real-world infrared images with heavy clutter and dim targets.

The ability to accurately map the seabed sediments plays an important role in seabed habitat development and stakeholder decision-making. In conventional seabed sediment classification methods, maps of seabed sediment are provided in categorical form (sediment classes). Therefore, the prediction of the sediment compositions in multibeam observational units has become a difficult issue in using conventional methods. To tackle this challenge, a new strategy is developed to realize the subpixel decomposition of seabed sediments. A key attribute of the proposed sediment decomposition model is that it utilizes spatial-spectral information provided by multibeam backscatter angular responses (ARs). First, an AR feature extraction method utilizing a bidirectional sliding window is proposed and a K-means clustering algorithm is used for segmentation. Second, a deep-sea sediment decomposition model based on the fuzzy method is constructed by selecting experimental samples that are distributed within a single clustering region. This model inverts the abundance of each sediment composition in the form of membership degrees. Finally, deep-sea multibeam survey data collected from the central Philippine Sea are used for verification. The overall mean square error and coefficient of determination reach 0.043 and 0.856, respectively. The experimental results show that the new method can accurately decompose deep-sea sediment compositions, thus providing a new technique for deep-sea acoustic sediment remote sensing and quantitative analysis.

Characterizing the subpixel heterogeneity within satellite pixels is a key issue in validation. Nevertheless, it is challenging due to multi-scale problems in the geological description based on remote sensing. Based on an airborne platform, the multi-scale variation laws of several key indicators in validation, including spatial heterogeneity, representativeness errors, and representative area with subpixel size, were analyzed and discussed. Furthermore, this paper discussed the optimal subpixel size to assess spatial heterogeneity within a coarse pixel and the optimal footprint of in situ measurements for building dense and sparse validation networks. Spatial heterogeneity decreases with the increase of subpixel size. And a reduction about 10% can be obtained from 5 m×5 m to 150 m×150 m subpixel size, depending on the degree of spatial heterogeneity within the typical satellite pixels. And the sensitiveness of spatial heterogeneity to subpixel size decreases gradually with the increasing of subpixel size. Ideally, spatial heterogeneity should be assessed using maps with pixel sizes corresponding to the footprint of in situ measurements. Regarding the deployment of future validation networks, the footprint of in situ sites should be designed at least larger than 25 m for dense networks. And much larger footprints (e.g., 100 m) are preferred in designing sparse networks. The representativeness error is not fully related to subpixel sizes because it is affected by many factors. The findings are also transferable to model evaluation when comparing model grid values to local observations.

Subpixel mapping (SPM) based on multisource remote sensing fusion data (MRSFD) for land-cover classes, called SPM-MRSFD, is proposed in this letter. First, the original hyperspectral image and the auxiliary panchromatic image are fused to produce the high spatial and spectral resolution fused image by pan-sharpening technology. Second, the fused image with spatial-spectral information and the auxiliary digital surface model (DSM) of light detection and ranging (LiDAR) with elevation information are fused to obtain the MRSFD with spatial-spectral elevation information by feature fusion. Finally, the fractional images with the proportions of subpixels belonging to land-cover classes are derived by unmixing the MRSFD, and the classes labels are allocated to subpixels to obtain the final SPM result according to these proportions' information. The main contribution of this work is that multiple types of auxiliary information (i.e., spatial-spectral elevation information) from MRSFD is fully utilized and the accuracy of SPM result is improved. Experimental results show that SPM-MRSFD obtains more accurate mapping results than state-of-the-art SPM methods.

Remote-sensing image registration is a pivotal preprocessing step for earth observation data analytics. In this regard, georeferencing corrects the systematic geometric degradation in the image. However, it is difficult to achieve subpixel geometric accuracy across multitemporal scenes. This article focuses on hindmost part of geometric correction that uses reference layer and feature detection in hierarchical stages to improve the geometric fidelity of images at subpixel level. The methodology developed is based on patch affine-oriented fast and brief with mode-guided tiled scale invariant feature transform (MT-SIFT) techniques in a coordinate manner at a multistage processing architecture, which we refer to as FIRM. Motion smoothness constraint (MSC) keypoint correspondence optimization is used in FIRM to remove the outliers at gross stage and estimate segmented affine transformation parameters at finer stage. The automatic coregistration pipeline is evaluated in Indian Resourcesat multispectral camera images covering diverse landscapes. The capability of the designed framework is demonstrated to handle relatively large geometrical error. With more than a decade difference in acquisitions, multitemporal images are superimposed over each other and compared with state-of-the-art feature-based methods. The potential of the proposed approach FIRM is assessed on multisatellite imagery acquired from Resourcesat-2 and Landsat -8. It is observed that the root mean square error (RMSE) between coregistered images is 0.12 pixel at a spatial resolution of 5 m.

Mixed pixel problem is omnipresent in remote sensing images for urban land use interpretation due to the hardware limitations. Subpixel mapping (SPM) is a usual way to solve this problem by improving the observation scale and realizing a finer spatial resolution land cover mapping. Recently, deep learning-based subpixel mapping network (DLSMNet) was proposed, benefited from its strong representation and learning ability, to restore a visually pleasing finer mapping. However, the spatial context features of artifacts are usually aggregated and progressively lost during the forward pass of the network without sufficient representation, which make it difficult to be learned and restored. In this article, a semantic information modulated (SIM) deep subpixel mapping network (SIMNet) is proposed, which uses low-resolution semantic images as prior, to reinforce the representation of spatial context features. In SIMNet, SIM module is proposed to parametrically incorporate the semantic prior into the state-of-the-art (SOTA) feed forward network architecture in an end-to-end training fashion. Furthermore, stacked SIM module with residual blocks (SIM_ResBlock) is adopted to pass the representation of spatial context feature to the deep layers, to get it fully learned during backpropagation. Experiments have been implemented on three public urban scenario data sets, and the SIMNet generates a clearer outline of artificial facilities with sufficient spatial context, and is distinctive for even individual building, which is challenging for other SOTA DLSMNet. The results demonstrate that the proposed SIMNet is a promising way for high-resolution urban land use mapping from easily available lower resolution remote sensing images.Mixed pixel problem is omnipresent in remote sensing images for urban land use interpretation due to the hardware limitations. Subpixel mapping (SPM) is a usual way to solve this problem by improving the observation scale and realizing a finer spatial resolution land cover mapping. Recently, deep learning-based subpixel mapping network (DLSMNet) was proposed, benefited from its strong representation and learning ability, to restore a visually pleasing finer mapping. However, the spatial context features of artifacts are usually aggregated and progressively lost during the forward pass of the network without sufficient representation, which make it difficult to be learned and restored. In this article, a semantic information modulated (SIM) deep subpixel mapping network (SIMNet) is proposed, which uses low-resolution semantic images as prior, to reinforce the representation of spatial context features. In SIMNet, SIM module is proposed to parametrically incorporate the semantic prior into the state-of-the-art (SOTA) feed forward network architecture in an end-to-end training fashion. Furthermore, stacked SIM module with residual blocks (SIM_ResBlock) is adopted to pass the representation of spatial context feature to the deep layers, to get it fully learned during backpropagation. Experiments have been implemented on three public urban scenario data sets, and the SIMNet generates a clearer outline of artificial facilities with sufficient spatial context, and is distinctive for even individual building, which is challenging for other SOTA DLSMNet. The results demonstrate that the proposed SIMNet is a promising way for high-resolution urban land use mapping from easily available lower resolution remote sensing images.

Subpixel mapping (SPM) is an effective way to solve the mixed pixel problem, which is a ubiquitous phenomenon in remotely sensed imagery, by characterizing subpixel distribution within the mixed pixels. In fact, the majority of the classical and state-of-the-art SPM algorithms can be viewed as a convolution process, but these methods rely heavily on fixed and handcrafted kernels that are insufficient in characterizing a geographically realistic distribution image. In addition, the traditional SPM approach is based on the prerequisite of abundance images derived from spectral unmixing (SU), during which process uncertainty inherently exists and is propagated to the SPM. In this article, a kernel-learnable convolutional neural network (CNN) framework for subpixel mapping (SPMCNN-F) is proposed. In SPMCNN-F, the kernel is learnable during the training stage based on the given training sample pairs of low- and high-resolution patches for learning a geographically realistic prior, instead of fixed priors. The end-to-end mapping structure enables direct subpixel information extraction from the original coarse image, avoiding the uncertainty propagation from the SU. In the experiments undertaken in this study, two state-of-the-art super-resolution networks were selected as application demonstrations of the proposed SPMCNN-F method. In experiment part, three hyperspectral image data sets were adopted, two in a synthetic coarse image approach and one in a real coarse image approach, for the validation. Additionally, a new data set with pairs of Moderate-resolution Imaging Spectroradiometer (MODIS) and Landsat images were adopted in a real coarse image approach, for further validation of SPMCNN-F in large-scale area. The restored fine distribution images obtained in all the experiments showed a perceptually better reconstruction quality, both qualitatively and quantitatively, confirming the superiority of the proposed SPM framework.

We propose a deep learning framework by the probability of the predicting semantic spatial position distribution for remote sensing image registration. Traditional matching methods optimize similarity metrics with pixel-by-pixel searching, which is time consuming and sensitive to radiometric differences. Driven by learning-based methods, we take the reference and template images as inputs and map them to the semantic distribution position of the corresponding reference image. We apply the affine invariant to obtain a correspondence between the overlap of the barycenter position of the semantic template and the center pixel point, which transforms the semantic boundary alignment into a point-to-point matching problem. Additionally, two loss functions are proposed, one for optimizing the subpixel matching position and the other for determining the semantic spatial probability distribution of the matching template. In this work, we explore the influence of the template radius size, the filling form of training labels, and the weighted combination of loss function on the matching accuracy. Our experiments show that the trained model is robust to template deformation while still operating orders of magnitude faster. Furthermore, our proposed method implements high matching accuracy in four large scene images with radiometric differences. This ensures the improved speed of remote sensing image analysis and pipeline processing while facilitating novel directions in learning-based registration. Our code is freely available at https://github.com/liliangzhi110/semantictemplatematching .

Satellite stereo mapping, together with laser altimetry, can be used to obtain three-dimensional geospatial information. Spaceborne laser altimeter can provide high-accuracy elevation information; however, due to the lack of detailed intensity information, its planimetric accuracy is usually worse than the ranging accuracy. The Chinese Gaofen-7 (GF-7) satellite, which was designed for civilian mapping application, was launched on November 3, 2019. The GF-7 satellite’s main payloads are a laser altimeter system (with footprint camera) and a dual-linear charge-coupled device (CCD) mapping camera. According to the pixel coordinate of the laser footprint in the stereo image, the laser altimeter together with the footprint camera can provide planimetric geodetic coordinates for the control points of a higher accuracy than the other traditional satellite laser altimeters, and represents a new technology for satellite mapping. In this article, a laser footprint planimetric location method for the GF-7 satellite is proposed. The method is designed based on the main payload characteristics of GF-7 and the working modes of the laser altimeter by the combined use of subpixel phase correlation image matching and four types of laser spot center detection methods. The planimetric positioning accuracies of the laser spots in urban, suburban, farmland, forest, mountainous, and ice sheet areas were also analyzed. The experimental results show that the accuracy of planimetric location relative to stereo image for the laser footprint is 0.3–1.0 m (except for ice sheets ~12 m) when the footprint camera works under the synchronous mode, and 0.2–0.4 m when the footprint camera works under asynchronous mode (AM).

Denoising of hyperspectral images (HSIs) is an important preprocessing step to enhance the performance of its analysis and interpretation. In reality, a remotely sensed HSI experiences disturbance from different sources and therefore gets affected by multiple noise types. However, most of the existing denoising methods concentrates in removal of a single noise type ignoring their mixed effect. Therefore, a method developed for a particular noise type doesn’t perform satisfactorily for other noise types. To address this limitation, a denoising method is proposed here, that effectively removes multiple frequently encountered noise patterns from HSI including their combinations. The proposed dual branch deep neural network based architecture works on wavelet transformed bands. The first branch of the network uses deep convolutional skip connected layers with residual learning for extracting local and global noise features. The second branch includes layered autoencoder together with subpixel upsampling that performs repeated convolution in each layer to extract prominent noise features from the image. Two hyperspectral datasets are used in the experiment to evaluate the performance of the proposed method for denoising of Gaussian, stripe and mixed noises. Experimental results demonstrate the superior performance of the proposed network compared to other state-of-the-art denoising methods with PSNR 36.74, SSIM 0.97 and overall accuracy 94.03 %.

This article presents new developments in a multiframe blind deconvolution algorithm for imaging low earth orbiting objects during flyover. The foundational aspects of the algorithm rely on the constrained maximum likelihood (ML) formulation in the presence of Poisson noise, previously developed in Schulz et al. (2018). The new algorithm achieves real time evaluation capability at over two frames per second, which is allowed by two novel aspects. First, the prototype algorithms are transferred to highly parallelized computations on graphical processing units (GPUs) with CUDA implementation that reduce the computational time of a single iteration by a factor of up to 100. Second, new numerical optimization strategies are developed and demonstrated to accelerate the convergence of the algorithm by a factor of 5 to 10. Several other new capabilities are also demonstrated in this article. We derive and implement a modified variation of the algorithm that achieves subpixel resolution, which is shown effective on real and simulated data. Finally, a new post-processing visual enhancement technique is proposed with several examples, which in part helps deal with the dynamic range degradation due to glint.

Super-resolution (SR) has attracted increasing attention as it can improve the quality of video satellite images. Most previous studies only consider several integer magnification factors and focus on obtaining a specific SR model for each scale factor. However, in the real world, it is a common requirement to zoom the videos arbitrarily by rolling the mouse wheel. In this article, we propose a unified network for arbitrary scale SR (ASSR) of video satellite images. The proposed ASSR consists of two modules, i.e., feature learning module and arbitrary upscale module. The feature learning module accepts multiple low-resolution (LR) frames and extracts useful features of those frames by using many 3-D residual blocks. The arbitrary upscale module takes the extracted features as input and enhances the spatial resolution by subpixel convolution and bicubic-based adjustment. Different from existing video satellite image SR methods, ASSR can continuously zoom LR video satellite images with arbitrary integer and noninteger scale factors in a single model. Experiments have been conducted on real video satellite images acquired by Jilin-1 and OVS-1. Quantitative and qualitative results have demonstrated that ASSR has superior reconstruction performance compared with the state-of-the-art SR methods.

We consider the problem of visual odometry for a sequence of video frames using a camera directed perpendicularly downward. We propose an adaptive two-stage visual odometry technology based on sequential determination of interframe shifts and regular correction of current coordinate estimates. At the first stage, the shift between two consecutive frames is determined by the correlation method, with the compared video frames being aligned using the found shift parameters up to a pixel. At the second stage, the shifts are refined with subpixel precision using the optical flow method. To improve reliability, the most consistent estimates of the optical flow are selected. We present the results of experimental studies on publicly available survey data, which confirm the high reliability and accuracy of the estimates.

Hyperspectral unmixing is an important problem for remotely sensed data interpretation. It amounts at estimating the spectral signatures of the pure spectral constituents in the scene (endmembers) and their corresponding subpixel fractional abundances. Although the unmixing problem is inherently nonlinear (due to multiple scattering), the nonlinear unmixing of hyperspectral data has been a very challenging problem. This is because nonlinear models require detailed knowledge about the physical interactions between the sunlight scattered by multiple materials. In turn, bilinear mixture models (BMMs) can reach good accuracy with a relatively simple model for scattering. In this article, we develop a new BMM and a corresponding unsupervised unmixing approach which consists of two main steps. In the first step, a deep autoencoder is used to linearly estimate the endmember signatures and their associated abundance fractions. The second step refines the initial (linear) estimates using a bilinear model, in which another deep autoencoder (with a low-rank assumption) is adapted to model second-order scattering interactions. It should be noted that in our developed BMM model, the two deep autoencoders are trained in a mutually interdependent manner under the multitask learning framework, and the relative reconstruction error is used as the stopping criterion. The effectiveness of the proposed method is evaluated using both synthetic and real hyperspectral data sets. Our experimental results indicate that the proposed approach can reasonably estimate the nature of nonlinear interactions in real scenarios. Compared with other state-of-the-art unmixing algorithms, the proposed approach demonstrates very competitive performance.

Orthogonal subspace projection (OSP) has been widely used in many applications for hyperspectral data exploitation. However, its performance is sensitive to its used prior target knowledge, which is significantly affected by target background (BKG). To resolve this issue, this article develops three approaches to extend OSP in improving its performance. One is data sphering which can suppress BKG via removing the first- and second-order data statistics. Another takes advantage of a recently developed low-rank and sparse matrix decomposition (LRaSMD) to separate BKG and target signal sources in two subspaces characterized by the low-rank matrix and sparse matrix, respectively, and then annihilates BKG via the low-rank matrix, referred to as BKG-annihilated OSP (BA-OSP). A third approach combines data sphering and LRaSMD to further improve OSP over target detectability and BKG suppressibility. Experiments show that implementing OSP in conjunction with data sphering and LRaSMD significantly improves OSP in target detection and BKG suppression, and also performs as well as a widely used constrained energy minimization (CEM)-based subpixel target detection.

Archival aerial imagery (AAI) represents a unique and relatively unexploited resource for assessing long-term environmental changes at a very high spatial resolution. A major constraint for the wider use of AAI often lies in the difficulties of establishing precise geo-referencing, namely in the difficult and time-consuming task of assigning ground reference through manual digitization of Ground control points (GCPs). We present a highly automated photogrammetric workflow for orientation of AAI. The workflow substitutes manual GCP measurements by generating image matches to a digital reference. The resulting abundant observations are algorithmically filtered and used in a bundle block adjustment (BBA) to obtain final image orientations. The proposed workflow has successfully been employed to process a complete coverage of AAI over the territory of Switzerland based on images acquired between 1985 and 1991. The accuracies obtained from the orientation process are very satisfying and allow for generating meaningful 2D and 3D products. The absolute accuracy for derived orthophotos and their mosaics is about 1 m. The relative accuracies are in the subpixel range and allow for generation of country-wide Digital surface models (DSMs) through dense-image matching. The obtained accuracies are comparable to those obtained at the authors’ affiliation using classical workflows that involve manual GCP identification from digital reference data. With regard to human working time, the workflow has, in our case, proven to be at least five times more efficient than classical workflows whilst the required computational resources are very moderate.

In this paper, the interpolation sub-pixel mapping (ISM) based on multiple sub-pixel shifted images (MSSIs) based on a consideration of the point spread function (PSF) effect, named as MSSI-PSF, is...

Ultrasound localization microscopy can image microvasculature in vivo without sacrificing imaging penetration depth. However, the reliance on super-resolution inference limits the applicability of the technique as subpixel tissue motion can corrupt microvascular reconstruction. Consequently, the majority of previous pre-clinical research on this super-resolution procedure has been restricted to low motion experimental models with ample motion correction and/or data rejection, which precludes the imaging of organ sites with that exhibit a high degree of respiratory and other motion. In this paper, we present a novel anesthesia protocol in rabbits that induces safe controllable periods of apnea to enable the long imaging acquisition times required for ultrasound localization microscopy. We apply this protocol to a VX2 liver tumor model undergoing sorafenib therapy and compare the results to super-resolution images from a conventional high-dose isoflurane anesthesia. We found that the apneic protocol was necessary to correctly identify the poorly vascularized tumor cores, as verified by immunohistochemistry, and to reveal the tumoral microvascular architecture.

This article presents a novel multiaxis $\mu$ Force sensor with a range-to-resolution ratio of 55 000, which makes the sensing device highly useful for wide range of applications. The presented device relies on a sensing strategy where a 2-D encoded micropattern is encrypted to the mobile part of the compliant platform. This encoded pattern allows multiaxis visual sensing of displacements over a long range, whereas Fourier computations ensure subpixel interpolation leading to the high resolution. The device is fabricated on a silicon wafer by means of clean room technology to comply with both precision encoding requirements and reliable mechanical behavior. This article presents the design, modeling, and fabrication as well as the experimental validation of the multiaxis $\mu$ Force-Torque sensor. A modeling strategy is proposed for an effective estimation of force/torque with its capability to adapt nonlinear stiffness evolution during the external loading. Experimentation demonstrates a sensing resolution of 2 $\mu$ N over a validated range of 110 mN. The proposed sensor measures the force along the two planar axes, and also the torque along the axis orthogonal to the plane, thus, validating its axial decoupling capabilities.

Commonly available digital image correlation (DIC) tools are designed to evaluate displacement/deformation fields using well-optimized correlation functions and computational procedures. However, real-time monitoring at high sampling rates or tracking motion at predefined locations from large sequence images can be hardly accomplished due to extreme computational costs. As shown in this paper, a lightweight approach based on the registration of subpixel subset shifts using discrete Fourier transform (DFT) can easily tackle large sets of images if the measurement is limited to discrete locations. A simple open-source python software designed for evaluation of displacements using virtual extensometers is described in this paper, along with the analysis of its performance. It is shown that the DFT-based algorithms outperform a commonly used DIC package in terms of computational efficiency and required user interactions, without sacrificing precision and accuracy.

The dropper is one of the core components of the high speed railway catenary and dropper failure will lead to serious transportation accidents. It is very important to carry out dropper failure and defect detection. There are a large number of droppers installed in the catenary. The dropper images are collected by the high-definition camera installed on the top of the moving catenary inspection vehicles. The image quality and image consistency are poor, and it is very difficult to identify dropper defects automatically. The railway department and companies lack efficient and intelligent detection methods. This article innovatively proposes a multialgorithm fusion image processing technology, and builds a dropper recognition and failure–defect detection model based on a deep learning algorithm and subpixel level dropper defect detection model, and achieves high accuracy dropper failure and defect detection. The detection model based on the Faster R-CNN algorithm is studied to realize the positioning and recognition of the dropper and the failure detection of bending slack and broken dropper. The subpixel level dropper defect detection algorithm is based on the fusion of the image preprocessing, dropper fine positioning algorithm, edge fitting and bending zoom algorithms, the Hough transform algorithm, and so on. These can be used to realize detection of defects, such as microdeformation, dropper-strands loosened, dropper-strands broken, and foreign body adhesions. The test is verified by the catenary images taken from a practical high speed railway. The detection accuracy, real-time performance, and stability of the algorithm meet the needs of inspection and maintenance for a high speed railway.

We present the results of simulation of image recognition using invariant correlation filters. The possibilities of exploiting color information and preprocessing for quality improvement of subpixel image correlation matching are shown.

Photoacoustic dermoscopy (PAD) has been proven to visualize the microvascular network within the dermis noninvasively, and has great application potential and advantages. In practice, some subjects are children or unwell and may have difficulty in controlling their breathing or trembling. The artifacts caused by the uncontrollable trembling will degrade the image quality, resulting in deviation of the pathophysiological features. This work introduces a subpixel and on-line motion correction method for clinical application of PAD. The high-accuracy motion correction was realized by the subpixel motion estimation, which was achieved by five-fold upsampling the cross-correlation matrix between A-lines or B-scan images. Since the motion correction algorithm only takes the acquired data as a priori, it can be processed immediately when a B-scan is input, so it can be embedded in the acquisition program, leading to the motion correction during the data acquisition. Moreover, the algorithm was verified by the experimental data of human skin. The results demonstrated that the images after sub-pixel motion correction are obviously improved visually, and the structural similarity index measurement (SSIM) and peak signal-to-noise ratio (PSNR) between adjacent B-scan images after correction are increased by 31.6% and 47.7% compared to the uncorrected images. To conclude, the proposed algorithm can make it robust in terms of accuracy and reliability and will help PAD fulfill its clinical promise.

Turbulent heat flux (THF) over leads is an important variable used for monitoring climate change in the Arctic. Presently, THF over leads is often calculated from satellite imagery. The accuracy of the estimated THF is low for mixed pixels that consist of ice and leads, because the mixed pixels along lead boundaries will lower the accuracy of the surface temperature measured over leads and the corresponding lead map. To address this problem, a deep residual convolutional neural network (CNN)-based framework is proposed to estimate THF over leads at the subpixel scale (DeepSTHF) with remotely sensed imagery. The DeepSTHF allows the production of a sea surface temperature (SST) image and a corresponding lead map with a finer spatial resolution than the input SST image using two CNNs, so that the subpixel scale THF can be estimated from them. The proposed approach is assessed using simulated and real MODIS imagery and compared against the conventional bicubic interpolation and pixel-based methods. The results demonstrate that the proposed CNN-based method can effectively estimate subpixel-scale information from the coarse data and performs well in producing fine spatial resolution SST images and lead maps, thereby allowing researchers to obtain more accurate and reliable THF over leads.

We have developed a new high-speed driving circuit of a multi-color LED panel by the use of FPGA for subjective super-resolution display. In the subjective super-resolution display, the coded display image is switched and displayed at high speed. The viewer makes an unconscious correction by fixational eye movement and perceives the resolution higher than the physical number of pixels of the LED panel. Our developed LED display drive circuit shows multi-color images at a sufficiently high frame rate enough to perform the subjective super-resolution effect. The maximum frequency is 8.5 kHz for 8 multi-color representations. We have designed spatiotemporal coding to show nine adjacent subpixel information for 9 multi-color subframes. The coded 9 subframes are alternately shown on the LED panel by the use of our developed FPGA circuit. We have verified that the image was updated at high speed. The unconscious correction by the fixational eye movement allows the viewer to perceive a higher resolution than the actual number of LED lamps. Through an evaluation experiment, we have verified that the subjective super-resolution effect was induced in a multi-color image, that is, the displayed image in the proposed method looked smoother than that in the normal display method.

The disadvantage of low resolution and low signal-to-noise ratio (SNR) of thermal infrared (TIR) data makes it difficult to be applied in practice, although it has been studied for decades. In contrast, visible and near infrared (VNIR) and shortwave infrared (SWIR) data have higher resolution and SNR. In this letter, an ensemble extreme learning machine (EELM) is proposed, which is based on the extreme learning machine (ELM), to fuse together VNIR, SWIR, and TIR to improve TIR resolution and SNR. The associated ELM nets are trained with correlated training data which are selected from the random forest method. The temperature estimation cube can be obtained after putting SWIR data and VNIR data into these networks. Finally, the cube structure is adjusted by an ensemble rule to obtain the final temperature estimation results. We test the EELM algorithm on the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) L1B data set and compare it with other scholars’ algorithms. The experimental results show that the EELM algorithm can not only improve the resolution of TIR data, but also estimate the subpixel temperature of TIR more accurately.

Digital image correlation (DIC) has become a superior tool for surface deformation measurement so far. In order to improve its measurement accuracy, different error models have been developed to reveal the cause of various error. An error model unifying the random error and systematic error should be valuable. In this paper, a frequency-domain error model for 2D-DIC is established based on Fourier analysis. Using the model, the error caused by random noise, subpixel interpolation and the combination of the two can be analyzed and predicted. Numerical experiments were conducted to verify the effectiveness of the proposed models. The results from theoretical models and numerical experiments are in a good agreement. It is demonstrated that the proposed model is a generalization of several previously established models.

The requirement for increased resolution has created the concept of displays with nanoscale pixels; that is, each subpixel consists of multiple or even a single nanolight source, which is considered the ultimate light source for light field, near-eye, and implantable displays. However, related research is still at an early stage, and further insights into this future display concept should be provided. In this Perspective, we provide our proposed term for this future display, namely, nanopixel light-emitting display (NLED). We present an overview of nanolight-emitting diodes, which are considered the core component of NLEDs. Then, a roadmap to realize NLEDs from the view of material design is provided. Finally, we introduce our proposed operation mode (nonelectrical contact and noncarrier injection mode) for NLEDs and recommend possible nanopixel-level drive approaches. We hope that this Perspective will be helpful in designing innovative display technologies.

Timely mapping of corn and soybean plays an important role in food security in the USA. A subpixel fraction derived from Land Remote-Sensing Satellite (System, Landsat) imagery during growing seaso...

The low spatial resolution associated with imaging spectrometers has caused subpixel target detection to become a special problem in hyperspectral image (HSI) processing that poses considerable challenges. In subpixel target detection, the size of the target is smaller than that of a pixel, making the spatial information of the target almost useless so that a detection algorithm must rely on the spectral information of the image. To address this problem, this article proposes a subpixel target detection algorithm for hyperspectral remote sensing imagery based on background endmember extraction. First, we propose a background endmember extraction algorithm based on robust nonnegative dictionary learning to obtain the background endmember spectrum of the image. Next, we construct a hyperspectral subpixel target detector based on pixel reconstruction (HSPRD) to perform pixel-by-pixel target detection on the image to be tested using the background endmember spectral matrix and the spectra of known ground targets. Finally, the subpixel target detection results are obtained. The experimental results show that, compared with other existing subpixel target detection methods, the algorithm proposed here can provide the optimum target detection results for both synthetic and real-world data sets.

Due to the influences of imaging conditions, spectral imagery can be coarse and contain a large number of mixed pixels. These mixed pixels can lead to inaccuracies in the land-cover class (LC) mapping. Super-resolution mapping (SRM) can be used to analyze such mixed pixels and obtain the LC mapping information at the subpixel level. However, traditional SRM methods mostly rely on spatial correlation based on linear distance, which ignores the influences of nonlinear imaging conditions. In addition, spectral unmixing errors affect the accuracy of utilized spectral properties. In order to overcome the influence of linear and nonlinear imaging conditions and utilize more accurate spectral properties, the SRM based on spatial–spectral correlation (SSC) is proposed in this work. Spatial correlation is obtained using the mixed spatial attraction model (MSAM) based on the linear Euclidean distance. Besides, a spectral correlation that utilizes spectral properties based on the nonlinear Kullback–Leibler distance (KLD) is proposed. Spatial and spectral correlations are combined to reduce the influences of linear and nonlinear imaging conditions, which results in an improved mapping result. The utilized spectral properties are extracted directly by spectral imagery, thus avoiding the spectral unmixing errors. Experimental results on the three spectral images show that the proposed SSC yields better mapping results than state-of-the-art methods.

Mapping and monitoring of the glacier changes over different regions of Earth surface is a challenging task due to regional rugged topography and climate conditions. This study focused on the monitoring of snow or ice cover changes over Chhota-Shigri glacier, Western Himalaya, India. A subpixel-based change detection (SCD) approach is proposed, aiming to identify the transition zones (mixed pixels) between the two class categories. The SCD approach involves the integration of subpixel classification and change vector analysis (CVA) to define the changes in the form of magnitude and direction between two multitemporal dates at the subpixel level. To check the efficacy of proposed SCD, experimental outcomes have also been compared with existing neural-network (NN) based SCD (NN‒SCD). The result analysis has shown that proposed SCD achieved better accuracy (84.80%) as compared to NN‒SCD (78.80%). In addition, a time series data was acquired using the Landsat series (Landsat 5, 7 and 8 as per availability) to perform the trend analysis over Chhota-Shigri glacier, during the period 2001–2019. This study offers the effective way of estimating the bi-temporal snow/ice changes especially over rugged terrains around the globe.

Surface water plays a vital role in natural environment and human development. The research of water extraction method using remote sensing image is a hot topic, which has been widely developed in water index, classification, subpixel, and other aspects. Compared with other methods, a water-index based method has the advantages of fast speed and convenience. The characteristics of surface water, such as wide coverage and instability, make the water index stand out in monitoring large area of surface water. However, land surface in water environment is complex, and the main factors that reduce water extraction accuracy are also different, such as shadow in urban areas and water leakage in unshaded areas. The current index is bound to weaken the information in the water body when suppressing shadows, and vice versa. To address these issues, contrast difference water index (CDWI) and shadow difference water index (SDWI) are proposed in this paper by improving the modified normalized difference water index (MNDWI). CDWI is used to enhance water information, which is suitable for areas without building shadows. SDWI is used to eliminate the shadow of buildings, which is suitable for urban areas. Moreover, background difference water index (BDWI) was proposed by combining the advantages of CDWI and SDWI through a background regularizer B, which is used to extract surface water under complex background. The regularizer B represents the similarity between local background features of the image and the reference urban area, which is used to locally weight SDWI and CDWI, so that BDWI can automatically enhance the water body in the shadowless area and eliminate the shadow of buildings. The water extraction results of BDWI, MNDWI, the tasseled cap wetness index (TCW), the automatic water extraction index (AWEInsh, AWEIsh), and the water index 2015 (WI2015) were used for comparison. Other methods tend to perform well only in built-up areas or non-built-up areas, while the BDWI can extract surface water under various backgrounds with high accuracy and stability. The overall accuracy produced by the BDWI was 91.58–97.57%, CDWI was 84.85–97.09%, SDWI was 81.63–94.40%, MNDWI was 80.19–95.64%, TCW was 82.33–95.98%, AWEIsh was 87.50–96.37%, AWEInsh was 80.59–98.78%, and WI2015 was 78.24–98.38%. Combining water index with image local information is helpful to improve the accuracy of water extraction in large and complex environment. Finally, surface water in Jiangsu Province, China was extracted through BDWI and the changes in 1985, 2000, and 2015 were analyzed.

The super-resolution reconstruction of X-ray images is one of the hot issues in the field of medical imaging. Due to the limitations of X-ray machines, the acquired images often have some problems, such as blurred details, unclear edges and low contrast, which seriously affect doctors’ interpretations of X-ray images. In view of the above problems, an X-ray image super-resolution reconstruction method based on a multiple distillation feedback network is proposed. In the feature extraction stage, the shallow features of X-ray images are extracted through 3 × 3 and 1 × 1 convolutional layers, and a multiple distillation feedback module is designed, which iteratively upsamples and downsamples to fully extract the texture details of X-ray images. Subpixel convolution is used to improve the resolution and the residual convolution is used to predict the corresponding residual image. In the image reconstruction stage, the residual image is fused with the transposed convolution upsampled image, and the Laplacian pyramid structure is used to progressively reconstruct high-resolution X-ray images. The experimental results show that the proposed method can suppress noise and reduce artifacts. Its peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and information fidelity criteria (IFC) were all higher than those of the comparison methods, and its subjective visual was better.

Phase-shifting profilometry (PSP) via sequential fringe projection is applicable for high-accuracy 3-D measurements of static objects. However, moving objects cannot be static or quasi-static during one projection cycle, thereby causing surface misalignment and phase errors in 3-D measurements. This article proposes a motion-estimated PSP based on the fringe-oriented synthetic phase correlation algorithm (FO-SPCA), which achieves automatic 3-D measurements of objects in nonuniform 2-D motion. To robustly estimate the motion of objects among fringe images, the proposed fringe-oriented feature extraction technique obtains the surface features by eliminating the modulation influence of sinusoidal illumination. Furthermore, the FO-SPCA is developed to accurately detect the subpixel displacements of moving objects according to object surface features. With the subpixel displacements, the alignment procedure is completed through a reverse 2-D transformation and then the phase errors are compensated accordingly. The simulation and experimental results verify the feasibility and robustness of the motion-estimated PSP for the in situ 3-D measurements of nonuniform moving objects.

A method to construct a large transmission matrix (TM) of highly scattering media based on an algorithm is presented. For imaging through scattering media, measuring a large TM enables image reconstruction with high spatial resolution; however, it is difficult to operate in practice because the time required for the calibration of the TM is proportional to the number of calibrated input modes. Instead of directly achieving calibration using advanced hardware, a large TM is defined and constructed based on a multiframe image super-resolution algorithm, which traditionally aims to recover a high-resolution (HR) image of the original object from several low-resolution images. With sufficient size improvement, the defined large TM is constructed from several naturally measured complementary small TMs, each of them is measured using a subpixel-shifted projector phase mask. The realized large TM, which is constructed through subpixel registration and interpolation operations at the help of a hypothetical HR mask, enables HR image reconstruction from the original object’s speckle signal. The feasibility of the proposed method is proven via optical experiments involving statistical analysis and image reconstruction. The proposed method benefits existing TM-based image reconstruction applications and offers a new perspective on the size of TM measurements and the imaging resolution.

Hyperspectral images stand out from other remote sensing images in anomaly target detection because they contain unique distinguishing spectral information and attract great interest in applications of search and rescue. However, most of the popular techniques for hyperspectral anomaly detection tasks focus on improving accuracy with complicated algorithms and face difficulty in efficiently balancing performance and complexity. In this paper, we propose a novel anomaly detection approach using a selected band image extracted from the band selection model combined with an image filter. Singular value decomposition (SVD) is adopted for spectral dimensionality reduction. A dual-window guided filter is constructed to highlight the potential anomaly targets. To quickly calculate the abnormity degree, we design an efficient diagonal matrix operation to achieve the energy of each pixel, and a spatial regulation model is designed to enhance the subpixel target detection performance. Extensive experiments conducted on two real-world hyperspectral datasets demonstrate that, compared with the existing relevant state-of-the-art approaches, the proposed method requires less detection time and achieves higher detection accuracy.

Railway wheels are one of the important parts of railway vehicles, and it is necessary to inspect their status frequently. To measure the wheel tread geometric parameters dynamically and accurately in a train running environment, a line-structured light vision sensor-based measurement system was demonstrated in this paper. In the system, a subpixel laser stripe centre extraction algorithm based on skeleton extraction, which can significantly reduce the interference of complex lighting environments in the field and greatly improve the extraction accuracy, was proposed. To further improve the measurement accuracy and stability, the influence of the eccentricity error caused by the dynamic measurement on the results was analyzed, a mathematical model was established, and the deformed profile was corrected. The system has been successfully applied to railway maintenance sections and has become a part of train safety inspection systems. Field tests were conducted to verify the performance of the system, and the results showed that the measurement accuracy and stability are marked improved after eccentric error correction, especially for the flange height and QR value.

Efficient accurate Gaussian localization is an important topic in many applications, e.g. localization based super-resolution microscopy and image scanning microscopy, which requires large-scale Gaussian patterns localization for accurate super-resolution image reconstruction. Existing Gaussian localization methods usually require high signal-to-noise image and the existing standard fitting algorithm usually requires manually inputting a good initial value for all parameters, which could be not convenient to use and difficult to guarantee high robustness for large-scale Gaussian localizations with a computer. It would be even more challenge to detect all the Gaussian patterns with high-dynamic-range of amplitudes, as well as to estimate a good initial value for all parameters for efficient Gaussian fitting and guarantee high robustness of the localization algorithm for low signal-to-noise ratio image data with strong background. In this paper, we propose an efficient Gaussian patterns detection technique and a robust Gaussian fitting method for accurate Gaussian fitting without initial estimation. In our technique, a fast Pearson correlation algorithm is proposed to improve the efficiency of the calculation of normalized cross correlation for large scale object detection with template matching. By introducing blind background estimation, a modified iterative least-squares Gaussian fitting algorithm without initials estimation is proposed for robust Gaussian fitting with noisy data with strong background. The simulation shows that the performance of the proposed detection technique is high for low SNR image and an efficiency improvement of 27% can be achieved; the proposed Gaussian fitting algorithm is capable of calculating all parameters without initial estimation, and the resulting fitting accuracy is very close to exiting standard methods, which indicates that image signal-to-noise ratio higher than 10dB is required to obtain subpixel accuracy.

With the launch of Sentinel-2 new opportunities for large scale urban mapping arise. However, the spectral information embedded in the Sentinel-2 20 m spatial resolution bands cannot yet be fully explored in heterogeneous urban landscapes. The 20 m image pixels are often composed of different land covers, resulting in a difficult to interpret mixed pixel spectrum. Here, we propose an unmixing-based image fusion algorithm (UnFuSen2) that self-adapts to the spectral variability of varying land covers and improves the image fusion accuracy by constraining the unmixing equations on the basis of spectral mixing models and the correlation between spectral bands of coarse and fine spatial resolution, respectively. When compared to alternative state-of-the-art downscaling methods UnFuSen2 consistently showed the highest accuracy when applied across test sites in three different European cities (RMSEUnFuSen2 = 203 vs RMSEalternatives = [252, 337]). In a next step, we applied Multiple Endmember Spectral Mixture Analysis (MESMA) on the downscaled Sentinel-2 image cube (i.e. ten 10 m bands) to generate subpixel urban land cover fractions. We compared our MESMA results against the traditional MESMA output as applied on the original Sentinel-2 image cube (i.e. four 10 m bands and six 20 m bands) and tested its robustness against reference data obtained over all three study sites. Results revealed an average decrease in RMSE of respectively 18% and 8% for impervious surface and vegetation fractions when our approach was compared to the traditional MESMA outcomes.