Search

PURPOSE: To derive a multinomial probability function and quantitative measures of the data and epistemic uncertainty as direct output of a 3D U-Net segmentation network. APPROACH: A set of T1 brain MRI images were downloaded from the Connectome Project and segmented using FMRIB’s FAST algorithm to be used as ground truth. A 3D U-Net neural network was trained with sample sizes of 200, 500, and 898 T1 brain images using a loss function defined as the negative logarithm of the likelihood based on a derivation of the definition of the multinomial probability function. From this definition, the epistemic and aleatoric uncertainty equations were derived and used to quantify maps of the uncertainty along with tissue segmentations. RESULTS: Maps of the tissue segmentation along with the epistemic and aleatoric uncertainty, per voxel, are presented. The uncertainty decreased based on the increasing number of training data used to train the neural network. The neural network trained with 898 volumes resulted in uncertainty maps that were high primarily in the tissue boundary regions. The epistemic and aleatoric uncertainty were averaged over all test data (connectome and tumor separately), and the epistemic uncertainty showed a decreasing trend, as expected, with increasing numbers of data used to train the model. The aleatoric uncertainty showed a similar trend which was also expected as the aleatoric uncertainty is not expected to be as dependent on the number of training data. CONCLUSION: The derived uncertainty equations from a multinomial probability distribution were able to quantify the aleatoric and epistemic uncertainty per voxel and are applicable for all two-dimensional and three-dimensional neural networks.

Purpose: Our goal is to propose a landmark- and contour-matching (LCM) registration method that uses both landmark information and approximate point correspondences to boost the similarity between image pairs with sparse landmark information. Approach: A model for registering two-dimensional (2D) medical images with landmark information and contour-approximating landmarks was proposed. The model was also extended to accommodate the registration of three-dimensional (3D) cardiac images. We validated the LCM method on 2D hand x-rays and 3D porcine cardiac magnetic resonance images. The following metrics were used to assess the quality of specific aspects of the registered images: Dice similarity coefficient for the overall image overlap, target registration error for pointwise correspondence, and interior angle for local curvature. Results: Target registrations were reduced from 27.12 to 0.01 mm post-LCM registration. Implementing the proposed algorithm also led to a 112% average improvement in image similarity in terms of Dice coefficients. In addition, interior angle measurements indicate that the proposed method preserved the local curvature at major reference landmarks and mitigated the appearance of deformities in the registered images. Conclusions: The proposed method addressed several issues associated with purely landmark-based techniques, such as iterative closest point registration and thin plate spline interpolation. Furthermore, it provided accurate registration results even in the presence of landmark localization errors.

Currently, robust low-rank tensor reconstruction based on tensor singular value decomposition (t-SVD) has made remarkable achievements in the fields of computer vision, image processing, etc. However, existing works mainly concentrate on third-order tensors while order-d (d ≥ 4) tensors are commonly encountered in practical applications, such as fourth-order color videos, fourth-order facial images, fifth-order light-field images and sixth-order bidirectional texture functions. Aiming at this critical issue, we establish a robust order-d tensor reconstruction framework including the model, algorithm and theory by developing a novel algebraic foundation of order-d t-SVD with any invertible linear transforms. Equipped with the newly developed framework, any order-d low-rank tensors can be accurately reconstructed from a sample of its entries corrupted by arbitrary outliers. The superiority and effectiveness of the proposed framework are substantiated in comparison with state-of-the-art alternatives by conducting experiments on both synthetic and real-world tensor data.

In this research, a new image fusion technique was developed to merge the spatial details in the panchromatic (PAN) image and the spectral details in the multispectral (MS) image to produce fused images with the highest spatial and spectral qualities. This fusion technique is superior to the other existing fusion techniques in that it can produce sharp images with no color distortion for all types of land cover. This is because of using weighting parameters for all MS bands in the calculation of the I component according to the intersecting areas between the MS bands and the PAN band, and the I component will be an average of the four modified MS bands, which minimizes the gray level differences between PAN and I images. Moreover, using the variable vegetation coefficient β for the NIR band makes this technique produce the highest spectral and spatial qualities for each land cover class. To evaluate the performance of the ELSHORA fusion technique, it was compared to the traditional intensity-hue-saturation (IHS) fusion technique, the fast IHS (FIHS) fusion technique, and four other standard image fusion techniques; principal component analysis, hyper-spherical color space, multiplicative resolution merge and Brovey transform. These fusion techniques were applied to three sets of GeoEye-1 PAN and MS images covering different land covers in Tanta city, Egypt. The fused images are compared statistically and visually to the original PAN and MS images. The results showed that the ELSHORA fusion technique was chosen as the best method for all datasets. This is because it has produced the highest spectral quality in which no color distortion can be detected and has provided significant spatial quality, visually and analytically. To test the spatial and temporal transferability of ELSHORA fusion technique, it was compared to the original FIHS fusion technique to fuse three different datasets covering different classes of land cover (urban, agriculture, and mixed) located in different places and acquired at different times. The results indicate that ELSHORA fusion technique is spectrally and spatially in the lead for all datasets. This study proved the efficiency of ELSHORA fusion technique to produce images of high spectral and spatial resolution at different times and places.

Few-shot image classification commits to recognizing new concepts from limited annotated samples. Our insight is to obtain a sufficiently powerful embedding network (PEN) to solve few-shot classification tasks. We propose a method to tackle the few-shot classification tasks, namely PENs for few-shot image classification. The key core of PEN is gaining a well-trained embedding network that is capable of extracting strong discriminating representations to represent an image by utilizing two strategies. One strategy is that the multi-scale feature maps are fused instead of only utilizing the final top-level feature maps. We consider that low-level features also play an important role instead of only utilizing top-level representations. Another significant strategy is knowledge distillation (KD). The characteristics of KD can help us get better performance of an embedding network to extract features. Finally, a distance function is employed to classify unlabeled samples. Comprehensive experiments are conducted on few-shot benchmarks. Our method achieves promising performances. The results demonstrate that KD and future fusion are beneficial to gain an expected embedding network for few-shot classification tasks.

Background: As the design layout of integrated circuits (ICs) is continually scaling down, sub-resolution assist features (SRAF) have been extensively used in resolution enhancement technique applications to enhance lithography printing fidelity and widen the manufacturing process window (PW). With conventional SRAF insertion techniques, rule-based SRAF (RB-SRAF) and model-based SRAF (MB-SRAF) methods have since been widely adopted. Aim: The typical RB-SRAF is an efficient method to generate SRAFs consistently for simple designs but cannot be optimized for multiple critical patterns or complex layout schemes. Although MB-SRAF is able to achieve better PW as well as reducing conflicts between placement rules and clean-up rules, many iterations for convergence and extremely high computational costs are required. The explosion of machine learning techniques could facilitate the complex processes of mask optimization, such as SRAF insertion. Approach: Generative adversarial network was studied on a Via layer of advanced 3D NAND flash memory, by training target images and Inverse Lithography Technology (ILT) images of target patterns. GAN models, pix2pix, and CycleGAN, were first trained and then utilized to synthesize realistic ILT images. Results: These ILT images were eventually translated to polygons of SRAF with simplification process and mask manufacturing rules check constraints. The simulation results demonstrate that CycleGAN approach can place SRAF with comparable performance to mask optimization (MO) result which was optimized by the Tachyon source-mask optimizer (SMO). Conclusions: Methodologies of SRAF placement based on GANs were demonstrated in this paper. While traditional MO approach showed the best lithography performance, CycleGAN in particular was close behind and had much better performance than conventional RB-SRAF approach, whilst achieving good stability for different layout environments. Most importantly, the efficiency of SRAF insertion can be enhanced significantly through GANs.

A superpixel segmentation algorithm called multi-density peaks clustering (MDPC) is proposed. By selecting a sufficient number of local density maximum pixels from the image as cluster centers to depict the image texture, the boundary of the object can be captured very accurately. The algorithm framework of MDPC is divided into three steps. First, the local density of pixel is defined, and the local density maximum pixels are calculated. Then, the local density maximum pixels are used as cluster centers, and the global optimal search, which is based on the path-to-point idea, is used to complete the clustering of the remaining non-cluster center pixels to realize the initial segmentation. Finally, superpixels are obtained by merging the initial segments according to the size of the segments and the distance between adjacent segments. In quantitative comparisons, MDPC is compared with 13 state-of-the-art superpixel segmentation algorithms in three image segmentation datasets. The experimental results show that MDPC achieves better performance in terms of boundary recall, boundary precision, achievable segmentation accuracy, undersegmentation error, and explained variation. And the qualitative comparisons show that the proposed algorithm has obvious advantages over other superpixel segmentation algorithms in image detail description and boundary adherence. Finally, the practicability and stability of MDPC are further demonstrated by the application of image segmentation. The source code of MDPC will be available at https://github.com/zhaojianaaa.

Background: Systematic/stochastic pattern defects affect the production yield of integrated circuits (IC) containing trillions of 10-nm level features. Aim: We detect pattern anomalies/defects from images obtained from scanning electron microscopes (SEM) for random/arbitrary IC patterns without using design data. Approach: We decompose SEM images into small sub-images and apply an identical autoencoder to each of them to detect anomalies. The astronomical varieties in random IC patterns are reduced into limited varieties in elementary patterns, which are coded onto limited dimension latent vectors in autoencoder. The discrepancy between autoencoder input and output represents a deviation of local pattern shapes from ideal or allowed ones and is used as an index of anomaly. Results: A wide variety of anomalies/defects are detected in regular and random IC patterns fabricated by extreme ultra-violet lithography without prior knowledge about anomalies at a high signal-to-noise ratio within a time shorter than the typical image acquisition time of SEMs. They include missing/necking in holes/trenches, collapsing/breaking in lines, various local pattern distortion/deformation, and tiny particles. Frequency and spatial distributions of the discrepancy index are sensitive to process changes and can be used for visualizing the sign or causes of anomalies. Conclusions: The method is effective for inspecting memory and random logic ICs with high-speed SEMs.

A conventional possibilistic c-mean (PCM)-based algorithm uses means or variance–covariance statistical variables computed from training data sets for classification. Two objectives were covered; in the first objective, the PCM classifier using the training samples as “individual sample as mean” (PCM_IM) in place of “mean” was experimented with handling heterogeneity within the class. The second objective, the PCM-based local convolutional method, was tested with different parameters to handle the noisy pixel. Local convolution algorithms reduce the effect of noise, and PCM_IM handles heterogeneity to enhance the classification performance within the class. The optimized algorithm was first calculated in this research by applying the local convolutional methods for different distance measures and fuzziness factors. This optimized algorithm was obtained by classifying the dense forest, eucalyptus, riverine sand, water, wheat, and grassland classes using the multispectral images of Landsat-8 and Formosat-2 of the Haridwar area. It was observed that the Bray–Curtis distance measures at m = 2.9 (fuzziness factor) for possibilistic local information c-means using the base PCM classifier yields the highest overall accuracy (OA) (86.37%), irrespective of the classifier. Then, PCM_IM provides a classwise and OA that handles heterogeneity within the class. The classwise accuracy of dense forest, eucalyptus, grassland, sand, water, and wheat is 88.12%, 95.29%, 78.79%, 89.17%, 89.72%, and 92.43%, respectively, and the OA is 85.72%, whereas OA of the PCM classifier using the training sample as mean is 75.18%.

Image quality is greatly affected by noise in various applications such as machine vision and computer vision. Noisy images lead to inaccurate results as well as segmentation and enhancement errors, making denoising vital to efficient image processing. Existing denoising methods do not work effectively on both grayscale and color images. Thus an effective denoising method that works well on both image types is proposed. The proposed two-stage method involves: (i) noisy pixel detection and (ii) noisy pixel restoration. In the first stage, the conglomerate method detects the noisy pixel position; in the second stage, the corrupted value of the noisy pixel is restored using a conglomerate method. The proposed method was used to process a variety of grayscale and color images. The result analysis of image denoising was done in two stages: in the first stage, results are carried out using large standard datasets; comparison of these results with those of five other denoising methods showed that the proposed conglomerate denoising method outperformed these existing ones. The second stage of result analysis considers the state-of-the-art methods in comparison with the proposed method. The proposed method outperforms the state-of-the-art methods for images affected with noise ranging from 10% to 90%.

The super-resolution (SR) reconstruction of remote sensing images is a low-cost and efficient method to improve their resolution, and it is often used for further image analysis. To understand the development of SR reconstruction of remote sensing images and research hotspots and trends, we examined its history and reviewed existing methods categorized into traditional, learning-based, and deep-learning-based methods. To evaluate the reconstruction performance, we conducted experiments comparing various algorithms for the single- and multi-frame SR reconstruction of remote sensing images considering three datasets. The experimental results indicate the advantages and limitations of single- and multi-frame reconstruction, with the latter showing a higher performance. Finally, we provide directions for future development of this SR reconstruction.

Purpose: Developing, validating, and evaluating a method for measuring noise texture directly from patient liver CT images (i.e., in vivo). Approach: The method identifies target regions within patient scans that are least likely to have major contribution of patient anatomy, detrends them locally, and measures noise power spectrum (NPS) there using a previously phantom-validated technique targeting perceptual noise–non-anatomical fluctuations in the image that may interfere with the detection of focal lesions. Method development and validation used scanner-specific CT simulations of computational, anthropomorphic phantom (XCAT phantom, three phases of contrast-enhancement) with known ground truth of the NPS. Simulations were based on a clinical scanner (Definition Flash, Siemens) and clinically relevant settings (tube voltage of 120 kV at three dose levels). Images were reconstructed with filtered backprojection (kernel: B31, B41, and B50) and Sinogram Affirmed Iterative Reconstruction (kernel: I31, I41, and I50) using a manufacturer-specific reconstruction software (ReconCT, Siemens). All NPS measurements were made in the liver. Ground-truth NPS were taken as the sum of (1) a measurement in parenchymal regions of anatomy-subtracted (i.e., noise only) scans, and (2) a measurement in the same region of noise-free (pre-noise-insertion) images. To assess in vivo NPS performance, correlation of NPS average frequency (favg), was reported. Sensitivity of accuracy [root-mean-square-error (RMSE)] to number of pixels included in measurement was conducted via bootstrapped pixel-dropout. Sensitivity of NPS to dose and reconstruction kernel was assessed to confirm that ground truth NPS similarities were maintained in patient-specific measurements. Results: Pearson and Spearman correlation coefficients 0.97 and 0.96 for favg indicated good correlation. Results suggested accurate NPS measurements (within 5% total RMSE) could be acquired with ∼106 pixels. Conclusions: Relationships of similar NPS due to reconstruction kernel and dose were preserved between gold standard and observed in vivo estimations. The NPS estimation method was further deployed on clinical cases to demonstrate the feasibility of clinical analysis.

Significance: Fourier ptychography microscopy (FPM) is a computational optical imaging technology that employs angularly varying illuminations and a phase retrieval algorithm to achieve a wide field of view and high-resolution imaging simultaneously. In the FPM, LED position error will reduce the quality of the reconstructed high-resolution image. To correct the LED positions, current methods consider each of the LED positions as independent and use an optimization algorithm to get each of the positions. When the positional misalignment is large or the search position falls into a local optimal value, the current methods may lack stability and accuracy. Aim: We improve the model of the LED position and propose an accurate and stable two-step correction scheme (tcFPM) to calibrate the LED position error. Approach: The improved LED positions model combines the overall offset, which represents the relative deviation of the LED array and the optical axis, with the slight deviation of each LED’s independent position. In the tcFPM, the overall offset of the LED array is corrected at first, which obtains an approximate value of the overall offset of the LED array. Then the position of each LED is precisely adjusted, which obtains the slight offset of each LED. Results: This LED position error model is more in line with the actual situation. The simulation and experimental results show that the method has high accuracy in correcting the LED position. Furthermore, the reconstruction process of tcFPM is more stable and significantly improves the quality of the reconstruction results, which is compared with some LED position error correction methods. Conclusions: An LED position error correction technology is proposed, which has a stable iterative process and improves the reconstruction accuracy of complex amplitude.

Electro-optical testing and characterization of the Vera C. Rubin Observatory Legacy Survey of Space and Time Camera has shown that the focal plane charge-coupled devices developed by Imaging Technology Laboratory exhibit serial deferred charge signal in excess of the expected proportional deferred charge loss from global charge transfer inefficiency. We model the additional deferred charge as resulting from localized charge trapping in the serial register and from a signal-dependent electronic offset effect. A physically motivated forward deferred charge simulation algorithm and a reverse deferred charge correction algorithm were developed in order to correct for deferred charge during data reduction and reduce the possible impacts on Rubin Observatory project science goals. We apply the correction algorithm to calibration data and demonstrate the ability to reduce deferred charge signal measured in the overscan pixel regions and remove the serial pixel correlations resulting from the redistribution of charge between pixel neighbors during readout.

To effectively detect the surface cracks of subway tunnels, an automatic tunnel crack detection system based on machine vision is presented. Aiming at the problems of environmental complexity and low contrast in subway tunnels, the image texture feature is first enhanced by the methods of frequency domain filtering and spatial differencing. Then, depending on the characteristics of the tunnel cracks in question, the crack propagation method is used to extract the complete cracks. Finally, broken cracks are connected during processing, and the method of combining projection and threshold is used to determine the crack types. At the same time, characteristics such as the length, width, and area of the cracks are obtained. The experimental results show that the presented methods can effectively extract complete cracks in complex tunnel environments. The identification error of tunnel crack parameters meets the actual engineering requirements.

Demosaicking is a method to restore the mosaicked image generated by the color filter array to a full-color image. The performance of the demosaicking algorithm is crucial to the reconstruction of high-quality color images. However, current demosaicking algorithms inevitably cause zipper effects or false color artifacts in the process of demosaicking. We combine low-rank constraint based on spectral domain with image non-local similarity to correct the interpolation errors. Our method is applicable to various existing demosaicking algorithms and can further improve the quality of full color image reconstruction. Experimental results demonstrate that the proposed algorithm can improve the color peak signal to noise ratio of the reconstructed images.

Background: E-beam metrologies, both critical dimension scanning electron microscope metrology and defect scan metrology, have been playing a very critical role in gating patterning quality. SEM images can provide rich visual information for engineers to do qualitative and quantitative analyses. However, the low e-beam metrology tool throughput makes it impossible to obtain SEM images for larger area. Monte Carlo-based SEM image simulations or other SEM image simulations require postlithography or postetch pattern three-dimensional structures as prerequisite, and the simulation speed is not sufficiently fast for full chip implementation. Aim: We aim to develop machine learning SEM models with sufficient accuracy and speed for full chip application in semiconductor manufacturing environment. Approach: We have proposed a virtual SEM metrology solution based on U-Net neural network with physics-based feature maps as model input. With information in aerial image space encoded properly, SEM images of both postlithography and postetch can be predicted accurately enough for practical applications using our proposed virtual SEM metrology models. Equipped with GPUs, the machine learning-based SEM image models are fast enough to make it possible to realize full chip SEM generation from post-OPC data. Results: Our machine learning SEM image models can predict SEM images with normalized cross correlation around 0.95 in reference to ground truth SEM images, each SEM image (512 × 512 in size) takes about 800 ms using single CPU, and the speed can be accelerated to about 10 ms with single GPU. Conclusions: Using U-Net structure and physics-based feature maps as model inputs, machine learning-based SEM image models can be developed. The models are sufficiently accurate and fast to find their applications in semiconductor manufacturing, and they can be used as independent model for OPC data verification or generate SEM images as reference for SEM defect scan metrology.

Purpose: Magnetic resonance imaging has been recently used to examine the abnormalities of the placenta during pregnancy. Segmentation of the placenta and uterine cavity allows quantitative measures and further analyses of the organs. The objective of this study is to develop a segmentation method with minimal user interaction. Approach: We developed a fully convolutional neural network (CNN) for simultaneous segmentation of the uterine cavity and placenta in three dimensions (3D) while a minimal operator interaction was incorporated for training and testing of the network. The user interaction guided the network to localize the placenta more accurately. In the experiments, we trained two CNNs, one using 70 normal training cases and the other using 129 training cases including normal cases as well as cases with suspected placenta accreta spectrum (PAS). We evaluated the performance of the segmentation algorithms on two test sets: one with 20 normal cases and the other with 50 images from both normal women and women with suspected PAS. Results: For the normal test data, the average Dice similarity coefficient (DSC) was 92% and 82% for the uterine cavity and placenta, respectively. For the combination of normal and abnormal cases, the DSC was 88% and 83% for the uterine cavity and placenta, respectively. The 3D segmentation algorithm estimated the volume of the normal and abnormal uterine cavity and placenta with average volume estimation errors of 4% and 9%, respectively. Conclusions: The deep learning-based segmentation method provides a useful tool for volume estimation and analysis of the placenta and uterus cavity in human placental imaging.

Background: Finding optimal processing conditions to reduce defectivity is a major challenge in high-resolution lithographic tools such as directed self-assembly and extreme ultraviolet lithography. Aim: We aim to develop an efficient automated method that can detect defects and identify their types and locations, allowing for assessing the performance of lithographic processing conditions more easily. Approach: We propose a deep learning approach using an object recognition neural network for the classification and detection of defects in scanning electron microscopy (SEM) images featuring line-space patterns. We optimized our network by exploring its design variables and applied it on a case with limited numbers of SEM images available for training the network. Results: With an optimized network and data augmentation strategies (flipping and mixing images from simulations), it was possible to achieve a significant increase in the performance of the network. Transferability of the network was also proven when applied on a more diverse dataset of SEM images gathered from selected publications. Conclusions: The outcome of our work shows that the amalgamation of an optimal network design and augmentation strategies performs satisfactorily for defectivity analysis and is generic for data not constrained to fixed settings.

For a damaged image, the loss of pixel information can be roughly divided into two categories, random missing and non-random missing. The missing of an entire row or column of the image is a specific structural missing pattern that is extremely difficult to deal with. Although most of the existing methods have partially fixed this information missed problem, the diffusion-based methods tend to produce blur, the exemplar-based methods are prone to error filling, and the neural network-based methods are highly dependent on data, which cannot handle this special structural missing very well. Using the nonlocal self-similarity prior and the low-rank prior, we present multidirectional search and nonlocal low-rank tensor completion (MS-NLLRTC) algorithm based on the tensor ring (TR) decomposition and multidirectional search (MS). The MS method is a newly proposed method that can search similar patches much more diversified. Using MS method, we directly stack the similar patches into a three-dimensional similar tensor instead of pulling them into column vectors, then the similar tensor can be completed by TR decomposition. The optimization results can be obtained by leveraging the alternating direction method under the augmented Lagrangian multiplier framework. Moreover, we add a weighted nuclear norm to the tensor completion model (WNLLRTC), achieving a better inpainting performance. We also combine a noise removal method with WNLLRTC algorithm, which can handle image random missing and image noise removal simultaneously. Experimental results indicate that our proposed algorithms are competitive with some state-of-the-art inpainting algorithms in terms of both numerical evaluation and visual quality.

Deep neural networks have been successfully used in image processing and have benefitted image compression technology. On this basis, deeper and more extensive use of compression algorithms in remote sensing applications was investigated in our study. As a general residual block does not effectively utilize the connections between channels and spatial regions in the reconstruction process, an attention mechanism codec framework based on the channel-spatial attention residual block was introduced. Consequently, the network can account for the interdependence between channels and enhance specific spatial areas. Simultaneously, the feature importance map makes the network focus on valuable information, especially for small objects and remote sensing images with many tiny details, which is crucial to describe their contours. Introducing the attention mechanism commonly used in target detection can effectively recover the missing detailed information for remote sensing compressed images. Moreover, to utilize the underlying information of the various scene categories and characterize the feature differences between scenes in remote sensing images, an adaptive scene-aware module was introduced to enhance the feature representation in remote sensing images of different scenes. The experimental results show that the proposed method has superior subjective and objective effects in remote sensing image compression tasks.

Realtime stereo processing, particularly in computationally constrained environments such as the Mars 2020 Perseverance rover, requires image rectification to enable efficient one-dimensional correlation. A camera model has been designed to perform better image rectification for fish-eye lenses with a particular emphasis on the engineering cameras on Perseverance. The primary innovation is the use of a unit projection sphere rather than an image plane as the rectifying surface. For stereo rectification, the rows (for a horizontal pair) or the columns (for a vertical pair) of both cameras must lie on the same epipolar plane. Thus, we use a pair of moving planes to define the rows and columns. Each plane rotates about a dedicated static axis passing through the sphere center. Virtual pixels are located where the planes intersect with each other and the unit sphere.

Sparse approximation is critical to the applications of signal or image processing, and it is conducive to estimate the sparse signals with the joint efforts of transformation analysis. In this study, a simultaneous Bayesian framework was extended for sparse approximation by structured shared support, and a simultaneous sparse learning algorithm of structured approximation (SSL-SA) is proposed with transformation analysis which leads to the feasible solutions more sensibly. Then the improvements of sparse Bayesian learning and iterative reweighting were embedded in the framework to achieve speedy convergence as well as high efficiency with robustness. Furthermore, the iterative optimization and transformation analysis were embedded in the overall learning process to obtain the relative optima for sparse approximation. Finally, compared to conventional reweighting algorithms for simultaneous sparse models with l1 and l2, simulation results present the preponderance of the proposed approach to solve the sparse structure and iterative redundancy in processing sparse signals. The fact indicates that proposed method will be effective to sparsely approximate the various signals and images, which does accurately analyse the target in optimal transformation. It is envisaged that the proposed model could be suitable for a wide range of data in sparse separation and signal denosing.

Remote sensing image processing has been widely used in environmental monitoring, terrain survey, military investigation, disaster early warning, and other fields. The remote sensing images matching is a key step, and the accuracy and real-time of the alignment have a great impact on these applications. Due to the high resolution and complexity of remote sensing images, scale-invariant feature transformation (SIFT) algorithm has the problems of high computational complexity and poor matching effect. Adaptive threshold adjustment algorithm has proposed in the remote sensing image matching, but minor changes in the contrast threshold can bring about drastic changes in the image matching quality, which will affect applications such as monitoring, measurement, and early warning. To improve the matching quality of a SIFT detector, an adaptive contrast threshold SIFT method based on image complexity calculation (CACT-SIFT) is proposed. The CACT-SIFT finds two contrast thresholds for the target image and the reference image, respectively, and achieves the target by minimizing the proposed criteria Cik. Experiments show that the method can be applied to the matching of remote sensing images. The information of reference images and target images can be detected at the same time, and the key points can be extracted in a robust way, with better accuracy and real-time accuracy of the alignment.

Anomaly detection (AD) is a crucial task for detecting salient objects in cluttered backgrounds. Classical AD algorithms based on statistical models or geometric models have achieved acceptable detection results. However, most of them do not hierarchically extract deep features or consider the spatial structure of the images. We propose a method that combines the reconstruction error of autoencoder (AE) and spatial morphological characteristics to estimate anomalousness. The reconstruction errors and the spatially dominant information are comprehensively considered. Specifically, given the compression capability of the AE, we use the dimensionality reduction results obtained by encoding for analyzing local pixel difference; an adaptive dual window is employed in this process. The morphological transformation commonly used in edge detection is utilized to refine the small space anomalies. Experimental results on different hyperspectral images show that the proposed AE and spatial morphology extraction-based approach significantly surpasses several traditional alternatives.

Hyperspectral images contain rich spectral–spatial information. Therefore exploring spectral–spatial classifiers has become a mainstream trend in the field of hyperspectral image classification. However, current studies seldom delve into the contribution of the extracted spectral and spatial features to the subsequent classification task. To further explore the contribution of spectral and spatial features, we propose a classification framework based on capsule network (CapsNet). The framework consists of two branches, which extract spectral and spatial features to form capsules respectively. Subsequently, the spectral capsules and spatial capsules are sent to the dynamic routing layer to generate higher-level spectral–spatial capsules. In addition, we set up a primary–secondary relationship for the two branches, which indirectly reflects the contribution made by lower capsules forming higher capsules in the feature fusion stage. Our experiments, conducted on three widely used hyperspectral datasets and two sampling strategies, demonstrate that our proposed model had 4.60% to 8.80% improvements in OA compared with the original CapsNet. Compared to the capsule-based improved model (iCapsNet and Conv-Caps), our model also has a 1.20% to 2.50% improvement.

In retinal optical coherence tomography (OCT), it is significant to focus the scan beam on the retina layers of the human eye fast to acquire high-quality OCT images. The refractive errors of the human eye are usually compensated by shifting the optical lens in the sample arm manually or electrically. Otherwise, refractive errors of the human eye will cause axial focal shift of the scanning beam and blur the OCT images. We propose a fast automatic focus method using the interference fringes magnitudes of the whole retinal layers to determine the best refractive correction position of motor-driven optical compensation lens without the aid of optical fundus imaging or OCT B-scan imaging. Also, the optical path difference between the sample arm and reference arm is adjusted in the spectral-domain OCT (SD-OCT) to reduce the effects of sensitivity roll-off. The experiments in the lab-built SD-OCT show that the average running time of the method is 15.2 s and the mean absolute deviation of each imaging is within 1 mm. The method is independent of extra optical setup and requires less computational consumption, which is suitable for low-cost and automatic retinal OCT devices.

Convolutional neural networks (CNNs) have achieved remarkable results in multispectral (MS) and panchromatic (PAN) image fusion (pansharpening) because of their strong image feature extraction ability. However, previous CNN-based pansharpening methods mostly use an ordinary convolution, which has a small receptive field in the convolution layer, has insufficient contextual information, and can only extract shallow features, which is not conducive to learning the complex nonlinear mapping relationship between the input image and the fused image. Therefore, this study proposes a pansharpening algorithm based on a multiscale densely convolutional neural network (MDCNN). First, a two-stream network is used for feature extraction, with two convolution layers to extract spectral information from MS images. The multiscale convolutional feature extraction module is designed to extract the spatial detail features of the PAN images. Second, the proposed multiscale densely connected modules and residual modules are used as the backbone of the fusion network. Finally, the deep features generated are reconstructed, and spectral mapping is used to retain spectral information to obtain a high-resolution fusion image. Experimental results using three satellite image datasets show that the proposed algorithm generates high-quality fusion images, and it outperforms most advanced pansharpening methods in subjective visual and objective evaluation indexes.

Spectral and spatial details abound in hyperspectral data. However, there are few labeled samples available, and obtaining them is expensive and time-consuming. As a result, improving the classification accuracy of HSI with limited training samples is a critical challenge. We propose a double-branch network with attention feature fusion (DBN-AFF) for HSI classification. One branch extracts spectral features, and the other extracts spatial features. It is capable of effectively preventing cross-branch interference. Furthermore, the AFF module is used instead of the channel concatenation method. It effectively fuses spectral and spatial features by considering the relationship between different channels. On the Indian Pines, Pavia University, and Salinas Valley datasets, DBN-AFF achieves 97.44%, 98.37%, and 97.60% on limited training samples, respectively.

The detection of urban land use is of great significance to urban planning. With the development of deep learning technology, convolutional neural networks (CNN) are widely used in remote sensing image processing. However, existing CNN models lack the ability to fuse high-dimensional semantic information and low-dimensional position information of images and thus perform poorly in feature restoration. To obtain better urban land use detection results, the cross-dimensional feature fusion network (CDFFNet) is proposed in this work. In the CDFFNet, Class Feature Attention (CFA), cross-level feature fusion (CLFF), and Same Level Feature Gate Fusion (SLFGF) are designed. The CFA aggregates multi-scale and multi-receptive field input features to capture high-dimensional semantic information. The CLFF is designed to fuse features of different resolutions. It can extract and fuse semantic information of different dimensions, aggregate contextual information of features and distinguish the contextual relevance of each pixel. The SLFGF dynamically fuses high-dimensional semantic information and low-dimensional spatial information, so it could enhance information recovery of different categories. Compared with existing methods, the CDFFNet is significantly improved in accuracy. Its mean intersection over union on the Aerial Image Segmentation Dataset reaches 0.765.

Remote sensing (RS) oblique imagery provides valuable information of buildings’ facades. Facade detection using spectral-, spatial-, and texture-only features does not accurately separate facade and nonfacade regions in a single-view oblique image. Therefore, a facade index and a unimodal thresholding method were proposed to characterize and detect facade regions. This new index, named the probability-spatial facade index (PSFI), first highlighted facade areas. Then, the facade map was created through a histogram-based thresholding. In unimodal histograms, the thresholding procedure was according to the roots of the second derivative of a fourth-degree polynomial model that is fitted to the PSFI’s histogram. All the steps of the proposed method were implemented in the Google Earth Engine cloud computing platform and could automatically handle very high resolution oblique imagery (in terms of both angle and direction) without any limitations. Furthermore, various high- and low-rise buildings could be effectively processed without any assumptions about the structure of facades. The evaluation showed the high performance of the proposed method in different test areas, in which the average overall accuracies in distinguishing facade and nonfacade regions and in separating facade and rooftop regions were 97% and 86%, respectively.

We propose a panchromatic (Pan) and multispectral (MS) image fusion method in order to synthesize an MS image with enhanced spatial resolution and preserved spectral information of the original MS image. The proposed method is based on the spatial frequency and the a trous wavelet transform (SFATWT). The a trous wavelet transform (ATWT) is used to decompose Pan and MS images into low and high frequencies. The spatial frequency is exploited in the image fusion rules to improve the details or the high-frequency information which will be injected into the MS image. The ATWT reconstruction makes it possible to produce MS image with high spatial resolution. To validate the proposed method, two image sets are used. The first one is acquired by the SPOT (Satellite Pour l’Observation de la Terre) satellite with a resolution ratio of 2. The second one is the Algerian satellite AlSat 2A images with a resolution ratio of 4. The fused results are evaluated by means of qualitative and quantitative comparisons. The assessment of fused products has shown that the proposed method is promising.

Recent technological advancements in satellite imagery have increased the production of remotely sensed images. Therefore, developing efficient methods for annotating these images has gained popularity. Most of the current state-of-the-art methods are based on supervised machine learning techniques. We propose a method called semisupervised multi-label annotizer (SS-MLA) that adapts vector-quantized temporal associative memory to annotate remotely sensed images. One of the advantages of SS-MLA over the supervised methods is that it extracts features not only from the given sample but also from similar samples that are previously seen without using an explicit attention mechanism. Thus SS-MLA enhances the learning efficiency of the training process. We conduct extensive performance comparisons with five different methods in the literature over four datasets. The comparison results indicate the success of the proposed method over the existing ones: SS-MLA generates the best results in 7 out of 11 comparisons.

A block color image encryption algorithm based on chaotic system, elementary cellular automata (ECA), and deoxyribonucleic acid (DNA) sequence operations is proposed. The architecture is comprised of forward-diffusion, permutation, and backward-diffusion. First, the red, green, and blue components of a color image are converted into three DNA matrices by DNA encoding rule. Exclusive OR (XOR) operation is applied between the DNA coding chaotic sequence generated by the improved Henon map and three DNA matrices. Second, the large DNA matrix, composed of three DNA matrices, is divided into several small blocks with unequal sizes. ECA is employed to scramble the positions of these blocks. Finally, DNA computations are introduced to diffuse three permutated DNA matrices. In addition, initial values and system parameters of the chaotic system are calculated from the secure hash algorithm (SHA-256) hash value of the plain image and the given parameters. Both DNA encoding and ECA order rules rely on plaintext information. Therefore, the proposed algorithm is highly sensitive to plain image. Simulation results and security analyses demonstrate that our algorithm achieves excellent security performance and acceptable running time.

We present an improved Criminisi image inpainting algorithm through kriging pretreatment and facet model. We propose three improvement strategies. First, we propose an improved priority. A priority used in Criminisi algorithm consists of confidence term and data term. As iteration progresses, the confidence term is updated and plays a key role in the confidence term of the next edge block. At the same time, the confidence term gradually tends to 0, leading to a weakening of the role of priority, and ultimately affecting the repair effect. We propose an improved priority, which is represented by a piecewise function. Importantly, this improved priority reduces the risk of being weakened during iteration. Second, Criminisi algorithm uses fixed-size sample blocks to repair damaged images, regardless of whether the image content is a textured area or a structure area. We introduce an adaptive method to select sample block size based on the facet model. The size of the sample block is adaptively adjusted for different image contents, thereby improving the quality of the repaired image. Third, we show a weighted sum of squares differences matching principle based on the facet model. The matching formula is determined by the pixel gray value of a sample block and the structure value of four directions, which improves matching accuracy between target block and optimal matching block. Finally, experimental results show that the proposed algorithm is competitive with some state-of-the-art inpainting techniques in terms of both objective metrics and subjective visual inspection.

In this work, we present a sound mathematical result that may correct the pixelization bias in the calculation of the line edge roughness (LER) even when the pixel size of SEM images is much larger than its rms value. This result can be useful in the LER metrology of current and future lithographic patterns where the desired rms value of LER is in the deep sub-nm range and smaller than the used pixel sizes. We computationally justify the predictions of the mathematical result and then demonstrate its application in synthesized SEM images where we can separate the impact of noise and focus on the pixelization effects in LER measurements. Furthermore, this result emphasizes the power of mathematical and computational methods to enhance the metrological output of SEM images in deep nanometer scale.

Monocular depth estimation, which provides a critical method for understanding 3D scene geometry, is an ill-posed problem. Recent research studies have achieved significant progress by reliable network architecture and optimized constraints, such as spatial propagation network and depth metrics. We propose an effective generative adversarial network for fast and accurate monocular depth estimation. Our approach demonstrates the feasibility of applying a dense-connected UNet for reducing information transmission loss and then fine-tuning the blur depth by the high-order convolutional spatial propagation network (CSPN) that used a modified loss function of discriminator. Furthermore, we modify the loss function of discriminator by adding the correlation loss that is used to measure the similarity of real and fake labels. Compared with the original CSPN, the high-order CSPN reduces the computation complexity and accelerates the convergence of the generator network by increasing the order of kernel, which emphasizes the weight of kernels in the update formula. With these modifications, our generative adversarial second-order convolutional spatial propagation network (GA-CSPN) achieves more accurate results against state-of-the-art methods in both indoor and outdoor scenes on Make3D, KITTI 2015, and NYUv2 datasets.

In the task of interactive image segmentation, the user initially indicates some pixels as target seeds and background seeds and propagates the seeds’ labels to the rest of the image. We propose an equidistant seed generation method for interactive image segmentation that combines saliency detection and random walk (RW) with restart with optimal restarting probability. Our method includes two stages: first, we utilize the saliency detection method to obtain the initial target region and generate the initial seeds. Then we relabel the seeds using a Gaussian mixture model with limited user interaction to obtain accurate seeds. The relabeled seeds can limit the range of object; therefore, in the second stage, we can select an optimal restarting probability by calculating the area of the RW that exceeds the limited range of the object seeds. Our method can generate effective seeds and thus reduce the dependence on user interaction. Furthermore, using the optimal restarting probability, we can obtain higher segmentation accuracy. Extensive experiments on the Berkeley segmentation dataset, GrabCut dataset, and MSRA1000 dataset demonstrate that our method can reduce the dependence on user interaction and achieve better performance than the latest interactive image segmentation methods, such as RW, RW with restart, normalized RW, sub-Markov RW, and label propagation through complex networks.

Small object detection is always a hot research direction in the field of computer vision, and widely used in traffic, production, industry, and other fields. A neural network for detecting small objects based on original Cascade RCNN is proposed. First, we modify the traditional feature pyramid network and introduce multi-branch dilated convolutions to enhance the feature information of the small target. Second, the feature extraction networks in original Cascade RCNN are replaced with multi-layer deformable convolution networks, which can better adapt to the geometric variations of detection target and huge size span of the objects in the same scene. Finally, Soft non-maximum suppression is also integrated into the network to avoid problems in dense object detection. To verify the practicability of our proposed network, we apply it to the garbage detection system. The experimental results show that compared with original Cascade RCNN and other commonly used target detection networks, our proposed method, with a recall rate >0.98 as well as mean average precision up to 0.964, performs better not only in small object detection but also in industrial applications.

Principal component analysis (PCA) has been successfully employed for face recognition. However, if the training process occurs frequently, owing to the update or downdate of the face images used for training, batch PCA becomes prohibitively expensive to recalculate. To overcome this limitation, incremental principal component analysis (IPCA) and decremental principal component analysis (DPCA) can be utilized as a good alternative to PCA because it reuses their previous results for its updates. Many IPCA or DPCA algorithms have been proposed; however, inaccurate tracking of the mean values of the face image data accumulates decomposition errors, which results in poor performance compared with batch PCA. We proposed faster and more accurate algorithms for IPCA and DPCA that maintain accurate decomposition results. The experimental results reveal that the proposed algorithms produce eigenvectors that are significantly close to the eigenvectors of batch PCA and exhibit faster execution speed for face recognition.

The study of image segmentation is a crucial impression in image processing and computer vision appliances, which segments an input image into distinct non-overlapping homogenous divisions and helps to depict the image more conveniently. To improve the performance of the segmentation of gray-level images by introducing a normalized graph cut measure as a thresholding principle to separate an object from the background based on the neutrosophic membership function. We proposed an innovative neutrosophic approach combining a set of features of some efficient algorithms. The implementation of the proposed algorithm is known as the neutrosophic normalized graph cut (NNGC) method. In the gray image segmentation, the problem of wrongly segmentation and segmentation with low accuracy can identify by using this approach. This proposed algorithm is compared with the fuzzy entropy method, neutrosophic graph cut (NGC), classical graphcut, Otsu, and Kittler method. Moreover, we examine that in most cases, our algorithm gives the lowest absolute error, misclassification error, and higher values of signal to noise ratio values that improve the segmentation process of gray images. Finally, we analyze the change of different parameter values in the NNGC and the effect of the substitutes. Also, we discuss the computational complexity of neutrosophic weight matrix results with a weight matrix (classical) results.

We present methods enabling rapid non-uniformity and range walk error correction of 3D flash LiDAR imagers that exhibit electronic crosstalk caused by simultaneously triggering too many detectors. This additional electronic crosstalk is referred to as simultaneous ranging crosstalk noise (SRCN). Using a method in which the 3D flash LiDAR imager views a checkerboard target downrange, the SRCN is largely mitigated. Additionally, processing techniques for computing the non-uniformity correction (NUC) and range walk error correction are described; these include an in-situ thermally compensated dark-frame non-uniformity correction, image processing and filtering techniques for the creation of a photo-response non-uniformity correction, and characterization and correction of the range walk error using data collected across the full focal plane array without the need for sampling or windowing. These methods result in the ability to correct noisy test validation data to a range precision of 8.04 cm and a range accuracy of 1.73 cm and to improve the signal-to-noise ratio of the intensity return by 15 to 49 dB. Visualization of a 3D scene corrected by this process is additionally presented.

When deep convolutional neural networks perform feature extraction, the features computed at each layer express different abstractions of visual information. The earlier layers extract highly compact low-level features such as bandpass and directional primitives, whereas deeper layers extract structural features of increasing abstraction, similar to contours, shapes, and edges, becoming less effable as the depth increases. We propose a different kind of end-to-end no-reference (NR) image quality assessment (IQA) model, which is defined as a multi-depth output convolutional neural network (MoNET). It accomplishes this by mapping both shallow and deep features to perceived quality. MoNET delivers three outputs that express shallow (lower-level) and deep (high-level) features, and maps them to subjective quality scores. The multiple outputs are combined into a single, final quality score. MoNET does this by combining the responses of three learning machines, so it may be viewed as a form of ensemble learning. The experimental results on three public image quality databases show that our proposed model achieves better performance than other state-of-the-art NR IQA algorithms.

The hyperspectral image (HSI) has the characteristics of high resolution, a large amount of data, and a high correlation of bands. In the many HSI processing technologies, image classification is the most basic one. Supervised classification is the most effective and common classification method. However, to achieve the ideal classification effect, supervised classification needs a large number of labeled samples, which requires a lot of time and labor. To solve this problem, we combine active learning (AL) and transfer learning and propose an iterative weighted framework based on active transfer learning. First, we solve the optimal reconstruction matrix and projection matrix by minimizing the reconstruction error. Then, we project labeled samples from the source and target domains into the common subspace. In the iteration of the common subspace, the classifier performance will be better with the increase of iteration number. In each iteration, the weighted strategy is adopted to weigh the samples of the source domain. In this way, valuable source domain labeled samples will get a larger weight, so as to help the classification of target domain samples better. At the same time, AL is used to screen out a certain number of samples of the target domain for manual labeling, which is added to the labeled samples set. Experiments on three data sets demonstrate the effectiveness and reliability of the proposed method.

A method of monitoring produce freshness with hyperspectral imaging and machine learning is described as a way to reduce food waste in grocery stores. The method relies on hyperspectral reflectance images in the visible–near-infrared spectral range from 387.12 to 1023.5 nm with a 2.12-nm spectral resolution. The images were recorded in a laboratory with the imager viewing produce samples illuminated by broadband halogen lights, but we also recorded and discussed the implications of the illumination spectrum of lights found in a variety of grocery stores. A convolutional neural network was used to perform freshness classification for potatoes, bananas, and green peppers. Additionally, a genetic algorithm (GA) was used to determine the wavelengths carrying the most useful information for age classification, with an eye toward a future multispectral imager. Hyperspectral images were processed to explore the use of RGB images, GA-selected multispectral images, and full-spectrum hyperspectral images. The GA-based feature selection method outperformed RGB images for all tested produce, outperformed hyperspectral imagery for bananas, and matched hyperspectral imagery performance for green peppers. This feature selection method is being used to develop a low-cost multispectral imager for use in monitoring produce in grocery stores.

Purpose: Experienced radiologists have enhanced global processing ability relative to novices, allowing experts to rapidly detect medical abnormalities without performing an exhaustive search. However, evidence for global processing models is primarily limited to two-dimensional image interpretation, and it is unclear whether these findings generalize to volumetric images, which are widely used in clinical practice. We examined whether radiologists searching volumetric images use methods consistent with global processing models of expertise. In addition, we investigated whether search strategy (scanning/drilling) differs with experience level. Approach: Fifty radiologists with a wide range of experience evaluated chest computed-tomography scans for lung nodules while their eye movements and scrolling behaviors were tracked. Multiple linear regressions were used to determine: (1) how search behaviors differed with years of experience and the number of chest CTs evaluated per week and (2) which search behaviors predicted better performance. Results: Contrary to global processing models based on 2D images, experience was unrelated to measures of global processing (saccadic amplitude, coverage, time to first fixation, search time, and depth passes) in this task. Drilling behavior was associated with better accuracy than scanning behavior when controlling for observer experience. Greater image coverage was a strong predictor of task accuracy. Conclusions: Global processing ability may play a relatively small role in volumetric image interpretation, where global scene statistics are not available to radiologists in a single glance. Rather, in volumetric images, it may be more important to engage in search strategies that support a more thorough search of the image.

Building damage assessment plays an essential role during post-disaster rescue operations. Given that labeled samples are difficult to timely obtain after a disaster, transfer learning attracts increasing attention. However, different sensors employed cause considerable discrepancies not only between historical and current scenes but also among historical scenes, which could exert an effect on transfer performance. Therefore, a multi-source adversarial domain adaptation (MADA) method is proposed in this paper to fulfill the task of post-disaster building assessment. This method consists of two phases. First, imageries of several historical scenes are transformed into the same style of the current scene through the CycleGAN model with a classifier, ensuring class invariance, to be fused to make an adapted source domain. Second, feature alignment between adapted source and target domains is executed based on adversarial discriminative domain adaptation. The MADA method enhances the transformed image quality, fully utilizes relevant information in historical scenes, solves inter-scene interference problems among historical images, and improves the transfer efficiency from historical to the current disaster scene. Two experiments are conducted with Hurricane Sandy, Irma, and Maria datasets as multi-source and target domains to validate MADA’s effectiveness. Results show that the classification performance is better than other methods.

The automatic detection of defects is an essential part of the printed circuit board (PCB) production process. In recent years, while great progress has been made in the detection of PCB defects, there are still various problems in traditional defect detection methods, for example, over-reliance on the perfect template, difficult to achieve precise image registration, and highly vulnerable to environmental factors such as light, noise, and reflectivity. We propose a fast defect detection network. On one hand, this algorithm solved the problems of traditional methods. On the other hand, this algorithm solved the problems of large model size and poor real-time of existing deep learning methods. First of all, the k-means clustering algorithm is used to obtain more reasonable anchors boxes; second, an improved MobileNetV2 is used as the backbone network; after the feature extraction network, the spatial pyramid pooling (SPP) structure is introduced to increase the receptive field of the image; then, we use complete intersection over union to optimize the loss function; finally, we build an enhanced feature extraction network based on the feature pyramid network for multi-scale feature fusion. The experimental results show that this method has small model size, good real-time, and good portability, which is suitable for practical production.

With the rapid development of optical remote sensing, it is urgent to find a reliable target detection method. Compared with traditional detection algorithms, a convolutional neural network has attracted considerable attention owing to its efficiency and high transitivity. However, different from general images, remote sensing images contain complex background information and dense small targets with changeable directions that make detection very challenging. To solve these problems and provide a stable and high-performance detection method, a rotated saliency fusion object detection (RSD) model based on “you only look once” (YOLO)v4 is established. First, salient image fusion technology is used to magnify target information. Second, the angle variable and rotated non-maximal suppression is introduced to improve the accuracy of rotated object detection by including the detection of dense objects. Third, the network structure is enhanced to improve the performance of small-target detection. Finally, the k-means algorithm and data enhancement are introduced to increase the robustness of the model. Extensive experiments demonstrate the superiority of the proposed model in detection speed and accuracy. The mean average precision of the proposed RSD model reaches 97.32% for the remote sensing images in a harbor area with an average detection speed of 13.41 s − 1.

Full-field optical coherence tomography (FFOCT) can be used to acquire cellular-level morphological images of nonfixed and unstained biological tissues. However, the speckle noise and vignetting of FFOCT images are drawbacks in terms of pathological application. We present a novel adaptive nonlocal image enhancement algorithm that uses a fluorescence microscopy (FM) image of the nucleus to achieve cytoplasm image enhancement under FFOCT. The algorithm comprises a denoising module and an adaptive nonlocal enhancement module. The denoising module is an end-to-end residual network with skip connections and dilated convolution. The adaptive nonlocal enhancement module enhances images to segment tissues, nontissues, and transition areas so that the images retain the details of the original cell structure but do not contain noise. The proposed algorithm was implemented on a dual-modality scanner system that combines FFOCT and FM. A total of 12 pig liver and pig kidney tissues were sectioned into 10,236 samples for experiments. The peak signal-to-noise ratio and structural similarity index value increased from 30.8 and 0.58, respectively, for the aforementioned system without the proposed algorithm to 34.2 and 0.86, respectively, for the aforementioned system with the proposed algorithm. Ten samples of human breast tissues were used to validate the proposed algorithm. The average contrast-to-noise ratio was 1.69 and 11.73 when the aforementioned system was used with and without the proposed algorithm, respectively. The results of this study revealed that the quality of scanned images considerably improved when the proposed system was used and that the proposed system can facilitate clinical pathological diagnosis.