Search

Image deblurring is a classic problem in low-level computer vision with the aim to recover a sharp image from a blurred input image. Advances in deep learning have led to significant progress in solving this problem, and a large number of deblurring networks have been proposed. This paper presents a comprehensive and timely survey of recently published deep-learning based image deblurring approaches, aiming to serve the community as a useful literature review. We start by discussing common causes of image blur, introduce benchmark datasets and performance metrics, and summarize different problem formulations. Next, we present a taxonomy of methods using convolutional neural networks (CNN) based on architecture, loss function, and application, offering a detailed review and comparison. In addition, we discuss some domain-specific deblurring applications including face images, text, and stereo image pairs. We conclude by discussing key challenges and future research directions. [ABSTRACT FROM AUTHOR]

Rain is a common natural phenomenon. Taking images in the rain however often results in degraded quality of images, thus compromises the performance of many computer vision systems. Most existing de-rain algorithms use only one single input image and aim to recover a clean image. Few work has exploited stereo images. Moreover, even for single image based monocular deraining, many current methods fail to complete the task satisfactorily because they mostly rely on per pixel loss functions and ignore semantic information. In this paper, we present a Paired Rain Removal Network (PRRNet), which exploits both stereo images and semantic information. Specifically, we develop a Semantic-Aware Deraining Module (SADM) which solves both tasks of semantic segmentation and deraining of scenes, and a Semantic-Fusion Network (SFNet) and a View-Fusion Network (VFNet) which fuse semantic information and multi-view information respectively. In addition, we also introduce an Enhanced Paired Rain Removal Network (EPRRNet) which exploits semantic prior to remove rain streaks from stereo images. We first use a coarse deraining network to reduce the rain streaks on the input images, and then adopt a pre-trained semantic segmentation network to extract semantic features from the coarse derained image. Finally, a parallel stereo deraining network fuses semantic and multi-view information to restore finer results. We also propose new stereo based rainy datasets for benchmarking. Experiments on both monocular and the newly proposed stereo rainy datasets demonstrate that the proposed method achieves the state-of-the-art performance. https://github.com/HDCVLab/Stereo-Image-Deraining. [ABSTRACT FROM AUTHOR]

Facial expression is related to facial muscle contractions and different muscle movements correspond to different emotional states. For micro-expression recognition, the muscle movements are usually subtle, which has a negative impact on the performance of current facial emotion recognition algorithms. Most existing methods use self-attention mechanisms to capture relationships between tokens in a sequence, but they do not take into account the inherent spatial relationships between facial landmarks. This can result in sub-optimal performance on micro-expression recognition tasks. Therefore, learning to recognize facial muscle movements is a key challenge in the area of micro-expression recognition. In this paper, we propose a Hierarchical Transformer Network (HTNet) to identify critical areas of facial muscle movement. HTNet includes two major components: a transformer layer that leverages the local temporal features and an aggregation layer that extracts local and global semantical facial features. Specifically, HTNet divides the face into four different facial areas: left lip area, left eye area, right eye area and right lip area. The transformer layer is used to focus on representing local minor muscle movement with local self-attention in each area. The aggregation layer is used to learn the interactions between eye areas and lip areas. The experiments on four publicly available micro-expression datasets show that the proposed approach outperforms previous methods by a large margin. The codes and models are available at: https://github.com/wangzhifengharrison/HTNet. [ABSTRACT FROM AUTHOR]

Image restoration in adverse weather conditions is a difficult task in computer vision. In this paper, we propose a novel transformer-based framework called GridFormer which serves as a backbone for image restoration under adverse weather conditions. GridFormer is designed in a grid structure using a residual dense transformer block, and it introduces two core designs. First, it uses an enhanced attention mechanism in the transformer layer. The mechanism includes stages of the sampler and compact self-attention to improve efficiency, and a local enhancement stage to strengthen local information. Second, we introduce a residual dense transformer block (RDTB) as the final GridFormer layer. This design further improves the network’s ability to learn effective features from both preceding and current local features. The GridFormer framework achieves state-of-the-art results on five diverse image restoration tasks in adverse weather conditions, including image deraining, dehazing, deraining & dehazing, desnowing, and multi-weather restoration. The source code and pre-trained models will be released. [ABSTRACT FROM AUTHOR]

Current deep deblurring methods pay main attention to learning a transferring network to transfer synthetic blurred images to clean ones. Though achieving significant performance on the training datasets, they still suffer from a weaker generalization capability from training datasets to others with different synthetic blurs, thus resulting in significantly inferior performance on testing datasets. In order to alleviate this problem, we propose a latent contrastive model, Blur Distilling and Information Reconstruction Networks (BDIRNet), to learn image prior and improve the robustness of deep deblurring. The proposed BDIRNet consists of a blur removing network (DistillNet) and a reconstruction network (RecNet). Two kinds of images with almost the same information but different qualities are input into DistillNet to extract identical structure information via contrast latent information and purify the perturbations from other unimportant information like blur. While the RecNet is utilized to reconstruct sharp images based on the extracted information. In addition, inside the DistillNet and RecNet, a statistical anti-interference distilling (SAID) and anti-interference reconstruction (SAIR) modules are proposed to further enhance the robustness of our methods, respectively. Extensive experiments on different datasets show that the proposed methods achieve improved and robust results compared to recent state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Images captured in snowy days suffer from noticeable degradation of scene visibility, which degenerates the performance of current vision-based intelligent systems. Removing snow from images thus is an important topic in computer vision. In this paper, we propose a Deep Dense Multi-Scale Network (DDMSNet) for snow removal by exploiting semantic and depth priors. As images captured in outdoor often share similar scenes and their visibility varies with depth from camera, such semantic and depth information provides a strong prior for snowy image restoration. We incorporate the semantic and depth maps as input and learn the semantic-aware and geometry-aware representation to remove snow. In particular, we first create a coarse network to remove snow from the input images. Then, the coarsely desnowed images are fed into another network to obtain the semantic and depth labels. Finally, we design a DDMSNet to learn semantic-aware and geometry-aware representation via a self-attention mechanism to produce the final clean images. Experiments evaluated on public synthetic and real-world snowy images verify the superiority of the proposed method, offering better results both quantitatively and qualitatively. https://github.com/HDCVLab/Deep-Dense-Multi-scale-Network https://github.com/HDCVLab/Deep-Dense-Multi-scale-Network. [ABSTRACT FROM AUTHOR]

Automatic hand-drawn sketch recognition is an important task in computer vision. However, the vast majority of prior works focus on exploring the power of deep learning to achieve better accuracy on complete and clean sketch images, and thus fail to achieve satisfactory performance when applied to incomplete or destroyed sketch images. To address this problem, we first develop two datasets that contain different levels of scrawl and incomplete sketches. Then, we propose an angular-driven feedback restoration network (ADFRNet), which first detects the imperfect parts of a sketch and then refines them into high quality images, to boost the performance of sketch recognition. By introducing a novel “feedback restoration loop” to deliver information between the middle stages, the proposed model can improve the quality of generated sketch images while avoiding the extra memory cost associated with popular cascading generation schemes. In addition, we also employ a novel angular-based loss function to guide the refinement of sketch images and learn a powerful discriminator in the angular space. Extensive experiments conducted on the proposed imperfect sketch datasets demonstrate that the proposed model is able to efficiently improve the quality of sketch images and achieve superior performance over the current state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Blind Face Restoration (BFR) aims to generate high-quality face images from low-quality inputs. However, existing BFR methods often use private datasets for training and evaluation, making it challenging for future approaches to compare fairly. To address this issue, we introduce two benchmark datasets, BFRBD128 and BFRBD512, for evaluating state-of-the-art methods in five scenarios: blur, noise, low resolution, JPEG compression artifacts, and full degradation. We use seven standard quantitative metrics and two task-specific metrics, AFLD and AFICS. Additionally, we propose an efficient baseline model called Swin Transformer U-Net (STUNet), which outperforms state-of-the-art methods in various BFR tasks. The codes, datasets, and trained models are publicly available at: https://github.com/bitzpy/Blind-Face-Restoration-Benchmark-Datasets-and-a-Baseline-Model. [ABSTRACT FROM AUTHOR]

Despite significant progress has been made in image and video deblurring, much less attention has been paid to process ultra high-definition (UHD) videos (e.g., 4K resolution). In this work, we propose a novel deep model for fast and accurate UHD video deblurring (UHDVD). The proposed UHDVD is achieved by a depth-wise separable-patch architecture, which operates with a multi-scale integration scheme to achieve a large receptive field without adding the number of generic convolutional layers and kernels. Additionally, we adopt the temporal feature attention module to effectively exploit the temporal correlation between video frames to obtain clearer recovered images. We design an asymmetrical encoder–decoder architecture with residual channel-spatial attention blocks to improve accuracy and reduce the depth of the network appropriately. Consequently, the proposed UHDVD achieves real-time performance on 4K videos at 30 fps. To train the proposed model, we build a new dataset comprised of 4K blurry videos and corresponding sharp frames using three different smartphones. Extensive experimental results show that our network performs favorably against the state-of-the-art methods on the proposed 4K dataset and existing 720p and 2K benchmarks in terms of accuracy, speed, and model size. [ABSTRACT FROM AUTHOR]

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and the temporal domain (i.e., neighboring frames) and 2) how to restore sharp image details with respect to the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a deblurring network (DBLRNet) for spatial-temporal learning by applying a 3D convolution to both the spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to the improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the generative adversarial network (GAN) architecture and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as deblurring GAN, is tested on two standard benchmarks and achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

One key challenging issue of facial expression recognition is to capture the dynamic variation of facial physical structure from videos. In this paper, we propose a part-based hierarchical bidirectional recurrent neural network (PHRNN) to analyze the facial expression information of temporal sequences. Our PHRNN models facial morphological variations and dynamical evolution of expressions, which is effective to extract “temporal features” based on facial landmarks (geometry information) from consecutive frames. Meanwhile, in order to complement the still appearance information, a multi-signal convolutional neural network (MSCNN) is proposed to extract “spatial features” from still frames. We use both recognition and verification signals as supervision to calculate different loss functions, which are helpful to increase the variations of different expressions and reduce the differences among identical expressions. This deep evolutional spatial-temporal network (composed of PHRNN and MSCNN) extracts the partial-whole, geometry-appearance, and dynamic-still information, effectively boosting the performance of facial expression recognition. Experimental results show that this method largely outperforms the state-of-the-art ones. On three widely used facial expression databases (CK+, Oulu-CASIA, and MMI), our method reduces the error rates of the previous best ones by 45.5%, 25.8%, and 24.4%, respectively. [ABSTRACT FROM AUTHOR]

[Display omitted] • Ultrathin metal leaf to achieve conformal interface and uniform carbon precipitation. • BLG grown on dielectric substrates shows sub-nm surface roughness. • The interface-grown graphene exhibits nearly zero strain. • BLG grown on sapphire shows long-range scattering mobility of ∼1000 cm2V−1 s−1. • Facile, versatile and scalable synthesis route for wrinkle-free graphene production. Wrinkles invariably form during graphene growth and post-growth transfer, limiting graphene films' large-scale uniformity for electronic applications. We report a transfer-free synthesis route for highly-uniform bilayer graphene directly on dielectric substrates—SiO 2 , sapphire, and MgO—by interfacial carbon precipitation. Ultrathin Pd leaves having a thickness of 150 nm and grain size up to 100 µm are laminated onto the target dielectric substrate, followed by annealing and press rolling to form a uniform Pd-substrate interface. Rapid heating in a hydrocarbon atmosphere causes carbon diffusion through the Pd layer; upon cooling, precipitation of carbon results in graphene growth at the Pd-substrate interface. The interface-grown graphene remains on the substrate after removing the Pd layer by wet etching. It exhibits sub-nm surface roughness without wrinkles or folds. Over 94 % of the interface-grown area is dominated by bilayer graphene with low twist angles. In addition, the interface-grown graphene is nearly strain-free. From Raman characterization, an average long-range scattering mobility of ∼1000 cm2 V−1 s−1 was estimated for as-grown bilayer graphene on sapphire (0001) at room temperature. This technique shows promise to achieve device scale, ultra-uniform graphene fabrication directly on dielectric substrates, with the potential to accelerate graphene applications in electronics, photonics, and sensing. [ABSTRACT FROM AUTHOR]

Facial expression recognition has a wide range of applications in the real world. Although many existing deep learning methods have achieved remarkable success, weak expression recognition remains a challenging task because of the significant domain gap between a weak expression and its peak expression counterpart. One idea to solve this problem is to find an effective way to bridge the gap between the two domains by either transfer learning or cross-domain image synthesis. In this paper, we propose a Group Generative Adversarial Network (GroupGAN) that recognizes weak facial expression by magnifying the expressions to stronger or peak ones. Different from the traditional GAN which typically has only one generator and one discriminator, the proposed GroupGAN has one generator, one extractor and two discriminators. Similar to the "two-player game" analogy of the traditional GAN, in our setting the generator along with feature extractor act as one group to compete with the other group of the two distinct discriminators. Extensive experiments show that the proposed GroupGAN significantly improves the performance of weak expression recognition, and is able to magnify weak expressions, thus facilitating many expression-related vision tasks like sketch recognition. [ABSTRACT FROM AUTHOR]

As an unnoticed specialized task in image restoration, rain-by-snow weather removal aims to eliminate the complicated coexisting rain streaks and snow particles. In this work, we propose a simple attention-based sampling cross-hierarchy Transformer (SASCFormer). Initially, we explore the proximity of convolution network and Transformer in hierarchical architectures and experimentally find they perform approximately for intra-stage feature representation. On this basis, we utilize a Transformer-like convolution block (TCB) to replace the computation-heavy self-attention while preserving the attention characteristics for adapting to the input content. Meanwhile, we demonstrate that cross-stage sampling progression is critical for the performance improvement in rain-by-snow weather removal, and propose a global–local self-attention sampling mechanism (GLSASM) that samples the features while preserving both the global and local dependencies. Finally, we synthesize two novel rain-by-snow weather-degraded benchmarks, RSCityscapes and RS100K datasets. Extensive experiments verify that our proposed SASCFormer achieves the best trade-off between the performance and inference time. In particular, our approach advances existing methods by 1. 14 dB ∼ 4. 89 dB in peak signal-to-noise ratio. Related resources are available at https://github.com/chdwyb/Rain-by-snow. • We first focus on rain-by-snow weather-degraded image restoration. • We propose a simple attention-based sampling cross-hierarchy Transformer. • We demonstrate that cross-stage progression is crucial for performance improvement. • Two novel rain-by-snow datasets are synthesized for rain-by-snow weather removal. [ABSTRACT FROM AUTHOR]

Gilding is the ancient process of coating intricate artifacts with precious metals. Fascinating Egyptian and Chinese sculptures, coated with <200 nm thin metal films by this process, have resisted corrosion, wear, and other environmental degradations for thousands of years. Here, 150 nm thin palladium leaves are enriched by doped with a single layer of graphene. Commercially available Pd leaves are uniquely suited for graphene synthesis by a highly dynamic chemical vapor deposition process. The Pd leaves made by high strain rate beating are stable at high synthesis temperature, resisting solid‐state dewetting owing to their extremely low grain triple junctions density (0.017 µm−1). Mathematical models of growth kinetics guide the development of extremely rapid synthesis conditions, resulting in the formation of high‐quality graphene on Pd in less than a minute, owing to the graphene grains growing twice as fast as copper‐catalyzed growth. The graphene monolayer on the leaf increases the effective surface modulus by 59% to 236 GPa. Uniaxial strain testing with Raman spectroscopy reveals the excellent crystallinity of graphene by probing the stress‐induced phonon shifts. This new material could open exciting opportunities in utilizing high‐quality 2D materials to coat large structures. Gilding is the ancient process of coating sculptures with ultrathin metal films to enrich them and protect them against corrosion and wear. In this study, commercial palladium leaves with 150 nm thickness are used as ultrathin catalyst for high‐quality monolayer graphene growth. The graphene‐Pd leaves show advantageous mechanical behaviors, which make them suitable for advanced gilding. [ABSTRACT FROM AUTHOR]

After Chemical Vapor Deposition (CVD), faceted structures are routinely observed on a variety of metal catalyst surfaces in the graphene-covered regions. In spite of having its bare surface flattened through high diffusivity and surface pre-melting at high temperatures, the graphene-covered copper surface still presents faceted structures. Using atomistic simulations, we show the role of graphene in the preservation of the faceted surface morphology at the graphene–copper interface, manifesting as a suppressant against surface melting and surface-specific diffusion. The results of our molecular dynamics simulations are consistent with our experimental observations and demonstrate the thermo-mechanical interfacial surface stabilization role of graphene. Our study provides an understanding applicable to most metal–graphene interfaces and is especially relevant to most metallic catalysts for graphene growth by CVD. Understanding the interaction between graphene and the catalyst surface structure is critical for producing ultra-flat and defect-free graphene. [ABSTRACT FROM AUTHOR]

High-temperature components utilised in practical service are often subjected to multiaxial non-proportional loading due to their complex geometry. In the present study, a set of strain-controlled fatigue tests were conducted on P92 steel at 600 °C, including uniaxial, multiaxial proportional, and multiaxial non-proportional loading conditions. Results reveal that the axial and shear stresses exhibit varied responses to the different non-proportionalities, with proportional loading resulting in the shortest fatigue life when the strain ratio is small. The non-proportional loading induces accelerated cyclic softening at axial direction while the shear stress is enhanced. Electron backscatter diffraction (EBSD) and transmission electron microscope (TEM) analysis indicate that the increased role of shear stress promotes the dislocation motion from the planar slip to cross-slip under non-proportional loading, which in turn facilitates the microstructure evolution. It is also important to note that the evident development of microstructure and the combined effect of more crack initiations and more secondary cracks contribute to the lowest fatigue life under proportional loading. • The non-proportional loading accelerates axial cyclic softening while the shear stress is enhanced. • The lowest fatigue life of P92 steel appears under proportional loading when the strain ratio is small. • The increased role of shear stress promotes the dislocation motion from the planar slip to cross-slip. [ABSTRACT FROM AUTHOR]

Although deep learning-based methods have dominated stereo matching leaderboards by yielding unprecedented disparity accuracy, their inference time is typically slow, i.e., less than 4 FPS for a pair of 540p images. The main reason is that the leading methods employ time-consuming 3D convolutions applied to a 4D feature volume. A common way to speed up the computation is to downsample the feature volume, but this loses high-frequency details. To overcome these challenges, we propose a displacement-invariant cost computation module to compute the matching costs without needing a 4D feature volume. Rather, costs are computed by applying the same 2D convolution network on each disparity-shifted feature map pair independently. Unlike previous 2D convolution-based methods that simply perform context mapping between inputs and disparity maps, our proposed approach learns to match features between the two images. We also propose an entropy-based refinement strategy to refine the computed disparity map, which further improves the speed by avoiding the need to compute a second disparity map on the right image. Extensive experiments on standard datasets (SceneFlow, KITTI, ETH3D, and Middlebury) demonstrate that our method achieves competitive accuracy with much less inference time. On typical image sizes (e.g., 540 × 960 ), our method processes over 100 FPS on a desktop GPU, making our method suitable for time-critical applications such as autonomous driving. We also show that our approach generalizes well to unseen datasets, outperforming 4D-volumetric methods. We will release the source code to ensure the reproducibility. [ABSTRACT FROM AUTHOR]

The profound accumulation of precipitation during intense rainfall events can markedly degrade the quality of images, leading to the erosion of textural details. Despite the improvements observed in existing learning-based methods specialized for heavy rain removal, it is discerned that a significant proportion of these methods tend to overlook the precise reconstruction of the intricate details. In this work, we introduce a simple dual-stage progressive enhancement network, denoted as DPENet, aiming to achieve effective deraining while preserving the structural accuracy of rain-free images. This approach comprises two key modules, a rain streaks removal network (R 2 Net) focusing on accurate rain removal, and a details reconstruction network (DRNet) designed to recover the textural details of rain-free images. Firstly, we introduce a dilated dense residual block (DDRB) within R 2 Net, enabling the aggregation of high-level and low-level features. Secondly, an enhanced residual pixel-wise attention block (ERPAB) is integrated into DRNet to facilitate the incorporation of contextual information. To further enhance the fidelity of our approach, we employ a comprehensive loss function that accentuates both the marginal and regional accuracy of rain-free images. Extensive experiments conducted on publicly available benchmarks demonstrates the noteworthy efficiency and effectiveness of our proposed DPENet. The source code and pre-trained models are currently available at https://github.com/chdwyb/DPENet. • Dual-stage deraining network performs better than the single-stage network. • Multi-level feature fusion is significant for rain streaks removal. • Contextual information aggregation facilitates textural details reconstruction. [ABSTRACT FROM AUTHOR]

Image restoration under hazy weather condition, which is called single image dehazing, has been of significant interest for various computer vision applications. In recent years, deep learning-based methods have achieved success. However, existing image dehazing methods typically neglect the hierarchy of features in the neural network and fail to exploit their relationships fully. To this end, we propose an effective image dehazing method named Hierarchical Contrastive Dehazing (HCD), which is based on feature fusion and contrastive learning strategies. HCD consists of a hierarchical dehazing network (HDN) and a novel hierarchical contrastive loss (HCL). Specifically, the core design in the HDN is a hierarchical interaction module, which utilizes multi-scale activation to revise the feature responses hierarchically. To cooperate with the training of HDN, we propose HCL which performs contrastive learning on hierarchically paired exemplars, facilitating haze removal. Extensive experiments on public datasets, RESIDE, HazeRD, and DENSE-HAZE, demonstrate that HCD quantitatively outperforms the state-of-the-art methods in terms of PSNR, SSIM and achieves better visual quality. • Our method employs contrastive learning to enhance feature representation ability. • Hierarchical interaction module allows information flow to exchange efficiently across different branches. • Hierarchical contrastive loss effectively guides the model to learn the valid features for image dehazing. [ABSTRACT FROM AUTHOR]

The skin of the cephalopod is a 3D display, where the papillae muscles control the protrusion of each voxel by several millimeters out of the skin plane, create hierarchical textures, and collectively change the overall skin pattern in a fraction of a second. A material system capable of mimicking this response using electromechanical actuation of twisted spiral artificial muscles (TSAMs) is presented in this study. TSAMs leverage the mechanics of their twisted geometry to extend out of plane by 8 mm, corresponding to 2000% strain using a voltage of only 0.02 V mm−1. They are made of polymer fibers wrapped with a helical metal wire. These actuators are assembled on a stretchable skin with the required flexible electrical connections to form an array of digital texture voxels (DTVs). The DTV array produces arbitrary 3D surface patterns on‐demand, and provides opportunities to control hydrodynamic drag, camouflage, and haptic displays. [ABSTRACT FROM AUTHOR]