Search

Abstract We present a lightweight and efficient semi-supervised video object segmentation network based on the space-time memory framework. To some extent, our method solves the two difficulties encountered in traditional video object segmentation: one is that the single frame calculation time is too long, and the other is that the current frame’s segmentation should use more information from past frames. The algorithm uses a global context (GC) module to achieve high-performance, real-time segmentation. The GC module can effectively integrate multi-frame image information without increased memory and can process each frame in real time. Moreover, the prediction mask of the previous frame is helpful for the segmentation of the current frame, so we input it into a spatial constraint module (SCM), which constrains the areas of segments in the current frame. The SCM effectively alleviates mismatching of similar targets yet consumes few additional resources. We added a refinement module to the decoder to improve boundary segmentation. Our model achieves state-of-the-art results on various datasets, scoring 80.1% on YouTube-VOS 2018 and a $${\cal J}{\rm{\& }}{\cal F}$$ J & F score of 78.0% on DAVIS 2017, while taking 0.05 s per frame on the DAVIS 2016 validation dataset.

Background: Monocular depth estimation aims to predict the dense depth map from a single RGB image, which has important applications in 3D reconstruction, automatic driving, and augmented reality. However, existing methods directly feed the original RGB image into the model to extract depth features without avoiding the interference of depth-irrelevant information on depth estimation accuracy, which leads to inferior performance. To remove the influence of depth-irrelevant information and improve depth prediction accuracy, we propose RADepthNet, a novel reflectance-guided network fusing boundary features. Specifically, our method predicts depth maps using three steps: 1) Intrinsic Image Decomposition. We propose a Reflectance extraction module consisting of an encoder-decoder structure to extract depth-related reflectance. We demonstrate that the module can reduce the influence of illumination on depth estimation through an ablation study. 2) Boundary Detection. Boundary extraction module, consisting of an encoder, a refinement block, and an upsample block, is proposed to better predict depth at object boundaries utilizing gradient constraints. 3) Depth Prediction Module. Use a different encoder from 2) to obtain depth features from the reflectance map and fuse boundary features to predict depth. Besides, we proposed FIFADataset, a depth estimation dataset applied in soccer scenarios. Extensive experiments on the public dataset and our proposed FIFADataset show that our method achieves state-of-the-art performance.

Background: Exploring the correspondences across multi-view images is the basis of many computer vision tasks. However, most existing methods are limited on accuracy under challenging conditions. In order to learn more robust and accurate correspondences, we propose the DSD-MatchingNet for local feature matching in this paper. First, we develop a deformable feature extraction module to obtain multi-level feature maps, which harvests contextual information from dynamic receptive fields. The dynamic receptive fields provided by deformable convolution network ensures our method to obtain dense and robust correspondences. Second, we utilize the sparse-to-dense matching with the symmetry of correspondence to implement accurate pixel-level matching, which enables our method to produce more accurate correspondences. Experiments have shown that our proposed DSD-MatchingNet achieves a better performance on image matching benchmark, as well as on visual localization benchmark. Specifically, our method achieves 91.3% mean matching accuracy on HPatches dataset and 99.3% visual localization recalls on Aachen Day-Night dataset.

Background: The imperfect material effect is one of the most important themes to obtain photo-realistic results in rendering. Textile material rendering has always been a key area in the field of computer graphics. So far, a great deal of effort has been invested in its unique appearance and physicsbased simulation. The appearance of the dyeing effect commonly found in textiles has received little attention. This paper introduces techniques for simulation of staining effects on textiles. Pulling, wearing, squeezing, tearing, and breaking effects are more common imperfect effects of fabrics, these external forces will cause changes in the fabric structure, thus affecting the diffusion effect of stains. Based on the microstructure of yarn, we handle the effect of the stain on the imperfect textile surface. Our simulation results can achieve a photo-realistic effect.

Background: The interaction of gas and liquid can produce many interesting phenomena, such as bubbles rising from the bottom of the liquid. The simulation of two-phase fluids is a challenging topic in computer graphics. To animate the interaction of a gas and liquid, MultiFLIP samples the two types of particles, and a Euler grid is used to track the interface of the liquid and gas. However, MultiFLIP uses the fluid implicit particle (FLIP) method to interpolate the velocities of particles into the Euler grid, which suffer from additional noise and instability. Methods: To solve the problem caused by fluid implicit particles (FLIP), we present a novel velocity transport technique for two individual particles based on the affine particle-in-cell (APIC) method. First, we design a weighed coupling method for interpolating the velocities of liquid and gas particles to the Euler grid such that we can apply the APIC method to the simulation of a two-phase fluid. Second, we introduce a narrowband method to our system because MultiFLIP is a time-consuming approach owing to the large number of particles. Results: Experiments show that our method is well integrated with the APIC method and provides a visually credible two-phase fluid animation. Conclusions: The proposed method can successfully handle the simulation of a twophase fluid.

Abstract Compared with video de‐raining, single image de‐raining is more technically difficult due to the lack of temporally redundant information. This paper proposes a new two‐stage method for single image de‐raining. In the first stage, the authors develop an effective two‐step model to detect the rain streaks by taking pixel intensity, and direction of rain streaks as priors. In the second stage, the rain repair process is performed at the patch level. The authors first define a way to search for similar patches of each patch, and then group the similar patches together to form a matrix. Finally, a low‐rank matrix completion technique is utilized to recover the rain‐stained pixels based on the rain map obtained from the first stage. Compared with several state‐of‐the‐art methods, authors' proposed method is competitive in terms of the abilities of removing rain streaks, and preserving image details.

The epicardial fat plays a key role in the development of many cardiovascular diseases. It is necessary and useful to precisely segment this fat from CT scans in clinical studies. However, it is not feasible to manually segment this fat in clinical practice, as the workload and cost for technicians or physicians is quite high. In this work, we propose a novel method for automatic segmentation and quantification of epicardial fat from CT scans accurately. In detail, dual U-Nets with the morphological processing layer is used for this goal. The first network is based on the U-Net framework to detect the pericardium, before segmenting its inside region. A morphological layer is concatenated as the following layer of the first network, to refine and obtain the ideal inside region of the pericardium. While the second network is also applied using U-Net as its backbone to find and segment the epicardial fat of the processed inside region from the pericardium using the first network. Our proposed method obtains the highest mean Dice similarity (91.19%), correlation coefficient (0.9304) compared to other state-of-art methods on a cardiac CT dataset with 20 patients. The results indicate our proposed method is effective for quantifying epicardial fat automatically.

Automating the production of 2D hand-drawn animations is a significant and interesting component in computer graphics and vision. However, traditional methods in animation production pipeline always use physically or geometrically based models which are consuming due to complicated and massive computations, reducing their practicability. In this paper, we propose an efficient data-driven approach to create hand-drawn animations in an automatic manner. The key idea is to employ a correspondence match-based random search process to extract the geometry samples and the global motion pattern in an input animation sequence and then to generate a new output sequence through a coarse-to-fine sample-based synthesis algorithm. Our experiments demonstrate that our method achieves good results with high quality and performance, producing a range of artistic effects that previously required disparate and professional techniques.

We propose a new photo-guided drawing tool for the easy generation of artistic strokes in the style of flower painting, which is the most representative one in Chinese paintings, through stylizing rough sketch lines over a photoimage. While existing style migration methods aim at processing the entire image, we depict that a stroke-based stylization framework better performs detailed style features and catches the user's intent. The steps in our technique are first to correct user inaccurate strokes that mark candidates of style patterns on a painting and skeletal paths to be stylized over a photograph, then to select the optimal pattern for every skeletal stroke, and finally, to map the pattern onto skeletal stroke by shape calculation and texture synthesis. This tool can tolerate errors from user inputs and work well for touch devices. Our experiments prove that the paintings our technique helped to generate show close to the specified style and are aesthetically satisfied by normal users.

Recently, single-image rain removal has attracted sustained attention. In this paper, we utilize a simple shape prior, the fact that the rain streaks often appear in an elongated elliptical shape, in a particular way for single image de-raining. With this prior, the patches of the rain layer are forced to have a high correlation with our synthetic ellipse matrix. Then, the rain layer is refined during each layer decomposition process with the correlation constraints. The experiments show that the proposed method outperforms the existing single image de-raining methods on both real and synthetic data. Moreover, our approach runs much faster than the other state-of-the-art de-raining methods.