Search

Abstract: Blind deconvolution is an effective fluorescence microscopic image processing technique to improve the quality of degraded digital images resulting from photon counting noise and out-of-focus blur. For solving the severely ill-posed problem of deconvolution, in this paper we propose an alternate minimized blind deconvolution method which considers sparse representation as constraint condition to confine the solution space of traditional Richardson–Lucy method and Gaussian model as initial value of point spread function (PSF). We assume that Poisson noise is dominating during the course of imaging. The maximum-likelihood estimation on a fluorescence image and corresponding point spread function (PSF) is developed. By solving the Euler–Lagrange equation of the total cost function, including the data term obtained by the hypothetical Poisson noise distribution model and the regularized term corresponding to the sparse representation constraint, and using gradient descent method we can get the iterative equations of the original fluorescence image and PSF respectively. Compared with the related blind deconvolution methods, our model shows superior performance in terms of both objective criteria and subjective human vision via processing simulated and real fluorescence microscopic degraded images. [Copyright &y& Elsevier]

Three-dimensional motion estimation from multiview video sequences is of vital importance to achieve high-quality dynamic scene reconstruction. In this paper, we propose a new 3-D motion estimation method based on matrix completion. Taking a reconstructed 3-D mesh as the underlying scene representation, this method automatically estimates motions of 3-D objects. A “separating + merging” framework is introduced to multiview 3-D motion estimation. In the separating step, initial motions are first estimated for each view with a neighboring view. Then, in the merging step, the motions obtained by each view are merged together and optimized by low-rank matrix completion method. The most accurate motion estimation for each vertex in the recovered matrix is further selected by three spatiotemporal criteria. Experimental results on data sets with synthetic motions and real motions show that our method can reliably estimate 3-D motions. [ABSTRACT FROM AUTHOR]

Extracting motion layers from videos is an important task for video representation, analysis, and compression. For videos with large interframe motions, motion layer extraction is challenging in two respects: the estimation of large disparity motions and the awareness of large occluded regions. In this paper, we propose an effective method for motion layer extraction under large disparity motions. To robustly estimate large displacement motions, we have developed an efficient voting-based method that estimates planar homographies from sparse feature matches. To handle occlusions, we first integrate color and motion consistency into a Markov random field framework to achieve per-pixel assignment with occlusion detection. Then, we perform motion-color segmentation and an earth mover's distance-based comparison to determine motion labels for occluded pixels. Experimental results show that our proposed method achieves good performance in automatically extracting multiple moving objects under large disparity motions while maintaining a low computational cost. [ABSTRACT FROM AUTHOR]

This paper proposes a spectral-graph-based algorithm for face image repairing, which can improve the recognition performance on occluded faces. The face completion algorithm proposed in this paper includes three main procedures: 1) sparse representation for partially occluded face classification; 2) image-based data mining; and 3) graph Laplace (GL) for face image completion. The novel part of the proposed framework is GL, as named from graphical models and the Laplace equation, and can achieve a high-quality repairing of damaged or occluded faces. The relationship between the GL and the traditional Poisson equation is proven. We apply our face repairing algorithm to produce completed faces, and use face recognition to evaluate the performance of the algorithm. Experimental results verify the effectiveness of the GL method for occluded face completion. [ABSTRACT FROM AUTHOR]

We propose a new markerless shape and motion capture approach from multiview video sequences. The shape recovery method consists of two steps: separating and merging. In the separating step, the depth map represented with a point cloud for each view is generated by solving a proposed variational model, which is regularized by four constraints to ensure the accuracy and completeness of the reconstruction. Then, in the merging step, the point clouds of all the views are merged together and reconstructed into a 3-D mesh using a marching cubes method with silhouette constraints. Experiments show that the geometric details are faithfully preserved in each estimated depth map. The 3-D meshes reconstructed from the estimated depth maps are watertight and present rich geometric details, even for non-convex objects. Taking the reconstructed 3-D mesh as the underlying scene representation, a volumetric deformation method with a new positional-constraint computation scheme is proposed to automatically capture motions of the 3-D object. Our method can capture non-rigid motions even for loosely dressed humans without the aid of markers. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes a collaborative color calibration method for multi-camera systems. The multi-camera color calibration problem is formulated as an overdetermined linear system, in which a dynamic range shaping is incorporated to ensure the high contrasts for captured images. The cameras are calibrated with the parameters obtained by solving the linear system. For non-planar multi-camera systems, we design a novel omnidirectional color checker and present a method for establishing global correspondences to facilitate automatic color calibration without manual adjustment. According to experimental results on both synthetic and real-system datasets, the proposed method shows high performance in achieving inter-camera color consistency and high dynamic range. Thanks to the generality of the linear system formulation and the flexibility of the designed color checker, the proposed method is applicable to various multi-camera systems. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we propose a view-based 3D model retrieval algorithm, where many-to-many matching method, weighted bipartite graph matching, is employed for comparison between two 3D models. In this work, each 3D model is represented by a set of 2D views. Representative views are first selected from the query model and the corresponding initial weights are provided. These initial weights are further updated based on the relationship among these representative views. The weighted bipartite graph is built with these selected 2D views, and the matching result is used to measure the similarity between two 3D models. Experimental results and comparison with existing methods show the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes a 3D model comparison algorithm based on a 3D model descriptor: spatial structure circular descriptor (SSCD). The spatial structure is important in content-based 3D model analysis. Within the SSCD, the spatial structure of a 3D model is described by 2D images, and the attribute values of each pixel represent 3D spatial information. Hence, SSCD can preserve the global spatial structure of 3D models, and is invariant to rotation and scaling. In addition, by using 2D images to describe the spatial information of 3D models, all spatial information of the 3D models can be represented by SSCD without redundancy. Thus, SSCD can be applied to many scenarios which utilize spatial information. In this paper, an SSCD-based 3D model comparison algorithm is presented. The proposed algorithm has been tested on 3D model retrieval experiments. Experimental results demonstrate the effectiveness of the proposed algorithm. [Copyright &y& Elsevier]

Abstract: We address the task of view-based 3D object retrieval, in which each object is represented by a set of views taken from different positions, rather than a geometrical model based on polygonal meshes. As the number of views and the view point setting cannot always be the same for different objects, the retrieval task is more challenging and the existing methods for 3D model retrieval are infeasible. In this paper, the information in the sets of views is exploited from two aspects. On the one hand, the form of histogram is converted from vector to state sequence, and Markov chain (MC) is utilized for modeling the statistical characteristics of all the views representing the same object. On the other hand, the earth mover''s distance (EMD) is involved to achieve many-to-many matching between two sets of views. For 3D object retrieval, by combining the above two aspects together, a new distance measure is defined, and a novel approach to automatically determine the edge weights in graph-based semi-supervised learning is proposed. Experimental results on different databases demonstrate the effectiveness of our proposal. [Copyright &y& Elsevier]

Abstract: Histogram is a useful feature for image content analysis and has been widely used in many methods for image categorization. Most of the existing classifiers usually cannot distinguish the effects of different bins in histogram, except for setting different weights. However, these weights are often difficult to be exactly determined in advance. To further mine the information in histogram, in this paper, we propose a method to represent the histogram in another form called quasi-histogram, which can be thought as the state sequence of a Markov chain (MC). By modeling the quasi-histogram of each image as having been stochastically generated by an MC corresponding to its category, we can take the characteristic of each bin into account. Improved image categorization performance can be obtained through combining the results of the traditional classifier with those of MC. Experimental results show the effectiveness of our proposal. [Copyright &y& Elsevier]

Presents a study that examined the singular value decomposition (SVD) of the extended matrix. Information on SVD-based techniques; Definition of extended matrix; Findings.

Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the ‘incompleteness’ and the ‘inaccuracy’ problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting. [ABSTRACT FROM AUTHOR]

Event cameras or dynamic vision sensors (DVS) record asynchronous response to brightness changes instead of conventional intensity frames, and feature ultra-high sensitivity at low bandwidth. The new mechanism demonstrates great advantages in challenging scenarios with fast motion and large dynamic range. However, the recorded events might be highly sparse due to either limited hardware bandwidth or extreme photon starvation in harsh environments. To unlock the full potential of event cameras, we propose an inventive event sequence completion approach conforming to the unique characteristics of event data in both the processing stage and the output form. Specifically, we treat event streams as 3D event clouds in the spatiotemporal domain, develop a diffusion-based generative model to generate dense clouds in a coarse-to-fine manner, and recover exact timestamps to maintain the temporal resolution of raw data successfully. To validate the effectiveness of our method comprehensively, we perform extensive experiments on three widely used public datasets with different spatial resolutions, and additionally collect a novel event dataset covering diverse scenarios with highly dynamic motions and under harsh illumination. Besides generating high-quality dense events, our method can benefit downstream applications such as object classification and intensity frame reconstruction. [ABSTRACT FROM AUTHOR]

Fluorescence microscopes are indispensable tools for the life science research community. Nevertheless, the presence of optical component limitations, coupled with the maximum photon budget that the specimen can tolerate, inevitably leads to a decline in imaging quality and a lack of useful signals. Therefore, image restoration becomes essential for ensuring high-quality and accurate analyses. This paper presents the Wavelet-Enhanced Convolutional-Transformer (WECT), a novel deep learning technique developed specifically for the purpose of reducing noise in microscopy images and attaining super-resolution. Unlike traditional approaches, WECT integrates wavelet transform and inverse-transform for multi-resolution image decomposition and reconstruction, resulting in an expanded receptive field for the network without compromising information integrity. Subsequently, multiple consecutive parallel CNN-Transformer modules are utilized to collaboratively model local and global dependencies, thus facilitating the extraction of more comprehensive and diversified deep features. In addition, the incorporation of generative adversarial networks (GANs) into WECT enhances its capacity to generate high perceptual quality microscopic images. Extensive experiments have demonstrated that the WECT framework outperforms current state-of-the-art restoration methods on real fluorescence microscopy data under various imaging modalities and conditions, in terms of quantitative and qualitative analysis. [ABSTRACT FROM AUTHOR]

Along with broad applications of the linear Doppler effect, the rotational Doppler effect (RDE) of a structured light source carrying orbital angular momentum (OAM) has attracted significant attention for applications ranging from optical sensors to Doppler cooling. However, the high‐purity structured source's low energy efficiency and unknown optimal OAM parameters have significantly degraded the RDE's performance in previous work. Here, instead of utilizing an optical vortex source, the Doppler features are analyzed in scattered twisted photons carrying OAM with a common light source, and it is demonstrated that the RDE induced by a typical rotator can be extracted using a spiral phase spatial filter (SPSF) at the receiver. This model reveals that a rotating rough surface scatters abundant twisted photons carrying varied OAM values, and the OAM spectrum distribution is modulated by its angular coherence of spatial signature. Furthermore, common surfaces with different autocorrelation structures on received signals using the SPSF method are analyzed theoretically and experimentally. Such a scheme facilitates rotator detection with a generalized nonvortex simple source and dozen–fold improved efficiency with robustness against noncoaxial problems. These demonstrations open a path for studying and applying scattered twisted photons during light detection. [ABSTRACT FROM AUTHOR]

Light field (LF) stitching is a potential solution to improve the field of view (FOV) for hand-held plenoptic cameras. Existing LF stitching methods cannot provide accurate registration for scenes with large depth variation. In this paper, a novel LF stitching method is proposed to handle parallax in the LFs more flexibly and accurately. First, a depth layer map (DLM) is proposed to guarantee adequate feature points on each depth layer. For the regions of nondeterministic depth, superpixel layer map (SLM) is proposed based on LF spatial correlation analysis to refine the depth layer assignments. Then, DLM-SLM-based LF registration is proposed to derive the location dependent homography transforms accurately and to warp LFs to its corresponding position without parallax interference. 4D graph-cut is further applied to fuse the registration results for higher LF spatial continuity and angular continuity. Horizontal, vertical and multi-LF stitching are tested for different scenes, which demonstrates the superior performance provided by the proposed method in terms of subjective quality of the stitched LFs, epipolar plane image consistency in the stitched LF, and perspective-averaged correlation between the stitched LF and the input LFs. [ABSTRACT FROM AUTHOR]

Research in light field reconstruction focuses on synthesizing novel views with the assistance of depth information. In this paper, we present a learning-based light field reconstruction approach by fusing a set of sheared epipolar plane images (EPIs). We start by showing that a patch in a sheared EPI will exhibit a clear structure when the sheared value equals the depth of that patch. By taking advantage of this pattern, a convolutional neural network (CNN) is then trained to evaluate the sheared EPIs, and output a reference score for fusing the sheared EPIs. The proposed CNN is elaborately designed to learn the similarity degree between the input sheared EPI and the ground truth EPI. Therefore, no depth information is required for network training and reasoning. We demonstrate the high performance of the proposed method through evaluations on synthetic scenes, real-world scenes, and challenging microscope light fields. We also show a further application of our proposed network for depth inference. [ABSTRACT FROM AUTHOR]

Reliable object detection under dark environment is of wide applications but severely challenged by heavy noise washing out informative features and uneven radiance caused by nighttime illuminations. These unique features of dark videos would largely degenerate the performance of existing detectors. To address this issue, specially designed algorithms being able to extract and fuse the weak features buried in the low-quality videos are of vital importance. Bearing these in mind, we propose illumination-aware spatio-temporal feature fusion modules for low-light video object detection and implement a Dark Video Detector under a TRansformer network structure, dubbed as DVD-TR. Firstly, we use a dual-backbone Transformer to extract separate complementary features and fuse them to strengthen the network's feature extraction capability. Secondly, we incorporate a spatio-temporal sampling mechanism to aggregate features from multiple frames, which can enhance detection accuracy in dark videos. Thirdly, we use a small encoder–decoder network to obtain irradiance distribution which is further incorporated for illumination-aware feature fusion. Extensive experiments on large-scale multi-illuminance dark video benchmark show that DVD-TR outperforms state-of-the-art video detectors by a large margin and validate the effectiveness of the proposed approach. • Dual-backbone structure strengthens feature extraction capability for low-light images/videos. • Powerful temporal feature aggregation fuses information of multiple frames in dark videos. • Illumination-aware feature fusion adapts to dark data with high dynamic range and variant darkness levels. • Overall hybrid-backbone model achieves superior performance to existing works for object detection in extremely dark videos. [ABSTRACT FROM AUTHOR]

The prediction of RNA structure with pseudoknots is a nondeterministic polynomial-time hard (NP-hard) problem; according to minimum free energy models and computational methods, we investigate the RNA-pseudoknotted structure. Our paper presents an efficient algorithm for predicting RNA structure with pseudoknots, and the algorithm takes O(n3) time and O(n2) space, the experimental tests in Rfam10.1 and PseudoBase indicate that the algorithm is more effective and precise. The predicting accuracy, the time complexity and space complexity outperform existing algorithms, such as Maximum Weight Matching (MWM) algorithm, PKNOTS algorithm and Inner Limiting Layer (ILM) algorithm, and the algorithm can predict arbitrary pseudoknots. And there exists a 1+ε (ε>0) polynomial time approximation scheme in searching maximum number of stackings, and we give the proof of the approximation scheme in RNA-pseudoknotted structure. We have improved several types of pseudoknots considered in RNA folding structure, and analyze their possible transitions between types of pseudoknots. [ABSTRACT FROM AUTHOR]

Videos captured under low light conditions suffer from severe noise. A variety of efforts have been devoted to image/video noise suppression and made large progress. However, in extremely dark scenarios, extensive photon starvation would hamper extracting latent structures buried in noise. Instead, developing an imaging system collecting more photons is a more effective way for high-quality video capture in dark environments. In this paper, we propose to build a dual-sensor camera to additionally collect the photons in NIR wavelength, and make use of the correlation between RGB and near-infrared (NIR) spectrum to perform high-quality reconstruction from noisy dark video pairs. In hardware, we build a compact dual-sensor camera capturing parallax-free RGB and NIR videos simultaneously. Computationally, we propose a dual-channel multi-frame attention network (DCMAN) utilizing spatial–temporal-spectral priors to mutually boost the quality of low-light RGB and NIR videos. In addition, we build a high-quality paired RGB and NIR video dataset, based on which the approach can be applied to different sensors easily by training the DCMAN model with simulated noisy input following a physical-process-based CMOS noise model. Both experiments on synthetic and real videos validate the performance of this compact dual-sensor camera design and the corresponding reconstruction algorithm in dark videography. • Build a compact RGB-NIR dual sensor camera for high quality dark videography. • Design a Dual-Channel Network to utilize spatial–temporal-spectral prior jointly. • Collect a high quality RGB + NIR video dataset on real scenes. • Propose a learning strategy adaptable to different sensors and noise levels easily. • Achieve superior performance to existing work in detail recovery and color fidelity. [ABSTRACT FROM AUTHOR]

Emerging sensor designs increasingly rely on novel color filter arrays (CFAs) to sample the incident spectrum in unconventional ways. In particular, capturing a near-infrared (NIR) channel along with conventional RGB color is an exciting new imaging modality. RGB+NIR sensing has broad applications in computational photography, such as low-light denoising, it has applications in computer vision, such as facial recognition and tracking, and it paves the way toward low-cost single-sensor RGB and depth imaging using structured illumination. However, cost-effective commercial CFAs suffer from severe spectral cross talk. This cross talk represents a major challenge in high-quality RGB+NIR imaging, rendering existing spatially multiplexed sensor designs impractical. In this work, we introduce a new approach to RGB+NIR image reconstruction using learned convolutional sparse priors. We demonstrate high-quality color and NIR imaging for challenging scenes, even including high-frequency structured NIR illumination. The effectiveness of the proposed method is validated on a large data set of experimental captures, and simulated benchmark results which demonstrate that this work achieves unprecedented reconstruction quality. [ABSTRACT FROM AUTHOR]

Development in low-cost depth acquisition devices pushes the relevant applications forward, which requires higher quality of the depth maps. In this paper, a novel depth de-noising algorithm is proposed, which dithers the depth map according to its spatial and temporal depth classification around the extracted texture edges. After roughly filling the holes in the depth map by joint bilateral filter, the depth map is partitioned into edge region and non-edge region according to the extracted texture edges. The depth values on each side of the mapped texture edges are dithered adapting to depth classification. Joint spatio-temporal dithering approach is proposed to align the depth edges with the texture edges and to suppress the depth artifacts. Experimental results demonstrate that compared with existing depth de-noising approaches, the proposed algorithm provides much sharper and clearer depth edges and it improves the spatial and temporal consistency of the processed depth, which further benefits the applications requiring the depth of high quality, e.g., 3D rendering. [ABSTRACT FROM AUTHOR]

Site-specific recombination systems are widely used as bioengineering tools. However, the traditional site-specific recombination system requires a consensus sequence for the specific site. Such sequence-level constraints limit effective recombination between sites. Therefore, in order to develop an efficient site-specific recombination system, we investigated the attC site of the bacterial integration subsystem and built a predictive model to infer the important features that contribute to recombination. Here, we design an attC site prediction algorithm based on a combination optimisation strategy. Based on the structural features of attC sites, the prediction algorithm realises the high-precision prediction of the recombination frequencies between sites and the screening of the top 20 important features that play a role in recombination, which are effective for improving the design method of attC sites. The algorithm has better portability and higher prediction accuracy compared with the existing advanced algorithms, among which the Pearson correlation coefficient is 0.87, explained variance score is 0.73, root mean square error is 0.006 and mean absolute error is 0.041. This can not only provide ideas for the research of efficient recombination systems but also provide a theoretical basis for developing genetic engineering further. [ABSTRACT FROM AUTHOR]

Multiview stereopsis (MVS) methods, which can reconstruct both the 3D geometry and texture from multiple images, have been rapidly developed and extensively investigated from the feature engineering methods to the data-driven ones. However, there is no dataset containing both the 3D geometry of large-scale scenes and high-resolution observations of small details to benchmark the algorithms. To this end, we present GigaMVS, the first gigapixel-image-based 3D reconstruction benchmark for ultra-large-scale scenes. The gigapixel images, with both wide field-of-view and high-resolution details, can clearly observe both the Palace-scale scene structure and Relievo-scale local details. The ground-truth geometry is captured by the laser scanner, which covers ultra-large-scale scenes with an average area of 8667 m $^2$ 2 and a maximum area of 32007 m $^2$ 2 . Owing to the extremely large scale, complex occlusion, and gigapixel-level images, GigaMVS exposes problems that emerge from the poor scalability and efficiency of the existing MVS algorithms. We thoroughly investigate the state-of-the-art methods in terms of geometric and textural measurements, which point to the weakness of the existing methods and promising opportunities for future works. We believe that GigaMVS can benefit the community of 3D reconstruction and support the development of novel algorithms balancing robustness, scalability and accuracy. [ABSTRACT FROM AUTHOR]

We consider the reconstruction problem of video snapshot compressive imaging (SCI), which captures high-speed videos using a low-speed 2D sensor (detector). The underlying principle of SCI is to modulate sequential high-speed frames with different masks and then these encoded frames are integrated into a snapshot on the sensor and thus the sensor can be of low-speed. On one hand, video SCI enjoys the advantages of low-bandwidth, low-power and low-cost. On the other hand, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging and one of the bottlenecks lies in the reconstruction algorithm. Existing algorithms are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play (PnP) framework. In addition to the PnP-ADMM method, we further propose the PnP-GAP (generalized alternating projection) algorithm with a lower computational workload. We first employ the image deep denoising priors to show that PnP can recover a UHD color video with 30 frames from a snapshot measurement. Since videos have strong temporal correlation, by employing the video deep denoising priors, we achieve a significant improvement in the results. Furthermore, we extend the proposed PnP algorithms to the color SCI system using mosaic sensors, where each pixel only captures the red, green or blue channels. A joint reconstruction and demosaicing paradigm is developed for flexible and high quality reconstruction of color video SCI systems. Extensive results on both simulation and real datasets verify the superiority of our proposed algorithm. [ABSTRACT FROM AUTHOR]

Noninvasive optical imaging through dynamic scattering media has numerous important biomedical applications but still remains a challenging task. While standard diffuse imaging methods measure optical absorption or fluorescent emission, it is also well‐established that the temporal correlation of scattered coherent light diffuses through tissue much like optical intensity. Few works to date, however, have aimed to experimentally measure and process such temporal correlation data to demonstrate deep‐tissue video reconstruction of decorrelation dynamics. In this work, a single‐photon avalanche diode array camera is utilized to simultaneously monitor the temporal dynamics of speckle fluctuations at the single‐photon level from 12 different phantom tissue surface locations delivered via a customized fiber bundle array. Then a deep neural network is applied to convert the acquired single‐photon measurements into video of scattering dynamics beneath rapidly decorrelating tissue phantoms. The ability to reconstruct images of transient (0.1–0.4 s) dynamic events occurring up to 8 mm beneath a decorrelating tissue phantom with millimeter‐scale resolution is demonstrated, and it is highlighted how the model can flexibly extend to monitor flow speed within buried phantom vessels. [ABSTRACT FROM AUTHOR]

Link prediction aims at inferring missing links or predicting future ones based on the currently observed network. This topic is important for many applications such as social media, bioinformatics and recommendation systems. Most existing methods focus on homogeneous settings and consider only low-order pairwise relations while ignoring either the heterogeneity or high-order complex relations among different types of nodes, which tends to lead to a sub-optimal embedding result. This paper presents a method named Heterogeneous Hypergraph Variational Autoencoder (HeteHG-VAE) for link prediction in heterogeneous information networks (HINs). It first maps a conventional HIN to a heterogeneous hypergraph with a certain kind of semantics to capture both the high-order semantics and complex relations among nodes, while preserving the low-order pairwise topology information of the original HIN. Then, deep latent representations of nodes and hyperedges are learned by a Bayesian deep generative framework from the heterogeneous hypergraph in an unsupervised manner. Moreover, a hyperedge attention module is designed to learn the importance of different types of nodes in each hyperedge. The major merit of HeteHG-VAE lies in its ability of modeling multi-level relations in heterogeneous settings. Extensive experiments on real-world datasets demonstrate the effectiveness and efficiency of the proposed method. [ABSTRACT FROM AUTHOR]

Markov random field (MRF), as one of special undirected graphs, is widely used in modeling priors of natural images. Targeting to learn better prior models from a given database, we explore the natural image statistics at different scales and build normalized filter pool, a kind of high-order MRF, for prior learning of nature images. The main contribution of the proposed model is that we construct a multi-scale MRF model through constraining the norms of filters in kernel space and integrate all the filtering responses in a unified framework. We formulate both learning and inference as constrained optimization problems and solve them using augmented Lagrange method. The experiment results demonstrate that the normalization of filters at different scales helps to achieve fast convergence in learning stage and obtain superior performance in image restoration, e.g., image denoising and image inpainting. [ABSTRACT FROM AUTHOR]

In this letter, we consider improving the image categorization performance by exploiting multiple local descriptors on the image. To achieve this goal, a novel deep learning configuration called maximum margin multimodal deep neural network (3mDNN) is proposed to learn joint feature from different data views. The local feature representations encoded by 3mDNN exhibit two significant advantages: (1) involving the information of multiple descriptors and (2) exhibiting discriminative ability. The whole deep architecture is well solved by the typical back propagation (BP) method and its performances are verified on three benchmark image datasets. [ABSTRACT FROM AUTHOR]

Identifying prostate cancer patients that are harboring aggressive forms of prostate cancer remains a significant clinical challenge. Here we develop an approach based on multispectral deep-ultraviolet (UV) microscopy that provides novel quantitative insight into the aggressiveness and grade of this disease, thus providing a new tool to help address this important challenge. We find that UV spectral signatures from endogenous molecules give rise to a phenotypical continuum that provides unique structural insight (i.e., molecular maps or "optical stains") of thin tissue sections with subcellular (nanoscale) resolution. We show that this phenotypical continuum can also be applied as a surrogate biomarker of prostate cancer malignancy, where patients with the most aggressive tumors show a ubiquitous glandular phenotypical shift. In addition to providing several novel "optical stains" with contrast for disease, we also adapt a two-part Cycle-consistent Generative Adversarial Network to translate the label-free deep-UV images into virtual hematoxylin and eosin (H&E) stained images, thus providing multiple stains (including the gold-standard H&E) from the same unlabeled specimen. Agreement between the virtual H&E images and the H&E-stained tissue sections is evaluated by a panel of pathologists who find that the two modalities are in excellent agreement. This work has significant implications towards improving our ability to objectively quantify prostate cancer grade and aggressiveness, thus improving the management and clinical outcomes of prostate cancer patients. This same approach can also be applied broadly in other tumor types to achieve low-cost, stain-free, quantitative histopathological analysis. [ABSTRACT FROM AUTHOR]

Modeling 3D humans accurately and robustly from a single image is very challenging, and the key for such an ill-posed problem is the 3D representation of the human models. To overcome the limitations of regular 3D representations, we propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. In our PaMIR-based reconstruction framework, a novel deep neural network is proposed to regularize the free-form deep implicit function using the semantic features of the parametric model, which improves the generalization ability under the scenarios of challenging poses and various clothing topologies. Moreover, a novel depth-ambiguity-aware training loss is further integrated to resolve depth ambiguities and enable successful surface detail reconstruction with imperfect body reference. Finally, we propose a body reference optimization method to improve the parametric model estimation accuracy and to enhance the consistency between the parametric model and the implicit function. With the PaMIR representation, our framework can be easily extended to multi-image input scenarios without the need of multi-camera calibration and pose synchronization. Experimental results demonstrate that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types. [ABSTRACT FROM AUTHOR]

We present a novel iterative feedback approach for the simultaneous estimation of depth and all-in-focus (AIF) videos from a defocused video pair by joint spatiotemporal optimization. Depth and AIF videos benefit each other in the iterative optimization. First, for the recovery of AIF video, the sparse prior of natural video is incorporated to ensure a high-quality defocus blur removal even under inaccurate depth estimation. Second, in depth estimation step, we feed back the spatial and temporal constraints from the high-quality AIF video and adopt a numerical solution, which is robust to the inaccuracy of AIF recovery to further boost the performance of depth from the defocus algorithm. Benefitting from the incorporation of AIF video priors and the temporal consistency constraint, the proposed framework can effectively reconstruct the depth of the textureless region and is insensitive to camera parameter changes. Our approach provides better temporal consistency and higher depth accuracy than the conventional method that applies postsmoothing to the sequential frame estimation. We not only demonstrate the feasibility of our approach via real experimentation but also provide visual and quantitative evaluation on synthetic data. [ABSTRACT FROM AUTHOR]

We proposed an approach to create plausible free-viewpoint relighting video using multi-view cameras array under general illumination. Given the multi-view video dataset recorded using a set of industrial cameras under general uncontrolled and unknown illumination, we first reconstruct 3D model of the captured target using existing multi-view stereo approach. Using the coarse geometry reconstruction, we estimate the spatially varying surface reflectance in the spherical harmonics domain considering the spatial and temporal coherence. With the estimated geometry and reflectance, the 3D target is relit to the novel illumination with the environment map of the target environment. Relit performance is enhanced using a flow- and quotient-based transfer strategy to achieve detailed and plausible performance relighting. Finally, the free-viewpoint video is generated using a view-dependent rendering strategy. Experimental results on various dataset show that our approach enables plausible free-view relighting, and opens up a path towards relightable free-viewpoint video using less complex acquisition setups. [ABSTRACT FROM AUTHOR]

Recent development of depth acquiring technique has accelerated the progress of 3D video in the market. Utilizing the acquired depth, arbitrary view frames can be generated based on depth image based rendering (DIBR) technique in free viewpoint video system. Different from texture video, depth sequence is mainly utilized for virtual view generation rather than viewing. Inspired by this, a depth frame interpolation scheme using texture information is proposed in this paper. The proposed scheme consists of a texture aided motion estimation (TAME) and texture aided motion compensation (TAMC) to fully explore the correlation between depth and the accompanying textures. The optimal motion vectors in TAME and the best interpolation weights in TAMC are respectively selected taking the geometric mapping relationship between depth and the accompanying texture frames into consideration. The proposed scheme is able to not only maintain the temporal consistency among interpolated depth sequence but also improve the quality of virtual frames generated by interpolated depth. Besides, it can be easily applied to arbitrary motion compensation based frame interpolation scheme. Experimental results demonstrate that the proposed depth frame interpolation scheme is able to improve the quality of virtual view texture frames in both subjective and objective criterions compared with existing schemes. [ABSTRACT FROM AUTHOR]

Site-specific recombination systems are widely used as bioengineering tools. However, the traditional site-specific recombination system requires a consensus sequence for the specific site. Such sequence-level constraints limit effective recombination between sites. Therefore, in order to develop an efficient site-specific recombination system, we investigated the attC site of the bacterial integration subsystem and built a predictive model to infer the important features that contribute to recombination. Here, we design an attC site prediction algorithm based on a combination optimisation strategy. Based on the structural features of attC sites, the prediction algorithm realises the high-precision prediction of the recombination frequencies between sites and the screening of the top 20 important features that play a role in recombination, which are effective for improving the design method of attC sites. The algorithm has better portability and higher prediction accuracy compared with the existing advanced algorithms, among which the Pearson correlation coefficient is 0.87, explained variance score is 0.73, root mean square error is 0.006 and mean absolute error is 0.041. This can not only provide ideas for the research of efficient recombination systems but also provide a theoretical basis for developing genetic engineering further. [ABSTRACT FROM AUTHOR]

Abstract: Removing non-uniform blur caused by camera shaking is troublesome because of its high computational cost. We analyze the efficiency bottlenecks of a non-uniform deblurring algorithm and propose an efficient optical computation deblurring framework that implements the time-consuming and repeatedly required modules, i.e., non-uniform convolution and perspective warping, by light transportation. Specifically, the non-uniform convolution and perspective warping are optically computed by a hybrid system that is composed of an off-the-shelf projector and a camera mounted on a programmable motion platform. Benefitting from the high speed and parallelism of optical computation, our system has the potential to accelerate existing non-uniform motion deblurring algorithms significantly. To validate the effectiveness of the proposed approach, we also develop a prototype system that is incorporated into an iterative deblurring framework to effectively address the image blur of planar scenes that is caused by 3D camera rotation around the x-, y- and z-axes. The results show that the proposed approach has a high efficiency while obtaining a promising accuracy and has a high generalizability to more complex camera motions. [Copyright &y& Elsevier]

Abstract: Detecting the visually identical regions among successive frames for noisy videos, called visual identicalness detection (VID) in this paper, is a fundamental tool in video applications for lower power consumption and higher efficiency. In this paper, instead of performing VID on the original video signal or on the de-noised video signal, a Retinex based VID approach is proposed to perform VID on the Retinex signal to eliminate the noise influence introduced by imaging system. Several Retinex output generation approaches are compared, within which the proposed Cohen–Daubechies–Feauveau wavelet based approach is demonstrated to have better efficiency in detection and higher adaptability to the video content and noise severity. Compared with approaches performing detection in the de-noised images, the proposed algorithm presents up to 4.78 times higher detection rate for the videos with moving objects and up to 30.79 times higher detection rate for the videos with static scenes, respectively, at the same error rate. Also, an application of this technique is provided by integrating it into an H.264/AVC video encoder. Compared with compressing the de-noised videos using the existing fast algorithm, an average of 1.7dB performance improvement is achieved with up to 5.47 times higher encoding speed. Relative to the reference encoder, up to 32.47 times higher encoding speed is achieved without sacrificing the subjective quality. [Copyright &y& Elsevier]

As an emerging imaging modality, transient imaging that records the transient information of light transport has significantly shaped our understanding of scenes. In spite of the great progress made in computer vision and optical imaging fields, commonly used multi-frequency time-of-flight (ToF) sensors are still afflicted with the band-limited modulation frequency and long acquisition process. To overcome such barriers, more effective image-formation schemes and reconstruction algorithms are highly desired. In this paper, we propose a compressive transient imaging model, without any priori knowledge, by constructing a near-tight-frame based representation of the ToF imaging principle. We prove that the compressibility of sensor measurements can be presented in the Fourier domain and held in the frame, and the ToF measurements possess multi-scale characteristics. Solving the inverse problems in transient imaging with our proposed model consists of two major steps, including a compressed-sensing-based approach for full measurement recovery, which essentially reduces the capture time, and a wavelet-based transient image reconstruction framework, which realizes adaptive transient image reconstruction and achieves highly accurate reconstruction results. The compressive transient imaging model is suitable for various existing multi-frequency ToF sensors and requires no hardware modifications. Experimental results using synthetic and real online datasets demonstrate its promising performance. [ABSTRACT FROM AUTHOR]

To improve free viewpoint video (FVV) coding efficiency and optimize the quality of the synthesized virtual view video, this paper proposes a depth-assisted FVV coding framework and analyzes the rate-distortion (R-D) property of the synthesized virtual view video in FVV coding. In the depth-assisted FVV coding framework, the depth assigned disparity compensated prediction is introduced to exploit the correlation between multiview video (MVV) and depth. To model the R-D property of the synthesized virtual view video, a region-based view synthesis distortion estimation approach is investigated with respect to the distortion of MVV and depth. Subsequently, the general R-D property estimation models of MVV and depth are analyzed. Finally, a rate-allocation scheme is designed to optimize the quantization parameter pair of MVV and depth in FVV coding. The simulation results demonstrate that the proposed depth-assisted FVV coding framework can improve the FVV coding efficiency. The region-based view synthesis distortion estimation approach and the general R-D model are able to precisely approximate the R-D property of synthesized virtual view video in the multiview video plus depth based FVV coding frameworks. The proposed rate-allocation scheme can optimize the overall FVV coding efficiency to achieve a high-quality reconstructed video at the desired viewpoint with a given rate constraint. [ABSTRACT FROM AUTHOR]

Perfect compressed sensing (CS) recovery can be achieved when a certain basis space is found to sparsely represent the original signal. However, due to the diversity of the signals, there does not exist a universal predetermined basis space that can sparsely represent all kinds of signals, which results in an unsatisfying performance. To improve the accuracy of recovered signal, this paper proposes an adaptive basis CS reconstruction algorithm by minimizing the rank of an accumulated matrix (MRAM), whose eigenvectors approximate the optimal basis sparsely representing the original signal. The accumulated matrix is constructed to efficiently exploit the second-order statistical property of the signal's autocorrelations. Based on the theory of matrix completion, MRAM reconstructs the original signal from its random projections under the observation that the constructed accumulated matrix is of low rank for most natural signals such as periodic signals and those coming from an autoregressive stationary process. Experimental results show that the proposed MRAM efficiently improves the reconstruction quality compared with the existing algorithms. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we propose an image-driven regional fusion method based on a specific region partition strategy according to the redundant and complementary correlation of the input images. Different from the traditional regional fusion approaches dividing one or more input images, our final region map is generated from the similarity comparisons between source images. Inspired by the success of structural similarity index (SSIM), the similarity characteristics of source images are represented by luminance, contrast, and structure comparisons. To generate redundant and complementary regions, we over segment the SSIM map using watershed, and merge the small homogeneous regions with close correlation based on the similarity components. In accordance with the concentrated similarity of different regions, the fusion principles for special regions are constructed to combine the redundant or complementary property. In our method, the redundant and complementary regions of input images are distinguished effectively, which can aid in the sequent fusion process. Experimental results demonstrate that our approach achieve superior results in the different fusion applications. Compared with the existing work, the proposed approach outperforms in both visual presentation and objective evaluation. [Copyright &y& Elsevier]

Estimating 3D information from an image sequence has long been a challenging problem, especially for dynamic scenes. In this paper, a novel semi-automatic 2D-to-3D conversion method is presented to estimate the disparity maps for regular 2D video shots. Our method requires only a few user-scribbles on very sparse key frames, and then other frames of the video are converted to 3D automatically. Multiple objects are first segmented by the input user-scribbles. Then, the initial disparity map is assigned to each key frame with the aid of various preset disparity models for each object. After the disparity assignment step, disparity maps for other frames of the video are obtained through a disparity propagation strategy taking into account both color similarity and motion information. Finally, the 3D video is synthesized according to the type of 3D display device. Our method is verified on different kinds of challenging sequences containing occlusion, textureless regions, color ambiguity, large displacement movements, etc. The experimental results show that our method has better performance than the state-of-the-art 2D-to-3D conversion systems. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we describe a video-object segmentation and 3D-trajectory estimation method for the analysis of dynamic scenes from a monocular uncalibrated view. Based on the color and motion information among video frames, our proposed method segments the scene, calibrates the camera, and calculates the 3D trajectories of moving objects. It can be employed for video-object segmentation, 2D-to-3D video conversion, video-object retrieval, etc. In our method, reliable 2D feature motions are established by comparing SIFT descriptors among successive frames, and image over-segmentation is achieved using a graph-based method. Then, the 2D motions and the segmentation result iteratively refine each other in a hierarchically structured framework to achieve video-object segmentation. Finally, the 3D trajectories of the segmented moving objects are estimated based on a local constant-velocity constraint, and are refined by a Hidden Markov Model (HMM)-based algorithm. Experiments show that the proposed framework can achieve a good performance in terms of both object segmentation and 3D-trajectory estimation. [ABSTRACT FROM AUTHOR]

Abstract: The most widely used basis selection algorithm for adaptive wavelet packets is the optimal basis selection method, which first grows a full wavelet packet tree and then prunes it into the optimal tree that gives the minimum cost. We observed that there exists the interscale embedding property in wavelet packet decomposition structures between interscale subbands along the same orientation. Based on this observation, we propose a fast basis selection algorithm by first decomposing a dyadic wavelet subband with the decomposition structure of its parent subband, before applying the optimal basis selection method for further decomposition. Experiments show that the proposed algorithm generates almost the same wavelet packet decomposition structures as the optimal basis selection method while significantly reducing the computational complexity for some image classes. [Copyright &y& Elsevier]

Abstract: This paper presents a neural model which learns low-dimensional nonlinear manifolds embedded in higher-dimensional data space based on mixtures of local linear manifolds under a self-organizing framework. Compared to other similar networks, the local linear manifolds learned by our network have a more localized representation of local data distributions thanks to a new distortion measure, which removes confusion between sub-models that exists in many similar mixture models. Each neuron in the network asymptotically learns a mean vector and a principal subspace of the data in its local region. It is proved that there is no local extremum for each sub-model. Experiments show that the new mixture model is better adapted to nonlinear manifolds of various data distributions than other similar models. The online-learning property of this model is desirable when the data set is very large, when computational efficiency is of paramount importance, or when data are sequentially input. We further show an application of this model to recognition of handwritten digit images based on mixtures of local linear manifolds. [Copyright &y& Elsevier]

Content-based image retrieval (CBIR) has been more and more important in the last decade, and the gap between high-level semantic concepts and low-level visual features hinders further performance improvement. The problem of online feature selection is critical to really bridge this gap. In this paper, we investigate online feature selection in the relevance feedback learning process to improve the retrieval performance of the region-based image retrieval system. Our contributions are mainly in three areas. 1) A novel feature selection criterion is proposed, which is based on the psychological similarity between the positive and negative training sets. 2) An effective online feature selection algorithm is implemented in a boosting manner to select the most representative features for the current query concept and combine classifiers constructed over the selected features to retrieve images. 3) To apply the proposed feature selection method in region-based image retrieval systems, we propose a novel region-based representation to describe images in a uniform feature space with real-valued fuzzy features. Our system is suitable for online relevance feedback learning in CBIR by meeting the three requirements: learning with small size training set, the intrinsic asymmetry property of training samples, and the fast response requirement. Extensive experiments, including comparisons with many state-of-the-arts, show the effectiveness of our algorithm in improving the retrieval performance and saving the processing time. [ABSTRACT FROM AUTHOR]

Abstract: Hidden annotation (HA) is an important research issue in content-based image retrieval (CBIR). We propose to incorporate long-term relevance feedback (LRF) with HA to increase both efficiency and retrieval accuracy of CBIR systems. The work contains two parts. (1) Through LRF, a multi-layer semantic representation is built to automatically extract hidden semantic concepts underlying images. HA with these concepts alleviates the burden of manual annotation and avoids the ambiguity problem of keyword-based annotation. (2) For each learned concept, semi-supervised learning is incorporated to automatically select a small number of candidate images for annotators to annotate, which improves efficiency of HA. [Copyright &y& Elsevier]