Search

We propose a novel monocular depth estimator, which improves the prediction accuracy on human regions by utilizing pose information. The proposed algorithm consists of two networks — PoseNet and DepthNet — to estimate keypoint heatmaps and a depth map, respectively. We incorporate the pose information from PoseNet to improve the depth estimation performance of DepthNet. Specifically, we develop the feature blending block, which fuses the features from PoseNet and DepthNet and feeds them into the next layer of DepthNet, to make the networks learn to predict the depths of human regions more accurately. Furthermore, we develop a novel joint training scheme using partially labeled datasets, which balances multiple loss functions effectively by adjusting weights. Experimental results demonstrate that the proposed algorithm can improve depth estimation performance significantly, especially around human regions. For example, the proposed algorithm improves the depth estimation performance on the human regions of ResNet-50 by 2.8% and 7.0% in terms of $\delta _{1}$ and RMSE, respectively, on the proposed HD + P dataset.

A novel light field super-resolution algorithm to improve the spatial and angular resolutions of light field images is proposed in this work. We develop spatial and angular super-resolution (SR) networks, which can faithfully interpolate images in the spatial and angular domains regardless of the angular coordinates. For each input image, we feed adjacent images into the SR networks to extract multi-view features using a trainable disparity estimator. We concatenate the multi-view features and remix them through the proposed adaptive feature remixing (AFR) module, which performs channel-wise pooling. Finally, the remixed feature is used to augment the spatial or angular resolution. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art algorithms on various light field datasets. The source codes and pre-trained models are available at https://github.com/keunsoo-ko/ LFSR-AFR

A novel video interpolation network to improve the temporal resolutions of video sequences is proposed in this work. We develop a multi-scale warping module to interpolate intermediate frames robustly for both small and large motions. Specifically, the proposed multi-scale warping module deals with large motions between two consecutive frames using coarse-scale features, while estimating detailed local motions by exploring fine-scale features. To this end, it takes multi-scale features from the encoder and estimates kernel weights and offset vectors for each scale. Finally, it synthesizes multi-scale warping frames and combines them to obtain an intermediate frame. Extensive experimental results demonstrate that the proposed algorithm outperforms state-of-the-art video interpolation algorithms on various benchmark datasets.

In this paper, a temporal superpixel algorithm using proximity-weighted patch matching (PPM) is proposed to yield temporally consistent superpixels for image and video processing. PPM estimates the motion vector of a superpixel robustly, by considering the patch matching distances of neighboring superpixels as well as the superpixel itself. In each frame, we initialize superpixels by transferring the superpixel labels of the previous frame using PPM motion vectors. Then, we update the superpixel labels of boundary pixels by minimizing a cost function, which is composed of feature distance, compactness, contour, and temporal consistency terms. Finally, we carry out superpixel splitting, merging, and relabeling to regularize superpixel sizes and correct inaccurate labels. Extensive experimental results confirm that the proposed algorithm outperforms the state-of-the-art conventional algorithms significantly. Also, it is demonstrated that the proposed algorithm can be applied to video object segmentation and video saliency detection tasks.

A novel algorithm to estimate instance-level future motion (FM) in a single image is proposed in this paper. First, the FM of an instance is defined with its direction, speed, and action classes. Then, a deep neural network, called FM-Net, is developed to determine the FM of the instance. More specifically, the multi-context pooling layer is proposed to exploit both object and global context features, and the cyclic ordinal regression scheme is developed using binary classifiers for effective FM classification. Also, the proposed FM-Net is trained in a semi-supervised domain adaptation setting to obtain reliable FM estimation results, even when a source domain in the training process and a target domain in the inference process are different. Extensive experimental results demonstrate that the proposed algorithm provides remarkable performance and thus can be used effectively for computer vision applications, including single object tracking, multiple object tracking, and crowd analysis. Furthermore, the FM dataset, collected from diverse sources and annotated manually, is released as a benchmark for single-image FM estimation.

How to make an online tracking model effectively adapt to newly appearing objects and object disappearance as well as appearance variations of target objects from few examples is an essential issue in multiple object tracking (MOT). Learning target appearances from few examples is a few-shot classification problem, while identifications of newly appearing objects and object disappearance has the aspect of open-set classification. In this work, we regard online MOT as open-set few-show classification to address both learning from few examples (few-shot classification) and unknown classes such as new objects (open-set classification). Specifically, we develop an embedding neural network, called VOFNet, consisting of convolutional and recurrent parts, to perform open-set few-shot classification. The convolutional part constructs a feature from an example of a target object and the recurrent part determines a representative feature of a target object from few examples. Then VOFNet is trained to provide effective features for open-set few-shot classification. Finally, we develop an online multiple object tracker based on the combination of VOFNet and the bipartite matching. The proposed tracker achieves 49.2 multiple object tracking accuracy (MOTA) with 28.9 frames per second on MOT17 dataset, which shows a significantly better trade-off between the accuracy and the speed than the existing algorithms. For example, the proposed algorithm yields about 3.17 times faster speed with 0.99 times lower accuracy than recent existing MOT algorithm [1] .

We propose an object tracking method under large motion in image sequences. Dense sampling and particle filtering have been widely applied to cope with this problem; however, the former is computationally expensive, and the latter is sensitive to local minima. By introducing a novel search method based on coarse-to-fine strategy and image superpixels , we try to solve both drawbacks. In the coarse step, we first extract superpixels associated with a target object on the entire search region by using a simple generative appearance model. In the fine step, we perform a sampling and similarity measurement process within the selected superpixels to find the most accurate location of the target object, also suggest a way to use both a discriminative appearance model and a sophisticated generative appearance model simultaneously. Extensive experiments on popular benchmark dataset demonstrate that the proposed method outperforms other competitive approaches, and also show better results in challenging scenarios such as occlusion, deformation, out-of-view, and in-plane/out-of-plane rotation.

We propose the property-specific aesthetic assessment (PSAA) algorithm with unsupervised aesthetic property discovery. The proposed PSAA algorithm uses an aesthetic feature extractor, an aesthetic property classifier, and multiple property-specific assessment networks. The aesthetic feature extractor analyzes aesthetics of images to generate features. Using such aesthetic features, we discover diverse aesthetic properties in an unsupervised manner and develop the aesthetic property classifier to predict the aesthetic property of each image. For each discovered aesthetic property, we train a property-specific assessment network. Thus, we can assess the aesthetic quality of an image using the property-specific network that corresponds to its property. Experimental results on a large dataset show that the proposed PSAA algorithm achieves state-of-the-art aesthetic assessment performance. Furthermore, we demonstrate that PSAA is useful for improving aesthetic qualities of images in two applications: contrast enhancement and image cropping.

We propose a deep learning framework for various 3D vision tasks, which takes a point cloud as input. The convolution is a basic operator for feature extraction in deep learning. However, it is not directly applicable to a point cloud, which is an irregular, unordered point set. This makes deep learning on point clouds challenging. To address this issue, we propose the extended Laplacian filter (ELF) for point clouds, which adopts the design principles of discrete Laplacian filters in 2D image processing. In other words, ELF extends the Laplacian filters and has the following two properties: 1) it is a two-state filter using two filter matrices (one for a center point and the other for neighboring points), and 2) it employs a scalar weighting function to predict the relative importance of the neighboring points. Then, we develop ELF-Nets, which consist of ELF convolution layers and fully connected layers. Experimental results demonstrate that the proposed ELF-Nets are capable of recognizing the 3D shape of a point cloud effectively and efficiently. In particular, ELF-Nets provide better or comparable performances than the state-of-the-art techniques in both object classification and part segmentation tasks.

A novel algorithm to detect an optimal set of semantic lines is proposed in this work. We develop two networks: selection network (S-Net) and harmonization network (H-Net). First, S-Net computes the probabilities and offsets of line candidates. Second, we filter out irrelevant lines through a selection-and-removal process. Third, we construct a complete graph, whose edge weights are computed by H-Net. Finally, we determine a maximal weight clique representing an optimal set of semantic lines. Moreover, to assess the overall harmony of detected lines, we propose a novel metric, called HIoU. Experimental results demonstrate that the proposed algorithm can detect harmonious semantic lines effectively and efficiently. Our codes are available at https://github.com/dongkwonjin/Semantic-Line-MWCS., Accepted to CVPR2021

Smart farm refers to a farm that can remotely and automatically maintain proper growth and management of crops and livestock by integrating technology with agriculture. Currently, smart farms are concentrated in the field of smart horticulture, and although spreading research is being conducted in limited spaces. In addition, it is difficult to obtain a sufficient amount of data to be used for learning, and there is a problem that data imbalance occurs because it is difficult to obtain a similar amount for each class. In this paper, we propose a method to amplify a small amount of data and to solve the problems of imbalance data by using a feature that can learn to mimic the data of a generative adversarial network. The proposed method can create dataset of various crops and also show high hit rate. Dataset generated from crops would be used to solve problems of data imbalance by learning.

We propose a novel guided interactive segmentation (GIS) algorithm for video objects to improve the segmentation accuracy and reduce the interaction time. First, we design the reliability-based attention module to analyze the reliability of multiple annotated frames. Second, we develop the intersection-aware propagation module to propagate segmentation results to neighboring frames. Third, we introduce the GIS mechanism for a user to select unsatisfactory frames quickly with less effort. Experimental results demonstrate that the proposed algorithm provides more accurate segmentation results at a faster speed than conventional algorithms. Codes are available at https://github.com/yuk6heo/GIS-RAmap., accepted to CVPR2021 (oral)

A procedure of interactive video object segmentation is cutting out a target object in a video by repeatedly providing user-annotations (points, lines, etc.) to an algorithm. In this paper, the procedure is achieved by generating the inter-image affinity matrix between the target frame and the annotated frame, and the other matrix between the previous frame and the target frame. By utilizing the two inter-image affinity matrices, we transport encoded features of the annotated frame and the previous frame for the target object. Through the experimental result, we show that exploiting the inter-image affinity matrices are effective for conducting the interactive video object segmentation.

In this paper, we propose a novel semantic line detection algorithm. For an input image, we first detect semantic lines using a semantic line detector by classifying candidate lines. Then, we predict scores indicating whether they are harmonized or not between the detected lines. To this end, we develop a score prediction network (SPNet). Finally, we construct a graph consisting of the detected lines and the predicted scores between them and iteratively select the reliable semantic lines. Experimental results demonstrate that the proposed algorithm detects semantic lines accurately.

We propose an optimized color contrast enhancement algorithm for dichromats with color vision deficiency using local and global information. Based on the fact that dichromats perceive colors projected onto a 2D plane, we first measure the color differences on the plane. Then, we formulate an optimization problem to minimize the perceived color differences subject to constraints on the projected plane. Finally, by solving the optimization problem, we obtain the optimal plane and perform color conversion. Simulation results show that the proposed algorithm outperforms the state-of-the-art algorithms in preserving both local details and naturalness of images.

A novel approach for image demosaicking based on adaptive lattice-aware filter (ALF) and global refinement unit (GRU) is proposed in this work. We generate ALFs dynamically, which are adaptive to positions of pixels within color lattices in a color filter array, to obtain a locally demosaicked image. We then refine the locally demosaicked image using GRU to exploit global information, as well as local information. To extend the receptive fields efficiently, we adopt dilated convolutions in GRU. Experimental results demonstrate that the proposed algorithm provides the state-of-the-art performances in standard demosaicking datasets.

OBJECTIVE. The purposes of this study were to assess the performance of a 3D convolutional neural network (CNN) for automatic segmentation of prostates on MR images and to compare the volume estimates from the 3D CNN with those of the ellipsoid formula. MATERIALS AND METHODS. The study included 330 MR image sets that were divided into 260 training sets and 70 test sets for automated segmentation of the entire prostate. Among these, 162 training sets and 50 test sets were used for transition zone segmentation. Assisted by manual segmentation by two radiologists, the following values were obtained: estimates of ground-truth volume (VGT), software-derived volume (VSW), mean of VGT and VSW (VAV), and automatically generated volume from the 3D CNN (VNET). These values were compared with the volume calculated with the ellipsoid formula (VEL). RESULTS. The Dice similarity coefficient for the entire prostate was 87.12% and for the transition zone was 76.48%. There was no significant difference between VNET and VAV (p = 0.689) in the test sets of the entire prostate, whereas a significant difference was found between VEL and VAV (p < 0.001). No significant difference was found among the volume estimates in the test sets of the transition zone. Overall intraclass correlation coefficients between the volume estimates were excellent (0.887-0.995). In the test sets of entire prostate, the mean error between VGT and VNET (2.5) was smaller than that between VGT and VEL (3.3). CONCLUSION. The fully automated network studied provides reliable volume estimates of the entire prostate compared with those obtained with the ellipsoid formula. Fast and accurate volume measurement by use of the 3D CNN may help clinicians evaluate prostate disease.

A novel algorithm to detect semantic lines is proposed in this paper. We develop three networks: detection network with mirror attention (D-Net) and comparative ranking and matching networks (R-Net and M-Net). D-Net extracts semantic lines by exploiting rich contextual information. To this end, we design the mirror attention module. Then, through pairwise comparisons of extracted semantic lines, we iteratively select the most semantic line and remove redundant ones overlapping with the selected one. For the pairwise comparisons, we develop R-Net and M-Net in the Siamese architecture. Experiments demonstrate that the proposed algorithm outperforms the conventional semantic line detector significantly. Moreover, we apply the proposed algorithm to detect two important kinds of semantic lines successfully: dominant parallel lines and reflection symmetry axes. Our codes are available at https://github.com/dongkwonjin/Semantic-Line-DRM.

Video interpolation increases the temporal resolution of a video sequence by synthesizing intermediate frames between two consecutive frames. We propose a novel deep-learning-based video interpolation algorithm based on bilateral motion estimation. First, we develop the bilateral motion network with the bilateral cost volume to estimate bilateral motions accurately. Then, we approximate bi-directional motions to predict a different kind of bilateral motions. We then warp the two input frames using the estimated bilateral motions. Next, we develop the dynamic filter generation network to yield dynamic blending filters. Finally, we combine the warped frames using the dynamic blending filters to generate intermediate frames. Experimental results show that the proposed algorithm outperforms the state-of-the-art video interpolation algorithms on several benchmark datasets. The source codes and pre-trained models are available at https://github.com/JunHeum/BMBC.

An interactive video object segmentation algorithm, which takes scribble annotations on query objects as input, is proposed in this paper. We develop a deep neural network, which consists of the annotation network (A-Net) and the transfer network (T-Net). First, given user scribbles on a frame, A-Net yields a segmentation result based on the encoder-decoder architecture. Second, T-Net transfers the segmentation result bidirectionally to the other frames, by employing the global and local transfer modules. The global transfer module conveys the segmentation information in an annotated frame to a target frame, while the local transfer module propagates the segmentation information in a temporally adjacent frame to the target frame. By applying A-Net and T-Net alternately, a user can obtain desired segmentation results with minimal efforts. We train the entire network in two stages, by emulating user scribbles and employing an auxiliary loss. Experimental results demonstrate that the proposed interactive video object segmentation algorithm outperforms the state-of-the-art conventional algorithms. Codes and models are available at https://github.com/yuk6heo/IVOS-ATNet.

Image enhancement is an inherently subjective process since people have diverse preferences for image aesthetics. However, most enhancement techniques pay less attention to the personalization issue despite its importance. In this paper, we propose the first deep learning approach to personalized image enhancement, which can enhance new images for a new user, by asking him or her to select about 10–20 preferred images from a random set of images. First, we represent various users’ preferences for enhancement as feature vectors in an embedding space, called preference vectors. We construct the embedding space based on metric learning. Then, we develop the personalized image enhancement network (PieNet) to enhance images adaptively using each user’s preference vector. Experimental results demonstrate that the proposed algorithm is capable of achieving personalization successfully, as well as outperforming conventional general image enhancement algorithms significantly. The source codes and trained models are available at https://github.com/hukim1124/PieNet.

A novel approach for paired and unpaired image enhancement is proposed in this work. First, we develop global enhancement network (GEN) and local enhancement network (LEN), which can faithfully enhance images. The proposed GEN performs the channel-wise intensity transforms that can be trained easier than the pixel-wise prediction. The proposed LEN refines GEN results based on spatial filtering. Second, we propose different training schemes for paired learning and unpaired learning to train GEN and LEN. Especially, we propose a two-stage training scheme based on generative adversarial networks for unpaired learning. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-arts in paired and unpaired image enhancement. Notably, the proposed unpaired image enhancement algorithm provides better results than recent state-of-the-art paired image enhancement algorithms. The source codes and trained models are available at https://github.com/hukim1124/GleNet.

Despite the practical importance of photographic composition for improving or assessing the aesthetical quality of photographs, only a few simple composition rules have been considered for its classification. In this work, we propose novel techniques to classify photographic composition rules of outdoor scenes and detect dominant geometric elements, called composition elements, for each composition class. Specifically, we first categorize composition rules of outdoor photographs into nine classes: RoT, center, horizontal, symmetric, diagonal, curved, vertical, triangle, and pattern. Then, we develop a photographic composition classification algorithm using a convolutional neural network (CNN). To train the CNN, we construct a photographic composition database, which is publicly available. Finally, for each composition class, we propose an effective scheme to locate composition elements, i.e., bounding boxes for main subjects, leading lines, axes of symmetry, triangles, and sky regions. Extensive experimental results demonstrate that the proposed algorithm classifies composition classes reliably and detects composition elements accurately.

A tracking-by-segmentation algorithm, which tracks and segments a target object in a video sequence, is proposed in this paper. In the first frame, we segment out the target object in a user-annotated bounding box. Then, we divide subsequent frames into superpixels. We develop a superpixel-wise neural network for tracking-by-segmentation, called TBSNet, which extracts multi-level convolutional features of each superpixel and yields the foreground probability of the superpixel as the output. We train TBSNet in two stages. First, we perform offline training to enable TBSNet to discriminate general objects from the background. Second, during the tracking, we fine-tune TBSNet to distinguish the target object from non-targets and adapt to color change and shape variation of the target object. Finally, we perform conditional random field optimization to improve the segmentation quality further. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art trackers on four challenging data sets.

We propose four efficient feature extractors based on convolutional neural networks for stereo matching cost computation. Two of them generate multiscale features with diverse receptive field sizes. These multiscale features are used to compute the corresponding multiscale matching costs. We then determine an optimal cost by combining the multiscale costs using edge information. On the other hand, the other two feature extractors produce uni-scale features by combining multiscale features directly through fully connected layers. Finally, after obtaining matching costs using one of the four extractors, we determine optimal disparities based on the cross-based cost aggregation and the semiglobal matching. Extensive experiments on the Middlebury stereo data sets demonstrate the effectiveness and efficiency of the proposed algorithm. Specifically, the proposed algorithm provides competitive matching performance with the state of the arts, while demanding lower computational complexity.

A deep learning framework for 3D point cloud processing is proposed in this work. In a point cloud, local neighborhoods have various shapes, and the semantic meaning of each point is determined within the local shape context. Thus, we propose shape-adaptive filters (SAFs), which are dynamically generated from the distributions of local points. The proposed SAFs can extract robust features against noise or outliers, by employing local shape contexts to suppress them. Also, we develop the SAF-Nets for classification and segmentation using multiple SAF layers. Extensive experimental results demonstrate that the proposed SAF-Nets significantly outperform the state-of-the-art conventional algorithms on several benchmark datasets. Moreover, it is shown that SAFs can improve scene flow estimation performance as well.

We propose a novel subpixel rendering algorithm for diamond-shaped PenTile displays, which reduces color distortions while improving apparent resolutions. We develop two types of subpixel rendering filters: main filter and color distortion reduction (CDR) filters. To derive the filters, we formulate a quadratic program to minimize the difference between an original input image and a virtual image that the human visual system perceives. By imposing two constraints for filter size and coefficients, we obtain the main filter, which has a suitable size and is normalized. Then, we design the CDR filters based on the analysis of various patch patterns for image areas. We define the patch patterns to classify local areas with possible color distortions. By imposing additional constraints according to the patch patterns, we derive the CDR filters. Lastly, by matching local areas in the input image into the pre-defined patch patterns, we render the image using the main filter and the CDR filters, which are applied adaptively to the local areas. Experimental results demonstrate that the proposed subpixel rendering algorithm improves apparent resolutions and suppresses color distortions effectively, thereby outperforming conventional algorithms.

The image quality of an in-vehicle black box camera is often degraded by the reflections of internal objects, dirt, and dust on the windshield. In this paper, we propose a novel algorithm that simultaneously removes the reflections and small dirt artifacts from in-vehicle black box videos under fast forward camera motion. The algorithm exploits the spatiotemporal coherence of the reflection and dirt, which remain stationary relative to the fast-moving background. Unlike previous algorithms, the algorithm first separates stationary reflection and then restores the background scene. To this end, we propose an average image prior, thereby imposing spatiotemporal coherence. The separation model is a two-layer model composed of stationary and background layers, where different gradient sparsity distributions are utilized in a region-based manner. Motion compensation in postprocessing is proposed to alleviate layer jitter due to vehicle vibrations. In evaluation experiments, the proposed algorithm successfully extracts the stationary layer from several real and synthetic black box videos.

We develop an efficient structure-texture-noise decomposition method.We propose the image enhancement algorithm tailored for noisy low-light images.The proposed algorithm provides robust performance under various conditions. A noisy low-light image enhancement algorithm based on structure-texture-noise (STN) decomposition is proposed in this work. We split an input image into structure, texture, and noise components, and enhance the structure and texture components separately. More specifically, we first enhance the contrast of the structure image, by extending a 2D-histogram-based image enhancement scheme based on the characteristics of low-light images. Then, we reconstruct the texture image by retrieving residual texture components from the noise image and enhance it by exploiting the perceptual response of the human visual system (HVS). Experimental results on both synthetic and real-world images demonstrate that the proposed STN algorithm sharpens the texture and enhances the contrast more effectively than conventional algorithms, while providing robust performance under various noise and illumination conditions.

We propose a deep learning-based frame rate upconversion algorithm using bilateral motion estimation. We first estimate bilateral motion fields by employing a convolutional neural network. Also, we approximate intermediate bi-directional motion fields, assuming linear motions between successive frames. Finally, we develop the synthesis network to produce an intermediate frame by merging the warped frames, which are obtained using the two kinds of motion fields. Experimental results demonstrate that the proposed algorithm generates high-quality intermediate frames on challenging sequences with large motions and occlusion, and outperforms state-of-the-art conventional algorithms.

Ahstract-A robust change detection algorithm for high resolution satellite images, which are not perfectly registered, is proposed in this work. To achieve this goal, a change detection technique for registered images and an image registration technique are employed in a cooperative way. Specifically, we use not only hand-crafted features but also change detection results to match keypoints extracted from two images. We then align the images using the matching pairs of keypoints. Finally, we obtain a change map from the aligned images. These steps of image registration and change detection are alternately iterated until the convergence. Experimental results demonstrate that proposed algorithm outperforms the conventional change detection technique significantly, when there are geometric distortions between temporal satellite images.

We propose a unified approach to three tasks of aesthetic score regression, binary aesthetic classification, and personalized aesthetics. First, we develop a comparator to estimate the ratio of aesthetic scores for two images. Then, we construct a pairwise comparison matrix for multiple reference images and an input image, and predict the aesthetic score of the input via the eigenvalue decomposition of the matrix. By varying the reference images, the proposed algorithm can be used for binary aesthetic classification and personalized aesthetics, as well as generic score regression. Experimental results demonstrate that the proposed unified algorithm provides the state-of-the-art performances in all three tasks of image aesthetics.

A novel algorithm to estimate instance-level future motion in a single image is proposed in this paper. We first represent the future motion of an instance with its direction, speed, and action classes. Then, we develop a deep neural network that exploits different levels of semantic information to perform the future motion estimation. For effective future motion classification, we adopt ordinal regression. Especially, we develop the cyclic ordinal regression scheme using binary classifiers. Experiments demonstrate that the proposed algorithm provides reliable performance and thus can be used effectively for vision applications, including single and multi object tracking. Furthermore, we release the future motion (FM) dataset, collected from diverse sources and annotated manually, as a benchmark for single-image future motion estimation.

An interactive image segmentation algorithm, which accepts user-annotations about a target object and the background, is proposed in this work. We convert user-annotations into interaction maps by measuring distances of each pixel to the annotated locations. Then, we perform the forward pass in a convolutional neural network, which outputs an initial segmentation map. However, the user-annotated locations can be mislabeled in the initial result. Therefore, we develop the backpropagating refinement scheme (BRS), which corrects the mislabeled pixels. Experimental results demonstrate that the proposed algorithm outperforms the conventional algorithms on four challenging datasets. Furthermore, we demonstrate the generality and applicability of BRS in other computer vision tasks, by transforming existing convolutional neural networks into user-interactive ones.

We propose a novel algorithm for monocular depth estimation using relative depth maps. First, using a convolutional neural network, we estimate relative depths between pairs of regions, as well as ordinary depths, at various scales. Second, we restore relative depth maps from selectively estimated data based on the rank-1 property of pairwise comparison matrices. Third, we decompose ordinary and relative depth maps into components and recombine them optimally to reconstruct a final depth map. Experimental results show that the proposed algorithm provides the state-of-art depth estimation performance.

We propose a compressed domain video saliency detection algorithm.The proposed algorithm extracts features from the partially decoded data.The proposed algorithm performs the detection in real-time with good performance. A compressed domain video saliency detection algorithm, which employs global and local spatiotemporal (GLST) features, is proposed in this work. We first conduct partial decoding of a compressed video bitstream to obtain motion vectors and DCT coefficients, from which GLST features are extracted. More specifically, we extract the spatial features of rarity, compactness, and center prior from DC coefficients by investigating the global color distribution in a frame. We also extract the spatial feature of texture contrast from AC coefficients to identify regions, whose local textures are distinct from those of neighboring regions. Moreover, we use the temporal features of motion intensity and motion contrast to detect visually important motions. Then, we generate spatial and temporal saliency maps, respectively, by linearly combining the spatial features and the temporal features. Finally, we fuse the two saliency maps into a spatiotemporal saliency map adaptively by comparing the robustness of the spatial features with that of the temporal features. Experimental results demonstrate that the proposed algorithm provides excellent saliency detection performance, while requiring low complexity and thus performing the detection in real-time.

In this paper, we propose a novel change detection algorithm for high resolution satellite images using convolutional neural networks (CNNs), which does not require any preprocessing, such as ortho-rectification and classification. When analyzing multi-temporal satellite images, it is crucial to distinguish viewpoint or color variations of an identical object from actual changes. Especially in urban areas, the registration difficulty due to high-rise buildings makes conventional change detection techniques unreliable, if they are not combined with preprocessing schemes using digital surface models or multi-spectral information. We design three encoder-decoder-structured CNNs, which yield change maps from an input pair of RGB satellite images. For the supervised learning of these CNNs, we construct a large fully-labeled dataset using Google Earth images taken in different years and seasons. Experimental results demonstrate that the trained CNNs detect actual changes successfully, even though image pairs are neither perfectly registered nor color-corrected. Furthermore, an ensemble of the three CNNs provides excellent performance, outperforming each individual CNN.

Image aesthetic assessment is important for finding well taken and appealing photographs but is challenging due to the ambiguity and subjectivity of aesthetic criteria. We develop the pairwise aesthetic comparison network (PAC-Net), which consists of two parts: aesthetic feature extraction and pairwise feature comparison. To alleviate the ambiguity and subjectivity, we train PAC-Net to learn the relative aesthetic ranks of two images by employing a novel loss function, called aesthetic-adaptive cross entropy loss. Then, we develop simple schemes for using PAC-Net in the tasks of aesthetic ranking and aesthetic classification, respectively. Experimental results demonstrate that PAC-Net achieves the state-of-the-art performances in both the ranking and classification applications.

A reliable algorithm for rendering depth of field (DoF) effects using estimated depth maps, obtained through stereo matching, is proposed in this paper. The proposed algorithm generates blurring to simulate images spontaneously seen by human vision systems. We develop two types of windows : circle of confusion (CoC) blurring window and peripheral blurring window. First, the CoC blurring window is determined by comparing the depth values of a gazing point and each sample point. Second, the peripheral blurring window is obtained by calculating the distance between the gazing and sample points. Then, we combine the two windows to make the total blurring window. Finally, through a masking process, we modulate the total blurring window to provide a more natural DoF. Experimental results demonstrate that the proposed algorithm provides realistic blurring, by preserving edges clearly as well as blurring far points from the gazing point effectively.

A novel algorithm to segment out objects in a video sequence is proposed in this work. First, we extract object instances in each frame. Then, we select a visually important object instance in each frame to construct the salient object track through the sequence. This can be formulated as finding the maximal weight clique in a complete k-partite graph, which is NP hard. Therefore, we develop the sequential clique optimization (SCO) technique to efficiently determine the cliques corresponding to salient object tracks. We convert these tracks into video object segmentation results. Experimental results show that the proposed algorithm significantly outperforms the state-of-the-art video object segmentation and video salient object detection algorithms on recent benchmark datasets.

We propose a video stabilization algorithm, which extracts a guaranteed number of reliable feature trajectories for robust mesh grid warping. We first estimate feature trajectories through a video sequence and transform the feature positions into rolling-free smoothed positions. When the number of the estimated trajectories is insufficient, we generate virtual trajectories by augmenting incomplete trajectories using a low-rank matrix completion scheme. Next, we detect feature points on a large moving object and exclude them so as to stabilize camera movements, rather than object movements. With the selected feature points, we set a mesh grid on each frame and warp each grid cell by moving the original feature positions to the smoothed ones. For robust warping, we formulate a cost function based on the reliability weights of each feature point and each grid cell. The cost function consists of a data term, a structure-preserving term, and a regularization term. By minimizing the cost function, we determine the robust mesh grid warping and achieve the stabilization. Experimental results demonstrate that the proposed algorithm reconstructs videos more stably than the conventional algorithms.

A novel algorithm to remove rain or snow streaks from a video sequence using temporal correlation and low-rank matrix completion is proposed in this paper. Based on the observation that rain streaks are too small and move too fast to affect the optical flow estimation between consecutive frames, we obtain an initial rain map by subtracting temporally warped frames from a current frame. Then, we decompose the initial rain map into basis vectors based on the sparse representation, and classify those basis vectors into rain streak ones and outliers with a support vector machine. We then refine the rain map by excluding the outliers. Finally, we remove the detected rain streaks by employing a low-rank matrix completion technique. Furthermore, we extend the proposed algorithm to stereo video deraining. Experimental results demonstrate that the proposed algorithm detects and removes rain or snow streaks efficiently, outperforming conventional algorithms.

A novel saliency detection algorithm for video sequences based on the random walk with restart (RWR) is proposed in this paper. We adopt RWR to detect spatially and temporally salient regions. More specifically, we first find a temporal saliency distribution using the features of motion distinctiveness, temporal consistency, and abrupt change. Among them, the motion distinctiveness is derived by comparing the motion profiles of image patches. Then, we employ the temporal saliency distribution as a restarting distribution of the random walker. In addition, we design the transition probability matrix for the walker using the spatial features of intensity, color, and compactness. Finally, we estimate the spatiotemporal saliency distribution by finding the steady-state distribution of the walker. The proposed algorithm detects foreground salient objects faithfully, while suppressing cluttered backgrounds effectively, by incorporating the spatial transition matrix and the temporal restarting distribution systematically. Experimental results on various video sequences demonstrate that the proposed algorithm outperforms conventional saliency detection algorithms qualitatively and quantitatively.

%&'()* + ,-./012 3. 45 6789:; 0 ?- DE3. F.-G @A H IJ H 67H K$ LM 3. N IO P QR0STH K$ UVW 3. XY0ST12Z* AP [I ,-./[\4$K0] 3.AbstractIn this paper, we propose a precise frame-level near-duplicate video matching algorithm. First, a binary frame descriptor for near-duplicate video matching is proposed. The binary frame descriptor divides a frame into patches and represent the relations between patches in bits. Seconds, we formulate a cost function for the matching, composed of matching costs and compensatory costs. Then, we roughly determine initial matchings and refine the matchings iteratively to minimize the cost function. Experimental results demonstrate that the proposed algorithm provides efficient performance for frame-level near duplicate video matching.Keyword : Near-duplicate video, frame-level video matching, and binary frame descriptor.

We propose a fast quality metric for depth maps, called fast depth quality metric (FDQM), which efficiently evaluates the impacts of depth map errors on the qualities of synthesized intermediate views in multiview video plus depth applications. In other words, the proposed FDQM assesses view synthesis distortions in the depth map domain, without performing the actual view synthesis. First, we estimate the distortions at pixel positions, which are specified by reference disparities and distorted disparities, respectively. Then, we integrate those pixel-wise distortions into an FDQM score by employing a spatial pooling scheme, which considers occlusion effects and the characteristics of human visual attention. As a benchmark of depth map quality assessment, we perform a subjective evaluation test for intermediate views, which are synthesized from compressed depth maps at various bitrates. We compare the subjective results with objective metric scores. Experimental results demonstrate that the proposed FDQM yields highly correlated scores to the subjective ones. Moreover, FDQM requires at least 10 times less computations than conventional quality metrics, since it does not perform the actual view synthesis.

In this paper, the concept of cubic set to subalgebras, ideals and closed ideals of B -algebras are introduced. Relations among cubic subalgebras with cubic ideals and cubic closed ideals of B -algebras investigated. The homomorphic image and inverse image of cubic subalgebras, ideals are studied and some related properties are investigated. Also, the product of cubic B -algebras are investigated.

An adaptive range image coding algorithm for the geometry compression of large-scale 3D point clouds (LS3DPCs) is proposed in this work. A terrestrial laser scanner generates an LS3DPC by measuring the radial distances of objects in a real world scene, which can be mapped into a range image. In general, the range image exhibits different characteristics from an ordinary luminance or color image, and thus the conventional image coding techniques are not suitable for the range image coding. We propose a hybrid range image coding algorithm, which predicts the radial distance of each pixel using previously encoded neighbors adaptively in one of three coordinate domains: range image domain, height image domain, and 3D domain. We first partition an input range image into blocks of various sizes. For each block, we apply multiple prediction modes in the three domains and compute their rate-distortion costs. Then, we perform the prediction of all pixels using the optimal mode and encode the resulting prediction residuals. Experimental results show that the proposed algorithm provides significantly better compression performance on various range images than the conventional image or video coding techniques.