Search

Accurate retinal fluid segmentation on Optical Coherence Tomography (OCT) images plays an important role in diagnosing and treating various eye diseases. The art deep models have shown promising performance on OCT image segmentation given pixel-wise annotated training data. However, the learned model will achieve poor performance on OCT images that are obtained from different devices (domains) due to the domain shift issue. This problem largely limits the real-world application of OCT image segmentation since the types of devices usually are different in each hospital. In this paper, we study the task of cross-domain OCT fluid segmentation, where we are given a labeled dataset of the source device (domain) and an unlabeled dataset of the target device (domain). The goal is to learn a model that can perform well on the target domain. To solve this problem, in this paper, we propose a novel Structure-guided Cross-Attention Network (SCAN), which leverages the retinal layer structure to facilitate domain alignment. Our SCAN is inspired by the fact that the retinal layer structure is robust to domains and can reflect regions that are important to fluid segmentation. In light of this, we build our SCAN in a multi-task manner by jointly learning the retinal structure prediction and fluid segmentation. To exploit the mutual benefit between layer structure and fluid segmentation, we further introduce a cross-attention module to measure the correlation between the layer-specific feature and the fluid-specific feature encouraging the model to concentrate on highly relative regions during domain alignment. Moreover, an adaptation difficulty map is evaluated based on the retinal structure predictions from different domains, which enforces the model focus on hard regions during structure-aware adversarial learning. Extensive experiments on the three domains of the RETOUCH dataset demonstrate the effectiveness of the proposed method and show that our approach produces state-of-the-art performance on cross-domain OCT fluid segmentation. [ABSTRACT FROM AUTHOR]

Milano Retinex is a family of spatial color algorithms inspired by Retinex and mainly devoted to the image enhancement. In the so-called point-based sampling Milano Retinex algorithms, this task is accomplished by processing the color of each image pixel based on a set of colors sampled in its surround. This paper presents STAR, a segmentation based approximation of the point-based sampling Milano Retinex approaches: it replaces the pixel-wise image sampling by a novel, computationally efficient procedure that detects once for all the color and spatial information relevant to image enhancement from clusters of pixels output by a segmentation. The experiments reported here show that STAR performs similarly to previous point-based sampling Milano Retinex approaches and that STAR enhancement improves the accuracy of the well-known algorithm scale-invariant feature transform on the description and matching of photographs captured under difficult light conditions. [ABSTRACT FROM AUTHOR]

Video object segmentation (VOS) is important for various computer vision problems, and handling it with minimal human supervision is highly desired for the large-scale applications. To bring down the supervision, existing approaches largely follow a data mining perspective by assuming the availability of multiple videos sharing the same object categories. It, however, would be problematic for the tasks that consume a single video. To address this problem, this paper proposes a novel approach that explores weakly labeled images to solve video object segmentation. Given a video labeled with a target category, images labeled with the same category are collected, from which noisy object exemplars are automatically discovered. After that the proposed approach extracts a set of region proposals on various frames and efficiently matches them with massive noisy exemplars in terms of appearance and spatial context. We then jointly select the best proposals across the video by solving a novel submodular problem that combines region voting and global region matching. Finally, the localization results are leveraged as strong supervision to guide pixel-level segmentation. Extensive experiments are conducted on two challenging public databases: Youtube-Objects and DAVIS. The results suggest that the proposed approach improves over previous weakly supervised/unsupervised approaches significantly, showing a performance even comparable with the several approaches supervised by the costly manual segmentations. [ABSTRACT FROM AUTHOR]

Driven by recent vision and graphics applications such as image segmentation and object recognition, computing pixel-accurate saliency values to uniformly highlight foreground objects becomes increasingly important. In this paper, we propose a unified framework called pixelwise image saliency aggregating (PISA) various bottom-up cues and priors. It generates spatially coherent yet detail-preserving, pixel-accurate, and fine-grained saliency, and overcomes the limitations of previous methods, which use homogeneous superpixel based and color only treatment. PISA aggregates multiple saliency cues in a global context, such as complementary color and structure contrast measures, with their spatial priors in the image domain. The saliency confidence is further jointly modeled with a neighborhood consistence constraint into an energy minimization formulation, in which each pixel will be evaluated with multiple hypothetical saliency levels. Instead of using global discrete optimization methods, we employ the cost-volume filtering technique to solve our formulation, assigning the saliency levels smoothly while preserving the edge-aware structure details. In addition, a faster version of PISA is developed using a gradient-driven image subsampling strategy to greatly improve the runtime efficiency while keeping comparable detection accuracy. Extensive experiments on a number of public data sets suggest that PISA convincingly outperforms other state-of-the-art approaches. In addition, with this work, we also create a new data set containing 800 commodity images for evaluating saliency detection. [ABSTRACT FROM PUBLISHER]

A holistic understanding of dynamic scenes is of fundamental importance in real-world computer vision problems such as autonomous driving, augmented reality and spatio-temporal reasoning. In this paper, we propose a new computer vision benchmark: Video Panoptic Segmentation (VPS). To study this important problem, we present two datasets, Cityscapes-VPS and VIPER together with a new evaluation metric, video panoptic quality (VPQ). We also propose VPSNet++, an advanced video panoptic segmentation network, which simultaneously performs classification, detection, segmentation, and tracking of all identities in videos. Specifically, VPSNet++ builds upon a top-down panoptic segmentation network by adding pixel-level feature fusion head and object-level association head. The former temporally augments the pixel features while the latter performs object tracking. Furthermore, we propose panoptic boundary learning as an auxiliary task, and instance discrimination learning which learns spatio-temporally clustered pixel embedding for individual thing or stuff regions, i.e., exactly the objective of the video panoptic segmentation problem. Our VPSNet++ significantly outperforms the default VPSNet, i.e., FuseTrack baseline, and achieves state-of-the-art results on both Cityscapes-VPS and VIPER datasets. The datasets, metric, and models are publicly available at https://github.com/mcahny/vps. [ABSTRACT FROM AUTHOR]

Self-attention is widely explored to model long-range dependencies in semantic segmentation. However, this operation computes pair-wise relationships between the query point and all other points, leading to prohibitive complexity. In this paper, we propose an efficient Sampling-based Attention Network which combines a novel sample method with an attention mechanism for semantic segmentation. Specifically, we design a Stochastic Sampling-based Attention Module (SSAM) to capture the relationships between the query point and a stochastic sampled representative subset from a global perspective, where the sampled subset is selected by a Stochastic Sampling Module. Compared to self-attention, our SSAM achieves comparable segmentation performance while significantly reducing computational redundancy. In addition, with the observation that not all pixels are interested in the contextual information, we design a Deterministic Sampling-based Attention Module (DSAM) to sample features from a local region for obtaining the detailed information. Extensive experiments demonstrate that our proposed method can compete or perform favorably against the state-of-the-art methods on the Cityscapes, ADE20K, COCO Stuff, and PASCAL Context datasets. [ABSTRACT FROM AUTHOR]

Weakly supervised semantic segmentation (WSSS) based on bounding box annotations has attracted considerable recent attention and has achieved promising performance. However, most of existing methods focus on generation of high-quality pseudo labels for segmented objects using box indicators, but they fail to fully explore and exploit prior from bounding box annotations, which limits performance of WSSS methods, especially for fine parts and boundaries. To overcome above issues, this paper proposes a novel Pixel-as-Instance Prior (PIP) for WSSS methods by delving deeper into pixel prior from bounding box annotations. Specifically, the proposed PIP is built on two important observations on pixels around bounding boxes. First, since objects are usually irregularity and tightly close to bounding boxes (dubbed irregular-filling prior), so each row or column of bounding boxes basically have at least one pixel belonging to foreground objects and background, respectively. Second, pixels near the bounding boxes tend to be highly ambiguous and more difficult to classify (dubbed label-ambiguity prior). To implement our PIP, a constrained loss alike multiple instance learning (MIL) and a labeling-balance loss are developed to jointly train WSSS models, which regards each pixel as a weighted positive or negative instance while considering more effective prior (i.e., irregular-filling and label-ambiguity priors) from bounding box annotations in an efficient way. Note that our PIP can be flexibly integrated with various WSSS methods, while clearly improving their performance with negligible computational overload in training stage. The experiments are conducted on most widely used PASCAL VOC 2012 and Cityscapes benchmarks, and the results show that our PIP has a good ability to improve performance of various WSSS methods, while achieving very competitive results. [ABSTRACT FROM AUTHOR]

This paper addresses semi-supervised semantic segmentation by exploiting a small set of images with pixel-level annotations (strong supervisions) and a large set of images with only image-level annotations (weak supervisions). Most existing approaches aim to generate accurate pixel-level labels from weak supervisions. However, we observe that those generated labels still inevitably contain noisy labels. Motivated by this observation, we present a novel perspective and formulate this task as a problem of learning with pixel-level label noise. Existing noisy label methods, nevertheless, mainly aim at image-level tasks, which can not capture the relationship between neighboring labels in one image. Therefore, we propose a graph-based label noise detection and correction framework to deal with pixel-level noisy labels. In particular, for the generated pixel-level noisy labels from weak supervisions by Class Activation Map (CAM), we train a clean segmentation model with strong supervisions to detect the clean labels from these noisy labels according to the cross-entropy loss. Then, we adopt a superpixel-based graph to represent the relations of spatial adjacency and semantic similarity between pixels in one image. Finally we correct the noisy labels using a Graph Attention Network (GAT) supervised by detected clean labels. We comprehensively conduct experiments on PASCAL VOC 2012, PASCAL-Context, MS-COCO and Cityscapes datasets. The experimental results show that our proposed semi-supervised method achieves the state-of-the-art performances and even outperforms the fully-supervised models on PASCAL VOC 2012 and MS-COCO datasets in some cases. [ABSTRACT FROM AUTHOR]

This paper pushes the envelope on decomposing camouflaged regions in an image into meaningful components, namely, camouflaged instances. To promote the new task of camouflaged instance segmentation of in-the-wild images, we introduce a dataset, dubbed CAMO++, that extends our preliminary CAMO dataset (camouflaged object segmentation) in terms of quantity and diversity. The new dataset substantially increases the number of images with hierarchical pixel-wise ground truths. We also provide a benchmark suite for the task of camouflaged instance segmentation. In particular, we present an extensive evaluation of state-of-the-art instance segmentation methods on our newly constructed CAMO++ dataset in various scenarios. We also present a camouflage fusion learning (CFL) framework for camouflaged instance segmentation to further improve the performance of state-of-the-art methods. The dataset, model, evaluation suite, and benchmark will be made publicly available on our project page. [ABSTRACT FROM AUTHOR]

Existing color sampling-based alpha matting methods use the compositing equation to estimate alpha at a pixel from the pairs of foreground ( F ) and background ( B ) samples. The quality of the matte depends on the selected ( $F,B$ ) pairs. In this paper, the matting problem is reinterpreted as a sparse coding of pixel features, wherein the sum of the codes gives the estimate of the alpha matte from a set of unpaired F and B samples. A non-parametric probabilistic segmentation provides a certainty measure on the pixel belonging to foreground or background, based on which a dictionary is formed for use in sparse coding. By removing the restriction to conform to ( $F,B$ ) pairs, this method allows for better alpha estimation from multiple F and B samples. The same framework is extended to videos, where the requirement of temporal coherence is handled effectively. Here, the dictionary is formed by samples from multiple frames. A multi-frame graph model, as opposed to a single image as for image matting, is proposed that can be solved efficiently in closed form. Quantitative and qualitative evaluations on a benchmark dataset are provided to show that the proposed method outperforms the current stateoftheart in image and video matting [ABSTRACT FROM PUBLISHER]

Different from conventional instance segmentation, salient instance segmentation (SIS) faces two difficulties. The first is that it involves segmenting salient instances only while ignoring background, and the second is that it targets generic object instances without pre-defined object categories. In this paper, based on the state-of-the-art Mask R-CNN model, we propose to leverage complementary saliency and contour information to handle these two challenges. We first improve Mask R-CNN by introducing an interleaved execution strategy and proposing a novel mask head network to incorporate global context within each RoI. Then we add two branches to Mask R-CNN for saliency and contour detection, respectively. We fuse the Mask R-CNN features with the saliency and contour features, where the former supply pixel-wise saliency information to help with identifying salient regions and the latter provide a generic object contour prior to help detect and segment generic objects. We also propose a novel multiscale global attention model to generate attentive global features from multiscale representative features for feature fusion. Experimental results demonstrate that all our proposed model components can improve SIS performance. Finally, our overall model outperforms state-of-the-art SIS methods and Mask R-CNN by more than 6% and 3%, respectively. By using additional multitask training data, we can further improve the model performance on the ILSO dataset. [ABSTRACT FROM AUTHOR]

Few-shot semantic segmentation remains an open problem because limited support (training) images are insufficient to represent the diverse semantics within target categories. Conventional methods typically model a target category solely using information from the support image(s), resulting in incomplete semantic activation. In this paper, we propose a novel few-shot segmentation approach, termed harmonic feature activation (HFA), with the aim to implement dense support-to-query semantic transform by incorporating the features of both query and support images. HFA is formulated as a bilinear model, which takes charge of the pixel-wise dense correlation (bilinear feature activation) between query and support images in a systematic way. HFA incorporates a low-rank decomposition procedure, which speeds up bilinear feature activation with negligible performance cost. In addition, a semantic diffusion procedure is fused with HFA, which further improves the global harmony and local consistency of the feature activation. Extensive experiments on commonly used datasets (PASCAL VOC and MS COCO) show that HFA improves the state-of-the-arts with significant margins. Code is available at https://github.com/Bibikiller/HFA. [ABSTRACT FROM AUTHOR]

In this paper, we present a superpixel segmentation algorithm called linear spectral clustering (LSC), which is capable of producing superpixels with both high boundary adherence and visual compactness for natural images with low computational costs. In LSC, a normalized cuts-based formulation of image segmentation is adopted using a distance metric that measures both the color similarity and the space proximity between image pixels. However, rather than directly using the traditional eigen-based algorithm, we approximate the similarity metric through a deliberately designed kernel function such that pixel values can be explicitly mapped to a high-dimensional feature space. We then apply the conclusion that by appropriately weighting each point in this feature space, the objective functions of the weighted K-means and the normalized cuts share the same optimum points. Consequently, it is possible to optimize the cost function of the normalized cuts by iteratively applying simple K-means clustering in the proposed feature space. LSC possesses linear computational complexity and high memory efficiency, since it avoids both the decomposition of the affinity matrix and the generation of the large kernel matrix. By utilizing the underlying mathematical equivalence between the two types of seemingly different methods, LSC successfully preserves global image structures through efficient local operations. Experimental results show that LSC performs as well as or even better than the state-of-the-art superpixel segmentation algorithms in terms of several commonly used evaluation metrics in image segmentation. The applicability of LSC is further demonstrated in two related computer vision tasks. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new framework for hierarchical image segmentation based on iterative contraction and merging. In the proposed framework, we treat the hierarchical image segmentation problem as a sequel of optimization problems, with each optimization process being realized by a contraction-and-merging process to identify and merge the most similar data pairs at the current resolution. At the beginning, we perform pixel-based contraction and merging to quickly combine image pixels into initial region-elements with visually indistinguishable intra-region color difference. After that, we iteratively perform region-based contraction and merging to group adjacent regions into larger ones to progressively form a segmentation dendrogram for hierarchical segmentation. Comparing with the state-of-the-art techniques, the proposed algorithm can not only produce high-quality segmentation results in a more efficient way, but also keep a lot of boundary details in the segmentation results. [ABSTRACT FROM PUBLISHER]

Superpixels are widely used in computer vision applications, as they conserve the running costs of subsequent processing while preserving the original performance. In most of the existing algorithms, the boundary adherence and the compactness of superpixels are necessarily inter-inhibitive because the color/gradient information is balanced against the position constraints, and the set criteria define all pixels indiscriminately. In this paper, we present a two-phase superpixel segmentation method based on the watershed transformation. After designing a new approach for calculating the flooding priority, we propose a new strategy with two distinct criteria for global and local refinement of the boundary pixels. These criteria reduce the compromise between the boundary adherence and compactness. Unlike the indiscriminate standards, our method applies different treatments to pixels in different environments, preserving the color homogeneity in content-rich areas while improving the regularity of the superpixels in content-plain regions. The superior accuracy and computing time of our proposed method are verified in comparison experiments with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation for indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3~5) color images from uncalibrated sparse views, which significantly simplifies data acquisition and broadens applicable scenarios. To achieve promising 3D reconstruction from sparse views with limited overlap, our method first recovers the depth map and semantic information for each view, and then fuses the depth maps into a 3D scene. To this end, we design an iterative deep architecture, named IterNet, to estimate the depth map and semantic segmentation alternately. To obtain accurate alignment between views with limited overlap, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information. We also make available a new indoor synthetic dataset, containing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors, and better 3D reconstruction results over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper presents a new approach to recognizing vanishing-point-constrained building planes from a single image of street view. We first design a novel convolutional neural network (CNN) architecture that generates geometric segmentation of per-pixel orientations from a single street-view image. The network combines two-stream features of general visual cues and surface normals in gated convolution layers, and employs a deeply supervised loss that encapsulates multi-scale convolutional features. Our experiments on a new benchmark with fine-grained plane segmentations of real-world street views show that our network outperforms state-of-the-arts methods of both semantic and geometric segmentation. The pixel-wise segmentation exhibits coarse boundaries and discontinuities. We then propose to rectify the pixel-wise segmentation into perspectively-projected quads based on spatial proximity between the segmentation masks and exterior line segments detected through an image processing. We demonstrate how the results can be utilized to perspectively overlay images and icons on building planes in input photos, and provide visual cues for various applications. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel random walk model, called Dynamic Random Walk (DRW), which adds a new type of dynamic node to the original RW model and reduces redundant calculation by limiting the walk range. To solve the seed-lacking problem of the proposed DRW, we redefine the energy function of the original RW and use the first arrival probability among each node pair to avoid the interference for each partition. Relaxation of our DRW is performed with the help of a greedy strategy and the Weighted Random Walk Entropy(WRWE) that uses the gradient feature to approximate the stationary distribution. The proposed DRW not only can enhance the boundary adherence but also can run with linear time complexity. To extend our DRW for superpixel segmentation, a seed initialization strategy is proposed. It can evenly distribute seeds in both 2D and 3D space and generate superpixels in only one iteration. The experimental results demonstrate that our DRW is faster than existing RW models and better than the state-of-the-art superpixel segmentation algorithms with respect to both efficiency and segmentation effects. [ABSTRACT FROM AUTHOR]

Text instance provides valuable information for the understanding and interpretation of natural scenes. The rich precise high-level semantics embodied in the text could be beneficial for understanding the world around us, and empower a wide range of real-world applications. While most recent visual phrase grounding approaches focus on general objects, this paper explores extracting designated texts and predicting unambiguous scene text segmentation mask, i.e., scene text segmentation from natural language descriptions (referring expressions) like orange text on a little boy in black swinging a bat. The solution of this novel problem enables accurate segmentation of scene text instances from the complex background. In our proposed framework, a unified deep network jointly models visual and linguistic information by encoding both region-level and pixel-level visual features of natural scene images into spatial feature maps, and then decode them into saliency response map of text instances. To conduct quantitative evaluations, we establish a new scene text referring expression segmentation dataset: COCO-CharRef. Experimental results demonstrate the effectiveness of the proposed framework on the text instance segmentation task. By combining image-based visual features with language-based textual explanations, our framework outperforms baselines that are derived from state-of-the-art text localization and natural language object retrieval methods on COCO-CharRef dataset. [ABSTRACT FROM AUTHOR]

Image semantic segmentation tasks and methods based on weakly supervised conditions have been proposed and achieve better and better performance in recent years. However, the purpose of these tasks is mainly to simplify the labeling work. In this paper, we establish a new and more challenging task condition: weaklier supervision with one image level annotation per category, which only provides prior knowledge that humans need to recognize new objects, and aims to achieve pixel-level object semantic understanding. In order to solve this problem, a three-stage semantic segmentation framework is put forward, which realizes image level, pixel level, and object common features learning from coarse to fine grade, and finally obtains semantic segmentation results with accurate and complete object regions. Researches on PASCAL VOC 2012 dataset demonstrates the effectiveness of the proposed method, which makes an obvious improvement compared to baselines. Based on fewer supervised information, the method also provides satisfactory performance compared to weakly supervised learning-based methods with complete image-level annotations. [ABSTRACT FROM AUTHOR]

Object proposal is essential for current state-of-the-art object detection pipelines. However, the existing proposal methods generally fail in producing results with satisfying localization accuracy. The case is even worse for small objects, which, however, are quite common in practice. In this paper, we propose a novel scale-aware pixelwise object proposal network (SPOP-net) to tackle the challenges. The SPOP-net can generate proposals with high recall rate and average best overlap, even for small objects. In particular, in order to improve the localization accuracy, a fully convolutional network is employed which predicts locations of object proposals for each pixel. The produced ensemble of pixelwise object proposals enhances the chance of hitting the object significantly without incurring heavy extra computational cost. To solve the challenge of localizing objects at small scale, two localization networks, which are specialized for localizing objects with different scales are introduced, following the divide-and-conquer philosophy. Location outputs of these two networks are then adaptively combined to generate the final proposals by a large-/small-size weighting network. Extensive evaluations on PASCAL VOC 2007 and COCO 2014 show the SPOP network is superior over the state-of-the-art models. The high-quality proposals from SPOP-net also significantly improve the mean average precision of object detection with Fast-Regions with CNN features framework. Finally, the SPOP-net (trained on PASCAL VOC) shows great generalization performance when testing it on ILSVRC 2013 validation set. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a visual saliency detection algorithm from the perspective of reconstruction error. The image boundaries are first extracted via superpixels as likely cues for background templates, from which dense and sparse appearance models are constructed. First, we compute dense and sparse reconstruction errors on the background templates for each image region. Second, the reconstruction errors are propagated based on the contexts obtained from $K$ -means clustering. Third, the pixel-level reconstruction error is computed by the integration of multi-scale reconstruction errors. Both the pixel-level dense and sparse reconstruction errors are then weighted by image compactness, which could more accurately detect saliency. In addition, we introduce a novel Bayesian integration method to combine saliency maps, which is applied to integrate the two saliency measures based on dense and sparse reconstruction errors. Experimental results show that the proposed algorithm performs favorably against 24 state-of-the-art methods in terms of precision, recall, and F-measure on three public standard salient object detection databases. [ABSTRACT FROM PUBLISHER]

Linear spectral unmixing is a challenging problem in hyperspectral imaging that consists of decomposing an observed pixel into a linear combination of pure spectra (or endmembers) with their corresponding proportions (or abundances). Endmember extraction algorithms can be employed for recovering the spectral signatures while abundances are estimated using an inversion step. Recent works have shown that exploiting spatial dependencies between image pixels can improve spectral unmixing. Markov random fields (MRF) are classically used to model these spatial correlations and partition the image into multiple classes with homogeneous abundances. This paper proposes to define the MRF sites using similarity regions. These regions are built using a self-complementary area filter that stems from the morphological theory. This kind of filter divides the original image into flat zones where the underlying pixels have the same spectral values. Once the MRF has been clearly established, a hierarchical Bayesian algorithm is proposed to estimate the abundances, the class labels, the noise variance, and the corresponding hyperparameters. A hybrid Gibbs sampler is constructed to generate samples according to the corresponding posterior distribution of the unknown parameters and hyperparameters. Simulations conducted on synthetic and real AVIRIS data demonstrate the good performance of the algorithm. [ABSTRACT FROM PUBLISHER]

Segmenting semantically meaningful whole objects from images is a challenging problem, and it becomes especially so without higher level common sense reasoning. In this paper, we present an interactive segmentation framework that integrates image appearance and boundary constraints in a principled way to address this problem. In particular, we assume that small sets of pixels, which are referred to as seed pixels, are labeled as the object and background. The seed pixels are used to estimate the labels of the unlabeled pixels using Dirichlet process multiple-view learning, which leverages 1) multiple-view learning that integrates appearance and boundary constraints and 2) Dirichlet process mixture-based nonlinear classification that simultaneously models image features and discriminates between the object and background classes. With the proposed learning and inference algorithms, our segmentation framework is experimentally shown to produce both quantitatively and qualitatively promising results on a standard dataset of images. In particular, our proposed framework is able to segment whole objects from images given insufficient seeds. [ABSTRACT FROM AUTHOR]

This paper proposes the use of an adaptive locally oriented coordinate frame when calculating an optical flow field. The coordinate frame is aligned with the least curvature direction in a local window about each pixel. This has advantages to both fitting the flow field to the image data and in imposing smoothness constraints between neighboring pixels. In terms of fitting, robustness is obtained to a wider variety of image motions due to the extra invariance provided by the coordinate frame. Smoothness constraints are naturally propagated along image boundaries which often correspond to motion boundaries. In addition, moving objects can be efficiently segmented in the least curvature direction. We show experimentally the benefits of the method and demonstrate robustness to fast rotational motion, such as what often occurs in human motion. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of image segmentation by means of active contours, whose evolution is driven by the gradient flow derived from an energy functional that is based on the Bhattacharyya distance. In particular, given the values of a photo- metric variable (or of a set thereof), which is to be used for classifying the image pixels, the active contours are designed to converge to the shape that results in maximal discrepancy between the empirical distributions of the photometric variable inside and outside of the contours. The above discrepancy is measured by means of the Bhattacharyya distance that proves to be an extremely useful tool for solving the problem at hand. The proposed methodology can be viewed as a generalization of the segmentation methods, in which active contours maximize the difference between a finite number of empirical moments of the ‘inside’ and ‘outside’ distributions. Furthermore, it is shown that the proposed methodology is very versatile and flexible in the sense that it allows one to easily accommodate a diversity of the image features based on which the segmentation should be performed. As an additional contribution, a method for automatically adjusting the smoothness properties of the empirical distributions is proposed. Such a procedure is crucial in situations when the number of data samples (supporting a certain segmentation class) varies considerably in the course of the evolution of the active contour. In this case, the smoothness properties of the empirical distributions have to be properly adjusted to avoid either over-or underestimation artifacts. Finally, a number of relevant segmentation results are demonstrated and some further research directions are discussed. [ABSTRACT FROM AUTHOR]

In this paper, we present a new algorithm for real-time single-object tracking in videos in unconstrained environments. The algorithm comprises two different components that are trained “in one shot” at the first video frame: a detector that makes use of the generalized Hough transform with color and gradient descriptors and a probabilistic segmentation method based on global models for foreground and background color distributions. Both components work at pixel level and are used for tracking in a combined way adapting each other in a co-training manner. Moreover, we propose an adaptive shape model as well as a new probabilistic method for updating the scale of the tracker. Through effective model adaptation and segmentation, the algorithm is able to track objects that undergo rigid and non-rigid deformations and considerable shape and appearance variations. The proposed tracking method has been thoroughly evaluated on challenging benchmarks, and outperforms the state-of-the-art tracking methods designed for the same task. Finally, a very efficient implementation of the proposed models allows for extremely fast tracking. [ABSTRACT FROM PUBLISHER]

This paper proposes a novel saliency detection framework termed as saliency tree. For effective saliency measurement, the original image is first simplified using adaptive color quantization and region segmentation to partition the image into a set of primitive regions. Then, three measures, i.e., global contrast, spatial sparsity, and object prior are integrated with regional similarities to generate the initial regional saliency for each primitive region. Next, a saliency-directed region merging approach with dynamic scale control scheme is proposed to generate the saliency tree, in which each leaf node represents a primitive region and each non-leaf node represents a non-primitive region generated during the region merging process. Finally, by exploiting a regional center-surround scheme based node selection criterion, a systematic saliency tree analysis including salient node selection, regional saliency adjustment and selection is performed to obtain final regional saliency measures and to derive the high-quality pixel-wise saliency map. Extensive experimental results on five datasets with pixel-wise ground truths demonstrate that the proposed saliency tree model consistently outperforms the state-of-the-art saliency models. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new method of segmenting an image into several sets of pixels with similar intensity values called regions. A multiple-region segmentation problem is unstable because the result considerably depends on the number of regions given a priori. Therefore, one of the most important tasks in solving the problem is automatically finding the number of regions. The method we propose is able to find the reasonable number of distinct regions not only for clean images but also for noisy ones. Our method is made up of two procedures. First, we develop the adaptive global maximum clustering. In this procedure, we deal with an image histogram and automatically obtain the number of significant local maxima of the histogram. This number indicates the number of different regions in the image. Second, we derive a simple and fast calculation to segment an image composed of distinct multiple regions. Then, we split an image into multiple regions according to the previous procedure. Finally, we show the efficiency of our method by comparing it with other previous methods. [ABSTRACT FROM AUTHOR]

In this paper, a new method for supervised hyperspectral data classification is proposed. In particular, the notion of stochastic minimum spanning forest (MSF) is introduced. For a given hyperspectral image, a pixelwise classification is first performed. From this classification map, M marker maps are generated by randomly selecting pixels and labeling them as markers for the construction of MSFs. The next step consists in building an MSF from each of the M marker maps. Finally, all the M realizations are aggregated with a maximum vote decision rule in order to build the final classification map. The proposed method is tested on three different data sets of hyperspectral airborne images with different resolutions and contexts. The influences of the number of markers and of the number of realizations M on the results are investigated in experiments. The performance of the proposed method is compared to several classification techniques (both pixelwise and spectral–spatial) using standard quantitative criteria and visual qualitative evaluation. [ABSTRACT FROM AUTHOR]

Connections in image processing are an important notion that describes how pixels can be grouped together according to their spatial relationships and/or their gray-level values. In recent years, several works were devoted to the development of new theories of connections among which hyperconnection (h-connection) is a very promising notion. This paper addresses two major issues of this theory. First, we propose a new axiomatic that ensures that every h-connection generates decompositions that are consistent for image processing and, more precisely, for the design of h-connected filters. Second, we develop a general framework to represent the decomposition of an image into h-connections as a tree that corresponds to the generalization of the connected component tree. Such trees are indeed an efficient and intuitive way to design attribute filters or to perform detection tasks based on qualitative or quantitative attributes. These theoretical developments are applied to a particular fuzzy h-connection, and we test this new framework on several classical applications in image processing, i.e., segmentation, connected filtering, and document image binarization. The experiments confirm the suitability of the proposed approach: It is robust to noise, and it provides an efficient framework to design selective filters. [ABSTRACT FROM PUBLISHER]

The classic centroidal Voronoi tessellation (CVT) model and its generalizations work quite well at extracting uniformly colored objects, but often fail to handle images with distinct color distribution or strong inhomogeneous intensity. To resolve this problem within the CVT methodology, in this paper we incorporate the information of local variation of colors/intensities and the length of boundaries into the energy functional and develop a new model called the Local Variation and Edge-Weighted Centroidal Voronoi Tessellation (LVEWCVT) for image segmentation. Its mathematical formulation and practical implementations are also discussed and given. We test the LVEWCVT method on various type of segments and also compare it with several state-of-art algorithms using extensive segmentation examples, the results demonstrate excellent performance and competence of the proposed method. [ABSTRACT FROM PUBLISHER]

This paper describes a novel framework for detection and suppression of properly shadowed regions for most possible scenarios occurring in real video sequences. Our approach requires no prior knowledge about the scene, nor is it restricted to specific scene structures. Furthermore, the technique can detect both achromatic and chromatic shadows even in the presence of camouflage that occurs when foreground regions are very similar in color to shadowed regions. The method exploits local color constancy properties due to reflectance suppression over shadowed regions. To detect shadowed regions in a scene, the values of the background image are divided by values of the current frame in the RGB color space. We show how this luminance ratio can be used to identify segments with low gradient constancy, which in turn distinguish shadows from foreground. Experimental results on a collection of publicly available datasets illustrate the superior performance of our method compared with the most sophisticated, state-of-the-art shadow detection algorithms. These results show that our approach is robust and accurate over a broad range of shadow types and challenging video conditions. [ABSTRACT FROM AUTHOR]

Anatomical structure tracing on cephalograms is a significant way to obtain cephalometric analysis. Cephalometric analysis is divided in two categories, manual and automatic approaches. The manual approach is limited in accuracy and repeatability due to differences in inter- and intra-personal marking. In this paper, we have attempted to develop and test a novel method for automatic localization of craniofacial structures based on the detected edges in the region of interest. Before edge detection of the particular region, the region was filtered by adaptive non local filter for noise removal by keeping the edge information undisturbed. According to the gray-scale feature at the different regions of the cephalograms, modified Canny edge detection algorithm for obtaining tissue contour was proposed. With the application of morphological opening and edge linking approaches, an improved bidirectional contour tracing methodology was proposed by an interactive selection of the starting edge pixels, the tracking process searches repetitively for an edge pixel at the neighborhood of previously searched edge pixel to segment images, and then craniofacial structures are obtained. The effectiveness of the algorithm is demonstrated by the preliminary experimental results obtained with the proposed method. [ABSTRACT FROM PUBLISHER]

Stereo matching problem attempts to find corresponding locations between pairs of displaced images of the same scene. Correspondence estimation between pixels suffers from occlusions, noise, and bias. This paper introduces a novel approach to represent images by means of interval-valued fuzzy sets. These sets allow one to overcome the uncertainty due to the aforementioned problems. The aim is to take advantage of the new representation to develop a stereo matching algorithm. The interval-valued fuzzification process for images that is proposed here is based on image segmentation. Interval-valued fuzzy similarities are introduced to compare windows whose pixels are represented by intervals. To make use of color information, the similarities of the RGB channels were aggregated using the luminance formula. The experimental analysis makes a comparison with other methods. The new representation that is proposed together with the new similarity measure show a better overall behavior, providing more accurate correspondences, mainly near depth discontinuities and for images with a large amount of color. [ABSTRACT FROM AUTHOR]

A novel ellipse detector based upon edge following is proposed in this paper. The detector models edge connectivity by line segments and exploits these line segments to construct a set of elliptical-arcs. Disconnected elliptical-arcs which describe the same ellipse are identified and grouped together by incrementally finding optimal pairings of elliptical-arcs. We extract hypothetical ellipses of an image by fitting an ellipse to the elliptical-arcs of each group. Finally, a feedback loop is developed to sieve out low confidence hypothetical ellipses and to regenerate a better set of hypothetical ellipses. In this aspect, the proposed algorithm performs self-correction and homes in on “difficult” ellipses. Detailed evaluation on synthetic images shows that the algorithm outperforms existing methods substantially in terms of recall and precision scores under the scenarios of image cluttering, salt-and-pepper noise and partial occlusion. Additionally, we apply the detector on a set of challenging real-world images. Successful detection of ellipses present in these images is demonstrated. We are not aware of any other work that can detect ellipses from such difficult images. Therefore, this work presents a significant contribution towards ellipse detection. [ABSTRACT FROM AUTHOR]

This paper presents a technique for motion detection that incorporates several innovative mechanisms. For example, our proposed technique stores, for each pixel, a set of values taken in the past at the same location or in the neighborhood. It then compares this set to the current pixel value in order to determine whether that pixel belongs to the background, and adapts the model by choosing randomly which values to substitute from the background model. This approach differs from those based upon the classical belief that the oldest values should be replaced first. Finally, when the pixel is found to be part of the background, its value is propagated into the background model of a neighboring pixel. We describe our method in full details (including pseudo-code and the parameter values used) and compare it to other background subtraction techniques. Efficiency figures show that our method outperforms recent and proven state-of-the-art methods in terms of both computation speed and detection rate. We also analyze the performance of a downscaled version of our algorithm to the absolute minimum of one comparison and one byte of memory per pixel. It appears that even such a simplified version of our algorithm performs better than mainstream techniques. [ABSTRACT FROM AUTHOR]

The mixed raster content (MRC) standard (ITU-T T.44) specifies a framework for document compression which can dramatically improve the compression/quality tradeoff as compared to traditional lossy image compression algorithms. The key to MRC compression is the separation of the document into foreground and background layers, represented as a binary mask. Therefore, the resulting quality and compression ratio of a MRC document encoder is highly dependent upon the segmentation algorithm used to compute the binary mask. In this paper, we propose a novel multiscale segmentation scheme for MRC document encoding based upon the sequential application of two algorithms. The first algorithm, cost optimized segmentation (COS), is a blockwise segmentation algorithm formulated in a global cost optimization framework. The second algorithm, connected component classification (CCC), refines the initial segmentation by classifying feature vectors of connected components using an Markov random field (MRF) model. The combined COS/CCC segmentation algorithms are then incorporated into a multiscale framework in order to improve the segmentation accuracy of text with varying size. In comparisons to state-of-the-art commercial MRC products and selected segmentation algorithms in the literature, we show that the new algorithm achieves greater accuracy of text detection but with a lower false detection rate of nontext features. We also demonstrate that the proposed segmentation algorithm can improve the quality of decoded documents while simultaneously lowering the bit rate. [ABSTRACT FROM AUTHOR]

This paper presents a multiphase fuzzy region competition model that takes into account spatial and frequency information for image segmentation. In the proposed energy functional, each region is represented by a fuzzy membership function and a data fidelity term that measures the conformity of spatial and frequency data within each region to (generalized) Gaussian densities whose parameters are determined jointly with the segmentation process. Compared with the classical region competition model, our approach gives soft segmentation results via the fuzzy membership functions, and moreover, the use of frequency data provides additional region information that can improve the overall segmentation result. To efficiently solve the minimization of the energy functional, we adopt an alternate minimization procedure and make use of Chambolle's fast duality projection algorithm. We apply the proposed method to synthetic and natural textures as well as real-world natural images. Experimental results show that our proposed method has very promising segmentation performance compared with the current state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Human observers understand the content of an image intuitively. Based upon image content, they perform many image-related tasks, such as creating slide shows and photo albums, and organizing their image archives. For example, to select photos for an album, people assess image quality based upon the main objects in the image. They modify colors in an image based upon the color of important objects, such as sky, grass or skin. Serious photographers might modify each object separately. Photo applications, in contrast, use low-level descriptors to guide similar tasks. Typical descriptors, such as color histograms, noise level, JPEG artifacts and overall sharpness, can guide an imaging application and safeguard against blunders. However, there is a gap between the outcome of such operations and the same task performed by a person. We believe that the gap can be bridged by automatically understanding the content of the image. This paper presents algorithms for automatic tagging of perceptual objects in images, including sky, skin, and foliage, which constitutes an important step toward this goal. [ABSTRACT FROM AUTHOR]

A number of algorithms have been proposed for intelligent image/video retargeting with image content retained as much as possible. However, they usually suffer from some artifacts in the results, such as ridge or structure twist. In this paper, we present a structure-preserving media retargeting technique that preserves the content and image structure as best as possible. Different from the previous pixel or grid based methods, we estimate the image content saliency from the structure of the content. A block structure energy is introduced with a top-down strategy to constrain the image structure inside to deform uniformly in either x or y direction. However, the flexibilities for retargeting are quite different for different images. To cope with this problem, we propose a compressibility assessment scheme for media retargeting by combining the entropies of image gradient magnitude and orientation distributions. Thus, the resized media is produced to preserve the image content and structure as best as possible. Our experiments demonstrate that the proposed method provides resized images/videos with better preservation of content and structure than those by the previous methods. [ABSTRACT FROM AUTHOR]

Automatic image registration is still an actual challenge in several fields. Although several methods for automatic image registration have been proposed in the last few years, it is still far from a broad use in several applications, such as in remote sensing. In this paper, a method for automatic image registration through histogram-based image segmentation (HAIRIS) is proposed. This new approach mainly consists in combining several segmentations of the pair of images to be registered, according to a relaxation parameter on the histogram modes delineation (which itself is a new approach), followed by a consistent characterization of the extracted objects—through the objects area, ratio between the axis of the adjust ellipse, perimeter and fractal dimension—and a robust statistical based procedure for objects matching. The application of the proposed methodology is illustrated to simulated rotation and translation. The first dataset consists in a photograph and a rotated and shifted version of the same photograph, with different levels of added noise. It was also applied to a pair of satellite images with different spectral content and simulated translation, and to real remote sensing examples comprising different viewing angles, different acquisition dates and different sensors. An accuracy below 1^\circ for rotation and at the subpixel level for translation were obtained, for the most part of the considered situations. HAIRIS allows for the registration of pairs of images (multitemporal and multisensor) with differences in rotation and translation, with small differences in the spectral content, leading to a subpixel accuracy. [ABSTRACT FROM AUTHOR]

Dong et al. examined the ability of 51 computational feature sets to estimate human perceptual texture similarity; however, none performed well for this task. While it is well-known that the human visual system is extremely adept at exploiting longer-range aperiodic (and periodic) “contour” characteristics in images, none of the investigated feature sets exploit higher order statistics (HOS) over larger image regions ( $> 19\times 19$ pixels). We, therefore, hypothesise that long-range HOS, in the form of contour data, are useful for perceptual texture similarity estimation. We present the results of a psychophysical experiment that shows that contour data are more important, than local image patches, or global second-order data, to human observers for this task. Inspired by this finding, we propose a set of perceptually motivated image features (PMIF) that encode the long-range HOS computed from spatial and angular distributions of contour segments. We use two perceptual texture similarity estimation tasks to compare PMIF against the 51 feature sets referred to above and four commonly used contour representations. This new feature set is also examined in the context of two additional tasks: sketch-based image retrieval and natural scene recognition. The results show that the proposed feature set performs better, or at least comparably to, all the other feature sets. We attribute this promising performance to the fact that the proposed feature set exploits both short-range and long-range HOS. [ABSTRACT FROM PUBLISHER]

Existing self-supervised methods pose one-shot video object segmentation (O-VOS) as pixel-level matching to enable segmentation mask propagation across frames. However, the two tasks are not fully equivalent since O-VOS is more reliant on semantic correspondence rather than accurate pixel matching. To remedy this issue, we explore a new self-supervised framework that integrates pixel-level correspondence learning with semantic-level adaptation. The pixel-level correspondence learning is performed through photometric reconstruction of adjacent RGB frames during offline training, while semantic-level adaption operates at test-time by enforcing a bi-directional agreement of the predicted segmentation masks. In addition, we further propose a new network architecture with multi-perspective feature mining mechanism which can not only enhance reliable features but also suppress noisy ones to facilitate more robust image matching. By training the network using the proposed self-supervised framework, we achieve state-of-the-art performance on widely adopted datasets, further closing up the gap between self-supervised learning methods and their fully supervised counterparts. [ABSTRACT FROM AUTHOR]

We propose a new method for joint segmentation of monotonously growing or shrinking shapes in a time sequence of noisy images. The task of segmenting the image time series is expressed as an optimization problem using the spatio-temporal graph of pixels, in which we are able to impose the constraint of shape growth or of shrinkage by introducing monodirectional infinite links connecting pixels at the same spatial locations in successive image frames. The globally optimal solution is computed with a graph cut. The performance of the proposed method is validated on three applications: segmentation of melting sea ice floes and of growing burned areas from time series of 2D satellite images, and segmentation of a growing brain tumor from sequences of 3D medical scans. In the latter application, we impose an additional intersequences inclusion constraint by adding directed infinite links between pixels of dependent image structures. [ABSTRACT FROM AUTHOR]

We present the first ratio-based image segmentation method that allows imposing curvature regularity of the region boundary. Our approach is a generalization of the ratio framework pioneered by Jermyn and Ishikawa so as to allow penalty functions that take into account the local curvature of the curve. The key idea is to cast the segmentation problem as one of finding cyclic paths of minimal ratio in a graph where each graph node represents a line segment. Among ratios whose discrete counterparts can be globally minimized with our approach, we focus in particular on the elastic ratio $ \displaystyle\int_0^\cal L(C)\nabla I(C(s))\cdot(C^\prime(s))^\perp\, ds\over \displaystyle \nu\, \cal L(C)+\int_0^\cal L(C)\vert \kappa_C(s)\vert ^q\,ds $that depends, given an image I, on the oriented boundary C of the segmented region candidate. Minimizing this ratio amounts to finding a curve, neither small nor too curvy, through which the brightness flux is maximal. We prove the existence of minimizers for this criterion among continuous curves with mild regularity assumptions. We also prove that the discrete minimizers provided by our graph-based algorithm converge, as the resolution increases, to continuous minimizers. In contrast to most existing segmentation methods with computable and meaningful, i.e., nondegenerate, global optima, the proposed approach is fully unsupervised in the sense that it does not require any kind of user input such as seed nodes. Numerical experiments demonstrate that curvature regularity allows substantial improvement of the quality of segmentations. Furthermore, our results allow drawing conclusions about global optima of a parameterization-independent version of the snakes functional: the proposed algorithm allows determining parameter values where the functional has a meaningful solution and simultaneously provides the corresponding global solution. [ABSTRACT FROM PUBLISHER]

The authors present a computationally efficient technique for maximum a posteriori (MAP) estimation of images in the presence of both blur and noise. The image is divided into statistically independent regions. Each region is modelled with a WSS Gaussian prior. Classical Wiener filter theory is used to generate a set of convex sets in the solution space, with the solution to the MAP estimation problem lying at the intersection of these sets. The proposed algorithm uses an underlying segmentation of the image, and a means of determining the segmentation and refining it are described. The algorithm is suitable for a range of image restoration problems, as it provides a computationally efficient means to deal with the shortcomings of Wiener filtering without sacrificing the computational simplicity of the filtering approach. The algorithm is also of interest from a theoretical viewpoint as it provides a continuum of solutions between Wiener filtering and Inverse filtering depending upon the segmentation used. We do not attempt to show here that the proposed method is the best general approach to the image reconstruction problem. However, related work referenced herein shows excellent performance in the specific problem of demosaicing. [ABSTRACT FROM PUBLISHER]

Superpixels are widely used in computer vision applications. Most of the existing superpixel methods use established criteria to indiscriminately process all pixels, resulting in superpixel boundary adherence and regularity being unnecessarily inter-inhibitive. This study builds upon a previous work by proposing a new segmentation strategy that classifies image content into meaningful areas containing object boundaries and meaningless parts that include color-homogeneous and texture-rich regions. Based on this classification, we design two distinct criteria to process the pixels in different environments to achieve highly accurate superpixels in content-meaningful areas and keep the regularity of the superpixels in content-meaningless regions. Additionally, we add a group of weights when adopting the color feature, successfully reducing the undersegmentation error. The superior accuracy and the moderate compactness achieved by the proposed method in comparative experiments with several state-of-the-art methods indicate that the content-adaptive criteria efficiently reduce the compromise between boundary adherence and compactness. [ABSTRACT FROM AUTHOR]

Panoptic segmentation (PS) is a complex scene understanding task that requires providing high-quality segmentation for both thing objects and stuff regions. Previous methods handle these two classes with semantic and instance segmentation modules separately, following with heuristic fusion or additional modules to resolve the conflicts between the two outputs. This work simplifies this pipeline of PS by consistently modeling the two classes with a novel PS framework, which extends a detection model with an extra module to predict category- and instance-aware pixel embedding (CIAE). CIAE is a novel pixel-wise embedding feature that encodes both semantic-classification and instance-distinction information. At the inference process, PS results are simply derived by assigning each pixel to a detected instance or a stuff class according to the learned embedding. Our method not only demonstrates fast inference speed but also the first one-stage method to achieve comparable performance to two-stage methods on the challenging COCO benchmark. [ABSTRACT FROM AUTHOR]

This article proposes a hybrid multi-dimensional features fusion structure of spatial and temporal segmentation model for automated thermography defects detection. In addition, the newly designed attention block encourages local interaction among the neighboring pixels to recalibrate the feature maps adaptively. A Sequence-PCA layer is embedded in the network to provide enhanced semantic information. The final model results in a lightweight structure with smaller number of parameters and yet yields uncompromising performance after model compression. The proposed model allows better capture of the semantic information to improve the detection rate in an end-to-end procedure. Compared with current state-of-the-art deep semantic segmentation algorithms, the proposed model presents more accurate and robust results. In addition, the proposed attention module has led to improved performance on two classification tasks compared with other prevalent attention blocks. In order to verify the effectiveness and robustness of the proposed model, experimental studies have been carried out for defects detection on four different datasets. The demo code of the proposed method can be linked soon: http://faculty.uestc.edu.cn/gaobin/zh_CN/lwcg/153392/list/index.htm [ABSTRACT FROM AUTHOR]