Search

Specularity segmentation in colonoscopy images is a crucial pre-processing step for efficient computational diagnosis. The presence of these specular highlights could mislead the detectors that are intended towards precise identification of biomarkers. Conventional methods adopted so far do not provide satisfactory results, especially in the overexposed regions. The potential of deep learning methods is still unexplored in the related problem domain. Our work aims at providing a solution for more accurate highlights segmentation to assist surgeons. In this paper, we propose a novel deep learning based approach that performs segmentation following a multi-resolution analysis. This is achieved by introducing discrete wavelet transform (DWT) into the proposed model. We replace the standard pooling layers with DWTs, which helps preserve information and circumvent the effect of overexposed regions. All analytical experiments are performed using a publicly available benchmark dataset, and an F1-score (%) of 83.10 ± 0.14 is obtained on the test set. The experimental results show that this technique outperforms state-of-the-art methods and performs significantly better in overexposed regions. The proposed model also performed superior to some deep learning models (but applied in different domains) when tested with our problem specifications. Our method provides segmentation outcomes that are closer to the actual segmentation done by experts. This ensures improved pre-processed colonoscopy images that aid in better diagnosis of colorectal cancer. [ABSTRACT FROM AUTHOR]

Specular highlights detection and removal in images is a fundamental yet non-trivial problem of interest. Most modern techniques proposed are inadequate at dealing with real-world images taken under uncontrolled conditions with the presence of complex textures, multiple objects, and bright colours, resulting in reduced accuracy and false positives. To detect specular pixels in a wide variety of real-world images independent of the number, colour, or type of illuminating source, we propose an efficient Specular Segmentation (SpecSeg) network based on the U-net architecture that is expeditious to train on nominal-sized datasets. The proposed network can detect pixels strongly affected by specular highlights with a high degree of precision, as shown by comparison with the state-of-the-art methods. The technique proposed is trained on publicly available datasets and tested using a large selection of real-world images with highly encouraging results. [ABSTRACT FROM AUTHOR]

Specular highlights detection and removal in images is a fundamental yet non-trivial problem of interest. Most modern techniques proposed are inadequate at dealing with real-world images taken under uncontrolled conditions with the presence of complex textures, multiple objects, and bright colours, resulting in reduced accuracy and false positives. To detect specular pixels in a wide variety of real-world images independent of the number, colour, or type of illuminating source, we propose an efficient Specular Segmentation (SpecSeg) network based on the U-net architecture that is expeditious to train on nominal-sized datasets. The proposed network can detect pixels strongly affected by specular highlights with a high degree of precision, as shown by comparison with the state-of-the-art methods. The technique proposed is trained on publicly available datasets and tested using a large selection of real-world images with highly encouraging results.

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs. [ABSTRACT FROM AUTHOR]

Purpose: Live intra-operative functional imaging has multiple potential clinical applications, such as localization of ischemia, assessment of organ transplantation success and perfusion monitoring. Recent research has shown that live monitoring of functional tissue properties, such as tissue oxygenation and blood volume fraction, is possible using multispectral imaging in laparoscopic surgery. While the illuminant spectrum is typically kept constant in laparoscopic surgery and can thus be estimated from preoperative calibration images, a key challenge in open surgery originates from the dynamic changes of lighting conditions. Methods: The present paper addresses this challenge with a novel approach to light source calibration based on specular highlight analysis. It involves the acquisition of low-exposure time images serving as a basis for recovering the illuminant spectrum from pixels that contain a dominant specular reflectance component. Results: Comprehensive in silico and in vivo experiments with a range of different light sources demonstrate that our approach enables an accurate and robust recovery of the illuminant spectrum in the field of view of the camera, which results in reduced errors with respect to the estimation of functional tissue properties. Our approach further outperforms state-of-the-art methods proposed in the field of computer vision. Conclusion: Our results suggest that low-exposure multispectral images are well suited for light source calibration via specular highlight analysis. This work thus provides an important first step toward live functional imaging in open surgery. [ABSTRACT FROM AUTHOR]

Stereoscopic display technology provides immersive experiences in VR/AR/XR, but requires markedly higher bandwidth and is perceived differently than 2D content. Here we adapt the ISO/IEC 29170‐2 flicker paradigm for subjective assessment of low impairment stereoscopic image compression. We compared the performance VESA VDC‐M codec on stereoscopic images with 2D image performance. [ABSTRACT FROM AUTHOR]

Video streams are utilised to guide minimally-invasive surgery and diagnosis in a wide range of procedures, and many computer-assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modelling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames, where they are not present in the same location. This is achieved using in-vivo data from gastric endoscopy (Hyper Kvasir) in a fully unsupervised manner that relies on the automatic detection of specular highlights. System evaluations show significant improvements to other methods through direct comparison and ablation studies that depict the importance of the network's temporal and transfer learning components. The generalisability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in comparison to other methods on computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of our specular inpainting method on these tasks in a novel comprehensive analysis. Our code and dataset are made available at https://github.com/endomapper/Endo-STTN. [Display omitted] • A temporal learning-based solution to endoscopic specular highlight removal. • A pseudo ground truth dataset for unsupervised training and quantitative evaluation. • A quantitative and qualitative evaluation of our approach. • An evaluation of the effect of inpainting specular highlights on downstream tasks. [ABSTRACT FROM AUTHOR]

Specular highlights in images pose a significant challenge in algorithms for image segmentation, object detection and other image-based decision-making systems. However, most systems ignore this particular scenario and neglect input images with specular highlights instead of mitigating it in the pre-processing stage. In this paper, we leverage deep neural networks and take advantage of the varying illumination information in polarimetric images for synthesizing specular-free images. We propose a multi-domain Specular Highlight Mitigation Generative Adversarial Network (SHMGAN) with self-attention. SHMGAN consists of a single generator–discriminator pair trained simultaneously using polarimetric images. The proposed GAN uses a dynamically generated attention mask based on a specularity segmentation network without requiring additional manual input. The network is able to learn the illumination variation between the four polarimetric images and a pseudo-diffuse image. Once trained, SHMGAN is able to generate specular-free images from a single RGB image as input; without requiring any additional external labels. The proposed network is trained and tested publicly available datasets of real-world images. SHMGAN is able to accurately identify the specularity affected pixels and generates high visual quality images with mitigated specular reflections. The generated images are realistic and have very low noise, distortions and aberrations compared to the existing state-of-the-art methods for specular highlight removal. [ABSTRACT FROM AUTHOR]

A procedure for identifying apples in night-time orchard images was developed and tested on two datasets totalling over 550 images of Golden Delicious trees captured on two years with different cameras and lighting systems. The analysis started by detecting specular reflection highlights and extracting sub-images (101 by 101 pixels) centred at those local maxima. Each sub-image was reduced to a 676 Upright Speeded Up Robust Features (U-SURF) vector. Close to 20,000 sub-images from one dataset were manually labelled as “apple” or “not apple”. The latter group included parts of leaves, branches and other objects which exhibited strong specular reflection. Seventy-two classifiers were trained with the number of “apple” and “not apple” training samples ranging from 500 to 2000 and from 5000 to 10,000, respectively, and with vocabulary size ranging from 500 to 10,000. Misclassifications occurred mostly in dark and low contrast regions, which led to developing alternate models based on the posterior probability that the classification result was correct taking into account the sub-image entropy or intensity. Yield models were calibrated for each dataset, using 20 random trees. For both datasets the overall yield estimate was within 10% of the actual yield, and the standard deviation was around 30% of the average tree yield. These results are similar to those reported in previous studies, but while these previous studies used procedures calibrated and tested with images from the same dataset, in the present study the classifier trained with images from one dataset was successfully applied to the second dataset. [ABSTRACT FROM AUTHOR]

A novel specular highlights detection method in colonoscopy videos is presented. The method is based on a model of appearance defining specular highlights as bright spots which are highly contrasted with respect to adjacent regions. Our approach proposes two stages: segmentation and then classification of bright spot regions. The former defines a set of candidate regions obtained through a region growing process with local maxima as initial region seeds. This process creates a tree structure which keeps track, at each growing iteration, of the region frontier contrast; final regions provided depend on restrictions over contrast value. Non-specular regions are filtered through a classification stage performed by a linear SVM classifier using model-based features from each region. We introduce a new validation database with more than 25, 000 regions along with their corresponding pixel-wise annotations. We perform a comparative study against other approaches. Results show that our method is superior to other approaches, with our segmented regions being closer to actual specular regions in the image. Finally, we also present how our methodology can also be used to obtain an accurate prediction of polyp histology. [ABSTRACT FROM AUTHOR]

Specular BRDF rendering traditionally approximates surface microstructure using a smooth normal distribution, but this ignores glinty effects, easily observable in the real world. While modeling the actual surface microstructure is possible, the resulting rendering problem is prohibitively expensive. Recently, Yan et al. [2014] and Jakob et al. [2014] made progress on this problem, but their approaches are still expensive and lack full generality in their material and illumination support. We introduce an efficient and general method that can be easily integrated in a standard rendering system. We treat a specular surface as a four-dimensional position-normal distribution, and fit this distribution using millions of 4D Gaussians, which we call elements. This leads to closed-form solutions to the required BRDF evaluation and sampling queries, enabling the first practical solution to rendering specular microstructure. [ABSTRACT FROM AUTHOR]

We investigated whether specular highlights facilitate the perception of shape from shading in a search paradigm and how highlights interact with shading to facilitate this perception. Our results indicated that stimuli containing highlights led to shorter searching time with the dependence on the light source direction (top lights make searching faster), suggesting that highlights indeed facilitate shape-from-shading processing. To examine how highlight processing interacts with shading processing, we tested unnatural stimuli for which the lighting directions for shading and highlights were inconsistent. The results indicated that unnatural highlights (bright spots) placed in a direction inconsistent with the shading either decrease or do not alter searching time. This suggests that highlights may facilitate, and not suppress, shading processing. With more physically plausible highlights generated from image-based lighting, we also observed facilitation with consistent highlights, but no change with inconsistent highlights. Finally, we examined whether highlights indeed work to facilitate depth perception in a discrimination task. The results showed that correct discrimination of depth increases when highlights are added to shading even when their lighting directions are inconsistent. These results indicate that specular highlights facilitate shading processing, and do not suppress it even when the highlights are placed in a direction inconsistent with shading. The results also elucidate the lighting constraints of the visual system. [ABSTRACT FROM AUTHOR]

Detection of fruit in tree images has been the focus of numerous studies. Although most studies considered approaches based primarily on color analysis, the major drawback of such approaches is that the fruit apparent color depends not only on variety or physiological stage but also on illumination, which is inherently non-uniform within the canopy, even if artificial lighting is used. In the present work we developed a novel approach to detect apples in nighttime images by analyzing the spatial distribution of the light around highlights (“bright spots”). The approach is based on the observation that, under the artificial illumination used, apples exhibit strong specular reflection so that a small, but very bright, spot is visible on almost all apples. Each of these highlights serves as the center of a region of interest and is the seed of the investigated light patch. This patch is initially very small but its size is increased iteratively by annexing pixels with predefined decreasing gray level intensities. The evolution of the patch geometry is used to determine whether it corresponds to an apple. The approach was tested with two datasets containing over 360 images (close to 13,000 apples) acquired in the same ‘Golden Delicious’ orchard in July 2012 and August 2013. Twenty images from the 2012 dataset were randomly selected to develop and calibrate the procedure. The results of these 20 images were used to establish a linear relationship between the number of detected objects and the actual number of apples visible in the images ( R 2 ∼ 0.75). Applying the calibrated procedure to the remaining images of this dataset led to an estimate of 6739 apples compared to a visual count of 6195 apples (∼9% overestimate). Analysis of the 2013 dataset, in which the apparent size of the apples was smaller, required only adjustment of the two parameters related to apple size. Following this adjustment, 12 images were randomly selected to determine the relationship between the number of detected objects and the actual number of apples ( R 2 ∼ 0.74). Using this relationship, the estimated number of apples was 6687, compared to the visual count of 6713 fruits. [ABSTRACT FROM AUTHOR]

Complex specular surfaces under sharp point lighting show a fascinating glinty appearance, but rendering it is an unsolved problem. Using Monte Carlo pixel sampling for this purpose is impractical: the energy is concentrated in tiny highlights that take up a minuscule fraction of the pixel. We instead compute an accurate solution using a completely different deterministic approach. Our method considers the true distribution of normals on a surface patch seen through a single pixel, which can be highly complex. We show how to evaluate this distribution efficiently, assuming a Gaussian pixel footprint and Gaussian intrinsic roughness. We also take advantage of hierarchical pruning of position-normal space to rapidly find texels that might contribute to a given normal distribution evaluation. Our results show complex, temporally varying glints from materials such as bumpy plastics, brushed and scratched metals, metallic paint and ocean waves. [ABSTRACT FROM AUTHOR]

Visual inspection for a highly reflective surface is commonly faced with a serious limitation, which is that useful information on geometric construction and textural defects is covered by a parasitic image due to specular highlights. In order to solve the problem, we propose an effective method for removing the parasitic image. Specifically, a digital micromirror device (DMD) camera for programmable imaging is first described. The strength of the optical system is to process scene ray before image formation. Based on the DMD camera, an iterative algorithm of modulated region selection, precise region mapping, and multimodulation provides removal of the parasitic image and reconstruction of a correction image. Finally, experimental results show the performance of the proposed approach. [ABSTRACT FROM AUTHOR]

Once HDR displays were developed, a constant question persisted about how much dynamic range is needed for display. If one uses physical scene luminances or human visual system threshold detections to answer this question, the needed ranges are unachievable at exorbitant cost, and likely to remain so for decades. Therefore we designed studies to find the range that is preferred by human observers, and for suprathreshold appearances. Two studies address the diffuse reflective regions, and a third study tested preferences of highlight regions. Test images were specifically designed to test these limits without the perceptual conflicts that usually occur in these types of studies. For the diffuse range, we found displays capable of a dynamic range between 0.1 and 650 cd/m2 match the average preferences. However, to satisfy 90% of the population, a dynamic range from 0.005 to ∼3,000 cd/m2 is needed. Since a display should be able to produce values brighter than the diffuse white maximum, as in specular highlights and emissive sources, the highlight study concludes that the average preferred maximum luminance for highlight reproduction satisfying 50% of viewers is ∼2,500 cd/m2. This value increases to marginally over 20,000 cd/m2 when catering to 90%. Though there is some variability in preferred brightness between certain demographics, the call for a more capable display is still evident, as preferred luminances found in this study exceed even the best of consumer displays today. [ABSTRACT FROM AUTHOR]

This paper presents an algorithm that automatically detects and corrects specular reflections in thoracoscopic images and its application in the context of automatic segmentation of surgical tools. The detection is done by isolating the spike component of the specular reflection which is characterized by a bump at the end of the histogram of thoracoscopic images. The specular lobe is then extracted in the neighborhood of the spike component of the reflection. The result is a mask of the reflections positions in the image. Thereafter, the image is corrected using Oliveira et al.'s digital inpainting method. The automatic segmentation of surgical tools using the corrected images is then demonstrated. Results of the segmentation with and without the specular reflection elimination technique are compared. Moreover, 108 images extracted from 5 different surgeries performed under various conditions were considered to demonstrate the effectiveness of the proposed technique. [ABSTRACT FROM AUTHOR]

This article is a contribution to a recent debate in the philosophy of perception between Alva Noë and Sean Kelly. Noë (2004) has argued that the perspectival part of perception is simultaneously represented along with the non-perspectival part of perception. Kelly (2004) argues that the two parts of perception are not always simultaneously experienced. Here I focus on specular highlights as an example of the perspectival part of perception. First I give a priori motivation to think that specular highlights are experienced at the same time as non-perspectival properties, which challenges Kelly's position. Then I discuss psychophysical work by Andrew Blake and Heinrich Bulthoff (1990) which seems to show that specular highlights are not represented in the way that Noë (2004) would suggest. In the third section I suggest a compromise between Noë and Kelly: specular highlights are not represented, but rather play an evidentiary role in the representation of perspective-independent properties, like gloss and shape. I conclude with some thoughts about how this study can generalize to other kinds of experience. [ABSTRACT FROM AUTHOR]

In social interactions, our sense of when we have eye contact with another person relies on the distribution of luminance across their eye region, reflecting the position of the darker iris within the lighter sclera of the human eye. This distribution of luminance can be distorted by the lighting conditions, consistent with the fundamental challenge that the visual system faces in distinguishing the nature of a surface from the pattern of light falling upon it. Here we perform a set of psychophysics experiments in human observers to investigate how illumination impacts on the perception of eye contact. First, we find that simple changes in the direction of illumination can produce systematic biases in our sense of when we have eye contact with another person. Second, we find that the visual system uses information about the lighting conditions to partially discount or 'explain away' the effects of illumination in this context, leading to a significantly more robust sense of when we have eye contact with another person. Third, we find that perceived eye contact is affected by specular reflections from the eye surface in addition to shading patterns, implicating eye glint as a potential cue to gaze direction. Overall, this illustrates how our interpretation of social signals relies on visual mechanisms that both compensate for the effects of illumination on retinal input and potentially exploit novel cues that illumination can produce. [ABSTRACT FROM AUTHOR]