Search

Small moving objects at far distance always occupy only one or a few pixels in image and exhibit extremely limited visual features, which bring great challenges to motion detection. Highly evolved visual systems endow flying insects with remarkable ability to pursue tiny mates and prey, providing a good template to develop image processing method for small target motion detection. The insects’ excellent sensitivity to small moving objects is believed to come from a class of specific neurons called small target motion detectors (STMDs). However, existing STMD-based methods often experience performance degradation when coping with complex natural scenes. In this paper, we propose a bio-inspired visual system with spatio-temporal feedback mechanism (called Spatio-Temporal Feedback STMD) to suppress false positive background movement while enhancing system responses to small targets. Specifically, the proposed visual system is composed of two complementary subnetworks and a feedback loop. The first subnetwork is designed to extract spatial and temporal movement patterns of cluttered background by neuronal ensemble coding. The second subnetwork is developed to capture small target motion information where its output and signal from the first subnetwork are integrated together via the feedback loop to filter out background false positives in a recurrent manner. Experimental results demonstrate that the proposed spatio-temporal feedback visual system is more competitive than existing methods in discriminating small moving targets from complex natural environments. [ABSTRACT FROM AUTHOR]

High dynamic range (HDR) imaging is a technique that allows an extensive dynamic range of exposures, which is important in image processing, computer graphics, and computer vision. In recent years, there has been a significant advancement in HDR imaging using deep learning (DL). This study conducts a comprehensive and insightful survey and analysis of recent developments in deep HDR imaging methodologies. We hierarchically and structurally group existing deep HDR imaging methods into five categories based on (1) number/domain of input exposures, (2) number of learning tasks, (3) novel sensor data, (4) novel learning strategies, and (5) applications. Importantly, we provide a constructive discussion on each category regarding its potential and challenges. Moreover, we review some crucial aspects of deep HDR imaging, such as datasets and evaluation metrics. Finally, we highlight some open problems and point out future research directions. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application. [ABSTRACT FROM AUTHOR]

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA. [ABSTRACT FROM AUTHOR]

Exploring and establishing artificial neural networks with electrophysiological characteristics and high computational efficiency is a popular topic that has been explored for many years in the fields of pattern recognition and computer vision. Inspired by the working mechanism of the primary visual cortex, pulse-coupled neural networks (PCNNs) can exhibit the characteristics of synchronous oscillation, refractory period, and exponential decay. These characteristics empower the PCNN model to group pixels with similar spatiality and gray values and to process digital images without training. However, electrophysiological evidence shows that the neurons exhibit highly complex nonlinear dynamics when stimulated by external periodic signals. This chaos phenomenon, also known as the ‘butterfly effect,” cannot be explained by all PCNN models. In this work, we analyze the main obstacle preventing PCNN models from imitating a real primary visual cortex. We consider neuronal excitation as a stochastic process. We then propose a novel neural network of the primary visual cortex, called a continuous-coupled neural network (CCNN). Theoretical analysis indicates that the dynamic behavior of the CCNN is distinct from the PCNN. Numerical results show that the CCNN model exhibits periodic behavior under a DC stimulus, and exhibits chaotic behavior under an AC stimulus, which is consistent with the testing results of primary visual cortex neurons. Furthermore, the image and video processing mechanisms of the CCNN model are analyzed. For image processing tasks, this model encodes the pixel intensity as the frequency of output signals so that it can group pixels with similar gray values. This image processing method can reduce the local gray level difference of the image, and compensate for small local discontinuities in the image. For video processing tasks, the CCNN encodes changing pixels as non-periodic chaotic signals, and it encodes static pixels as periodic signals. It thus achieves the purpose of moving target object recognition by distinguishing the dynamic states corresponding to different neuron clusters in the video. Experimental results on image segmentation indicate that the CCNN model has better performance than the state-of-the-art of visual cortex neural network models. [ABSTRACT FROM AUTHOR]

Defect pattern detection and classification are challenging for thin-film-transistor liquid-crystal display (TFT-LCD) manufacturing. Limitations of the existing solutions for automatic optical inspection can be traced in part to the lack of a framework within which different existing and new defect patterns can be analyzed, while integrating domain knowledge and effective technologies. This study aims to develop a framework for image-based defect classification that employs the convolution neural networks without using complex and time-consuming image-processing processes in advance. An empirical study was conducted in a leading TFT-LCD manufacturing in Taiwan for validation. The results have shown that the defect patterns can be effectively classified by the proposed convolutional neural networks that outperform the existing approaches such as Support Vector Machine and Random Forest. The developed solution is implemented to effectively support the engineers. [ABSTRACT FROM AUTHOR]

With more sensors being installed by utilities for accurate control of power grids, there is a growing risk of vulnerability to sophisticated data integrity attacks such as false data injection (FDI), circumventing current bad data detection schemes resulting in inaccurate state estimation solutions. While diverse automated detectors to battle FDI have been grown, such methodologies underestimate the strong analytical abilities of humans. This is while most proposed automated methods need observant human control. Although automated methods provide opportunities to improve scalability, humans can cope with exceptions and new attack trends. In this paper, to address the ever-increasing cyber-attack challenge in power systems, a visualization based attack detection framework using deep learning techniques is developed to provide human security researchers with improved techniques to uncover trends, identify outliers, recognize correlations, and communicate their results. Specifically, we first encode multivariate systems state time-series data into 2D colored images and then utilize a carefully designed deep convolutional neural network (CNN) classifier. The proposed method is developed to allow network operators to immediately capture and visually understand the statistical features of a network attack at a glance. The proposed method has been evaluated on the IEEE 14-bus and IEEE 118-bus systems. Our experiments show that the proposed framework accomplishes high classification accuracy.

Due to numerous hardware shortcomings, medical image acquisition devices are susceptible to producing low-quality (i.e., low contrast, inappropriate brightness, noisy, etc.) images. Regrettably, perceptually degraded images directly impact the diagnosis process and make the decision-making manoeuvre of medical practitioners notably complicated. This study proposes to enhance such low-quality images by incorporating end-to-end learning strategies for accelerating medical image analysis tasks. To the best concern, this is the first work in medical imaging which comprehensively tackles perceptual enhancement, including contrast correction, luminance correction, denoising, etc., with a fully convolutional deep network. The proposed network leverages residual blocks and a residual gating mechanism for diminishing visual artefacts and is guided by a multi-term objective function to perceive the perceptually plausible enhanced images. The practicability of the deep medical image enhancement method has been extensively investigated with sophisticated experiments. The experimental outcomes illustrate that the proposed method could outperform the existing enhancement methods for different medical image modalities by 5.00 to 7.00 dB in peak signal-to-noise ratio (PSNR) metrics and 4.00 to 6.00 in DeltaE metrics. Additionally, the proposed method can drastically improve the medical image analysis tasks’ performance and reveal the potentiality of such an enhancement method in real-world applications. [ABSTRACT FROM AUTHOR]

Multi-focus image fusion (MFIF) is an important area in image processing. Since 2017, deep learning has been introduced to the field of MFIF and various methods have been proposed. However, there is a lack of survey papers that discuss deep learning-based MFIF methods in detail. In this study, we fill this gap by giving a detailed survey on deep learning-based MFIF algorithms, including methods, datasets and evaluation metrics. To the best of our knowledge, this is the first survey paper that focuses on deep learning-based approaches in the field of MFIF. Besides, extensive experiments have been conducted to compare the performance of deep learning-based MFIF algorithms with conventional MFIF approaches. By analyzing qualitative and quantitative results, we give some observations on the current status of MFIF and discuss some future prospects of this field. [ABSTRACT FROM AUTHOR]

Saliency detection by human refers to the ability to identify pertinent information using our perceptive and cognitive capabilities. While human perception is attracted by visual stimuli, our cognitive capability is derived from the inspiration of constructing concepts of reasoning. Saliency detection has gained intensive interest with the aim of resembling human ‘perceptual’ system. However, saliency related to human ‘cognition’, particularly the analysis of complex salient regions (‘cogitating’ process), is yet to be fully exploited. We propose to resemble human cognition, coupled with human perception, to improve saliency detection. We recognize saliency in three phases (‘Seeing’ - ‘Perceiving’ - ‘Cogitating), mimicking human's perceptive and cognitive thinking of an image. In our method, ‘Seeing’ phase is related to human perception, and we formulate the ‘Perceiving’ and ‘Cogitating’ phases related to the human cognition systems via deep neural networks (DNNs) to construct a new module (Cognitive Gate) that enhances the DNN features for saliency detection. To the best of our knowledge, this is the first work that established DNNs to resemble human cognition for saliency detection. In our experiments, our approach outperformed 17 benchmarking DNN methods on six well-recognized datasets, demonstrating that resembling human cognition improves saliency detection. [ABSTRACT FROM AUTHOR]

Image processing based vehicle tracking is a powerful tool for monitoring traffic, but it is error prone. Relatively small errors that impede measuring time-series speed and acceleration can be hard to detect, e.g., 1 m positioning error in a 100 m long trajectory. This paper presents an efficient approach to separate the positioning errors from vehicle travel for evaluating image processing based vehicle trajectories. The approach starts with a spatiotemporal slice, STS, which is effectively a visual time-space diagram sampled from the video. This work skews the STS to flatten a given trajectory, eliminating the vehicle travel recorded in the trajectory. Positioning errors that were imperceptible relative to the distance traveled become readily apparent in the flattened track. Thus, providing a means to quickly assess reported trajectories from almost any image processing system against the true vehicle positions in the original video data. Recognizing that the flattening process works both ways, if errors are evident in a given trajectory, the STS method can also be used to quickly fix them. Thereby providing a path to accurate instantaneous speed and acceleration throughout the given trajectory. Alternatively, one can use this process to generate vehicle trajectories directly from the STS. While the main focus is longitudinal tracking, the process can also be used to assess (extract) the lateral position of a given vehicle. The method is evaluated using the NGSIM, Cityflow and UA-DETRAC datasets, in each case it is shown how this work can increase the fidelity of the given dataset. [ABSTRACT FROM AUTHOR]

The measurement for levitation gap is a very important part of levitation control system. The traditional levitation gap sensors have some shortcomings such as small measurement range and specific installation requirements, and usually need nonlinear correction and temperature compensation to meet the control requirements of the magnetic levitation system. In our work, two novel measurement methods for levitation gap based on computer vision are proposed. First, the levitation gap is measured by calculating the image pixel area of the region of interest. The error of the pixel area model can be limited within ±0.250 mm, and the mean absolute error (MAE) is 0.095 mm for full scale (FS). Second, the model named SelfConvNet based on convolutional neural network is designed for measuring the levitation gap. The error of SelfConvNet model can be limited within ±0.048 mm, and the MAE is 0.013 mm for FS. The measurement results show that the SelfConvNet model is better than SqueezeNet and Visual Geometry Group 16 models, which has high measurement accuracy and strong anti-interference ability. The method based on pixel area has lower measurement accuracy but higher processing speed. Finally, the proposed gap measurement methods have been verified in closed-loop experiment of maglev ball control system. [ABSTRACT FROM AUTHOR]

Feature matching, which refers to identifying and then corresponding the same or similar visual pattern from two or more images, is a key technique in any image processing task that requires establishing good correspondences between images. Given potential correspondences (matches) in two scenes, a novel whole-part deep learning framework, termed as Context Structure Representation Network (CSR-Net), is designed to infer the probabilities of arbitrary correspondences being inliers. Traditional approaches commonly build the local relation between correspondences by manually engineered criteria. Different from existing attempts, the main idea of our work is to learn explicitly neighborhood structure of each correspondence, allowing us to formulate the matching problem into a dynamic local structure consensus evaluation in an end-to-end fashion. For this purpose, we propose a permutation-invariant STructure Representation (STR) learning module, which can easily merge different types of networks into a unified architecture to deal with sparse matches directly. By the collaborative use of STR, we introduce a Context-Aware Attention (CAA) mechanism to adaptively re-calibrate structure features via a rotation-invariant context aware encoding and simple feature gating, thus arising the ability of fine-grained patterns recognition. Moreover, to further weaken the cost of establishing reliable correspondences, the CSR-Net is formulated as whole-part consensus learning, where the aim of whole level is compensating rigid transformations. In order to demonstrate our CSR-Net can effectively boost the baselines, we intensively experiment on image matching and other visual tasks. The results of the experiment confirm that the matching performances of CSR-Net have significantly improved over nine state-of-the-art competitors. [ABSTRACT FROM AUTHOR]

Stereoscopic visual fatigue (SVF) due to prolonged immersion in the virtual environment can lead to negative user experience, thus hindering the development of virtual reality (VR) industry. Previous studies have focused on investigating the evaluation indicators associated with SVF, while few studies have been conducted to reveal the underlying neural mechanism, especially in VR applications. In this paper, a modified Go/NoGo paradigm was adopted to induce SVF in VR environment with Go trials for maintaining participants’ attention and NoGo trials for investigating the neural effects under SVF. Random dot stereograms (RDSs) with 11 disparities were presented to evoke the depth-related visual evoked potentials (DVEPs) during 64-channel EEG recordings. EEG datasets collected from 15 participants in NoGo trials were selected to conduct individual processing and group analysis, in which the characteristics of the DVEPs components for various fatigue degrees were compared and independent components were clustered to explore the original cortex areas related to SVF. Point-by-point permutation statistics revealed that DVEPs sample points from 230 ms to 280 ms (component P2) in most brain areas changed significantly when SVF increased. Additionally, independent component analysis (ICA) identified that component P2 which originated from posterior cingulate cortex and precuneus, was associated statistically with SVF. We believe that SVF is rather a conscious status concerning the changes of self-awareness or self-location awareness than the performance reduction of retinal image processing. Moreover, we suggest that indicators representing higher conscious state may be a better indicator for SVF evaluation in VR environments. [ABSTRACT FROM AUTHOR]

Over the recent years, a number of deep learning approaches are successfully introduced to tackle the problem of image in-painting for achieving better perceptual effects. However, there still exist obvious hole-edge artifacts in these deep learning-based approaches, which need to be rectified before they become useful for practical applications. In this article, we propose an iteration-driven in-painting approach, which combines the deep context model with the backpropagation mechanism to fine-tune the learning-based in-painting process and hence, achieves further improvement over the existing state of the arts. Our iterative approach fine tunes the image generated by a pretrained deep context model via backpropagation using a weighted context loss. Extensive experiments on public available test sets, including the CelebA, Paris Streets, and PASCAL VOC 2012 dataset, show that our proposed method achieves better visual perceptual quality in terms of hole-edge artifacts compared with the state-of-the-art in-painting methods using various context models. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) plays a central role in many image processing algorithms and systems. Although many popular IQA models achieves high performance on existing released databases, they are still not well accepted in practical applications due to the not-always satisfactory accuracy on real-world data and situations. In this paper, we revisit the IQA research, and point out an ignored but interesting problem in IQA: the coarse-grained (i.e., when quality variation is sufficiently big, as the setting of most IQA databases up to date) statistical results evaluated on existing databases mask the fine-grained differentiation. Accordingly, we present a survey on image quality assessment from a new perspective: fine-grained image quality assessment (FG-IQA). Recent FG-IQA research on five major kinds of images is introduced, and some popular IQA methods are analyzed from FG-IQA perspective. The potential problems for current IQA research based on existing coarse-grained databases are analyzed and the necessity of more FG-IQA research is justified. Finally, we discuss some challenges and possible directions for future works in FG-IQA. [ABSTRACT FROM AUTHOR]

Retrieving 3D shapes based on 2D images is a challenging research topic, due to the significant gap between different domains. Recently, various approaches have been proposed to handle this problem. However, the majority of methods target the cross-domain retrieval task as a pure domain adaptation problem, which focuses on the alignment but ignores the visual relevance between the 2D images and their corresponding 3D shapes. To fundamentally decrease the divergence between different domains, we propose a novel cross-domain learning network (CLN) for 2D image-based 3D shape retrieval task. First, we estimate the pose information from the 2D image to guide the view rendering of 3D shapes, which increases the visual correlations of the cross-domain data to eliminate the divergence between them. Second, we introduce a novel joint learning network, considering both the domain-specific characteristics and the cross-domain interactions for data alignment, which further compensates for the gap between different domains by controlling the distance of intra- and inter-classes. After the metric learning process, discriminative descriptors of images and shapes are generated for the cross-domain retrieval task. To prove the effectiveness and robustness of the proposed method, we conduct extensive experiments on the MI3DOR, SHREC’13, and SHREC’14 datasets. The experimental results demonstrate the superiority of our proposed method, and significant improvements have been achieved compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Classical redundancy-based fault detection techniques, such as Duplication with Comparison (DWC), rely on replicating the computation and comparing the replicas’ output at a bit-wise granularity. In many application environments these costs are prohibitive, especially when applications are characterized by an intrinsic level of tolerance. This article presents a novel fault-detection approach for the specific context of image filtering. Peculiarity of the proposed approach is that it estimates the impact of the fault on the processed output, in order to determine whether the image is usable or should be re-processed. To limit overheads, the proposed solution exploits Approximate Computing (AC), allowing the definition of disciplined AC strategies to trade-off between accuracy and costs. Core of our solution is the successful combination of Image Quality Assessment metrics and Machine Learning models to assess the visual impact of the fault in a lightweight manner. Extensive experimental campaigns demonstrate the effectiveness of the solution, achieving achieving a reduction in terms of execution time up to 44 percent with respect to the classical DWC, with a fault detection precision ranging from 94.58 to 96.70 percent, and recall ranging from 88.2 to 97.8 percent, depending on the adopted level of approximation. [ABSTRACT FROM AUTHOR]

Visual IoT (VIoT) is a promising IoT paradigm that visualizes sensing data from massive numbers of dispersed devices. A key objective in VIoT is to efficiently manage the devices to perform complex task-related visual data processing. Prior multimedia IoT systems have mainly focused on the delivery of captured video to remote servers, without considering the video tasks’ characteristics and the devices’ heterogeneous capabilities. In this work, we propose an astute video transmission framework for such a VIoT system composed of heterogeneous visual devices. First, we formulate the problem of joint video task allocation and heterogeneous device management by constructing a device hypergraph (DH) structure, which enables devices with different capabilities to perform complex video tasks cooperatively. Second, we model the video transmission within a VIoT system by applying fractal theory considering the NP-hardness of the optimization. In particular, we construct a comprehensive fractal submodular optimization framework through a DH and explore the inner submodular property to effectively leverage both video-task complexity and device heterogeneity. Third, we consider the geographically dispersed characteristic of massive numbers of VIoT devices and propose a multi-hop dispersed transmission mechanism for achieving globally cooperative optimality. The proposed architecture has been evaluated under diverse parameter settings. Numerical results are provided to validate the proposed algorithm in terms of delay, computational efficiency, and bandwidth utilization. Simulation results confirm the effectiveness and superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Human-inspired neuromorphic computing system breaks the limit of “von Neumann bottleneck” and improves the computing efficiency. However, the processing of a large amount of non-structural visual data results in a considerable waste of computing resources. In human visual system, visual information is preprocessed by the retina before transmitted to the visual cortex, which effectively reduces data redundancy. Therefore, multifunctional intelligent devices which integrate sensing-memory-processing functions are desired for the artificial visual system. Herein, an image memory logic unit (IMLU) inspired by human retina is proposed. The IMLU is formed by a simple two-terminal photodetector and comparator module. The photodetector is based on the MoxW1−xS2 alloy with persistent photo conductivity effect. In addition, the IMLU exhibits capabilities of in situ sensing, memory and logical operations (“AND”, “OR”, and “XOR”). Furthermore, the IMLU array enables image processing functions such as dilation, erosion and difference recognition, which shows great potential to provide simplified information processing for subsequent tasks in machine vision. [ABSTRACT FROM AUTHOR]

In this work, we simulate the early visual information processing mechanisms in biological visual system (BVS) to solve the visible-infrared image fusion (VIIF) task. Concretely, both infrared and visible images are first processed with a dynamic receptive field (DRF), which is imitated by a Difference of Gaussian (DoG) function whose parameters are adjusted according to local image statistics (e.g., local edge responses). The DRF processing produces two components for each source image (e.g., the visible image or the infrared image) that respectively represent the results of On-center based DRF and Off-center based DRF. Then, the results of On-center based DRF for visible image are fused with the results of Off-center based DRF for infrared image and the results of On-center based DRF for infrared image are fused with the results of Off-center based DRF for visible image, according to the mechanisms of cortex-based center-surround fusion. Algorithmically, this step fuses the visible and infrared images processed by DRF with On-center and Off-center according to the different levels of local homogeneity. Moreover, a feedward signal acting as the sub-cortical flow is also introduced to adjust the results during the fusion. The final output image is simply obtained by the summation of two components after the cortex and sub-cortex based fusion. Qualitative and quantitative tests on four datasets demonstrate that our approach can fuse the infrared and visible images effectively with the good background details and discernible salient areas. We emphasize the importance of corticothalamic feedback, cortex and sub-cortex-based fusion, and the interactions between On-center pathway and Off-center pathway that are ubiquitous in BVS for producing the high-quality visual signals. [ABSTRACT FROM AUTHOR]

The digital imaging and vision applications in science (DIVAS) program was built to improve the computational self-efficacy and skill of first- and second-year college students majoring in biological and chemical sciences. Our three-year pilot study showed that the program could be successful in both fronts. The scholars, faculty, and staff who participated formed a community of practice that became the heart of the DIVAS program. Through this community, we expanded access to the image processing workshop in collaboration with The Carpentries, supported faculty and secondary educators in developing computing modules for their classrooms, and created and staffed a "writing center for computing" on the host campus. Overall, the DIVAS program has sparked a local computing culture. DIVAS interventions and resources are freely available for adoption by other institutions. We hope to grow the community in a way that builds student access and opportunities and supports educators in the process. [ABSTRACT FROM AUTHOR]

Cloud removal is a ubiquitous and important task in remote sensing image processing, which aims at restoring the ground regions shadowed by clouds. It is challenging to remove the clouds for a single satellite image due to the difficulty of distinguishing clouds from white objects on the ground and filling the irregular missing regions with visual consistency. In this article, we propose a novel two-stage cloud removal method. The first stage is cloud segmentation, i.e., extracting the clouds and removing the thin clouds directly using U-Net. The second stage is image restoration, i.e., removing the thick cloud and recovering the corresponding irregular missing regions using generative adversarial network (GAN). We evaluate the proposed scheme on both synthetic images and real satellite images (over $20\,000\, \times \,20\,000$ pixels). On synthetic images for cloud coverage less than 40%, the proposed scheme achieves improvements of 0.049–0.078 in Structural SIMilarity (SSIM) and 3.8–6.2 dB in peak signal-to-noise ratio (PSNR), while the $\ell _{1}$ -norm error reduces by 49%–78%, compared with a state-of-the-art deep learning method Pix2Pix. On real satellite images, we demonstrate the consistent visual results of the proposed scheme. [ABSTRACT FROM AUTHOR]

Boundary detection has long been a fundamental tool for image processing and computer vision, supporting the analysis of static and time-varying data. In this work, we built upon the theory of Graph Signal Processing to propose a novel boundary detection filter in the context of graphs, having as main application scenario the visual analysis of spatio-temporal data. More specifically, we propose the equivalent for graphs of the so-called Laplacian of Gaussian edge detection filter, which is widely used in image processing. The proposed filter is able to reveal interesting spatial patterns while still enabling the definition of entropy of time slices. The entropy reveals the degree of randomness of a time slice, helping users to identify expected and unexpected phenomena over time. The effectiveness of our approach appears in applications involving synthetic and real data sets, which show that the proposed methodology is able to uncover interesting spatial and temporal phenomena. The provided examples and case studies make clear the usefulness of our approach as a mechanism to support visual analytic tasks involving spatio-temporal data. [ABSTRACT FROM AUTHOR]

Contrast enhancement plays a vital role in image processing and computer vision applications. However, the existing work inevitably introduces some unwanted artifacts and often fails in revealing image details when enhancing the invisible images. This article presents a novel Bezier curve modification scheme to improve the visual quality of contrast degraded imperceptible images. First, a salp swarm algorithm is incorporated to compute an optimal threshold value using weighted cumulative distribution function for adjusting the transformation in bright and dark areas separately. Then, an optimized Bezier curve uses a novel regularization parameter to improve the fine details of bright and dark regions, respectively. The experiential results are provided to demonstrate that the proposed method yields better quantitative results compared with the existing methods. Extensive experiments on different data sets demonstrate that the proposed method produces an enhanced image with superior visual quality for all five levels of contrast distorted images. [ABSTRACT FROM AUTHOR]

Plant disease diagnosis is very critical for agriculture due to its importance for increasing crop production. Recent advances in image processing offer us a new way to solve this issue via visual plant disease analysis. However, there are few works in this area, not to mention systematic researches. In this paper, we systematically investigate the problem of visual plant disease recognition for plant disease diagnosis. Compared with other types of images, plant disease images generally exhibit randomly distributed lesions, diverse symptoms and complex backgrounds, and thus are hard to capture discriminative information. To facilitate the plant disease recognition research, we construct a new large-scale plant disease dataset with 271 plant disease categories and 220,592 images. Based on this dataset, we tackle plant disease recognition via reweighting both visual regions and loss to emphasize diseased parts. We first compute the weights of all the divided patches from each image based on the cluster distribution of these patches to indicate the discriminative level of each patch. Then we allocate the weight to each loss for each patch-label pair during weakly-supervised training to enable discriminative disease part learning. We finally extract patch features from the network trained with loss reweighting, and utilize the LSTM network to encode the weighed patch feature sequence into a comprehensive feature representation. Extensive evaluations on this dataset and another public dataset demonstrate the advantage of the proposed method. We expect this research will further the agenda of plant disease recognition in the community of image processing. [ABSTRACT FROM AUTHOR]

It is challenging to extend the support region of state-of-the-art local edge-preserving filtering approaches to the entire input image on account of huge memory cost and heavy computational burden. In this paper, we propose an O(N) time recursive non-local edge-aware filter. A novel graph and a linear time algorithm are presented to effectively propagate information across the entire image. In this graph, information is propagated along all directions to avoid visual artifacts. Both the intensity similarity and the spatial affinity are utilized to estimate the similarity of neighboring pixels, and the distance of any two pixels is the length of the path between these two pixels on a binary tree. Specifically, the input image is filtered at four directions, namely left-to-right, right-to-left, top-to-bottom and bottom-to-top. In each pass, the un-normalized output and the normalization constant of the root node are computed recursively from leaf nodes to the root node on a binary tree in linear time. The filtering output is the average of the outputs for these four directions. A comparison with other non-local edge-aware filters is presented to show the advantages of our approach. Extensive experiments demonstrate that our recursive non-local edge-preserving filter is effective in a variety of computer vision and image processing applications, including edge-aware filtering, detail enhancement, stylization, and stereo matching. [ABSTRACT FROM AUTHOR]

In this letter, we describe and validate the first fully automatic parameter optimization for thermal synthetic aperture visualization. It replaces previous manual exploration of the parameter space, which is time-consuming and error-prone. We prove that the visibility of targets in thermal integral images is proportional to the variance of the targets’ image. Since this is invariant to occlusion, it represents a suitable objective function for optimization. Our findings have the potential to enable fully autonomous search and recuse operations with camera drones. [ABSTRACT FROM AUTHOR]

The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this article, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering. [ABSTRACT FROM AUTHOR]

Fog removal from an image is an active research topic in computer vision. However, current literature is weak in the following two areas which in many ways are hindering progress for developing defogging algorithms. First, there is no true real-world and naturally occurring foggy image datasets suitable for developing defogging models. Second, there is no suitable mathematically simple and easy to use image quality assessment (IQA) methods for evaluating the visual quality of defogged images. We address these two aspects in this paper. We first introduce a new foggy image dataset called multiple real-world foggy image dataset (MRFID). MRFID contains foggy and clear images of 200 outdoor scenes. For each scene, one clear image and 4 foggy images of different densities defined as slightly foggy, moderately foggy, highly foggy, and extremely foggy, are manually selected from images taken from these scenes over the course of one calendar year. We then process the foggy images of MRFID using 16 defogging methods to obtain 12,800 defogged images (DFIs) and perform a comprehensive subjective evaluation of the visual quality of the DFIs. Through collecting the mean opinion score (MOS) of 120 subjects and evaluating a variety of fog-relevant image features, we have developed a new Fog-relevant Feature based SIMilarity index (FRFSIM) for assessing the visual quality of DFIs. We present extensive experimental results to show that our new visual quality assessment measure, the FRFSIM, is more consistent with the MOS than other IQA methods and is therefore more suitable for evaluating defogged images than other state-of-the-art IQA methods. Our dataset and relevant code are available at http://www.vistalab.ac.cn/MRFID-for-defogging/. [ABSTRACT FROM AUTHOR]

Aiming at the challenges of networked visual servo control systems, which rarely consider network communication duration and image processing computational cost simultaneously, we here propose a novel platform for networked inverted pendulum visual servo control using ${H_{\infty } }$ analysis. Unlike most of the existing methods that usually ignore computational costs involved in measuring, actuating, and controlling, we design a novel event-triggered sampling mechanism that applies a new closed-loop strategy to dealing with networked inverted pendulum visual servo systems of multiple time-varying delays and computational errors. Using the Lyapunov stability theory, we prove that the proposed system can achieve stability whilst compromising image-induced computational and network-induced delays and system performance. In the meantime, we use ${H_{\infty }}$ disturbance attenuation level $\gamma $ for evaluating the computational errors, whereas the corresponding ${H_{\infty }}$ controller is implemented. Finally, simulation analysis and experimental results demonstrate the proposed system performance in reducing computational errors whilst maintaining system efficiency and robustness. [ABSTRACT FROM AUTHOR]

The complexity of today's visualization applications demands specific visualization systems tailored for the development of these applications. Frequently, such systems utilize levels of abstraction to improve the application development process, for instance by providing a data flow network editor. Unfortunately, these abstractions result in several issues, which need to be circumvented through an abstraction-centered system design. Often, a high level of abstraction hides low level details, which makes it difficult to directly access the underlying computing platform, which would be important to achieve an optimal performance. Therefore, we propose a layer structure developed for modern and sustainable visualization systems allowing developers to interact with all contained abstraction levels. We refer to this interaction capabilities as usage abstraction levels, since we target application developers with various levels of experience. We formulate the requirements for such a system, derive the desired architecture, and present how the concepts have been exemplary realized within the Inviwo visualization system. Furthermore, we address several specific challenges that arise during the realization of such a layered architecture, such as communication between different computing platforms, performance centered encapsulation, as well as layer-independent development by supporting cross layer documentation and debugging capabilities. [ABSTRACT FROM AUTHOR]

The quality detection of pharmaceutical liquid products is inevitable and crucial in drug manufacture because drugs contaminated with foreign particles are definitely not to be used. However, with the current detection methods, it is still a challenge to detect and identify the small moving particles using an imaging system. In this article, a deep multimodel cascade method combining single-frame image and multiframe images processing method to detect and identify foreign particles is proposed. The proposed method consists of three stages. First, a Faster R-CNN convolutional neural network is adopted to detect and localize the multiple suspected foreign particles of each single-frame image. Then, the k-means clustering algorithm is used to cluster the trail of that detected multiple suspected foreign particles in the eight sequential images to obtain the moving object trajectory. Finally, trajectory features are extracted and the random forest (RF) classifier is used to distinguish noises and foreign particles according to the motion feature of the moving object trajectory. Experimental results demonstrate that the proposed multitask stepwise method improves the accuracy of foreign particles detection and reduces the rate of omission in the case of strong noise, which proves the effectiveness of this method. [ABSTRACT FROM AUTHOR]

With an increasing number of remote sensing images (RSIs), the automatic region of interest (ROI) extraction based on convolutional neural networks (CNNs) has attracted much interest in recent years. Although fully supervised CNN-based methods have shown superiority in the field of object extraction, pixelwise annotations are expensive and time-consuming. Moreover, due to the unstable distribution of ROIs in complex landscapes, the ratios of foreground and background areas are quite different in RSIs. Training CNNs with such unbalanced data sets lead to over-fitting and low accuracy. In this article, we propose a framework that combines multiview learning and attention mechanism (MLAM) to solve the above mentioned problems. First, we develop a CNN-based weakly supervised method with a weight-balanced loss function to solve the problems caused by an unbalanced data set. It also helps to generate imagewise saliency maps by computing the gradient maps with respect to the input images. Then, we design a multiview strategy to dramatically reduce the missing inspection. Finally, we design a feedback attention mechanism based on the stage neighbor binary pattern to further modify the extraction result. In summary, the proposed framework achieves pixelwise ROI extraction under imagewise annotations through data-driven ML and a knowledge-driven visual attention mechanism. We evaluate the performance of the MLAM framework on two challenging data sets with complex backgrounds. The experimental results indicate that the proposed framework can achieve better performance than other eight ROI extraction models for unbalanced remote sensing data sets. [ABSTRACT FROM AUTHOR]

Generating realistic images with the guidance of reference images and human poses is challenging. Despite the success of previous works on synthesizing person images in the iconic views, no efforts are made towards the task of pose-guided image synthesis in the non-iconic views. Particularly, we find that previous models cannot handle such a complex task, where the person images are captured in the non-iconic views by commercially-available digital cameras. To this end, we propose a new framework – Multi-branch Refinement Network (MR-Net), which utilizes several visual cues, including target person poses, foreground person body and scene images parsed. Furthermore, a novel Region of Interest (RoI) perceptual loss is proposed to optimize the MR-Net. Extensive experiments on two non-iconic datasets, Penn Action and BBC-Pose, as well as an iconic dataset – Market-1501, show the efficacy of the proposed model that can tackle the problem of pose-guided person image generation from the non-iconic views. The data, models, and codes are downloadable from https://github.com/loadder/MR-Net. [ABSTRACT FROM AUTHOR]

This paper presents a new approach to recognizing vanishing-point-constrained building planes from a single image of street view. We first design a novel convolutional neural network (CNN) architecture that generates geometric segmentation of per-pixel orientations from a single street-view image. The network combines two-stream features of general visual cues and surface normals in gated convolution layers, and employs a deeply supervised loss that encapsulates multi-scale convolutional features. Our experiments on a new benchmark with fine-grained plane segmentations of real-world street views show that our network outperforms state-of-the-arts methods of both semantic and geometric segmentation. The pixel-wise segmentation exhibits coarse boundaries and discontinuities. We then propose to rectify the pixel-wise segmentation into perspectively-projected quads based on spatial proximity between the segmentation masks and exterior line segments detected through an image processing. We demonstrate how the results can be utilized to perspectively overlay images and icons on building planes in input photos, and provide visual cues for various applications. [ABSTRACT FROM AUTHOR]

Image metrics based on Human Visual System (HVS) play a remarkable role in the evaluation of complex image processing algorithms. However, mimicking the HVS is known to be complex and computationally expensive (both in terms of time and memory), and its usage is thus limited to a few applications and to small input data. All of this makes such metrics not fully attractive in real-world scenarios. To address these issues, we propose Deep Image Quality Metric (DIQM), a deep-learning approach to learn the global image quality feature (mean-opinion-score). DIQM can emulate existing visual metrics efficiently, reducing the computational costs by more than an order of magnitude with respect to existing implementations. [ABSTRACT FROM AUTHOR]

Automatic prediction of the aesthetic quality of a photograph has been an important research problem in image processing and computer vision. While assessing the aesthetic quality of a photograph by human is highly subjective, most existing studies have considered only objective (or general) opinion of multiple viewers. In this paper, we provide a comprehensive investigation of the issue of subjectivity in aesthetic quality assessment using a large-scale database containing photos, user ratings, and user comments. First, we analyze how the mean aesthetic quality level and the level of subjectivity have evolved over time. Second, we examine the feasibility of automatic prediction of the level of subjectivity based on visual features, and identify which features are effective for the prediction. Third, we analyze the users’ comments given to photos to understand the sources of subjectivity of aesthetic quality rating. Our results show that several factors are simultaneously involved in determining the level of subjectivity of a photo, but it can be predicted with reasonable accuracy. We believe that our results provide insight toward personalized aesthetic photo applications. [ABSTRACT FROM AUTHOR]

Recently, food recognition has received more and more attention in image processing and computer vision for its great potential applications in human health. Most of the existing methods directly extracted deep visual features via convolutional neural networks (CNNs) for food recognition. Such methods ignore the characteristics of food images and are, thus, hard to achieve optimal recognition performance. In contrast to general object recognition, food images typically do not exhibit distinctive spatial arrangement and common semantic patterns. In this paper, we propose a multi-scale multi-view feature aggregation (MSMVFA) scheme for food recognition. MSMVFA can aggregate high-level semantic features, mid-level attribute features, and deep visual features into a unified representation. These three types of features describe the food image from different granularity. Therefore, the aggregated features can capture the semantics of food images with the greatest probability. For that solution, we utilize additional ingredient knowledge to obtain mid-level attribute representation via ingredient-supervised CNNs. High-level semantic features and deep visual features are extracted from class-supervised CNNs. Considering food images do not exhibit distinctive spatial layout in many cases, MSMVFA fuses multi-scale CNN activations for each type of features to make aggregated features more discriminative and invariable to geometrical deformation. Finally, the aggregated features are more robust, comprehensive, and discriminative via two-level fusion, namely multi-scale fusion for each type of features and multi-view aggregation for different types of features. In addition, MSMVFA is general and different deep networks can be easily applied into this scheme. Extensive experiments and evaluations demonstrate that our method achieves state-of-the-art recognition performance on three popular large-scale food benchmark datasets in Top-1 recognition accuracy. Furthermore, we expect this paper will further the agenda of food recognition in the community of image processing and computer vision. [ABSTRACT FROM AUTHOR]

This paper aims to investigate the method of autonomous hovering control for the biomimetic underwater robot named RobCutt-II. First, a modular configuration for mimicry of undulatory propulsion is presented, followed by a coordinated control of dual fins for enhancing the maneuverability and stability of the RobCutt-II. Then the vision-based hovering problem is formulated. A dual-loop hovering control framework integrating an image processing module, a tracking differentiator, a hovering controller, a velocity controller, and a fuzzy-based control allocation module is proposed to govern the locomotion of the RobCutt-II. Finally, simulations and experimental results demonstrate the effectiveness of the proposed mechanical design and control methods. [ABSTRACT FROM AUTHOR]

Age of information has been recently proposed to quantify the freshness of information, e.g., in cyber-physical systems, where it is of critical importance. Motivated by wireless camera networks where multi-view image processing is required, in this paper we propose to extend the concept of age of information to capture packets carrying correlated data. We consider a system consisting of wireless camera nodes with overlapping fields of view and a set of processing nodes, and address the problem of the joint optimization of processing node assignment and camera transmission scheduling, so as to minimize the maximum peak age of information from all sources. We formulate the multi-view age minimization (MVAM) problem, and prove its NP-hardness under the two widely used interference models as well as with given candidate transmitting groups. We provide fundamental results including tractable cases and optimality conditions of the MVAM problem for two baseline scenarios. To solve MVAM efficiently, we develop an optimization algorithm based on a decomposition approach. Numerical results show that by employing our approach the maximum peak age is significantly reduced in comparison to a traditional centralized solution with minimum-time scheduling. [ABSTRACT FROM AUTHOR]

The quality detection of pharmaceutical liquid products is inevitable and crucial in drug manufacture because drugs contaminated with foreign particles are definitely not to be used. However, with the current detection methods, it is still a challenge to detect and identify the small moving particles using an imaging system. In this article, a deep multimodel cascade method combining single-frame image and multiframe images processing method to detect and identify foreign particles is proposed. The proposed method consists of three stages. First, a Faster R-CNN convolutional neural network is adopted to detect and localize the multiple suspected foreign particles of each single-frame image. Then, the $k$ -means clustering algorithm is used to cluster the trail of that detected multiple suspected foreign particles in the eight sequential images to obtain the moving object trajectory. Finally, trajectory features are extracted and the random forest (RF) classifier is used to distinguish noises and foreign particles according to the motion feature of the moving object trajectory. Experimental results demonstrate that the proposed multitask stepwise method improves the accuracy of foreign particles detection and reduces the rate of omission in the case of strong noise, which proves the effectiveness of this method. [ABSTRACT FROM AUTHOR]

A persistent problem in the three-dimensional (3-D) reconstruction technique is to eliminate blank areas in the 3-D map, which commonly emerges after removing undesired objects, such as dynamic targets or occluded areas. This task is challenging for it is difficult to acquire the coherence between color and depth information, which are both lost for each pixel in the target region. Moreover, creating novel artifacts also needs to be avoided during the completion process. To address these problems, in this paper, we propose a collaborative method to complete the lost area in the color image and its corresponding depth map. In the proposed method, an examplar-based image inpainting technique combined with planarity knowledge is adopted to iteratively repair the texture and structure information. The color completion process provides candidates for plane-fitting on segmented superpixels. The depth filling step extracts image areas requiring reinpainting, which is defined by their gradient differences. The final 3-D map can then be reconstructed from both completed color and depth images. Experiments on real-world scenes demonstrate the success of our method in the completion of 3-D maps. [ABSTRACT FROM AUTHOR]

The goal of image pre-compensation is to process an image such that after being convolved with a known kernel, will appear close to the sharp reference image. In a practical setting, the pre-compensated image has significantly higher dynamic range than the latent image. As a result, some form of tone mapping is needed. In this paper, we show how global tone mapping functions affect contrast and ringing in image pre-compensation. We further enhance contrast and reduce ringing by considering the visual saliency. Specifically, we prioritize contrast preservation in salient regions while tolerating more blurriness elsewhere. For quantitative analysis, we design new metrics to measure the contrast of an image with ringing. Specifically, we set out to find its "equivalent ringing-free" image that matches its intensity histogram and uses its contrast as the measure. We illustrate our approach on projector defocus compensation and visual acuity enhancement. Compared with the state-of-the-art, our approach significantly improves the contrast. We also perform user studies to demonstrate that our method can effectively improve the viewing experience for users with impaired vision. [ABSTRACT FROM AUTHOR]

The internal inspection of large pipeline infrastructures, such as sewers and waterworks, is a fundamental task for the prevention of possible failures. In particular, visual inspection is typically performed by human operators on the basis of video sequences either acquired on-line or recorded for further off-line analysis. In this work, we propose a vision-based software approach to assist the human operator by conveniently showing the acquired data and by automatically detecting and highlighting the pipeline sections where relevant anomalies could occur. [ABSTRACT FROM AUTHOR]

Thermal infrared imaging is an emerging modality that has gained increasing interest in recent years, mostly due to technical advances resulting in the availability of affordable microbolometer-based IR imaging sensors. However, while sensors are widely available, algorithms for thermal image processing still lack robustness and accuracy when compared to their RGB counterparts. Current methods developed for RGB data make use of machine learning algorithms that require large amounts of labeled images which are currently not available for the thermal domain. In this paper, we address the question whether providing a large number of labeled images would allow the application of current image processing methods on the example of solving challenging face analysis tasks. We introduce a high-resolution thermal facial image database with extensive manual annotations and explore how it can be used to adapt methods from the visual domain for infrared images. In addition, we extend existing approaches for infrared landmark detection with a head pose estimation for improved robustness and analyze the performance of a deep learning method on this task. An evaluation of algorithm performance shows that learning algorithms either outperform available solutions or allow completely new applications that could previously not be addressed. As a conclusion, we prove that investing the effort into acquiring appropriate training data and adapting competitive algorithms is not only a viable approach in analysing thermal infrared images but can also allow outperforming specific task-designed solutions. The database is freely available for academic use at https://github.com/marcinkopaczka/thermalfaceproject. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNNs) have been applied to visual tracking with demonstrated success in recent years. Most CNN-based trackers utilize hierarchical features extracted from a certain layer to represent the target. However, features from a certain layer are not always effective for distinguishing the target object from the backgrounds especially in the presence of complicated interfering factors (e.g., heavy occlusion, background clutter, illumination variation, and shape deformation). In this work, we propose a CNN-based tracking algorithm which hedges deep features from different CNN layers to better distinguish target objects and background clutters. Correlation filters are applied to feature maps of each CNN layer to construct a weak tracker, and all weak trackers are hedged into a strong one. For robust visual tracking, we propose a hedge method to adaptively determine weights of weak classifiers by considering both the difference between the historical as well as instantaneous performance, and the difference among all weak trackers over time. In addition, we design a Siamese network to define the loss of each weak tracker for the proposed hedge method. Extensive experiments on large benchmark datasets demonstrate the effectiveness of the proposed algorithm against the state-of-the-art tracking methods. [ABSTRACT FROM AUTHOR]

In this paper, a deep mixture of diverse experts algorithm is developed to achieve more efficient learning of a huge (mixture) network for large-scale visual recognition application. First, a two-layer ontology is constructed to assign large numbers of atomic object classes into a set of task groups according to the similarities of their learning complexities, where certain degrees of inter-group task overlapping are allowed to enable sufficient inter-group message passing. Second, one particular base deep CNNs with $M+1$ M + 1 outputs is learned for each task group to recognize its $M$ M atomic object classes and identify one special class of “not-in-group”, where the network structure (numbers of layers and units in each layer) of the well-designed deep CNNs (such as AlexNet, VGG, GoogleNet, ResNet) is directly used to configure such base deep CNNs. For enhancing the separability of the atomic object classes in the same task group, two approaches are developed to learn more discriminative base deep CNNs: (a) our deep multi-task learning algorithm that can effectively exploit the inter-class visual similarities; (b) our two-layer network cascade approach that can improve the accuracy rates for the hard object classes at certain degrees while effectively maintaining the high accuracy rates for the easy ones. Finally, all these complementary base deep CNNs with diverse but overlapped outputs are seamlessly combined to generate a mixture network with larger outputs for recognizing tens of thousands of atomic object classes. Our experimental results have demonstrated that our deep mixture of diverse experts algorithm can achieve very competitive results on large-scale visual recognition. [ABSTRACT FROM AUTHOR]

We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being sufficiently grounded in vision to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person real-time chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and consists of $\sim$ ∼ 1.2M dialog question-answer pairs from 10-round, human-human dialogs grounded in $\sim$ ∼ 120k images from the COCO dataset. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders—Late Fusion, Hierarchical Recurrent Encoder and Memory Network (optionally with attention over image features)—and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank and recall $@k$ @ k of human response. We quantify the gap between machine and human performance on the Visual Dialog task via human studies. Putting it all together, we demonstrate the first ‘visual chatbot’! Our dataset, code, pretrained models and visual chatbot are available on https://visualdialog.org. [ABSTRACT FROM AUTHOR]