Search

Multispectral imaging is a technique that captures data across several bands of the light spectrum, and it can be useful in many computer vision fields, including color constancy. We propose a method that exploits multispectral imaging for illuminant estimation, and then applies illuminant correction in the raw RGB domain to achieve computational color constancy. Our proposed method is composed of two steps: first, a selected number of existing camera-independent algorithms for illuminant estimation, originally designed for RGB data, are applied in generalized form to work with multispectral data. We demonstrate that the sole multispectral extension of such algorithms is not sufficient to achieve color constancy, and thus we introduce a second step, in which we re-elaborate the multispectral estimations before conversion into raw RGB with the use of the camera response function. Our results on the NUS dataset show that an improvement of 60% in the color constancy performance, measured in terms of reproduction angular error, can be obtained according to our method when compared to the traditional raw RGB pipeline.

We introduce a method that enhances RGB color constancy accuracy by combining neural network and k-means clustering techniques. Our approach stands out from previous works because we combine multispectral and color information together to estimate illuminants. Furthermore, we investigate the combination of the illuminant estimation in the RGB color and in the spectral domains, as a strategy to provide a refined estimation in the RGB color domain. Our investigation can be divided into three main points: (1) identify the spatial resolution for sampling the input image in terms of RGB color and spectral information that brings the highest performance; (2) determine whether it is more effective to predict the illuminant in the spectral or in the RGB color domain, and finally, (3) assuming that the illuminant is in fact predicted in the spectral domain, investigate if it is better to have a loss function defined in the RGB color or spectral domain. Experimental results are carried out on NUS: a standard dataset of multispectral radiance images with an annotated spectral global illuminant. Among the several considered options, the best results are obtained with a model trained to predict the illuminant in the spectral domain using an RGB color loss function. In terms of comparison with the state of the art, this solution improves the recovery angular error metric by 66% compared to the best tested spectral method, and by 41% compared to the best tested RGB method.

Semantic segmentation consists of classifying each pixel of an image and constitutes an essential step towards scene recognition and understanding. Deep convolutional encoder–decoder neural networks now constitute state-of-the-art methods in the field of semantic segmentation. The problem of street scenes’ segmentation for automotive applications constitutes an important application field of such networks and introduces a set of imperative exigencies. Since the models need to be executed on self-driving vehicles to make fast decisions in response to a constantly changing environment, they are not only expected to operate reliably but also to process the input images rapidly. In this paper, we explore genetic programming (GP) as a meta-model that combines four different efficiency-oriented networks for the analysis of urban scenes. Notably, we present and examine two approaches. In the first approach, we represent solutions as GP trees that combine networks’ outputs such that each output class’s prediction is obtained through the same meta-model. In the second approach, we propose representing solutions as lists of GP trees, each designed to provide a unique meta-model for a given target class. The main objective is to develop efficient and accurate combination models that could be easily interpreted, therefore allowing gathering some hints on how to improve the existing networks. The experiments performed on the Cityscapes dataset of urban scene images with semantic pixel-wise annotations confirm the effectiveness of the proposed approach. Specifically, our best-performing models improve systems’ generalization ability by approximately 5% compared to traditional ensembles, 30% for the less performing state-of-the-art CNN and show competitive results with respect to state-of-the-art ensembles. Additionally, they are small in size, allow interpretability, and use fewer features due to GP’s automatic feature selection.

Algorithms for computational color constancy are usually compared in terms of the angular error between ground truth and estimated illuminant. Despite its wide adoption, there exists no well-defined consensus on acceptability and/or noticeability thresholds in angular errors. One of the main reasons for this lack of consensus is that angular error weighs all hues equally by performing the comparison in a non-perceptual color space, whereas the sensitivity of the human visual system is known to vary depending on the chromaticity. We therefore propose a visualization strategy that presents simultaneously the angular error (preserved due to its wide adoption in the field), and a perceptual error (to convey information about its actual perceived impact). This is achieved by exploiting the angle-retaining chromaticity diagram, which shows errors in chromaticities while encoding RGB angular distances as 2D Euclidean distances, and by embedding contour lines of perceptual color differences at standard predefined thresholds. Example applications are shown for different color constancy methods on two imaging devices.

This paper presents a review of the NTIRE 2023 challenge on night photography rendering. The goal of the challenge was to find solutions that process raw camera images taken in nighttime conditions conditions, and thereby produce a photo-quality output images in the standard RGB (sRGB) space. Unlike the previous year's competition, participants were not provided with a large training dataset for the target sensor. Instead, this time they were given images of a color checker illuminated by a known light source. To evaluate the results, a sufficient number of viewers were asked to assess the visual quality of the proposed solutions, considering the subjective nature of the task. The highest ranking solutions were further ranked by Richard Collins, a renowned photographer. The top ranking participants' solutions effectively represent the state-of-the-art in nighttime photography rendering. More results can be found at https://nightimaging.org

This paper reviews the Challenge on Super-Resolution of Compressed Image and Video at AIM 2022. This challenge includes two tracks. Track 1 aims at the super-resolution of compressed image, and Track 2 targets the super-resolution of compressed video. In Track 1, we use the popular dataset DIV2K as the training, validation and test sets. In Track 2, we propose the LDV 3.0 dataset, which contains 365 videos, including the LDV 2.0 dataset (335 videos) and 30 additional videos. In this challenge, there are 12 teams and 2 teams that submitted the final results to Track 1 and Track 2, respectively. The proposed methods and solutions gauge the state-of-the-art of super-resolution on compressed image and video. The proposed LDV 3.0 dataset is available at https://github.com/RenYang-home/LDV_dataset. The homepage of this challenge is at https://github.com/RenYang-home/AIM22_CompressSR.

The growing production of digital content and its dissemination across the worldwide web require eficient and precise management. In this context, image quality assessment measures (IQAMs) play a pivotal role in guiding the development of numerous image processing systems for compression, enhancement, and restoration. The structural similarity index (SSIM) is one of the most common IQAMs for estimating the similarity between a pristine reference image and its corrupted variant. The multi-scale SSIM is one of its most popular variants that allows assessing image quality at multiple spatial scales. This paper proposes a two-stage genetic programming (GP) approach to evolve novel multi-scale IQAMs, that are simultaneously more effective and efficient. We use GP to perform feature selection in the first stage, while the second stage generates the final solutions. The experimental results show that the proposed approach outperforms the existing MS-SSIM. A comprehensive analysis of the feature selection indicates that, for extracting multi-scale similarities, spatially-varying convolutions are more effective than dilated convolutions. Moreover, we provide evidence that the IQAMs learned for one database can be successfully transferred to previously unseen databases. We conclude the paper by presenting a set of evolved multi-scale IQAMs and providing their interpretation.

The recognition of logos can be useful in developing autonomous checkout systems, or monitoring brand presence and advertisement in shopping malls. The continuous generation and update of new brand logos imposes the definition of a flexible solution to the problem. We therefore define a two-stage logo recognition system composed of an agnostic logo detector, to locate image regions that possess generic logo-like characteristics, and an incremental logo classifier, to progressively update the set of known logo classes. We investigate our solution's sensitivity to regularization and availability of training samples, and we develop two alternative techniques for model compression. Results are presented and compared with state of the art solutions, showing promising results. Our code is made available for public download.

In this paper we propose the truncated edge-based color constancy. It is based on, and extends, the edge-based framework by introducing the use of truncated Gaussian filters. The truncation level can be controlled with the use of a dedicated parameter that is added to the other three parameters existing in the edge-based framework, namely the derivative order, the standard deviation of the Gaussian filter, and the Minkowski norm. Experimental results on two standard dataset for color constancy show that the truncated edge-based framework allows to achieve the same or higher illuminant estimation accuracy of the edge-based framework considerably reducing the number of operations.

Several contemporaneous image processing and computer vision systems rely upon the full-reference image quality assessment (IQA) measures. The single-scale structural similarity index (SS-SSIM) is one of the most popular measures, and it owes its success to the mathematical simplicity, low computational complexity, and implicit incorporation of Human Visual System's (HVS) characteristics. In this paper, we revise the original parameters of SSIM and its multi-scale counterpart (MS-SSIM) to increase their correlation with subjective evaluation. More specifically, we exploit the evolutionary computation and the swarm intelligence methods on five popular IQA databases, two of which are dedicated distance-changed databases, to determine the best combination of parameters efficiently. Simultaneously, we explore the effect of different scale selection approaches in the context of SS-SSIM. The experimental results show that with a proper fine-tuning (1) the performance of SS-SSIM and MS-SSIM can be improved, in average terms, by 8% and by 3%, respectively, (2) the SS-SSIM after the so-called standard scale selection achieves similar performance as if applying computationally more expensive state-of-the-art scale selection methods or MS-SSIM; moreover, (3) there is evidence that the parameters learned on a given database can be successfully transferred to other (previously unseen) databases; finally, (4) we propose a new set of reference parameters for SSIM's variants and provide their interpretation.

Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation - which we call Planckian Jitter - that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets. In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.

Food recognition is a major challenge in the field of computer vision, requiring models that can effectively handle the wide variability and complexity of food images. In this paper, we explore the use of vision transformers, a category of models based on self-attention mechanisms, to address the task of food recognition. We focus on training and fine-tuning different vision transformer architectures on Food2K, a large-scale dataset of food images with 2,000 categories. We compare the performance of vision transformers with convolutional neural networks (CNNs) on Food2K and Food101. In addition, we use state-of-the-art explainability techniques to highlight the regions of interest that vision transformers take into account when performing a prediction. Our results show that vision transformers can achieve competitive results on food recognition tasks, with the added benefit that pre-training on Food2K improve their generalization capabilities and interpretability. This study highlights the potential of vision transformers in food computing, paving the way for future research in this field.

Road direction signs depict textual and directional information to convey instructions about how to reach a given destination. Detecting and interpreting road direction signs constitutes a key component in self-driving vehicles, therefore we propose a hybrid pipeline that combines deep learning with traditional handcrafted image processing, aiming for a combination of effectiveness and efficiency. Our solution includes a procedure for the identification of the orientation of arrows, generalizing on a wide variety of pictorial styles for direction signs across the globe. Our pipeline is evaluated in terms of accuracy and inference time of its individual steps, demonstrating excellent performance. Experiments are performed over two variants of the pipeline, assuming the availability of different levels of computational resources. We also test the system's dependence on annotated supervision by performing evaluation with a varying number of training instances.

Enhancing night photography images is a challenging task that requires advanced processing techniques. While CNN-based methods have shown promising results, their high computational requirements and limited interpretability can pose challenges. To address these limitations, we propose a camera pipeline for rendering visually pleasing photographs in low-light conditions. Our approach is characterized by a shallow structure, explainable steps, and a low parameter count, resulting in computationally efficient processing. We compared the proposed pipeline with recent CNN-based state-of-the-art approaches for low-light image enhancement, showing that our approach produces more aesthetically pleasing results. The psycho-visual comparisons conducted in this work show how our proposed solution is preferred with respect to the other methods (in about 44% of the cases our solution has been chosen, compared to only about 15% of the cases for the state-of-the-art best method).

Rendering night photography pictures is a challenging task that requires advanced processing techniques. Although deep learning-based Image Signal Processing (ISP) pipelines have shown promising results, current limitations are set by the lack of proper nighttime image datasets, their high computational requirements, and low explainability. In this paper, we propose a traditional ISP pipeline for rendering visually pleasing photographs of night scenes. Our pipeline is comprised of various algorithms addressing the different challenges presented by night images, and it is characterized by a shallow structure, explainable steps, and a low parameter count, resulting in computationally efficient processing. Moreover, it does not require training data. Experiments show that our pipeline can produce more pleasing results compared to other deep learning-based ISP pipelines, as it won first place in people's choice track and third place in photographer's choice track in the NTIRE 2023 Night Photography Rendering Challenge.

We present a unified deep learning framework for the recognition of user identity and the recognition of imagined actions, based on electroencephalography (EEG) signals, for application as a brain-computer interface. Our solution exploits a novel shifted subsampling preprocessing step as a form of data augmentation, and a matrix representation to encode the inherent local spatial relationships of multielectrode EEG signals. The resulting image-like data are then fed to a convolutional neural network to process the local spatial dependencies, and eventually analyzed through a bidirectional long-short term memory module to focus on temporal relationships. Our solution is compared against several methods in the state of the art, showing comparable or superior performance on different tasks. Specifically, we achieve accuracy levels above 90% both for action and user classification tasks. In terms of user identification, we reach 0.39% equal error rate in the case of known users and gestures, and 6.16% in the more challenging case of unknown users and gestures. Preliminary experiments are also conducted in order to direct future works toward everyday applications relying on a reduced set of EEG electrodes.

Full-reference image quality measures are a fundamental tool to approximate the human visual system in various applications for digital data management: from retrieval to compression to detection of unauthorized uses. Inspired by both the effectiveness and the simplicity of hand-crafted Structural Similarity Index Measure (SSIM), in this work, we present a framework for the formulation of SSIM-like image quality measures through genetic programming. We explore different terminal sets, defined from the building blocks of structural similarity at different levels of abstraction, and we propose a two-stage genetic optimization that exploits hoist mutation to constrain the complexity of the solutions. Our optimized measures are selected through a cross-dataset validation procedure, which results in superior performance against different versions of structural similarity, measured as correlation with human mean opinion scores. We also demonstrate how, by tuning on specific datasets, it is possible to obtain solutions that are competitive with (or even outperform) more complex image quality measures.

Annotated datasets for automatic white balance (AWB) are used for the evaluation and, when necessary, the training, of AWB methods. Relying on such datasets requires awareness of the potential bias in their content and characteristics: some methods are designed to rely on the presence of particular elements, such as human skin, while other methods learn implicit relationships between image content and light properties from training data. The dependency on these relationships makes it fundamental to understand whether the available datasets are actually representative of common application scenarios, such as the presence of human subjects, the diversity of composition, or the illumination conditions. In this paper we overview the most common datasets for Automatic White Balance, including those for single as well as multiple illuminant estimation, providing a critical analysis on their characteristics. Furthermore, we identify a number of existing methods for single illuminant estimation, as a representative pool of approaches to the problem with various levels of complexity. We investigate how the performance of these correlate to the image content of common datasets.

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

The first paper investigating the use of machine learning to learn the relationship between an image of a scene and the color of the scene illuminant was published by Funt et al. in 1996. Specifically, they investigated if such a relationship could be learned by a neural network. During the last 30 years we have witnessed a remarkable series of advancements in machine learning, and in particular deep learning approaches based on artificial neural networks. In this paper we want to update the method by Funt et al. by including recent techniques introduced to train deep neural networks. Experimental results on a standard dataset show how the updated version can improve the median angular error in illuminant estimation by almost 51% with respect to its original formulation, even outperforming recent illuminant estimation methods.

A pipeline for the generation of synthetic dataset of spectral scenes, with corresponding sensor readings, is here proposed. The pipeline is composed of two main parts: Part 1: Image pixel reflectance assignment. Individual pixels from an input sRGB image dataset are replaced with appropriate reflectance spectra from a given non-image reflectance dataset. The resulting dataset of reflectance images is considered the starting point for simulated sensor acquisition. Part 2: Simulated sensor acquisition. Each spectral reflectance image in the dataset is illuminated with an illuminant spectra to produce a radiance image. The resulting dataset of radiance images is then synthetically read from the simulated sensors (camera and ambient multispectral sensor) of the Huawei P50 phone, using the corresponding sensors transmittance information. The capability of generating any large-scale, diverse, and annotated synthetic spectral datasets can facilitate the development of data-driven imaging algorithms, and foster reproducible research.

This paper presents a novel approach for spectral illuminant correction in smartphone imaging systems, aiming to improve color accuracy and enhance image quality. The methods introduced include Spectral Super Resolution and Weighted Spectral Color Correction (W-SCC). These techniques leverage the spectral information of both the image and the illuminant to perform effective color correction. Experimental evaluations were conducted using a dataset of 100 synthetic images, whose acquisition is simulated using the transmittance information of a Huawei P50 smartphone camera sensor and an Ambient light Multispectral Sensor (AMS). The results demonstrate the superiority of the proposed methods compared to traditional trichromatic pipelines, achieving significant reductions in colorimetric errors measured in terms of ∆E94 units. The W-SCC technique, in particular, incorporates per-wavelength weight optimization, further enhancing the accuracy of spectral illuminant correction. The presented approaches have valuable applications in various fields, including color analysis, computer vision, and image processing. Future research directions may involve exploring additional optimization techniques and incorporating advanced machine learning algorithms to further advance spectral illuminant correction in smartphone imaging systems.

Computational color constancy algorithms are commonly evaluated only through angular error analysis on annotated datasets of static images. The widespread use of videos in consumer devices motivated us to define a richer methodology for color constancy evaluation. To this extent, temporal and spatial stability are defined here to determine the degree of sensitivity of color constancy algorithms to variations in the scene that do not depend on the illuminant source, such as moving subjects or a moving camera. Our evaluation methodology is applied to compare several color constancy algorithms on stable sequences belonging to the Gray Ball and Burst Color Constancy video datasets. The stable sequences, identified using a general-purpose procedure, are made available for public download to encourage future research. Our investigation proves the importance of evaluating color constancy algorithms according to multiple metrics, instead of angular error alone. For example, the popular fully convolutional color constancy with confidence-weighted pooling algorithm is consistently the best performing solution for error evaluation, but it is often surpassed in terms of stability by the traditional gray edge algorithm, and by the more recent sensor-independent illumination estimation algorithm.

Color constancy algorithms are typically evaluated with a statistical analysis of the recovery angular error and the reproduction angular error between the estimated and ground truth illuminants. Such analysis provides information about only the magnitude of the errors, and not about their chromatic properties. We propose an Angle-Retaining Chromaticity diagram (ARC) for the visual analysis of the estimated illuminants and the corresponding errors. We provide both quantitative and qualitative proof of the superiority of ARC in preserving angular distances compared to other chromaticity diagrams, making it possible to quantify the reproduction and recovery errors in terms of Euclidean distances on a plane. We present two case studies for the application of the ARC diagram in the visualization of the ground truth illuminants of color constancy datasets, and the visual analysis of error distributions of color constancy algorithms.

The angle-retaining color space (ARC) and the corresponding chromaticity diagram encode information following a cylindrical color model. Their main property is that angular distances in RGB are mapped into Euclidean distances in the ARC chromatic components, making the color space suitable for data representation in the domain of color constancy. In this paper, we present an in-depth analysis of various properties of ARC: we document the variations in the numerical precisions of two alternative formulations of the ARC-to-RGB transformation and characterize how various perturbations in RGB impact the ARC representation. This was done empirically for the ARC diagram in a direct comparison against other commonly used chromaticity diagrams, and analytically for the ARC space with respect to its three components. We conclude by describing the color space in terms of perceptual uniformity, suggesting the need for new perceptual color metrics.

This paper reviews the NTIRE 2022 challenge on night photography rendering. The challenge solicited solutions that processed RAW camera images captured in night scenes to produce a photo-finished output image encoded in the standard RGB (sRGB) space. Given the subjective nature of this task, the proposed solutions were evaluated based on the mean opinions of viewers asked to judge the visual appearance of the results. Michael Freeman, a world-renowned photographer, further ranked the solutions with the highest mean opinion scores. A total of 13 teams competed in the final phase of the challenge. The proposed methods provided by the participating teams represent state-of-the-art performance in nighttime photography. Results from the various teams can be found here: https://nightimaging.org

This paper reviews the third biennial challenge on spectral reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3-channel RGB image. This challenge presents the "ARAD_1K" data set: a new, larger-than-ever natural hyperspectral image data set containing 1,000 images. Challenge participants were required to recover hyper-spectral information from synthetically generated JPEG-compressed RGB images simulating capture by a known calibrated camera, operating under partially known parameters, in a setting which includes acquisition noise. The challenge was attended by 241 teams, with 60 teams com-peting in the final testing phase, 12 of which provided de-tailed descriptions of their methodology which are included in this report. The performance of these submissions is re-viewed and provided here as a gauge for the current state-of-the-art in spectral reconstruction from natural RGB images.

We present an efficient method for the reconstruction of multispectral information from RGB images, as part of the NTIRE 2022 Spectral Reconstruction Challenge. Given an input image, our method determines a global RGB-to-spectral linear transformation matrix, based on a search through optimal matrices from training images that share low-level features with the input. The resulting spectral signatures are then adjusted by a global scaling factor, determined through a lightweight SqueezeNet-inspired neural network. By combining the efficiency of linear transformation matrices with the data-driven effectiveness of convolutional neural networks, we are able to achieve superior performance than winners of the previous editions of the challenge.

We present a general-purpose framework for the optimization of parametric contrast enhancement algorithms. We first define a regression module for image acceptability, which is based on deep neural features and which is trained on a large dataset of user-expressed preferences. This regression module is then used as the objective function of a Bayesian optimization process, guiding the search for the optimal parameters of a given contrast enhancement algorithm. In our experiments we optimize three different contrast enhancement algorithms of varying levels of complexity. The effectiveness of our optimization framework is experimentally confirmed by evaluating the output of the optimized contrast enhancement algorithms with respect to reference enhanced images.

We present an efficient combination strategy for color constancy algorithms. We define a compact neural network architecture to process and combine the illuminant estimations of individual algorithms, that may be based on different assumptions over the input scene content. Our solution can be specialized to the image domain, thus expecting a single frame input, and to the video domain, exploiting a Long Short-Term Memory module (LSTM) to handle varying-length sequences. To prove the effectiveness of our combining method we limit ourselves to combine only learning-free color constancy algorithms based on simple image statistics. We experiment on the standard Shi-Gehler and NUS datasets for still images, and on the recent Burst Color Constancy dataset for videos. Experimental results show that our method outperforms other combination strategies, and reaches an illuminant estimation accuracy comparable to more sophisticated and computationally-demanding solutions when the standard dataset split is used. Furthermore, our solution is proven to be effective even when the number of training instances available is reduced. As a further analysis, we assess the individual contribution of each underlying method towards the final illuminant estimation. IEEE

Digital raindrop removal is a branch of image restoration that aims at identifying adherent droplets on a glass surface and replacing them with plausible content. When successfully performed, raindrop removal was proven in the past to positively affect both the perceived appearance of the scene, and the performance of computer vision tasks such as semantic segmentation and object detection. In this paper, we design and implement a new encoder-decoder neural network for supervised raindrop removal. Our network, given a rainy input image, produces as output the Laplacian pyramid of a rain-free version of the input, making it possible to handle the variety of appearances of rain droplets by processing different frequency bands independently. To this end, we define and experimentally prove the effectiveness of a custom loss function that combines the errors of the different Laplacian frequency bands. We test our model for raindrop removal on a standard dataset, using multiple objective metrics to provide a detailed analysis of its performance. We confirm the superiority of our proposal in a comparison with other methods from the state of the art.

Video restoration techniques aim to remove artifacts, such as noise, blur, and compression, introduced at various levels within and outside the camera imaging pipeline during video acquisition. Although excellent results can be achieved by considering one artifact at a time, in real applications a given video sequence can be affected by multiple artifacts, whose appearance is mutually influenced. In this paper, we present Multi-distorted Video Restoration Network (MdVRNet), a deep neural network specifically designed to handle multiple distortions simultaneously. Our model includes an original Distortion Parameter Estimation sub-Network (DPEN) to automatically infer the intensity of various types of distortions affecting the input sequence, novel Multi-scale Restoration Blocks (MRB) to extract complementary features at different scales using two parallel streams, and implements a two-stage restoration process to focus on different levels of detail. We document the accuracy of the DPEN module in estimating the intensity of multiple distortions, and present an ablation study that quantifies the impact of the DPEN and MRB modules. Finally, we show the advantages of the proposed MdVRNet in a direct comparison with another existing state-of-the-art approach for video restoration. The code is available at https:// github.com/ claudiom4sir/ MdVRNet.

Several interesting libraries for optimization have been proposed. Some focus on individual optimization algorithms, or limited sets of them, and others focus on limited sets of problems. Frequently, the implementation of one of them does not precisely follow the formal definition, and they are difficult to personalize and compare. This makes it difficult to perform comparative studies and propose novel approaches. In this paper, we propose to solve these issues with the General Purpose Optimization Library (GPOL): a flexible and efficient multipurpose optimization library that covers a wide range of stochastic iterative search algorithms, through which flexible and modular implementation can allow for solving many different problem types from the fields of continuous and combinatorial optimization and supervised machine learning problem solving. Moreover, the library supports full-batch and mini-batch learning and allows carrying out computations on a CPU or GPU. The package is distributed under an MIT license. Source code, installation instructions, demos and tutorials are publicly available in our code hosting platform (the reference is provided in the Introduction).

Illumination estimation is the essential step of computational color constancy, one of the core parts of various image processing pipelines of modern digital cameras. Having an accurate and reliable illumination estimation is important for reducing the illumination influence on the image colors. To motivate the generation of new ideas and the development of new algorithms in this field, two challenges on illumination estimation were conducted. The main advantage of testing a method on a challenge over testing it on some of the known datasets is the fact that the ground‐truth illuminations for the challenge test images are unknown up until the results have been submitted, which prevents any potential hyperparameter tuning that may be biased. The First illumination estimation challenge (IEC#1) had only a single task, global illumination estimation. The second illumination estimation challenge (IEC#2) was enriched with two additional tracks that encompassed indoor and two‐illuminant illumination estimation. Other main features of it are a new large dataset of images (about 5000) taken with the same camera sensor model, a manual markup accompanying each image, diverse content with scenes taken in numerous countries under a huge variety of illuminations extracted by using the SpyderCube calibration object, and a contest‐like markup for the images from the Cube++ dataset. This article focuses on the description of the past two challenges, algorithms which won in each track, and the conclusions that were drawn based on the results obtained during the first and second challenge that can be useful for similar future developments.

We address the task of classifying car images at multiple levels of detail, ranging from the top-level car type, down to the specific car make, model, and year. We analyze existing datasets for car classification, and identify the CompCars as an excellent starting point for our task. We show that convolutional neural networks achieve an accuracy above 90% on the finest-level classification task. This high performance, however, is scarcely representative of real-world situations, as it is evaluated on a biased training/test split. In this work, we revisit the CompCars dataset by first defining a new training/test split, which better represents real-world scenarios by setting a more realistic baseline at 61% accuracy on the new test set. We also propagate the existing (but limited) type-level annotation to the entire dataset, and we finally provide a car-tight bounding box for each image, automatically defined through an ad hoc car detector. To evaluate this revisited dataset, we design and implement three different approaches to car classification, two of which exploit the hierarchical nature of car annotations. Our experiments show that higher-level classification in terms of car type positively impacts classification at a finer grain, now reaching 70% accuracy. The achieved performance constitutes a baseline benchmark for future research, and our enriched set of annotations is made available for public download.

Learning-based methods for Automatic White Balance (AWB) are trained on properly-annotated datasets, where each image is associated to a ground truth illuminant. The intrinsic characteristics of such datasets, therefore, play a fundamental role in the generalization capability of the resulting AWB model. In this paper we analyze the biases of commonly-used datasets for Automatic White Balance: ColorChecker, Cube+, Gray Ball, INTEL-TAU, and NUS from National University of Singapore. We describe each dataset in terms of employed cameras, distribution of the illuminants, shooting parameters, and image content. The resulting analysis highlights the individual shortcomings of each dataset, as well as the type of image that is under-represented by all analyzed datasets, such as artificial-light and low-light scenarios.

The Angle-Retaining Chromaticity diagram (ARC) is used to map tristimulus values in a two-dimensional representation, so that angular distances in the original three-dimensional space are preserved as Euclidean distances in ARC. This property makes the ARC diagram particularly useful for computational color constancy, where the illuminant intensity is purposely discarded and illuminant chromaticities are compared in terms of angular distances, through either the recovery error or the reproduction error. We expand the ARC diagram by designing a full-fledged color space that adopts a cylindricalcoordinate color model. The resulting ARC space incorporates a third dimension encoding the intensity information of the initial color, while maintaining the angle-retaining properties of the ARC diagram for the first two dimensions. We formalize the equations that describe the mutual conversion between RGB and ARC space. We illustrate the emerging geometric properties of the ARC space, and its relationship with angular distances. We present a number of potential applications of the ARC space related to color constancy, texture analysis, and image enhancement.

Recently, Computer Vision based image analysis techniques have attracted a lot of attention because they are used to develop automatic dietary monitoring applications. Food recognition is a quite challenging task: it is a non-rigid object, and is characterized by intrinsic high iter- and intra-class variability. The proper design of a food recognition system based on Computer Vision should contain several analysis stages. This paper reports on the most recent solutions in the field of automatic food recognition using computer vision developed at the Imaging and Vision Laboratory in the last 12 years. We present and discuss the main solutions developed and results achieved for food localization, segmentation, recognition and analysis. Food localization and segmentation aim at identifying the regions in the image corresponding to food items, food recognition aims at labeling each food region with the identity of the depicted food, and food analysis aims at determining properties of the food such as its quantity or ingredients.

We investigate the negative effects of rain streaks over the performance of a neural network for real time semantic segmentation of street scenes. This is done by synthetically augmenting the CityScapes dataset with artificial rain. We then define and train a generative adversarial network for rain removal, and quantify the benefits of its application as a pre-processing step to both rainy and “clean” images. Finally, we show that by retraining the semantic segmentation network on images processed for rain removal, it is possible to gain even more accuracy, with a model that produces stable results in all analyzed atmospheric conditions. For our experiments, we present a per-class analysis in order to provide deeper insights over the impact of rain on semantic segmentation.

In this article we show the change in paradigm occurred in color constancy algorithms: from a pre-processing step in image understanding, to the exploitation of image understanding and computer vision results and techniques. Since color constancy is an ill-posed problem, we give an overview of the assumptions on which classical color constancy algorithms are based in order to solve it. Then, we chronologically review the color constancy algorithms that exploit results and techniques borrowed from the image understanding research field in order to exploit assumptions that could be met in a larger number of images.

We present a method for illuminant estimation that exploits a generative adversarial network architecture to generate a spatially-varying illuminant map. This map is then transformed by consensus into a global illuminant estimation, in the form of a single RGB triplet. To this end, different consensus strategies are designed and compared in this paper. The best solution won second place in the 2nd International Illumination Estimation Challenge, specifically for the indoor track.

Assisted living technologies can be of great importance for taking care of elderly people and helping them to live independently. In this work, we propose a monitoring system designed to be as unobtrusive as possible, by exploiting computer vision techniques and visual sensors such as RGB cameras. We perform a thorough analysis of existing video datasets for action recognition, and show that no single dataset can be considered adequate in terms of classes or cardinality. We subsequently curate a taxonomy of human actions, derived from different sources in the literature, and provide the scientific community with considerations about the mutual exclusivity and commonalities of said actions. This leads us to collecting and publishing an aggregated dataset, called ALMOND (Assisted Living MONitoring Dataset), which we use as the training set for a vision-based monitoring approach. We rigorously evaluate our solution in terms of recognition accuracy using different state-of-the-art architectures, eventually reaching 97% on inference of basic poses, 83% on alerting situations, and 71% on daily life actions. We also provide a general methodology to estimate the maximum allowed distance between camera and monitored subject. Finally, we integrate the defined actions and the trained model into a computer-vision-based application, specifically designed for the objective of monitoring elderly people at their homes.

We present a review of methods for automatic estimation of visual saliency: the perceptual property that makes specific elements in a scene stand out and grab the attention of the viewer. We focus on domains that are especially recent and relevant, as they make saliency estimation particularly useful and/or effective: omnidirectional images, image groups for co-saliency, and video sequences. For each domain, we perform a selection of recent methods, we highlight their commonalities and differences, and describe their unique approaches. We also report and analyze the datasets involved in the development of such methods, in order to reveal additional peculiarities of each domain, such as the representation used for the ground truth saliency information (scanpaths, saliency maps, or salient object regions). We define domain-specific evaluation measures, and provide quantitative comparisons on the basis of common datasets and evaluation criteria, highlighting the different impact of existing approaches on each domain. We conclude by synthesizing the emerging directions for research in the specialized literature, which include novel representations for omnidirectional images, inter- and intra- image saliency decomposition for co-saliency, and saliency shift for video saliency estimation.

Saliency detection methods proposed in the literature exploit different rationales, visual clues, and assumptions, but there is no single best saliency detection algorithm that is able to achieve good results on all the different benchmark datasets. In this paper we show that fusing different saliency detection algorithms together by exploiting neural network architectures makes it possible to obtain better results. Designing the best architecture for a given task is still an open problem since the existing techniques have some limits with respect to the problem formulation, to the search space, and require very high computational resources. To overcome these problems, in this paper we propose a three-step fusion approach. In the first step, genetic programming techniques are exploited to combine the outputs of existing saliency algorithms using a set of provided operations. Having a discrete search space allows us a fast generation of the candidate solutions. In the second step, the obtained solutions are converted into backbone Convolutional Neural Networks (CNNs) where operations are all implemented with differentiable functions, allowing an efficient optimization of the corresponding parameters (in a continuous space) by backpropagation. In the last step, to enrich the expressiveness of the initial architectures, the networks are further extended with additional operations on intermediate levels of the processing that are once again efficiently optimized through backpropagation. Extensive experimental evaluations show that the proposed saliency fusion approach outperforms the state-of-the-art on the MSRAB dataset and it is able to generalize to unseen data of different benchmark datasets.

We exploit evolutionary computation to optimize the handcrafted Structural Similarity method (SSIM) through a datadriven approach. We estimate the best combination of luminance, contrast and structure components, as well as the sliding window size used for processing, with the objective of optimizing the similarity correlation with human-expressed mean opinion score on a standard dataset. We experimentally observe that better results can be obtained by penalizing the overall similarity only for very low levels of luminance similarity. Finally, we report a comparison of SSIM with the optimized parameters against other metrics for full reference quality assessment, showing superior performance on a different dataset.

La descrizione automatica delle immagini digitali consiste nell'estrarre informazioni che rappresentano in modo significativo gli elementi raffigurati e i loro attributi. Il concetto specifico di "significativo" può essere guidato e determinato dall'applicazione finale: nell'assistere persone ipovedenti, ad esempio, l'utente finale potrebbe voler riconoscere elementi familiari quali edifici e loghi. Nel contesto del supporto dei conducenti per le auto intelligenti, potrebbe essere utile riconoscere altri veicoli e pedoni e distinguere la distanza dall'auto stessa. Una pipeline generale per gli scenari previsti prevede tre passaggi: object proposal, classification, ed attributes extraction. Nel corso di questa tesi sono stati studiati e sviluppati diversi metodi per affrontare ciascuna di queste fasi, e gli stessi sono stati successivamente applicati a domini specifici con l'intento di confrontare le soluzioni prodotte con lavori esistenti in letteratura. Object proposal: una o più sottoregioni contenenti elementi di potenziale interesse vengono estratti a partire dall'immagine di input. In questa tesi, l'object proposal per singoli oggetti è realizzata utilizzando un'architettura neurale ottimizzata in maniera originale, combinando la programmazione genetica per l'ottimizzazione della struttura con la back-propagation per l'ottimizzazione dei parametri. In sovrapposizione sugli step di object proposal e classification, la sementazione sematica è affrontata mediante la definizione di un'architettura neurale originale che presti particolare attenzione all'efficienza computazionale per scenari di high-throughput. Classificazione: le sottoregioni generate dalla fase di object proposal vengono classificate in specifiche classi visive. Il riconoscimento di loghi è riportato come primo caso di studio. È stato raccolto un nuovo set di dati, estendendo di dieci volte lo standard esistente. La combinazione di quest'ultimo con forme sintetiche di data augmentation consen, Automatically describing digital images consists in extracting information that meaningfully represents the depicted elements and their attributes. The specific concept of "meaningful" can be determined by the final application: in assistance to visually impaired people, for example, the final user might want to recognize familiar elements such as landmarks and logos. In the context of driver support for smart cars, it could be useful to recognize other vehicles and pedestrians, and to tell their distance from the car itself. A general pipeline for the envisioned scenarios involves three steps: object proposal, classification, and attributes extraction. In this thesis, several methods have been studied and developed for each of these steps, and subsequently applied to specific domains with the intent of comparing the produced solutions with existing works. Object proposal: one or many subregions containing elements of potential interest are extracted from the input image. In this thesis, single-object proposal is achieved using a neural architecture that is optimized in a novel way, combining genetic programming for the structure optimization with back-propagation for parameters tuning. Crossing the gap between object proposal and classification, semantic segmentation is then addressed with the definition of an original neural architecture that pays particular attention to computational efficiency for high-throughput scenarios. Classification: the subregions generated by the object proposal phase are classified into visual classes. Logo recognition is reported as a first case study. A new dataset has been collected, extending tenfolds the existing standard. Its combination with synthetic forms of data augmentation allows to reach state of the art performance. Vegetables and fruits recognition is then chosen as a representative example for fine-grained visual classification problems. The task is addressed by preprocessing images with object proposal algorithms, and b

We propose a new representation of distance information that is independent from any specific acquisition device, based on the size of portrayed subjects. In this alternative description, each pixel of an image is associated with the size, in real life, of what it represents. Using our proposed representation, datasets acquired with different devices can be effortlessly combined to build more powerful models, and monocular distance estimation can be performed on images acquired from devices that were never used during training. To assess the advantages of the proposed representation, we used it to train a fully convolutional neural network that predicts with pixel-precision the size of different subjects depicted in the image, as a proxy for their distance. Experimental results show that our representation, allowing the combination of heterogeneous training datasets, makes it possible for the trained network to gain better results at test time.

We address the task of image saliency estimation through proper recombination of existing methods in the state of the art. We define a general scheme, which we then specialize to perform dataset-specific and image-specific recombination, based on either linear weight regression, or method selection. The advantage of this approach lies in the possibility of exploiting the different strengths of existing methods. Experiments are conducted with both deep learning and hand-crafted methods on a widely used dataset, using standard evaluation measures. The proposed recombination strategy allows us to improve upon the state of the art, by exploiting a linear combination of the saliency maps produced by existing methods. We also show that image-specific combination and selection of saliency maps is limited by the apparent lack of relevant information intrinsic in the image itself.

We propose a novel modular CNN architecture that provides semantic segmentation and understanding of outdoor street environment images. Our solution processes a 512x1024 resolution image on a single Titan Xp GPU at 37.4 FPS attaining 70.4% IoU on the Cityscapes test dataset.

We present a state of the art method for vegetable and fruit recognition based on convolutional neural networks. We developed our solution around the concept of a smart kitchen/refrigerator equipped with an on-board camera. With this objective in mind, we adopted a dataset that was specifically collected and annotated according to the eating characteristics of the portrayed items. We performed two types of experiment: we first trained and evaluated different state-of-the-art neural architectures on the task of vegetable and fruit recognition. Secondly, we designed and tested a solution that exploits the hierarchical nature of such classes to further improve the final performance of our system. Experimental results demonstrate the quantitative superiority of the proposed solution compared to existing approaches.