Search

Large-scale self-supervised pretraining of deep learning models is known to be critical in several fields, such as language processing, where its has led to significant breakthroughs. Indeed, it is often more impactful than architectural designs. However, the use of self-supervised pretraining lags behind in several domains, such as hyperspectral images, due to data scarcity. This paper addresses the challenge of data scarcity in the development of methods for spatial super-resolution of hyperspectral images (HSI-SR). We show that state-of-the-art HSI-SR methods are severely bottlenecked by the small paired datasets that are publicly available, also leading to unreliable assessment of the architectural merits of the models. We propose to capitalize on the abundance of high resolution (HR) RGB images to develop a self-supervised pretraining approach that significantly improves the quality of HSI-SR models. In particular, we leverage advances in spectral reconstruction methods to create a vast dataset with high spatial resolution and plausible spectra from RGB images, to be used for pretraining HSI-SR methods. Experimental results, conducted across multiple datasets, report large gains for state-of-the-art HSI-SR methods when pretrained according to the proposed procedure, and also highlight the unreliability of ranking methods when training on small datasets.

We propose a flexible framework that can be used to explore large-scale image datasets and summarize photo albums. Our proposed method first groups images based on their semantic content, and then selects the most diverse and aesthetically pleasing images to represent each category. To ensure the selection of high-quality images, we use features extracted from a Convolutional Neural Network to assess their diversity and perceptual properties. The effectiveness of our method is tested using benchmarking datasets and a qualitative study.

Multispectral imaging is a technique that captures data across several bands of the light spectrum, and it can be useful in many computer vision fields, including color constancy. We propose a method that exploits multispectral imaging for illuminant estimation, and then applies illuminant correction in the raw RGB domain to achieve computational color constancy. Our proposed method is composed of two steps: first, a selected number of existing camera-independent algorithms for illuminant estimation, originally designed for RGB data, are applied in generalized form to work with multispectral data. We demonstrate that the sole multispectral extension of such algorithms is not sufficient to achieve color constancy, and thus we introduce a second step, in which we re-elaborate the multispectral estimations before conversion into raw RGB with the use of the camera response function. Our results on the NUS dataset show that an improvement of 60% in the color constancy performance, measured in terms of reproduction angular error, can be obtained according to our method when compared to the traditional raw RGB pipeline.

We introduce a method that enhances RGB color constancy accuracy by combining neural network and k-means clustering techniques. Our approach stands out from previous works because we combine multispectral and color information together to estimate illuminants. Furthermore, we investigate the combination of the illuminant estimation in the RGB color and in the spectral domains, as a strategy to provide a refined estimation in the RGB color domain. Our investigation can be divided into three main points: (1) identify the spatial resolution for sampling the input image in terms of RGB color and spectral information that brings the highest performance; (2) determine whether it is more effective to predict the illuminant in the spectral or in the RGB color domain, and finally, (3) assuming that the illuminant is in fact predicted in the spectral domain, investigate if it is better to have a loss function defined in the RGB color or spectral domain. Experimental results are carried out on NUS: a standard dataset of multispectral radiance images with an annotated spectral global illuminant. Among the several considered options, the best results are obtained with a model trained to predict the illuminant in the spectral domain using an RGB color loss function. In terms of comparison with the state of the art, this solution improves the recovery angular error metric by 66% compared to the best tested spectral method, and by 41% compared to the best tested RGB method.

Perceiving image complexity is a crucial aspect of human visual understanding, yet explicitly assessing image complexity poses challenges. Historically, this aspect has been understudied due to its inherent subjectivity, stemming from its reliance on human perception, and the semantic dependency of image complexity in the face of diverse real-world images. Different computational models for image complexity estimation have been proposed in the literature. These models leverage a variety of techniques ranging from low-level, handcrafted features, to advanced machine learning algorithms. This paper explores the use of recent deep-learning approaches based on Visual Transformer to extract robust information for image complexity estimation in a transfer learning paradigm. Specifically, we propose to leverage three visual backbones, CLIP, DINO-v2, and ImageNetViT, as feature extractors, coupled with a Support Vector Regressor with Radial Basis Function kernel as an image complexity estimator. We test our approach on two widely used benchmark datasets (i.e. IC9600 and SAVOIAS) in an intra-dataset and inter-dataset workflow. Our experiments demonstrate the effectiveness of the CLIP-based features for accurate image complexity estimation with results comparable to end-to-end solutions.

Performance in Speech Emotion Recognition (SER) on a single language has increased greatly in the last few years thanks to the use of deep learning techniques. However, cross-lingual SER remains a challenge in real-world applications due to two main factors: the first is the big gap among the source and the target domain distributions; the second factor is the major availability of unlabeled utterances in contrast to the labeled ones for the new language. Taking into account previous aspects, we propose a Semi-Supervised Learning (SSL) method for cross-lingual emotion recognition when only few labeled examples in the target domain (i.e. the new language) are available. Our method is based on a Transformer and it adapts to the new domain by exploiting a pseudo-labeling strategy on the unlabeled utterances. In particular, the use of a hard and soft pseudo-labels approach is investigated. We thoroughly evaluate the performance of the proposed method in a speaker-independent setup on both the source and the new language and show its robustness across five languages belonging to different linguistic strains. The experimental findings indicate that the unweighted accuracy is increased by an average of 40% compared to state-of-the-art methods.

Semantic segmentation consists of classifying each pixel of an image and constitutes an essential step towards scene recognition and understanding. Deep convolutional encoder–decoder neural networks now constitute state-of-the-art methods in the field of semantic segmentation. The problem of street scenes’ segmentation for automotive applications constitutes an important application field of such networks and introduces a set of imperative exigencies. Since the models need to be executed on self-driving vehicles to make fast decisions in response to a constantly changing environment, they are not only expected to operate reliably but also to process the input images rapidly. In this paper, we explore genetic programming (GP) as a meta-model that combines four different efficiency-oriented networks for the analysis of urban scenes. Notably, we present and examine two approaches. In the first approach, we represent solutions as GP trees that combine networks’ outputs such that each output class’s prediction is obtained through the same meta-model. In the second approach, we propose representing solutions as lists of GP trees, each designed to provide a unique meta-model for a given target class. The main objective is to develop efficient and accurate combination models that could be easily interpreted, therefore allowing gathering some hints on how to improve the existing networks. The experiments performed on the Cityscapes dataset of urban scene images with semantic pixel-wise annotations confirm the effectiveness of the proposed approach. Specifically, our best-performing models improve systems’ generalization ability by approximately 5% compared to traditional ensembles, 30% for the less performing state-of-the-art CNN and show competitive results with respect to state-of-the-art ensembles. Additionally, they are small in size, allow interpretability, and use fewer features due to GP’s automatic feature selection.

This paper reports on the NTIRE 2023 Quality Assessment of Video Enhancement Challenge, which will be held in conjunction with the New Trends in Image Restoration and Enhancement Workshop (NTIRE) at CVPR 2023. This challenge is to address a major challenge in the field of video processing, namely, video quality assessment (VQA) for enhanced videos. The challenge uses the VQA Dataset for Perceptual Video Enhancement (VDPVE), which has a total of 1211 enhanced videos, including 600 videos with color, brightness, and contrast enhancements, 310 videos with deblurring, and 301 deshaked videos. The challenge has a total of 167 registered participants. 61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions. A total of 176 submissions were submitted by 37 participating teams during the final testing phase. Finally, 19 participating teams submitted their models and fact sheets, and detailed the methods they used. Some methods have achieved better results than baseline methods, and the winning methods have demonstrated superior prediction performance.

This paper presents a review of the NTIRE 2023 challenge on night photography rendering. The goal of the challenge was to find solutions that process raw camera images taken in nighttime conditions conditions, and thereby produce a photo-quality output images in the standard RGB (sRGB) space. Unlike the previous year's competition, participants were not provided with a large training dataset for the target sensor. Instead, this time they were given images of a color checker illuminated by a known light source. To evaluate the results, a sufficient number of viewers were asked to assess the visual quality of the proposed solutions, considering the subjective nature of the task. The highest ranking solutions were further ranked by Richard Collins, a renowned photographer. The top ranking participants' solutions effectively represent the state-of-the-art in nighttime photography rendering. More results can be found at https://nightimaging.org

In this paper we propose an adaptive deep neural architecture for the prediction of multiple soil characteristics from the analysis of hyperspectral signatures. The proposed method overcomes the limitations of previous methods in the state of art: (i) it allows to predict multiple soil variables at once; (ii) it is based on a flexible neural architecture capable of automatically adapting to the spectral library under analysis. As an additional feature, our method permits to backtrace the spectral bands that most contribute to the estimation of a given variable. The proposed architecture is experimented on LUCAS, a large laboratory dataset and on a dataset achieved by simulating PRISMA hyperspectral sensor. Results, compared with other state-of-the-art methods confirm the effectiveness of the proposed solution. In fact, the proposed method achieves an overall R2 of about 0.75. In detail, pHCaCl and CaCO3 are predicted with an R2 above 0.90, pHH, OC, N, clay, sand, CEC, silt are predicted with an R2 between 0.89 and 0.70, K and P with an R2 respectively of 0.54 and 0.37. The proposed method overcomes, in terms of R2, the state of the art of about 15% on the LUCAS dataset and of about 24% on the PRISMA simulated dataset.

Precision agriculture has emerged as a promising approach to improve crop productivity and reduce the environmental impact. However, effective decision making in precision agriculture relies on accurate and timely data acquisition, management, and analysis. The collection of multisource and heterogeneous data for soil characteristics estimation is a critical component of precision agriculture, as it provides insights into key factors, such as soil nutrient levels, moisture content, and texture. To address these challenges, this work proposes a software platform that facilitates the collection, visualization, management, and analysis of soil data. The platform is designed to handle data from various sources, including proximity, airborne, and spaceborne data, to enable precision agriculture. The proposed software allows for the integration of new data, including data that can be collected directly on-board the acquisition device, and it also allows for the incorporation of custom predictive systems for soil digital mapping. The usability experiments conducted on the proposed software platform demonstrate that it is easy to use and effective. Overall, this work highlights the importance of decision support systems in the field of precision agriculture and the potential benefits of using such systems for soil data management and analysis.

This paper reviews the Challenge on Super-Resolution of Compressed Image and Video at AIM 2022. This challenge includes two tracks. Track 1 aims at the super-resolution of compressed image, and Track 2 targets the super-resolution of compressed video. In Track 1, we use the popular dataset DIV2K as the training, validation and test sets. In Track 2, we propose the LDV 3.0 dataset, which contains 365 videos, including the LDV 2.0 dataset (335 videos) and 30 additional videos. In this challenge, there are 12 teams and 2 teams that submitted the final results to Track 1 and Track 2, respectively. The proposed methods and solutions gauge the state-of-the-art of super-resolution on compressed image and video. The proposed LDV 3.0 dataset is available at https://github.com/RenYang-home/LDV_dataset. The homepage of this challenge is at https://github.com/RenYang-home/AIM22_CompressSR.

In this paper, we propose an unsupervised method for hyperspectral remote sensing image segmentation. The method exploits the mean-shift clustering algorithm that takes as input a preliminary hyperspectral superpixels segmentation together with the spectral pixel information. The proposed method does not require the number of segmentation classes as input parameter, and it does not exploit any a-priori knowledge about the type of land-cover or land-use to be segmented (e.g. water, vegetation, building etc.). Experiments on Salinas, SalinasA, Pavia Center and Pavia University datasets are carried out. Performance are measured in terms of normalized mutual information, adjusted Rand index and F1-score. Results demonstrate the validity of the proposed method in comparison with the state of the art.

Speech emotion recognition is a significant source of information especially when other channels, like face or body, are hidden. The shape of the vocal tract, tone of the voice, pitch and other characteristics are influenced by human emotions. In this paper, we propose the use of static, dynamic and acceleration features which are very effective in encoding those characteristics of the speech that are influenced by human emotions. These features are based on the concatenation of three global measures of Mel-frequency Cepstral Coefficients (MFCCs) (the static part) and the first (the dynamic part) and second derivatives (the acceleration part) of MFCCs. The features are processed with a custom 1-D CNN suitable designed by the authors for emotion recognition. Experiments are performed on two publicly available speech datasets containing audio files from different people and language and several emotions. Our approach on average overcomes the state of the art on both datasets.

The management of people with long-term or chronic illness is one of the biggest challenges for national health systems. In fact, these diseases are among the leading causes of hospitalization, especially for the elderly, and huge amount of resources required to monitor them leads to problems with sustainability of the healthcare systems. The increasing diffusion of portable devices and new connectivity technologies allows the implementation of telemonitoring system capable of providing support to health care providers and lighten the burden on hospitals and clinics. In this paper, we present the implementation of a telemonitoring platform for healthcare, designed to capture several types of physiological health parameters from different consumer mobile and custom devices. Consumer medical devices can be integrated into the platform via the Google Fit ecosystem that supports hundreds of devices, while custom devices can directly interact with the platform with standard communication protocols. The platform is designed to process the acquired data using machine learning algorithms, and to provide patients and physicians the physiological health parameters with a user-friendly, comprehensive, and easy to understand dashboard which monitors the parameters through time. Preliminary usabil-ity tests show a good user satisfaction in terms of functionality and usefulness.

The growing production of digital content and its dissemination across the worldwide web require eficient and precise management. In this context, image quality assessment measures (IQAMs) play a pivotal role in guiding the development of numerous image processing systems for compression, enhancement, and restoration. The structural similarity index (SSIM) is one of the most common IQAMs for estimating the similarity between a pristine reference image and its corrupted variant. The multi-scale SSIM is one of its most popular variants that allows assessing image quality at multiple spatial scales. This paper proposes a two-stage genetic programming (GP) approach to evolve novel multi-scale IQAMs, that are simultaneously more effective and efficient. We use GP to perform feature selection in the first stage, while the second stage generates the final solutions. The experimental results show that the proposed approach outperforms the existing MS-SSIM. A comprehensive analysis of the feature selection indicates that, for extracting multi-scale similarities, spatially-varying convolutions are more effective than dilated convolutions. Moreover, we provide evidence that the IQAMs learned for one database can be successfully transferred to previously unseen databases. We conclude the paper by presenting a set of evolved multi-scale IQAMs and providing their interpretation.

Several contemporaneous image processing and computer vision systems rely upon the full-reference image quality assessment (IQA) measures. The single-scale structural similarity index (SS-SSIM) is one of the most popular measures, and it owes its success to the mathematical simplicity, low computational complexity, and implicit incorporation of Human Visual System's (HVS) characteristics. In this paper, we revise the original parameters of SSIM and its multi-scale counterpart (MS-SSIM) to increase their correlation with subjective evaluation. More specifically, we exploit the evolutionary computation and the swarm intelligence methods on five popular IQA databases, two of which are dedicated distance-changed databases, to determine the best combination of parameters efficiently. Simultaneously, we explore the effect of different scale selection approaches in the context of SS-SSIM. The experimental results show that with a proper fine-tuning (1) the performance of SS-SSIM and MS-SSIM can be improved, in average terms, by 8% and by 3%, respectively, (2) the SS-SSIM after the so-called standard scale selection achieves similar performance as if applying computationally more expensive state-of-the-art scale selection methods or MS-SSIM; moreover, (3) there is evidence that the parameters learned on a given database can be successfully transferred to other (previously unseen) databases; finally, (4) we propose a new set of reference parameters for SSIM's variants and provide their interpretation.

Affective computing is a key research topic in artificial intelligence which is applied to psychology and machines. It consists of the estimation and measurement of human emotions. A person’s body language is one of the most significant sources of information during job interview, and it reflects a deep psychological state that is often missing from other data sources. In our work, we combine two tasks of pose estimation and emotion classification for emotional body gesture recognition to propose a deep multi-stage architecture that is able to deal with both tasks. Our deep pose decoding method detects and tracks the candidate’s skeleton in a video using a combination of depthwise convolutional network and detection-based method for 2D pose reconstruction. Moreover, we propose a representation technique based on the superposition of skeletons to generate for each video sequence a single image synthesizing the different poses of the subject. We call this image: ‘history pose image’, and it is used as input to the convolutional neural network model based on the Visual Geometry Group architecture. We demonstrate the effectiveness of our method in comparison with other methods in the state of the art on the standard Common Object in Context keypoint dataset and Face and Body gesture video database.

Defect segmentation of apples is an important task in the agriculture industry for quality control and food safety. In this paper, we propose a deep learning approach for the automated segmentation of apple defects using convolutional neural networks (CNNs) based on a U-shaped architecture with skip-connections only within the noise reduction block. An ad-hoc data synthesis technique has been designed to increase the number of samples and at the same time to reduce neural network overfitting. We evaluate our model on a dataset of multi-spectral apple images with pixel-wise annotations for several types of defects. In this paper, we show that our proposal outperforms in terms of segmentation accuracy general-purpose deep learning architectures commonly used for segmentation tasks. From the application point of view, we improve the previous methods for apple defect segmentation. A measure of the computational cost shows that our proposal can be employed in real-time (about 100 frame-per-second on GPU) and in quasi-real-time (about 7/8 frame-per-second on CPU) visual-based apple inspection. To further improve the applicability of the method, we investigate the potential of using only RGB images instead of multi-spectral images as input images. The results prove that the accuracy in this case is almost comparable with the multi-spectral case.

Data augmentation is a fundamental technique in machine learning that plays a crucial role in expanding the size of training datasets. By applying various transformations or modifications to existing data, data augmentation enhances the generalization and robustness of machine learning models. In recent years, the development of several libraries has simplified the utilization of diverse data augmentation strategies across different tasks. This paper focuses on the exploration of the most widely adopted libraries specifically designed for data augmentation in computer vision tasks. Here, we aim to provide a comprehensive survey of publicly available data augmentation libraries, facilitating practitioners to navigate these resources effectively. Through a curated taxonomy, we present an organized classification of the different approaches employed by these libraries, along with accompanying application examples. By examining the techniques of each library, practitioners can make informed decisions in selecting the most suitable augmentation techniques for their computer vision projects. To ensure the accessibility of this valuable information, a dedicated public website named DALib has been created. This website serves as a centralized repository where the taxonomy, methods, and examples associated with the surveyed data augmentation libraries can be explored. By offering this comprehensive resource, we aim to empower practitioners and contribute to the advancement of computer vision research and applications through effective utilization of data augmentation techniques.

Enhancing night photography images is a challenging task that requires advanced processing techniques. While CNN-based methods have shown promising results, their high computational requirements and limited interpretability can pose challenges. To address these limitations, we propose a camera pipeline for rendering visually pleasing photographs in low-light conditions. Our approach is characterized by a shallow structure, explainable steps, and a low parameter count, resulting in computationally efficient processing. We compared the proposed pipeline with recent CNN-based state-of-the-art approaches for low-light image enhancement, showing that our approach produces more aesthetically pleasing results. The psycho-visual comparisons conducted in this work show how our proposed solution is preferred with respect to the other methods (in about 44% of the cases our solution has been chosen, compared to only about 15% of the cases for the state-of-the-art best method).

Rendering night photography pictures is a challenging task that requires advanced processing techniques. Although deep learning-based Image Signal Processing (ISP) pipelines have shown promising results, current limitations are set by the lack of proper nighttime image datasets, their high computational requirements, and low explainability. In this paper, we propose a traditional ISP pipeline for rendering visually pleasing photographs of night scenes. Our pipeline is comprised of various algorithms addressing the different challenges presented by night images, and it is characterized by a shallow structure, explainable steps, and a low parameter count, resulting in computationally efficient processing. Moreover, it does not require training data. Experiments show that our pipeline can produce more pleasing results compared to other deep learning-based ISP pipelines, as it won first place in people's choice track and third place in photographer's choice track in the NTIRE 2023 Night Photography Rendering Challenge.

In this work we propose a novel method to evaluate the quality of enhanced videos. Perceived quality of a video depends on both technical aspects, such as the presence of distortions like noise and blur, and non-technical factors, such as content preference and recommendation. Our approach involves the use of three deep learning based models that encode video sequences in terms of their overall technical quality, quality-related attributes, and aesthetic quality. The resulting feature vectors are adaptively combined and used as input to a Support Vector Regressor to estimate the video quality score. Quantitative results on the recently released VQA Dataset for Perceptual Video Enhancement (VDPVE) introduced for the NTIRE 2023 Quality Assessment of Video Enhancement Challenge demonstrates the effectiveness of the proposed method.

Full-reference image quality measures are a fundamental tool to approximate the human visual system in various applications for digital data management: from retrieval to compression to detection of unauthorized uses. Inspired by both the effectiveness and the simplicity of hand-crafted Structural Similarity Index Measure (SSIM), in this work, we present a framework for the formulation of SSIM-like image quality measures through genetic programming. We explore different terminal sets, defined from the building blocks of structural similarity at different levels of abstraction, and we propose a two-stage genetic optimization that exploits hoist mutation to constrain the complexity of the solutions. Our optimized measures are selected through a cross-dataset validation procedure, which results in superior performance against different versions of structural similarity, measured as correlation with human mean opinion scores. We also demonstrate how, by tuning on specific datasets, it is possible to obtain solutions that are competitive with (or even outperform) more complex image quality measures.

Video memorability prediction aims to quantify how much a given video content will be remembered over time. The main attributes affecting the prediction of memorability are not yet fully understood and many of the methods in the literature are based on features extracted from content recognition models. In this paper we demonstrate that features extracted from a model trained with natural language supervision are effective for estimating video memorability. The proposed method exploits a Vision Transformer pretrained using Contrastive Language-Image Pretraining (CLIP) for encoding video frames. A temporal attention mechanism is then used to select and aggregate relevant frame representations into a video-level feature vector. Finally, a multi-layer perceptron maps the video-level features into a score. We test several types of encoding and temporal aggregation modules and submit our best solution to the MediaEval 2022 Predicting Media Memorability task. We achieve a correlation of 0.707 in subtask 1 (i.e. the Memento10k dataset). In task 2 we obtain a Pearson correlation of 0.487 by training on Memento10k and testing on videoMem and of 0.529 by training on videoMem and testing on Memento10k.

Annotated datasets for automatic white balance (AWB) are used for the evaluation and, when necessary, the training, of AWB methods. Relying on such datasets requires awareness of the potential bias in their content and characteristics: some methods are designed to rely on the presence of particular elements, such as human skin, while other methods learn implicit relationships between image content and light properties from training data. The dependency on these relationships makes it fundamental to understand whether the available datasets are actually representative of common application scenarios, such as the presence of human subjects, the diversity of composition, or the illumination conditions. In this paper we overview the most common datasets for Automatic White Balance, including those for single as well as multiple illuminant estimation, providing a critical analysis on their characteristics. Furthermore, we identify a number of existing methods for single illuminant estimation, as a representative pool of approaches to the problem with various levels of complexity. We investigate how the performance of these correlate to the image content of common datasets.

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

The first paper investigating the use of machine learning to learn the relationship between an image of a scene and the color of the scene illuminant was published by Funt et al. in 1996. Specifically, they investigated if such a relationship could be learned by a neural network. During the last 30 years we have witnessed a remarkable series of advancements in machine learning, and in particular deep learning approaches based on artificial neural networks. In this paper we want to update the method by Funt et al. by including recent techniques introduced to train deep neural networks. Experimental results on a standard dataset show how the updated version can improve the median angular error in illuminant estimation by almost 51% with respect to its original formulation, even outperforming recent illuminant estimation methods.

A pipeline for the generation of synthetic dataset of spectral scenes, with corresponding sensor readings, is here proposed. The pipeline is composed of two main parts: Part 1: Image pixel reflectance assignment. Individual pixels from an input sRGB image dataset are replaced with appropriate reflectance spectra from a given non-image reflectance dataset. The resulting dataset of reflectance images is considered the starting point for simulated sensor acquisition. Part 2: Simulated sensor acquisition. Each spectral reflectance image in the dataset is illuminated with an illuminant spectra to produce a radiance image. The resulting dataset of radiance images is then synthetically read from the simulated sensors (camera and ambient multispectral sensor) of the Huawei P50 phone, using the corresponding sensors transmittance information. The capability of generating any large-scale, diverse, and annotated synthetic spectral datasets can facilitate the development of data-driven imaging algorithms, and foster reproducible research.

This paper presents a novel approach for spectral illuminant correction in smartphone imaging systems, aiming to improve color accuracy and enhance image quality. The methods introduced include Spectral Super Resolution and Weighted Spectral Color Correction (W-SCC). These techniques leverage the spectral information of both the image and the illuminant to perform effective color correction. Experimental evaluations were conducted using a dataset of 100 synthetic images, whose acquisition is simulated using the transmittance information of a Huawei P50 smartphone camera sensor and an Ambient light Multispectral Sensor (AMS). The results demonstrate the superiority of the proposed methods compared to traditional trichromatic pipelines, achieving significant reductions in colorimetric errors measured in terms of ∆E94 units. The W-SCC technique, in particular, incorporates per-wavelength weight optimization, further enhancing the accuracy of spectral illuminant correction. The presented approaches have valuable applications in various fields, including color analysis, computer vision, and image processing. Future research directions may involve exploring additional optimization techniques and incorporating advanced machine learning algorithms to further advance spectral illuminant correction in smartphone imaging systems.

We propose an anomaly detection based image quality assessment method which exploits the correlations between feature maps from a pre-trained Convolutional Neural Network (CNN). The proposed method encodes the intra-layer correlation through the Gram matrix and then estimates the quality score combining the average of the correlation and the output from an anomaly detection method. The latter evaluates the degree of abnormality of an image by computing a correlation similarity with respect to a dictionary of pristine images. The effectiveness of the method is tested on different benchmarking datasets (LIVE-itW, KONIQ, and SPAQ).

In this paper a semi-supervised method for the detection of anomalies in both texture- and object-based product images is presented. The method exploits a pre-trained Convolutional Neural Network (CNN) autoencoder that is blended with a statistical-based transformation of the neural network embedding layer in order to remove anomalies from the input image. The “cleaned” version of the input image is then compared with the input image itself in order to spatially localize the anomalies. The method does not require a specific training of the CNN to be applied to a new class of product, but it requires a very fast domain adaptation based on only “anomaly-free” examples. Experiments conducted on a publicly available dataset made of fifteen texture- and object-based classes show that overall performance is better than the state of the art of about 4%. In the case of texture-based classes the proposed method outperforms the state of the art of about 13%. In the case of object-based classes, the proposed method reaches overall the same performance of the state of the art. In this case, apart from 3 cases, that is “bottle”, “transistor” and “metal nut”, the proposed method performs better than the state of the art in 7 object classes out of 10.

In this paper we present T1K+, a very large, heterogeneous database of high-quality texture images acquired under variable conditions. T1K+ contains 1129 classes of textures ranging from natural subjects to food, textile samples, construction materials, etc. T1K+ allows the design of experiments especially aimed at understanding the specific issues related to texture classification and retrieval. To help the exploration of the database, all the 1129 classes are hierarchically organized in 5 thematic categories and 266 sub-categories. To complete our study, we present an evaluation of hand-crafted and learned visual descriptors in supervised texture classification tasks.

Color constancy algorithms are typically evaluated with a statistical analysis of the recovery angular error and the reproduction angular error between the estimated and ground truth illuminants. Such analysis provides information about only the magnitude of the errors, and not about their chromatic properties. We propose an Angle-Retaining Chromaticity diagram (ARC) for the visual analysis of the estimated illuminants and the corresponding errors. We provide both quantitative and qualitative proof of the superiority of ARC in preserving angular distances compared to other chromaticity diagrams, making it possible to quantify the reproduction and recovery errors in terms of Euclidean distances on a plane. We present two case studies for the application of the ARC diagram in the visualization of the ground truth illuminants of color constancy datasets, and the visual analysis of error distributions of color constancy algorithms.

We propose a deep residual autoencoder exploiting Residual-in-Residual Dense Blocks (RRDB) leveraging both the learning capacity of deep residual networks and prior knowledge of the JPEG compression pipeline. The proposed method is blind and universal, consisting of a unique model that effectively restores images with any level of compression. It operates in the YCbCr color space and performs JPEG restoration in two phases using two different autoencoders: the first one restores the luma channel exploiting 2D convolutions; the second one, using the restored luma channel as a guide, restores the chroma channels exploiting 3D convolutions. Extensive experimental results on four widely used benchmark datasets (i.e. LIVE1, BDS500, CLASSIC-5, and Kodak) show that our model outperforms state of the art methods, even those using a different set of weights for each compression quality, in terms of all the evaluation metrics considered (i.e. PSNR, PSNR-B, and SSIM). Furthermore, the proposed model shows a greater robustness than state-of-the-art methods when applied to compression qualities not seen during training.

The problem of food segmentation is quite challenging since food is characterized by intrinsic high intra-class variability. Also, segmentation of food images taken in-the-wild may be characterized by acquisition artifacts, and that could be problematic for the segmentation algorithms. A proper evaluating of segmentation algorithms is of paramount importance for the design and improvement of food analysis systems that can work in less-than-ideal real scenarios. In this paper, we evaluate the performance of different deep learning-based segmentation algorithms in the context of food. Due to the lack of large-scale food segmentation datasets, we initially create a new dataset composed of 5000 images of 50 diverse food categories. The images are accurately annotated with pixel-wise annotations. In order to test the algorithms under different conditions, the dataset is augmented with the same images but rendered under different acquisition distortions that comprise illuminant change, JPEG compression, Gaussian noise, and Gaussian blur. The final dataset is composed of 120,000 images. Using standard benchmark measures, we conducted extensive experiments to evaluate ten state-of-the-art segmentation algorithms on two tasks: food localization and semantic food segmentation.

Accurate semantic segmentation ground-truths are difficult and expensive to obtain. On the other hand, the most promising approaches to automatically tackle this task, i.e. Deep Convolutional Neural Networks (CNNs), require high volumes of labeled data. We propose a new method based on deep learning for data augmentation in the context of semantic segmentation of highly-textured images. The method exploits a Generative Adversarial Network (GAN) to produce a semantic layout, then a texture synthesizer, based on a CNN, generates a new image according to the generated semantic layout and a reference real image taken from the training set. Even though our method is general and it can be utilized on a broad set of problems, we employed it on the real-world problem of detecting and localizing defects and cracks in road asphalts. We show how, starting from few labeled images, it is possible to augment small and long-tail datasets by producing new images with the associated semantic layouts. We prove the effectiveness of our approach by evaluating the performance of three different CNNs for semantic segmentation on the German Pavement Distress dataset and on a novel asphalt dataset collected by us. Results show a remarkable increase in performance, especially with low cardinality classes, when CNNs are trained on the augmented datasets with respect to original datasets.

This paper reviews the NTIRE 2022 challenge on night photography rendering. The challenge solicited solutions that processed RAW camera images captured in night scenes to produce a photo-finished output image encoded in the standard RGB (sRGB) space. Given the subjective nature of this task, the proposed solutions were evaluated based on the mean opinions of viewers asked to judge the visual appearance of the results. Michael Freeman, a world-renowned photographer, further ranked the solutions with the highest mean opinion scores. A total of 13 teams competed in the final phase of the challenge. The proposed methods provided by the participating teams represent state-of-the-art performance in nighttime photography. Results from the various teams can be found here: https://nightimaging.org

This paper reviews the third biennial challenge on spectral reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3-channel RGB image. This challenge presents the "ARAD_1K" data set: a new, larger-than-ever natural hyperspectral image data set containing 1,000 images. Challenge participants were required to recover hyper-spectral information from synthetically generated JPEG-compressed RGB images simulating capture by a known calibrated camera, operating under partially known parameters, in a setting which includes acquisition noise. The challenge was attended by 241 teams, with 60 teams com-peting in the final testing phase, 12 of which provided de-tailed descriptions of their methodology which are included in this report. The performance of these submissions is re-viewed and provided here as a gauge for the current state-of-the-art in spectral reconstruction from natural RGB images.

We present an efficient method for the reconstruction of multispectral information from RGB images, as part of the NTIRE 2022 Spectral Reconstruction Challenge. Given an input image, our method determines a global RGB-to-spectral linear transformation matrix, based on a search through optimal matrices from training images that share low-level features with the input. The resulting spectral signatures are then adjusted by a global scaling factor, determined through a lightweight SqueezeNet-inspired neural network. By combining the efficiency of linear transformation matrices with the data-driven effectiveness of convolutional neural networks, we are able to achieve superior performance than winners of the previous editions of the challenge.