Search

Introduction of large training datasets was essential for the recent advancement and success of deep learning methods. Due to the difficulties related to biometric data collection, facial image datasets with biometric trait labels are scarce and usually limited in terms of size and sample diversity. Web-scraping approaches for automatic data collection can produce large amounts of weakly labeled and noisy data. This work is focused on picking out the bad apples from web-scraped facial datasets by automatically removing erroneous samples that impair their usability. The unsupervised facial biometric data filtering method presented in this work greatly reduces label noise levels in web-scraped facial biometric data. Experiments on two large state-of-the-art web-scraped datasets demonstrate the effectiveness of the proposed method with respect to real and apparent age estimation based on five different age estimation methods. Furthermore, we apply the proposed method, together with a newly devised strategy for merging multiple datasets, to data collected from three major web-based data sources (i.e., IMDb, Wikipedia, Google) and derive the new Biometrically Filtered Famous Figure Dataset or B3FD. The proposed dataset, which is made publicly available, enables considerable performance gains for all tested age estimation methods and age estimation tasks. This work highlights the importance of training data quality compared to data quantity and selection of the estimation method. Funding: The author K. Besenic receives Ph.D. scholarship from the company Visage Technologies.

The surveillance of wild animal populations is important for wildlife sustainability, conservation and management. It has been estimated that the UAV-based survey of 100 ha large territory is ~10 times less time-consuming in comparison to surveys based on traditional field visits. Aerial surveys using thermal and visible light cameras allow remote observation of wildlife over relatively large geographical areas where the thermal imager is often used as a primary sensor for the detection of animal shape similar hot-spot, but higher-resolution visible light imaging data is used for the reduction of false-positive detections. Recent developments in unmanned aerial vehicles (UAVs), artificial intelligence and miniaturized dual imaging systems made it more flexible, affordable and accurate for aerial surveillance of wild animals. This study was conducted as part of project “ICT-based wild animal census approach for sustainable wildlife management” co-financed by the ERDF program “Industry-Driven Research” (dnr 1.1.1.1/18/A/146) and managed by the Institute for Environmental Solutions, Latvia. One part of the project activity is to develop the detection and classification workflow of wild animals from areal imaging data. This study describing data acquisition, detection and automated data pre-processing of thermal and RGB image co-registration as input for the development of animal classification algorithm. The focus of the study was a detection of the four dominant even-toed ungulate species in Latvia - elk (Alces alces), red deer (Cervus elaphus), roe deer (Capreolus capreolus) and wild boar (Sus scrofa). The data acquisition was performed over the fenced deer garden and open forest pilot territory located in Ramuļi, Latvia. The chosen UAV system was a quadrocopter platform with a dual-camera on the board. Initially, the main focus in data acquisition was over-fenced deer garden at different day times, weather conditions to collect data with animal presence as well as test different data acquisition regimes, strategies and animal behavioral response. Three flights with total coverage were performed over the deer garden area. After the post-detection of individuals, the average estimated accuracy was 88% of the known reference number of deers. Further on, drone flights were conducted over the whole pilot territory to obtain data of other species and behavioral overview in open forest land conditions. All collected data were registered in the database to annotate the weather conditions and the presence of an animal in a certain minute. In total 10 flights (3 h) were performed over the deer garden and 93 (45 h) flights over the open forest land pilot territory. The capabilities of the drone-based monitoring system with a dual-camera imaging setup will be presented.Keywords: UAV, elk, deer, roe deer, areal imagery, dual-camera, detection

Face alignment is the process of determining a face shape given its location and size in an image. It is used as a basis for other facial analysis tasks and for human-machine interaction and augmented reality applications. It is a challenging problem due to the extremely high variability in facial appearance affected by many external (illumination, occlusion, head pose) and internal factors (race, facial expression). However, advances in deep learning combined with domain-related knowledge from previous research recently demonstrated impressive results nearly saturating the unconstrained benchmark data sets. The focus is shifting towards reducing the computational burden of the face alignment models since real-time performance is required for such a highly dynamic task. Furthermore, many applications target devices on the edge with limited computational power which puts even greater emphasis on computational efficiency. We present the latest development in regression-based approaches that have led towards nearly solving the face alignment problem in an unconstrained scenario. Various regression architectures are systematically explored and recent training techniques discussed in the context of face alignment. Finally, a benchmark comparison of the most successful methods is presented, taking into account execution time as well, to provide a comprehensive overview of this dynamic research field.

Facial expression recognition applications demand accurate and fast algorithms that can run in real time on platforms with limited computational resources. We propose an algorithm that bridges the gap between precise but slow methods and fast but less precise methods. The algorithm combines gentle boost decision trees and neural networks. The gentle boost decision trees are trained to extract highly discriminative feature vectors (local binary features) for each basic facial expression around distinct facial landmark points. These sparse binary features are concatenated and used to jointly optimize facial expression recognition through a shallow neural network architecture. The joint optimization improves the recognition rates of difficult expressions such as fear and sadness. Furthermore, extensive experiments in both within- and cross-database scenarios have been conducted on relevant benchmark data sets for facial expression recognition: CK+, MMI, JAFFE, and SFEW 2.0. The proposed method (LBF-NN) compares favorably with state-of-the-art algorithms while achieving an order of magnitude improvement in execution time.

Availability of large training datasets was essential for the recent advancement and success of deep learning methods. Due to the difficulties related to biometric data collection, datasets with age and gender annotations are scarce and usually limited in terms of size and sample diversity. Web- scraping approaches for automatic data collection can produce large amounts weakly labeled noisy data. The unsupervised facial biometric data filtering method presented in this paper greatly reduces label noise levels in web-scraped facial biometric data. Experiments on two large state-of-the-art web- scraped facial datasets demonstrate the effectiveness of the proposed method, with respect to training and validation scores, training convergence, and generalization capabilities of trained age and gender estimators.

Current best local descriptors are learned on a large dataset of matching and non-matching keypoint pairs. However, data of this kind is not always available since detailed keypoint correspondences can be hard to establish. On the other hand, we can often obtain labels for pairs of keypoint bags. For example, keypoint bags extracted from two images of the same object under different views form a matching pair, and keypoint bags extracted from images of different objects form a non-matching pair. On average, matching pairs should contain more corresponding keypoints than non-matching pairs. We describe an end-to-end differentiable architecture that enables the learning of local keypoint descriptors from such weakly-labeled data. Additionally, we discuss how to improve the method by incorporating the procedure of mining hard negatives. We also show how can our approach be used to learn convolutional features from unlabeled video signals and 3D models. Our implementation is available at https://github.com/nenadmarkus/wlrn, This version has been accepted for publication in IEEE Transactions on Image Processing (presents methodological and experimental improvements of our ICPR2016 paper)

We combine the near-sensor image processing concept with address-event representation leading to an intensity-ranking image sensor (IRIS) and show the benefits of using this type of sensor for image classification. The functionality of IRIS is to output pixel coordinates (X and Y values) continuously as each pixel has collected a certain number of photons. Thus, the pixel outputs will be automatically intensity ranked. By keeping track of the timing of these events, it is possible to record the full dynamic range of the image. However, in many cases, this is not necessary-the intensity ranking in itself gives the needed information for the task at hand. This paper describes techniques for classification and proposes a particular variant (groves) that fits the IRIS architecture well as it can work on the intensity rankings only. Simulation results using the CIFAR-10 dataset compare the results of the proposed method with the more conventional ferns technique. It is concluded that the simultaneous sensing and classification obtainable with the IRIS sensor yields both fast (shorter than full exposure time) and processing-efficient classification. Funding Agencies|Swedish Research Council (Vetenskapsradet) [2014-6227]

An image mosaic is an assembly of a large number of small images, usually called tiles, taken from a specific dictionary/codebook. When viewed as a whole, the appearance of a single large image emerges, i.e. each tile approximates a small block of pixels. ASCII art is a related and older graphic design technique for producing images from printable characters. Although automatic procedures for both of these visualization schemes have been studied in the past, some are computationally heavy and cannot offer real-time and interactive performance. We propose an algorithm able to reproduce the quality of existing non-photorealistic rendering techniques, in particular ASCII art and image mosaics, obtaining large performance speed-ups. The basic idea is to partition the input image into a rectangular grid and use a decision tree to assign a tile from a pre-determined codebook to each cell. Our implementation can process video streams from webcams in real time and it is suitable for modestly equipped devices. We evaluate our technique by generating the renderings of a variety of images and videos, with good results. The source code of our engine is publicly available.

Privacy protection may be defined as replacing the original content in an image region with a new (less intrusive) content having modified target appearance information to make it less recognizable by applying a privacy protection technique. Indeed the development of privacy protection techniques needs also to be complemented with an established objective evaluation method to facilitate their assessment and comparison. Generally, existing evaluation methods rely on the use of subjective judgements or assume a specific target type in image data and use target detection and recognition accuracies to assess privacy protection. This work proposes a new annotation-free evaluation method that is neither subjective nor assumes a specific target type. It assesses two key aspects of privacy protection: protection and utility. Protection is quantified as an appearance similarity and utility is measured as a structural similarity between original and privacy-protected image regions. We performed an extensive experimentation using six challenging datasets (having 12 video sequences) including a new dataset (having six sequences) that contains visible and thermal imagery. The new dataset, called TST-Priv, is made available online below for community. We demonstrate effectiveness of the proposed method by evaluating six image-based privacy protection techniques, and also show comparisons of the proposed method over existing methods. Funding agencies: Swedish Research Council through the project Learning Systems for Remote Thermography [D0570301]; European Community [312784]

Mammalian herbivores have important top-down effects on ecological processes and landscapes by generating vegetation changes through grazing and trampling. For free-ranging herbivores on large landscapes, trampling is an important ecological factor. However, whereas grazing is widely studied, low-intensity trampling is rarely studied and quantified. The cold-adapted northern tundra reindeer (Rangifer tarandus) is a wide-ranging keystone herbivore in large open alpine and Arctic ecosystems. Reindeer may largely subsist on different species of slow-growing ground lichens, particularly in winter. Lichen grows in dry, snow-poor habitats with frost. Their varying elasticity makes them suitable for studying trampling. In replicated factorial experiments, high-resolution 3D laser scanning was used to quantify lichen volume loss from trampling by a reindeer hoof. Losses were substantial, that is, about 0.3 dm3 per imprint in dry thick lichen, but depended on type of lichen mat and humidity. Immediate trampling volume loss was about twice as high in dry, compared to humid thin (2–3 cm), lichen mats and about three times as high in dry vs. humid thick (6–8 cm) lichen mats, There was no significant difference in volume loss between 100% and 50% wetted lichen. Regained volume with time was insignificant for dry lichen, whereas 50% humid lichen regained substantial volumes, and 100% humid lichen regained almost all lost volume, and mostly within 10–20 min. Reindeer trampling may have from near none to devastating effects on exposed lichen forage. During a normal week of foraging, daily moving 5 km across dry 6- to 8-cm-thick continuous lichen mats, one adult reindeer may trample a lichen volume corresponding to about a year's supply of lichen. However, the lichen humidity appears to be an important factor for trampling loss, in addition to the extent of reindeer movement. Funding agencies: Oslofjorden Regional Research Fund; University College of Southeast Norway

We address the problem of estimating atmosphere parameters (temperature and water vapor content) from data captured by an airborne thermal hyperspectral imager and propose a method based on linear and nonlinear optimization. The method is used for the estimation of the parameters (temperature and emissivity) of the observed object as well as sensor gain under certain restrictions. The method is analyzed with respect to sensitivity to noise and the number of spectral bands. Simulations with synthetic signatures are performed to validate the analysis, showing that the estimation can be performed with as few as 10-20 spectral bands at moderate noise levels. The proposed method is also extended to exploit additional knowledge, for example, measurements of atmospheric parameters and sensor noise. Additionally, we show how to extend the method in order to improve spectral calibration. Funding Agencies|Swedish Research Council under Project EMC2

Hyperspectral remote sensing based on unmanned airborne vehicles is a field increasing in importance. The combined functionality of simultaneous hyperspectral and geometric modeling is less developed. A configuration has been developed that enables the reconstruction of the hyperspectral three-dimensional (3D) environment. The hyperspectral camera is based on a linear variable filter and a high frame rate, high resolution camera enabling point-to-point matching and 3D reconstruction. This allows the information to be combined into a single and complete 3D hyperspectral model. In this paper, we describe the camera and illustrate capabilities and difficulties through real-world experiments.

We propose a novel method for iterative learning of point correspondences between image sequences. Points moving on surfaces in 3D space are projected into two images. Given a point in either view, the considered problem is to determine the corresponding location in the other view. The geometry and distortions of the projections are unknown as is the shape of the surface. Given several pairs of point-sets but no access to the 3D scene, correspondence mappings can be found by excessive global optimization or by the fundamental matrix if a perspective projective model is assumed. However, an iterative solution on sequences of point-set pairs with general imaging geometry is preferable. We derive such a method that optimizes the mapping based on Neyman's chi-square divergence between the densities representing the uncertainties of the estimated and the actual locations. The densities are represented as channel vectors computed with a basis function approach. The mapping between these vectors is updated with each new pair of images such that fast convergence and high accuracy are achieved. The resulting algorithm runs in real-time and is superior to state-of-the-art methods in terms of convergence and accuracy in a number of experiments. funding agencies|EC|215078247947|ELLIIT||Strategic Area for ICT research||CADICS||Swedish Government||Swedish Research Council||CUAS||FOCUS||Swedish Foundation for Strategic Research|| DIPLECS GARNICS ELLIIT ETT CUAS FOCUS CADICS

We address short-term, single-object tracking, a topic that is currently seeing fast progress for visual video, for the case of thermal infrared (TIR) imagery. The fast progress has been possible thanks to the development of new template-based tracking methods with online template updates, methods which have not been explored for TIR tracking. Instead, tracking methods used for TIR are often subject to a number of constraints, e.g., warm objects, low spatial resolution, and static camera. As TIR cameras become less noisy and get higher resolution these constraints are less relevant, and for emerging civilian applications, e.g., surveillance and automotive safety, new tracking methods are needed. Due to the special characteristics of TIR imagery, we argue that template-based trackers based on distribution fields should have an advantage over trackers based on spatial structure features. In this paper, we propose a template-based tracking method (ABCD) designed specifically for TIR and not being restricted by any of the constraints above. In order to avoid background contamination of the object template, we propose to exploit background information for the online template update and to adaptively select the object region used for tracking. Moreover, we propose a novel method for estimating object scale change. The proposed tracker is evaluated on the VOT-TIR2015 and VOT2015 datasets using the VOT evaluation toolkit and a comparison of relative ranking of all common participating trackers in the challenges is provided. Further, the proposed tracker, ABCD, and the VOT-TIR2015 winner SRDCFir are evaluated on maritime data. Experimental results show that the ABCD tracker performs particularly well on thermal infrared sequences.

The Thermal Infrared Visual Object Tracking challenge 2016, VOT-TIR2016, aims at comparing short-term single-object visual trackers that work on thermal infrared (TIR) sequences and do not apply pre-learned models of object appearance. VOT-TIR2016 is the second benchmark on short-term tracking in TIR sequences. Results of 24 trackers are presented. For each participating tracker, a short description is provided in the appendix. The VOT-TIR2016 challenge is similar to the 2015 challenge, the main difference is the introduction of new, more difficult sequences into the dataset. Furthermore, VOT-TIR2016 evaluation adopted the improvements regarding overlap calculation in VOT2016. Compared to VOT-TIR2015, a significant general improvement of results has been observed, which partly compensate for the more difficult sequences. The dataset, the evaluation kit, as well as the results are publicly available at the challenge website.

We address two problems related to large-scale aerial monitoring of district heating networks. First, we propose a classification scheme to reduce the number of false alarms among automatically detected leakages in district heating networks. The leakages are detected in images captured by an airborne thermal camera, and each detection corresponds to an image region with abnormally high temperature. This approach yields a significant number of false positives, and we propose to reduce this number in two steps; by (a) using a building segmentation scheme in order to remove detections on buildings, and (b) to use a machine learning approach to classify the remaining detections as true or false leakages. We provide extensive experimental analysis on real-world data, showing that this post-processing step significantly improves the usefulness of the system. Second, we propose a method for characterization of leakages over time, i.e., repeating the image acquisition one or a few years later and indicate areas that suffer from an increased energy loss. We address the problem of finding trends in the degradation of pipe networks in order to plan for long-term maintenance, and propose a visualization scheme exploiting the consecutive data collections. Funding Agencies|Swedish Research Council (Vetenskapsradet) through project Learning systems for remote thermography [621-2013-5703]; Swedish Research Council [2014-6227]

We propose a method for detecting obstacles on the railway in front of a moving train using a monocular thermal camera. The problem is motivated by the large number of collisions between trains and various obstacles, resulting in reduced safety and high costs. The proposed method includes a novel way of detecting the rails in the imagery, as well as a way to detect anomalies on the railway. While the problem at a first glance looks similar to road and lane detection, which in the past has been a popular research topic, a closer look reveals that the problem at hand is previously unaddressed. As a consequence, relevant datasets are missing as well, and thus our contribution is two-fold: We propose an approach to the novel problem of obstacle detection on railways and we describe the acquisition of a novel data set.

An image mosaic is an assembly of a large number of small images, usually called tiles, taken from a specific dictionary/codebook. When viewed as a whole, the appearance of a single large image emerges, i.e., each tile approximates a small block of pixels. ASCII art is a related (and older) graphic design technique for producing images from printable characters. Although automatic procedures for both of these visualization schemes have been studied in the past, some are computationally heavy and cannot offer real-time and interactive performance. We propose an algorithm able to reproduce the quality of existing non-photorealistic rendering techniques, in particular ASCII art and image mosaics, obtaining large performance speed-ups. The basic idea is to partition the input image into a rectangular grid and use a decision tree to assign a tile from a predetermined codebook to each cell. Our implementation can process video streams from webcams in real-time and it is suitable for modestly equipped devices. We evaluate our technique by generating the renderings of a variety of images and videos, with good results. The source code of our engine is publicly available.

Short-term single-object (STSO) tracking in thermal images is a challenging problem relevant in a growing number of applications. In order to evaluate STSO tracking algorithms on visual imagery, there are de facto standard benchmarks. However, we argue that tracking in thermal imagery is different than in visual imagery, and that a separate benchmark is needed. The available thermal infrared datasets are few and the existing ones are not challenging for modern tracking algorithms. Therefore, we hereby propose a thermal infrared benchmark according to the Visual Visual Object Tracking (VOT) protocol for evaluation of STSO tracking methods. The benchmark includes the new LTIR dataset containing 20 thermal image sequences which have been collected from multiple sources and annotated in the format used in the VOT Challenge. In addition, we show that the ranking of different tracking principles differ between the visual and thermal benchmarks, confirming the need for the new benchmark.

In this paper, we consider fitting a 3D deformable face model to continuous video sequences for the tasks of tracking and training. We propose two appearance-based methods that only require a simple statistical facial texture model and do not require any information about an empirical or analytical gradient matrix, since the best search directions are estimated on the fly. The first method computes the fitting using a locally exhaustive and directed search where the 3D head pose and the facial actions are simultaneously estimated. The second method decouples the estimation of these parameters. It computes the 3D head pose using a robust feature-based pose estimator incorporating a facial texture consistency measure. Then, it estimates the facial actions with an exhaustive and directed search. Fitting and tracking experiments demonstrate the feasibility and usefulness of the developed methods. A performance evaluation also shows that the proposed methods can outperform the fitting based on an active appearance model search adopting a pre-computed gradient matrix. Although the proposed schemes are not as fast as the schemes adopting a directed continuous search, they can tackle many disadvantages associated with such approaches.

In this paper, we address the 3D tracking of pose and animation of the human face in monocular image sequences using deformable 3D models. The main contributions of this paper are as follows. First, we show how the robustness and stability of the Active Appearance Algorithm can be improved through the inclusion of a simple motion compensation based on feature correspondence. Second, we develop a new method able to adapt a deformable 3D model to a face in the input image. Central to this method is the decoupling of global head movements and local non-rigid deformations/animations. This decoupling is achieved by, first, estimating the global (rigid) motion using robust statistics and a statistical model for face texture, and then, adapting the 3D model to possible local animations using the concept of the Active Appearance Algorithm. This proposed method constitutes a significant step towards reliable model-based face trackers since the strengths of complementary tracking methodologies are combined. Experiments evaluating the effectiveness of the methods are reported. Adaptation and tracking examples demonstrate the feasibility and robustness of the developed methods.

We present a face and facial feature tracking system able to extract animation parameters describing the motion and articulation of a human face in real-time on consumer hardware. The system is based on a statistical model of face appearance and a search algorithm for adapting the model to an image. Speed and robustness is discussed, and the system evaluated in terms of accuracy.

District heating is a common way of providing heat to buildings in urban areas. The heat is carried by hot water or steam and distributed in a network of pipes from a central power plant. It is of great interest to minimize energy losses due to bad pipe insulation or leakages in such district heating networks. As the pipes generally are placed underground, it may be difficult to establish the presence and location of losses and leakages. Toward this end, this work presents methods for large-scale monitoring and detection of leakages by means of remote sensing using thermal cameras, so-called airborne thermography. The methods rely on the fact that underground losses in district heating systems lead to increased surface temperatures. The main contribution of this work is methods for automatic analysis of aerial thermal images to localize leaking district heating pipes. Results and experiences from large-scale leakage detection in several cities in Sweden and Norway are presented.

The interest for thermography as a method for spot weld inspection has increased during the last years since it is a full-field method suitable for automatic inspection. Thermography systems can be developed in different ways, with different physical setups, excitation sources, and image analysis algorithms. In this paper we suggest a single-sided setup of a thermography system using a flash lamp as excitation source. The analysis algorithm aims to find the spatial region in the acquired images corresponding to the successfully welded area, i.e., the nugget size. Experiments show that the system is able to detect spot welds, measure the nugget diameter, and based on the information also separate a spot weld from a stick weld. The system is capable to inspect more than four spot welds per minute, and has potential for an automatic non-destructive system for spot weld inspection. The development opportunities are significant, since the algorithm used in the initial analysis is rather simplified. Moreover, further evaluation of alternative excitation sources can potentially improve the performance. Funding Agencies|Vinnova (the FFI program)

We describe a method for eye pupil localization based on an ensemble of randomized regression trees and use several publicly available datasets for its quantitative and qualitative evaluation. The method compares well with reported state-of-the-art and runs in real-time on hardware with limited processing power, such as mobile devices. HighlightsA framework for eye pupil localization that compares well with state-of-the-art.Randomization during runtime improves performance.The developed system works in real-time on mobile devices.

We address the problem of reducing the number offalse alarms among automatically detected leakages in districtheating networks. The leakages are detected in images capturedby an airborne thermal camera, and each detection correspondsto an image region with abnormally high temperature. Thisapproach yields a significant number of false positives, and wepropose to reduce this number in two steps. First, we use abuilding segmentation scheme in order to remove detectionson buildings. Second, we extract features from the detectionsand use a Random forest classifier on the remaining detections.We provide extensive experimental analysis on real-world data,showing that this post-processing step significantly improves theusefulness of the system.