Search

Video saliency prediction has recently attracted attention of the research community, as it is an upstream task for several practical applications. However, current solutions are particularly computationally demanding, especially due to the wide usage of spatio-temporal 3D convolutions. We observe that, while different model architectures achieve similar performance on benchmarks, visual variations between predicted saliency maps are still significant. Inspired by this intuition, we propose a lightweight model that employs multiple simple heterogeneous decoders and adopts several practical approaches to improve accuracy while keeping computational costs low, such as hierarchical multi-map knowledge distillation, multi-output saliency prediction, unlabeled auxiliary datasets and channel reduction with teacher assistant supervision. Our approach achieves saliency prediction accuracy on par or better than state-of-the-art methods on DFH1K, UCF-Sports and Hollywood2 benchmarks, while enhancing significantly the efficiency of the model. Code is on https://github.com/feiyanhu/tinyHD, Comment: WACV2023

HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient todefine a parametric pipeline, called “CLAIRE COVID-19 Universal Pipeline”,which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90% accuracy in detecting COVID-19 lesions in CT scans.

In this paper we present SurfaceNet, an approach for estimating spatially-varying bidirectional reflectance distribution function (SVBRDF) material properties from a single image. We pose the problem as an image translation task and propose a novel patch-based generative adversarial network (GAN) that is able to produce high-quality, high-resolution surface reflectance maps. The employment of the GAN paradigm has a twofold objective: 1) allowing the model to recover finer details than standard translation models; 2) reducing the domain shift between synthetic and real data distributions in an unsupervised way. An extensive evaluation, carried out on a public benchmark of synthetic and real images under different illumination conditions, shows that SurfaceNet largely outperforms existing SVBRDF reconstruction methods, both quantitatively and qualitatively. Furthermore, SurfaceNet exhibits a remarkable ability in generating high-quality maps from real samples without any supervision at training time.

Introduction This study aimed to test the accuracy of a new automatic deep learning–based approach on the basis of convolutional neural networks (CNN) for fully automatic segmentation of the sinonasal cavity and the pharyngeal airway from cone-beam computed tomography (CBCT) scans. Methods Forty CBCT scans from healthy patients (20 women and 20 men; mean age, 23.37 ± 3.34 years) were collected, and manual segmentation of the sinonasal cavity and pharyngeal subregions were carried out by using Mimics software (version 20.0; Materialise, Leuven, Belgium). Twenty CBCT scans from the total sample were randomly selected and used for training the artificial intelligence model file. The remaining 20 CBCT segmentation masks were used to test the accuracy of the CNN fully automatic method by comparing the segmentation volumes of the 3-dimensional models obtained with automatic and manual segmentations. The accuracy of the CNN-based method was also assessed by using the Dice score coefficient and by the surface-to-surface matching technique. The intraclass correlation coefficient and Dahlberg's formula were used to test the intraobserver reliability and method error, respectively. Independent Student t test was used for between-groups volumetric comparison. Results Measurements were highly correlated with an intraclass correlation coefficient value of 0.921, whereas the method error was 0.31 mm3. A mean difference of 1.93 ± 0.73 cm3 was found between the methodologies, but it was not statistically significant (P >0.05). The mean matching percentage detected was 85.35 ± 2.59 (tolerance 0.5 mm) and 93.44 ± 2.54 (tolerance 1.0 mm). The differences, measured as the Dice score coefficient in percentage, between the assessments done with both methods were 3.3% and 5.8%, respectively. Conclusions The new deep learning–based method for automated segmentation of the sinonasal cavity and the pharyngeal airway in CBCT scans is accurate and performs equally well as an experienced image reader.

Currently, advances and affordability in digital data have increased the demand for expedite and automate several diagnostic and clinical tasks. In this regard, Artificial Intelligence is an extraordinary tool destined to support diagnosis, treatment plan and prognosis and to monitoring the progresses of orthodontic treatments. This paper is intended to help orthodontists in getting familiar with and to provide a breakdown of several of the pioneering applications of AI with special regard to Deep Learning and Computer Vision in orthodontics for continued innovation.

This study shows the results of an interdisciplinary research aimed at devising artificial intelligence methods to support data understanding in Cultural Heritage. The objective is to create automated methods for automatically classifying and successively retrieving photographic data leveraging the current breed of AI methods based on Deep Learning paradigm. In this work, the lack of images to be used for the training of the AI system is addressed by testing an approach based on Generative Adversarial Networks (GANs) for automatically synthesizing unrealistic photos, which, in turn, can be used for training image classification and retrieval systems. Specifically, we propose a method to drive the generation of realistic classical order images using GAN approaches anchored to semantic ontology domain representation. More specifically, the proposed AI leverages the advantages of knowledge-based and data-driven methods and foresees a level image generation: the first one that synthesizes isolated objects corresponding to each ontology concept and the second one that, instead, combines the generated objects according to the spatial information provided by the ontology in order to generate realistic scenes. The resulting images can then be employed to train automated classifiers and more reliable retrieval methods by minimizing the efforts usually required by human operators to manually annotate data. In addition, the proposed generative strategy allows for a deeper understanding of cultural heritage visual data enabling the possibility to enhance design capabilities in simulation contexts.

Colorectal cancer, largely arising from precursor lesions called polyps, remains one of the leading causes of cancer-related death worldwide. Current clinical standards require the resection and histopathological analysis of polyps due to test accuracy and sensitivity of optical biopsy methods falling substantially below recommended levels. In this study, we design a novel capsule network architecture (D-Caps) to improve the viability of optical biopsy of colorectal polyps. Our proposed method introduces several technical novelties including a novel capsule architecture with a capsule-average pooling (CAP) method to improve efficiency in large-scale image classification. We demonstrate improved results over the previous state-of-the-art convolutional neural network (CNN) approach by as much as 43%. This work provides an important benchmark on the new Mayo Polyp dataset, a significantly more challenging and larger dataset than previous polyp studies, with results stratified across all available categories, imaging devices and modalities, and focus modes to promote future direction into AI-driven colorectal cancer screening systems. Code is publicly available at https://github.com/lalonderodney/D-Caps ., Accepted for publication at ISBI 2020 (IEEE International Symposium on Biomedical Imaging). Code is publicly available at https://github.com/lalonderodney/D-Caps

This chapter combines traditional artificial intelligence (AI) concepts, i.e., computational ontologies, with more recent trends, i.e., deep learning for content-based semantic retrieval in Cultural Heritage. More specifically, the proposed AI-empowered system employs computational ontologies for modelling photographs of religious historical buildings. The ontology, besides supporting data-modelling and concept-level annotation, guides a learning process – implemented through Convolutional Neural Network (CNN) – for automated image categorization and retrieval. The whole system has been tested on the ruins of the church of Santa Maria delle Grazie in Misterbianco, Catania, Italy, showing satisfactory performance.

Being able to estimate the traversability of the area surrounding a mobile robot is a fundamental task in the design of a navigation algorithm. However, the task is often complex, since it requires evaluating distances from obstacles, type and slope of terrain, and dealing with non-obvious discontinuities in detected distances due to perspective. In this paper, we present an approach based on deep learning to estimate and anticipate the traversing score of different routes in the field of view of an on-board RGB camera. The backbone of the proposed model is based on a state-of-the-art deep segmentation model, which is fine-tuned on the task of predicting route traversability. We then enhance the model's capabilities by a) addressing domain shifts through gradient-reversal unsupervised adaptation, and b) accounting for the specific safety requirements of a mobile robot, by encouraging the model to err on the safe side, i.e., penalizing errors that would cause collisions with obstacles more than those that would cause the robot to stop in advance. Experimental results show that our approach is able to satisfactorily identify traversable areas and to generalize to unseen locations., Accepted at IROS 2020

A key objective of modern medicine is precision medicine, whose purpose is to personalize the treatment based on the specific characteristics of the patients and their illness. To guide treatment decisions, it is generally necessary to have a sample of the neoplastic tissue, which is obtained only with biopsies or similar invasive surgical procedures. As tumors are heterogeneous in their volume and change over time, a dynamic analysis of diagnostic medical images can provide a better understanding of the entire tumor, both in the screening and follow-up phase. In this work, the authors proposed the use of a radiomics pipeline which is able to characterize the possible response of the oncological patients to the anti- programmed death-ligand protein 1 (PD-L1) immunotherapeutic treatment. The immunotherapeutic treatment consists of a modern therapeutic approach in which the physicians try to reactivate the patient's immune system so that it recognizes and destroys cancer cells. The oncological biomarkers capable of characterizing patients who can benefit from immunotherapy from those who would not, are being studied. One of them is related to the expression of the PD-L1 inhibitor in the surface of neoplastic cells which are analyzed in this paper, considering that the analyzed immunotherapeutic treatment is of the anti-PD-L1 type. In this context, the authors propose a pipeline for an immunotherapy response prediction based on the analysis of only CT-scan images of patients with metastatic bladder cancer. Using a framework based on the use of deep Autoeconder network, CT-scan images were analyzed to extract the features capable of discriminating the patient's response to anti-PD-L1 immunotherapy treatment from those who are not. The preliminary results obtained (accuracy of approximately 86% with a sensitivity of approximately 80% against a specificity of approximately 89%) on the analyzed patient dataset, allows the confirmation of the feasibility of the proposed method. Although validated in a dataset containing patients with only one tumor histology (bladder cancer), the proposed method shows how modern radiomics techniques can contribute significantly in the implementation of non-invasive predictive systems that support the physician in the therapeutic choice. The idea of the authors is to create a form of oncological point of care on an embedded platform that allows physicians to always have a support tool in choosing the best therapy to suggest to the patient.

In next-generation cars, safety equipment related to assisted driving systems commonly known as ADAS (advanced driver-assistance systems) are of particular interest for the major car-makers. When we talk about the &ldquo, ADAS system&rdquo, we mean the devices and sensors having the precise objective of improving and making car driving safer, and among which it is worth mentioning rain sensors, the twilight sensor, adaptive cruise control, automatic emergency braking, parking sensors, automatic signal recognition, and so on. All these devices and sensors are installed on the new homologated cars to minimize the risk of an accident and make life on board of the car easier. Some sensors evaluate the movement and the opening of the eyes, the position of the head and its angle, or some physiological signals of the driver obtainable from the palm of the hands placed in the steering. In the present contribution, the authors will present an innovative recognition and monitoring system of the driver&rsquo, s attention level through the study of the photoplethysmographic (PPG) signal detectable from the palm of the driver&rsquo, s hands through special devices housed in the steering of the car. Through a particular and innovative post-processing algorithm of the PPG signal through a hyper-filtering framework, then processed by a machine learning framework, the entire pipeline proposed will be able to recognize and monitor the attention level of the driver with high accuracy and acceptable timing.

AcknowledgementsThe organization of the PlantCLEF task is supported by the French project Floris’Tic (Tela Botanica, INRIA, CIRAD, INRA, IRD) funded in the context of the national investment program PIA. The organization of the BirdCLEF task is supported by the Xeno-Canto foundation for nature sounds as well as the French CNRS project SABIOD.ORG and EADM MADICS, and Floris’Tic. The annotations of some soundscapes were prepared with the late wonderful Lucio Pando at Explorama Lodges, with the support of Pam Bucur, Marie Trone and H. Glotin. The organization of the SeaCLEF task is supported by the Ceta-mada NGO and the French project Floris’Tic.; International audience; Identifying and naming living plants or animals is usually impossible for the general public and often a difficult task for professionals and naturalists. Bridging this gap is a key challenge towards enabling effective biodiversity information retrieval systems. This taxonomic gap was actually already identified as one of the main ecological challenges to be solved during the Rio de Janeiro United Nations “Earth Summit” in 1992. Since 2011, the LifeCLEF challenges conducted in the context of the CLEF evaluation forum have been boosting and evaluating the advances in this domain. Data collections with an unprecedented volume and diversity have been shared with the scientific community to allow repeatable and long-term experiments. This paper describes the methodology of the conducted evaluation campaigns as well as providing a synthesis of the main results and lessons learned along the years.

Video Capsule Endoscopy (VCE) is a diagnostic imaging technology, based on capsule with a built-in camera, that enables screening of the gastro-intestinal tract by reducing the invasiveness of traditional endoscopy procedures. Despite VCE has been designed mainly for investigations on small intestine, it is a powerful tool, because of its low invasiveness and usage simplicity, for supporting large scale screening. However, each VCE video is typically long about eight hours, and endoscopists usually take about two hours, using simple computing methods, for its analysis, thus limiting its application for large scale studies. In this paper, we propose a novel computational framework leveraging the recent advances in artificial intelligence based on the deep learning paradigm to support effectively the whole screening procedure from video transmission to automated lesion identification to reporting. More specifically, our approach handles multiple video uploads at the same time, processes them automatically with the objective of identifying key video frames with potential lesions (for subsequent analysis by endoscopists) and provides physicians with means to compare the findings with either previously detected lesions or with images and scientific information from relevant retrieved documents for a more accurate final diagnosis.

This paper proposes CulTO, a software tool relying on a computational ontology for Cultural Heritage domain modelling, with a specific focus on religious historical buildings, for supporting cultural heritage experts in their investigations. It is specifically thought to support annotation, automatic indexing, classification and curation of photographic data and text documents of historical buildings. CULTO also serves as a useful tool for Historical Building Information Modeling (H-BIM) by enabling semantic 3D data modeling and further enrichment with non-geometrical information of historical buildings through the inclusion of new concepts about historical documents, images, decay or deformation evidence as well as decorative elements into BIM platforms. CulTO is the result of a joint research effort between the Laboratory of Surveying and Architectural Photogrammetry “Luigi Andreozzi” and the PeRCeiVe Lab (Pattern Recognition and Computer Vision Lab) of the University of Catania

Recent GAN-based video generation approaches model videos as the combination of a time-independent scene component and a time-varying motion component, thus factorizing the generation problem into generating background and foreground separately. One of the main limitations of current approaches is that both factors are learned by mapping one source latent space to videos, which complicates the generation task as a single data point must be informative of both background and foreground content. In this paper we propose a GAN framework for video generation that, instead, employs two latent spaces in order to structure the generative process in a more natural way: (1) a latent space to generate the static visual content of a scene (background), which remains the same for the whole video, and (2) a latent space where motion is encoded as a trajectory between sampled points and whose dynamics are modeled through an RNN encoder (jointly trained with the generator and the discriminator) and then mapped by the generator to visual objects’ motion. Performance evaluation showed that our approach is able to control effectively the generation process as well as to synthesize more realistic videos than state-of-the-art methods.

Analysis of medical videos for detection of abnormalities and diseases requires both high precision and recall, but also real-time processing for live feedback and scalability for massive screening of entire populations. Existing work on this field does not provide the necessary combination of retrieval accuracy and performance.; AB@In this paper, a multimedia system is presented where the aim is to tackle automatic analysis of videos from the human gastrointestinal (GI) tract. The system includes the whole pipeline from data collection, processing and analysis, to visualization. The system combines filters using machine learning, image recognition and extraction of global and local image features. Furthermore, it is built in a modular way so that it can easily be extended. At the same time, it is developed for efficient processing in order to provide real-time feedback to the doctors. Our experimental evaluation proves that our system has detection and localisation accuracy at least as good as existing systems for polyp detection, it is capable of detecting a wider range of diseases, it can analyze video in real-time, and it has a low resource consumption for scalability.

Semantic segmentation has been a long standing challenging task in computer vision. It aims at assigning a label to each image pixel and needs a significant number of pixel-level annotated data, which is often unavailable. To address this lack of annotations, in this paper, we leverage, on one hand, a massive amount of available unlabeled or weakly labeled data, and on the other hand, non-real images created through Generative Adversarial Networks. In particular, we propose a semi-supervised framework – based on Generative Adversarial Networks (GANs) – which consists of a generator network to provide extra training examples to a multi-class classifier, acting as discriminator in the GAN framework, that assigns sample a label y from the K possible classes or marks it as a fake sample (extra class). The underlying idea is that adding large fake visual data forces real samples to be close in the feature space, which, in turn, improves multiclass pixel classification. To ensure a higher quality of generated images by GANs with consequently improved pixel classification, we extend the above framework by adding weakly annotated data, i.e., we provide class level information to the generator. We test our approaches on several challenging benchmarking visual datasets, i.e. PASCAL, SiftFLow, Stanford and CamVid, achieving competitive performance compared to state-of-the-art semantic segmentation methods.

Health care has a long history of adopting technology to save lives and improve the quality of living. Visual information is frequently applied for disease detection and assessment, and the established fields of computer vision and medical imaging provide essential tools. It is, however, a misconception that disease detection and assessment are provided exclusively by these fields and that they provide the solution for all challenges. Integration and analysis of data from several sources, real-time processing, and the assessment of usefulness for end-users are core competences of the multime- dia community and are required for the successful improvement of health care systems. For the benefit of society, the multimedia community should recognize the challenges of the medical world that they are uniquely qualified to address. We have conducted initial investigations into two use cases surrounding diseases of the gastrointestinal (GI) tract, where the detection of abnormali- ties provides the largest chance of successful treatment if the initial observation of disease indicators occurs before the patient notices any symptoms. Although such detection is typically provided vi- sually by applying an endoscope, we are facing a multitude of new multimedia challenges that differ between use cases. In real-time assistance for colonoscopy, we combine sensor information about camera position and direction to aid in detecting, investigate means for providing support to doctors in unobtrusive ways, and assist in reporting. In the area of large-scale capsular endoscopy, we investigate questions of scalability, performance and energy efficiency for the recording phase, and combine video summarization and retrieval questions for analysis.

International audience; Using multimedia identification tools is considered as one of the most promising solutions to help bridge the taxonomic gap and build accurate knowledge of the identity, the geographic distribution and the evolution of living species. Large and structured communities of nature observers (e.g., iSpot, Xeno-canto, Tela Botanica, etc.) as well as big monitoring equipment have actually started to produce outstanding collections of multimedia records. Unfortunately, the performance of the state-of-the-art analysis techniques on such data is still not well understood and is far from reaching real world requirements. The LifeCLEF lab proposes to evaluate these challenges around 3 tasks related to multimedia information retrieval and fine-grained classification problems in 3 domains. Each task is based on large volumes of real-world data and the measured challenges are defined in collaboration with biologists and environmental stakeholders to reflect realistic usage scenarios. For each task, we report the methodology, the data sets as well as the results and the main outcome

Human annotated data is a prerequisite for the training and evaluation of computer vision algorithms. Such data is referred to as “ground truth” data. In this chapter, we describe the strategies and systems we have devised in order to obtain the ground truth data required by each of the computer vision components within the Fish4Knowledge system, including fish detection, tracking, and recognition.

International audience; In this paper we investigate the fine-grained object categorization problem of determining fish species in low-quality visual data (images and videos) recorded in real-life settings. We first describe a new annotated dataset of about 35,000 fish images (MA-35K dataset), derived from the Fish4Knowledge project, covering 10 fish species from the Eastern Indo-Pacific bio-geographic zone. We then resort to a label propagation method able to transfer the labels from the MA-35K to a set of 20 million fish images in order to achieve variability in fish appearance. The resulting annotated dataset, containing over one million annotations (AA-1M), was then manually checked by removing false positives as well as images with occlusions between fish or showing partially fish. Finally, we randomly picked more than 30,000 fish images distributed among ten fish species and extracted from about 400 10-minute videos, and used this data (both images and videos) for the fish task of the LifeCLEF 2014 contest. Together with the fine-grained visual dataset release, we also present two approaches for fish species classification in, respectively, still images and videos. Both approaches showed high performance (for some fish species the precision and recall were close to one) in object classification and outperformed state-of-the-art methods. In addition, despite the fact that dataset is unbalanced in the number of images per species, both methods (especially the one operating on still images) appear to be rather robust against the long-tail curse of data, showing the best performance on the less populated object classes.

Label propagation consists in annotating an unlabeled dataset starting from a set of labeled items. However, most current methods exploit only image similarity between labeled and unlabeled images in order to find propagation candidates, which may result, especially in very large datasets, in retrieving mostly near-duplicate images. While such approaches are technically correct, as they maximize the propagation precision, the resulting annotated dataset may not be as useful, since they lack intra-class variability within the set of images sharing the same label. In this paper, we propose an approach for label propagation which favors the propagation of an object's label to a set of images representing as many different views of that object as possible, while at the same time preserving the relevance of the retrieved items to the query. Our method is based on a diversity-based clustering technique using a random forest framework and a label propagation approach which is able to effectively and efficiently propagate annotations using a similarity-based approach operating on clusters. The method was tested on a very large dataset of fish images achieving good performance in automated label propagation, ensuring diversification of the annotated items while preserving precision.

Aim of the paper is to illustrate the methodology used to implement an ubiquitous city platform called Wi-City-Plus provided with mobile and centralized Decision Support Systems (DSSs) taking advantage of all the data of city interest, including social data, and those sensed by networked ambient Cyber-Physical-Systems (CPSs). The paper proposes to model such data by a suitable RDF ontology specifically designed to support the main user scenarios in smart cities and to implement both the technical and the service interoperability envisaged by the CPS ideal model. This ontology is obtained by reusing existing ontologies and allows the DSSs to access by SPARQL queries all the relevant data independent of the proprietary technology of the CPSs and of the DBMS adopted by the data owners. Examples derived from the pilot version of the proposed platform allows us to point out advantages and perspectives of our approach.

In this paper we present an approach for segmenting objects in videos taken in complex scenes with multiple and different targets. The method does not make any specific assumptions about the videos and relies on how objects are perceived by humans according to Gestalt laws. Initially, we rapidly generate a coarse foreground segmentation, which provides predictions about motion regions by analyzing how superpixel segmentation changes in consecutive frames. We then exploit these location priors to refine the initial segmentation by optimizing an energy function based on appearance and perceptual organization, only on regions where motion is observed. We evaluated our method on complex and challenging video sequences and it showed significant performance improvements over recent state-of-the-art methods, being also fast enough to be used for “on-the-fly” processing.

International audience; Using multimedia identification tools is considered as one of the most promising solutions to help bridging the taxonomic gap and build accurate knowledge of the identity, the geographic distribution and the evolution of living species. Large and structured communities of nature observers (e.g. eBird, Xeno-canto, Tela Botanica, etc.) as well as big monitoring equipments have actually started to produce outstanding collections of multimedia records. Unfortunately, the performance of the state-of-the-art analysis techniques on such data is still not well understood and is far from reaching the real world's requirements. The LifeCLEF lab proposes to evaluate these challenges around 3 tasks related to multimedia information retrieval and fine-grained classification problems in 3 living worlds. Each task is based on large and real-world data and the measured challenges are defined in collaboration with biologists and environmental stakeholders in order to reflect realistic usage scenarios. This paper presents more particularly the 2015 edition of Life-CLEF. For each of the three tasks, we report the methodology and the data sets as well as the raw results and the main outcomes.

Chapitre : Information Access Evaluation. Multilinguality, Multimodality, and Interaction; International audience; Using multimedia identification tools is considered as one of the most promising solutions to help bridging the taxonomic gap and build accurate knowledge of the identity, the geographic distribution and the evolution of living species. Large and structured communities of nature observers (e.g. eBird, Xeno-canto, Tela Botanica, etc.) as well as big monitoring equipments have actually started to produce outstanding collections of multimedia records. Unfortunately, the performance of the state-of-the-art analysis techniques on such data is still not well understood and is far from reaching the real world’s requirements. The LifeCLEF lab proposes to evaluate these challenges around three tasks related to multimedia information retrieval and fine-grained classification problems in three living worlds. Each task is based on large and real-world data and the measured challenges are defined in collaboration with biologists and environmental stakeholders in order to reflect realistic usage scenarios. This paper presents more particularly the 2014 edition of LifeCLEF, i.e. the pilot one. For each of the three tasks, we report the methodology and the datasets as well as the official results and the main outcomes.

The use of low-cost devices for depth estimation, such as Microsoft Kinect, is becoming more and more popular in computer vision research. In this paper, we propose an algorithm for background modeling which exploits this kind of devices to make the background and foreground models more robust to effects such as camouflage and illumination changes. Our algorithm, after a preprocessing stage for aligning color and depth data and for filtering/filling noisy depth measurements, explicitly models the scene's background and foreground with a Kernel Density Estimation approach in a quantized x-y-hue-saturation-depth space. The results in three different indoor environments, with different lighting conditions, showed that our approach is able to achieve an accuracy in foreground segmentation over 90% and that the combination of depth data and illumination-independent color space proved to be very robust against noise and illumination changes.

The study of fish populations in their own natural environment is a task that has usually been tackled in invasive ways which inevitably influenced the behavior of the fish under observation. Recent projects involving the installation of permanent underwater cameras (e.g. the Fish4Knowledge (F4K) project, for the observation of Taiwan’s coral reefs) allow to gather huge quantities of video data, without interfering with the observed environment, but at the same time require the development of automatic processing tools, since manual analysis would be impractical for such amounts of videos. Event detection is one of the most interesting aspects from the biologists’ point of view, since it allows the analysis of fish activity during particular events, such as typhoons. In order to achieve this goal, in this paper we present an automatic video analysis approach for fish behavior understanding during typhoon events. The first step of the proposed system, therefore, involves the detection of “typhoon” events and it is based on video texture analysis and on classification by means of Support Vector Machines (SVM). As part of our behavior understanding efforts, trajectory extraction and clustering have been performed to study the differences in behavior when disruptive events happen. The integration of event detection with fish behavior understanding surpasses the idea of simply detecting events by low-level features analysis, as it supports the full semantic comprehension of interesting events.

We present a research tool that supports marine ecologists' research by allowing analysis of long-term and continuous fish monitoring video content. The analysis can be used for instance to discover ecological phenomena such as changes in fish abundance and species composition over time and area. Two characteristics set our system apart from traditional ecological data collecting and processing methods. First, the continuous video recording results in enormous data volumes of monitoring data. Currently around a year of video recordings (containing over the 4 million fish observations) have been processed. Second, different from traditional manual recording and analysing the ecological data, the whole recording, analysing and presentation of results is automated in this system. On one hand, it saves the effort of manually examining every video, which is infeasible. On the other hand, no automatic video analysis method is perfect, so the user interface provides marine ecologists with multiple options to verify the data. Marine ecologists can examine the underlying videos, check results of automatic video analysis at different certainty levels computed by our system, and compare results generated by multiple versions of automatic video analysis software to verify the data in our system. This research tool enables marine ecologists for the first time to analyse long-term and continuous underwater video records.

Understanding and analyzing fish behaviour is a fundamental task for biologists that study marine ecosystems because the changes in animal behaviour reflect environmental conditions such as pollution and climate change. To support investigators in addressing these complex questions, underwater cameras have been recently used. They can continuously monitor marine life while having almost no influence on the environment under observation, which is not the case with observations made by divers for instance. However, the huge quantity of recorded data make the manual video analysis practically impossible. Thus machine vision approaches are needed to distill the information to be investigated. In this paper, we propose an automatic event detection system able to identify solitary and pairing behaviours of the most common fish species of the Taiwanese coral reef. More specifically, the proposed system employs robust low-level processing modules for fish detection, tracking and recognition that extract the raw data used in the event detection process. Then each fish trajectory is modeled and classified using hidden Markov models. The events of interest are detected by integrating end-user rules, specified through an ad hoc user interface, and the analysis of fish trajectories. The system was tested on 499 events of interest, divided into solitary and pairing events for each fish species. It achieved an average accuracy of 0.105, expressed in terms of normalized detection cost. The obtained results are promising, especially given the difficulties occurring in underwater environments. And moreover, it allows marine biologists to speed up the behaviour analysis process, and to reliably carry on their investigations.

In this paper we present a flash game that aims at generating easily ground truth for testing object detection algorithms. Flash the Fish is an online game where the user is shown videos from underwater environments and has to take photos of fish by clicking on them. The initial ground truth is provided by object detection algorithms and, subsequent, cluster analysis and user evaluation techniques, allow for the generation of ground truth based on the weighted combination of these "photos". Evaluation of the platform and comparison of the obtained results against a hand drawn ground truth confirmed that reliable ground truth generation is not necessarily a cumbersome task both in terms of effort and time needed.

In this paper we present an algorithm for visual object detection in a underwater real-life context which explicitly models both the background and the foreground for each frame - thus helping to avoid foreground absorption into similar background -, and integrates both colour and texture features (which have proved effective in overcoming the limitations of colour-only appearance descriptors) into a covariance-based model, which provides an elegant way to merge multiple features together and enforce structural relationships. A joint domain-range model combined to a post-processing approach based on Markov Random Field takes into account the spatial dependency between pixels in the classification process, unlike the classical pixel-oriented modeling techniques. Our results show the effectiveness of this approach in the underwater environment, which presents a lot of variety in scene conditions, objects' motion patterns, shapes and colouring, and background activity.

Vascular cognitive impairment-no dementia (VCI-ND) is a condition at risk for future dementia and should be the target of preventive strategies. Recently, an enhanced intracortical facilitation observed in VCI-ND patients was proposed as a candidate neurophysiological marker of the disease process. The aim of this study was to monitor the excitability of the motor cortex and the functioning of excitatory/inhibitory intracortical circuits in patients with VCI-ND after a follow-up period of approximately 2 years, in order to pick out early markers of disease progression into dementia. Nine patients and 9 age-matched controls were re-evaluated for single and paired pulse TMS measures of cortical excitability, as well as for neuropsycological and functional assessment. Compared to the first evaluation, patients showed a decrease of the median resting motor threshold (rMT). Patients exhibited a significant worsening at Stroop Color-Word Test Interference scores without substantial functional impairment. Our study represents the first evidence of a decrease of rMT in VCI-ND patients during the progression of cognitive impairment. This result might be considered an index of motor cortex plasticity and interpreted as a compensatory mechanism for the loss of motor cortex neurons.