Search

Hyperspectral Imagery (HSI) has been used in many applications to non-destructively determine the material and/or chemical compositions of samples. There is growing interest in creating 3D hyperspectral reconstructions, which could provide both spatial and spectral information while also mitigating common HSI challenges such as non-Lambertian surfaces and translucent objects. However, traditional 3D reconstruction with HSI is difficult due to technological limitations of hyperspectral cameras. In recent years, Neural Radiance Fields (NeRFs) have seen widespread success in creating high quality volumetric 3D representations of scenes captured by a variety of camera models. Leveraging recent advances in NeRFs, we propose computing a hyperspectral 3D reconstruction in which every point in space and view direction is characterized by wavelength-dependent radiance and transmittance spectra. To evaluate our approach, a dataset containing nearly 2000 hyperspectral images across 8 scenes and 2 cameras was collected. We perform comparisons against traditional RGB NeRF baselines and apply ablation testing with alternative spectra representations. Finally, we demonstrate the potential of hyperspectral NeRFs for hyperspectral super-resolution and imaging sensor simulation. We show that our hyperspectral NeRF approach enables creating fast, accurate volumetric 3D hyperspectral scenes and enables several new applications and areas for future study., Comment: Main paper: 15 pages + 2 pages references. Supplemental/Appendix: 6 pages

Here, we introduce a collection of more than 83,000 light microscopy images of individual diatom valves or frustules, covering a broad range of taxa and morphology by more than 320 samples from 15 different river and lake ecotypes. The diatoms were imaged at high resolution (< 0.1 µm/pixel) by transmitted light bright-field microscopy, with focus stacking to artificially increase focal depth up to 25 µm, allowing simultaneous observation of valve ornamentation and shape. Taken from a real-world setting, the images partly also include debris, mineral particles or other diatoms. Four experts identified over 500 diatom species; more than 100 species are represented by at least 100 specimens and about 150 by at least 50 specimens each. This data set is about one order of magnitude larger than previously published diatom data sets, and its high interspecies similarity makes it a valuable resource e.g. for benchmarking fine-grained out-of-distribution (OOD) detection, on which we present a new approach.

This paper outlines a roadmap to effectively leverage shared mental models in multi-robot, multi-stakeholder scenarios, drawing on experiences from the BugWright2 project. The discussion centers on an autonomous multi-robot systems designed for ship inspection and maintenance. A significant challenge in the development and implementation of this system is the calibration of trust. To address this, the paper proposes that trust calibration can be managed and optimized through the creation and continual updating of shared and accurate mental models of the robots. Strategies to promote these mental models, including cross-training, briefings, debriefings, and task-specific elaboration and visualization, are examined. Additionally, the crucial role of an adaptable, distributed, and well-structured user interface (UI) is discussed., Comment: In SCRITA 2023 Workshop Proceedings (arXiv:2311.05401) held in conjunction with 32nd IEEE International Conference on Robot & Human Interactive Communication, 28/08 - 31/08 2023, Busan (Korea)

The quality of a wood log in the wood industry depends heavily on the presence of both outer and inner defects, including inner knots that are a result of the growth of tree branches. Today, locating the inner knots require the use of expensive equipment such as X-ray scanners. In this paper, we address the task of predicting the location of inner defects from the outer shape of the logs. The dataset is built by extracting both the contours and the knots with X-ray measurements. We propose to solve this binary segmentation task by leveraging convolutional recurrent neural networks. Once the neural network is trained, inference can be performed from the outer shape measured with cheap devices such as laser profilers. We demonstrate the effectiveness of our approach on fir and spruce tree species and perform ablation on the recurrence to demonstrate its importance.

Most of the research in content-based image retrieval (CBIR) focus on developing robust feature representations that can effectively retrieve instances from a database of images that are visually similar to a query. However, the retrieved images sometimes contain results that are not semantically related to the query. To address this, we propose a method for CBIR that captures both visual and semantic similarity using a visual hierarchy. The hierarchy is constructed by merging classes with overlapping features in the latent space of a deep neural network trained for classification, assuming that overlapping classes share high visual and semantic similarities. Finally, the constructed hierarchy is integrated into the distance calculation metric for similarity search. Experiments on standard datasets: CUB-200-2011 and CIFAR100, and a real-life use case using diatom microscopy images show that our method achieves superior performance compared to the existing methods on image retrieval., Comment: Accepted in ICVS 2023

© The Author(s), 2022. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Orenstein, E., Ayata, S., Maps, F., Becker, É., Benedetti, F., Biard, T., Garidel‐Thoron, T., Ellen, J., Ferrario, F., Giering, S., Guy‐Haim, T., Hoebeke, L., Iversen, M., Kiørboe, T., Lalonde, J., Lana, A., Laviale, M., Lombard, F., Lorimer, T., Martini, S., Meyer, A., Möller, K.O., Niehoff, B., Ohman, M.D., Pradalier, C., Romagnan, J.-B., Schröder, S.-M., Sonnet, V., Sosik, H.M., Stemmann, L.S., Stock, M., Terbiyik-Kurt, T., Valcárcel-Pérez, N., Vilgrain, L., Wacquet, G., Waite, A.M., & Irisson, J. Machine learning techniques to characterize functional traits of plankton from image data. Limnology and Oceanography, 67(8), (2022): 1647-1669, https://doi.org/10.1002/lno.12101., Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms., SDA acknowledges funding from CNRS for her sabbatical in 2018–2020. Additional support was provided by the Institut des Sciences du Calcul et des Données (ISCD) of Sorbonne Université (SU) through the support of the sponsored junior team FORMAL (From ObseRving to Modeling oceAn Life), especially through the post-doctoral contract of EO. JOI acknowledges funding from the Belmont Forum, grant ANR-18-BELM-0003-01. French co-authors also wish to thank public taxpayers who fund their salaries. This work is a contribution to the scientific program of Québec Océan and the Takuvik Joint International Laboratory (UMI3376; CNRS - Université Laval). FM was supported by an NSERC Discovery Grant (RGPIN-2014-05433). MS is supported by the Research Foundation - Flanders (FWO17/PDO/067). FB received support from ETH Zürich. MDO is supported by the Gordon and Betty Moore Foundation and the U.S. National Science Foundation. ECB is supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) under the grant agreement no. 88882.438735/2019-01. TB is supported by the French National Research Agency (ANR-19-CE01-0006). NVP is supported by the Spanish State Research Agency, Ministry of Science and Innovation (PTA2016-12822-I). FL is supported by the Institut Universitaire de France (IUF). HMS was supported by the Simons Foundation (561126) and the U.S. National Science Foundation (CCF-1539256, OCE-1655686). Emily Peacock is gratefully acknowledged for expert annotation of IFCB images. LS was supported by the Chair VISION from CNRS/Sorbonne Université.

© The Author(s), 2022. This article is distributed under the terms of the Creative Commons Attribution License. The definitive version was published in Orenstein, E., Ayata, S., Maps, F., Becker, É., Benedetti, F., Biard, T., Garidel‐Thoron, T., Ellen, J., Ferrario, F., Giering, S., Guy‐Haim, T., Hoebeke, L., Iversen, M., Kiørboe, T., Lalonde, J., Lana, A., Laviale, M., Lombard, F., Lorimer, T., Martini, S., Meyer, A., Möller, K.O., Niehoff, B., Ohman, M.D., Pradalier, C., Romagnan, J.-B., Schröder, S.-M., Sonnet, V., Sosik, H.M., Stemmann, L.S., Stock, M., Terbiyik-Kurt, T., Valcárcel-Pérez, N., Vilgrain, L., Wacquet, G., Waite, A.M., & Irisson, J. Machine learning techniques to characterize functional traits of plankton from image data. Limnology and Oceanography, 67(8), (2022): 1647-1669, https://doi.org/10.1002/lno.12101., Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms., SDA acknowledges funding from CNRS for her sabbatical in 2018–2020. Additional support was provided by the Institut des Sciences du Calcul et des Données (ISCD) of Sorbonne Université (SU) through the support of the sponsored junior team FORMAL (From ObseRving to Modeling oceAn Life), especially through the post-doctoral contract of EO. JOI acknowledges funding from the Belmont Forum, grant ANR-18-BELM-0003-01. French co-authors also wish to thank public taxpayers who fund their salaries. This work is a contribution to the scientific program of Québec Océan and the Takuvik Joint International Laboratory (UMI3376; CNRS - Université Laval). FM was supported by an NSERC Discovery Grant (RGPIN-2014-05433). MS is supported by the Research Foundation - Flanders (FWO17/PDO/067). FB received support from ETH Zürich. MDO is supported by the Gordon and Betty Moore Foundation and the U.S. National Science Foundation. ECB is supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) under the grant agreement no. 88882.438735/2019-01. TB is supported by the French National Research Agency (ANR-19-CE01-0006). NVP is supported by the Spanish State Research Agency, Ministry of Science and Innovation (PTA2016-12822-I). FL is supported by the Institut Universitaire de France (IUF). HMS was supported by the Simons Foundation (561126) and the U.S. National Science Foundation (CCF-1539256, OCE-1655686). Emily Peacock is gratefully acknowledged for expert annotation of IFCB images. LS was supported by the Chair VISION from CNRS/Sorbonne Université.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

We propose a novel hybrid cable-based robot with manipulator and camera for high-accuracy, medium-throughput plant monitoring in a vertical hydroponic farm and, as an example application, demonstrate non-destructive plant mass estimation. Plant monitoring with high temporal and spatial resolution is important to both farmers and researchers to detect anomalies and develop predictive models for plant growth. The availability of high-quality, off-the-shelf structure-from-motion (SfM) and photogrammetry packages has enabled a vibrant community of roboticists to apply computer vision for non-destructive plant monitoring. While existing approaches tend to focus on either high-throughput (e.g. satellite, unmanned aerial vehicle (UAV), vehicle-mounted, conveyor-belt imagery) or high-accuracy/robustness to occlusions (e.g. turn-table scanner or robot arm), we propose a middle-ground that achieves high accuracy with a medium-throughput, highly automated robot. Our design pairs the workspace scalability of a cable-driven parallel robot (CDPR) with the dexterity of a 4 degree-of-freedom (DoF) robot arm to autonomously image many plants from a variety of viewpoints. We describe our robot design and demonstrate it experimentally by collecting daily photographs of 54 plants from 64 viewpoints each. We show that our approach can produce scientifically useful measurements, operate fully autonomously after initial calibration, and produce better reconstructions and plant property estimates than those of over-canopy methods (e.g. UAV). As example applications, we show that our system can successfully estimate plant mass with a Mean Absolute Error (MAE) of 0.586g and, when used to perform hypothesis testing on the relationship between mass and age, produces p-values comparable to ground-truth data (p=0.0020 and p=0.0016, respectively)., Comment: 8 pages (6-content, 2-citations), 10 figures, 4 tables, submitted to ICRA 2023

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

We introduce Object-Guided Localization (OGuL) based on a novel method of local-feature matching. Direct matching of local features is sensitive to significant changes in illumination. In contrast, object detection often survives severe changes in lighting conditions. The proposed method first detects semantic objects and establishes correspondences of those objects between images. Object correspondences provide local coarse alignment of the images in the form of a planar homography. These homographies are consequently used to guide the matching of local features. Experiments on standard urban localization datasets (Aachen, Extended-CMU-Season, RobotCar-Season) show that OGuL significantly improves localization results with as simple local features as SIFT, and its performance competes with the state-of-the-art CNN-based methods trained for day-to-night localization.

Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab-based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.

Automatic classification of aquatic microorganisms is based on the morphological features extracted from individual images. The current works on their classification do not consider the inter-class similarity and intra-class variance that causes misclassification. We are particularly interested in the case where variance within a class occurs due to discrete visual changes in microscopic images. In this paper, we propose to account for it by partitioning the classes with high variance based on the visual features. Our algorithm automatically decides the optimal number of sub-classes to be created and consider each of them as a separate class for training. This way, the network learns finer-grained visual features. Our experiments on two databases of freshwater benthic diatoms and marine plankton show that our method can outperform the state-of-the-art approaches for classification of these aquatic microorganisms., Comment: 13th International Conference on Computer Vision Systems (2021)

The analysis of the internal structure of trees is highly important for both forest experts, biological scientists, and the wood industry. Traditionally, CT-scanners are considered as the most efficient way to get an accurate inner representation of the tree. However, this method requires an important investment and reduces the cost-effectiveness of this operation. Our goal is to design neural-network-based methods to predict the internal density of the tree from its external bark shape. This paper compares different image-to-image(2D), volume-to-volume(3D) and Convolutional Long Short Term Memory based neural network architectures in the context of the prediction of the defect distribution inside trees from their external bark shape. Those models are trained on a synthetic dataset of 1800 CT-scanned look-like volumetric structures of the internal density of the trees and their corresponding external surface.

The application of autonomous robots in agriculture is gaining increasing popularity thanks to the high impact it may have on food security, sustainability, resource use efficiency, reduction of chemical treatments, and the optimization of human effort and yield. With this vision, the Flourish research project aimed to develop an adaptable robotic solution for precision farming that combines the aerial survey capabilities of small autonomous unmanned aerial vehicles (UAVs) with targeted intervention performed by multi-purpose unmanned ground vehicles (UGVs). This paper presents an overview of the scientific and technological advances and outcomes obtained in the project. We introduce multi-spectral perception algorithms and aerial and ground-based systems developed for monitoring crop density, weed pressure, crop nitrogen nutrition status, and to accurately classify and locate weeds. We then introduce the navigation and mapping systems tailored to our robots in the agricultural environment, as well as the modules for collaborative mapping. We finally present the ground intervention hardware, software solutions, and interfaces we implemented and tested in different field conditions and with different crops. We describe a real use case in which a UAV collaborates with a UGV to monitor the field and to perform selective spraying without human intervention., Comment: Published in IEEE Robotics & Automation Magazine, vol. 28, no. 3, pp. 29-49, Sept. 2021

This paper introduces a novel feature detector based only on information embedded inside a CNN trained on standard tasks (e.g. classification). While previous works already show that the features of a trained CNN are suitable descriptors, we show here how to extract the feature locations from the network to build a detector. This information is computed from the gradient of the feature map with respect to the input image. This provides a saliency map with local maxima on relevant keypoint locations. Contrary to recent CNN-based detectors, this method requires neither supervised training nor finetuning. We evaluate how repeatable and how matchable the detected keypoints are with the repeatability and matching scores. Matchability is measured with a simple descriptor introduced for the sake of the evaluation. This novel detector reaches similar performances on the standard evaluation HPatches dataset, as well as comparable robustness against illumination and viewpoint changes on Webcam and photo-tourism images. These results show that a CNN trained on a standard task embeds feature location information that is as relevant as when the CNN is specifically trained for feature detection.

Most of the research effort on image-based place recognition is designed for urban environments. In bucolic environments such as natural scenes with low texture and little semantic content, the main challenge is to handle the variations in visual appearance across time such as illumination, weather, vegetation state or viewpoints. The nature of the variations is different and this leads to a different approach to describing a bucolic scene. We introduce a global image descriptor computed from its semantic and topological information. It is built from the wavelet transforms of the image semantic edges. Matching two images is then equivalent to matching their semantic edge descriptors. We show that this method reaches state-of-the-art image retrieval performance on two multi-season environment-monitoring datasets: the CMU-Seasons and the Symphony Lake dataset. It also generalises to urban scenes on which it is on par with the current baselines NetVLAD and DELF.

Deep learning generates state-of-the-art semantic segmentation provided that a large number of images together with pixel-wise annotations are available. To alleviate the expensive data collection process, we propose a semi-supervised domain adaptation method for the specific case of images with similar semantic content but different pixel distributions. A network trained with supervision on a past dataset is finetuned on the new dataset to conserve its features maps. The domain adaptation becomes a simple regression between feature maps and does not require annotations on the new dataset. This method reaches performances similar to classic transfer learning on the PASCAL VOC dataset with synthetic transformations.

Short-term changes in temporal and spatial distributions of the toxic cyanobacterium Planktothrix rubescens in Lake Zurich were investigated using high-resolution data sets acquired with an autonomous surface vessel (ASV). Data profiles were recorded with a multi-parameter probe while the ASV navigated along 1.5 km toward a waypoint located on the other side of the lake by using a global positioning system. We deployed the ASV seven times on five consecutive days during the stratification period (July 2011) to generate cross-sectional views of temperature, light, oxygen, and phycoerythrin and chlorophyll fluorescence from surface to 18 m. The data were also used to compute daily photosynthetic rates. Data showed a daily reshaping of the P. rubescens distribution in the metalimnion on both horizontal and vertical axes, from patches to a shore-to-shore spreading. A thermocline depression observed after 16 h of sustained winds forced the accumulation of P. rubescens on the downwind shore. The compression of the metalimnion and its downward shift by 6 m within 24 h suggested the modulation of a longitudinal seiche following the wind event. This passive transport of the metalimnetic P. rubescens population resulted in a 90% light reduction, and a decrease of the averaged daily photosynthetic rate from +21 mmol O2 m−2 d−1 to −10 mmol O2 m−2 d−1. Negative photosynthetic rates were computed on 2 d out of 5 d, meaning that the transport of P. rubescens by seiches significantly affected the balance between oxygen production and utilization in Lake Zurich, especially because it is the dominant primary producer.