Search

Pl@ntnet is a citizen observatory that relies on artificial intelligence (AI) technologies to help people identify plants with their smartphones (Joly 2014). Over the past few years, Pl@ntNet has become one of the largest plant biodiversity observatories in the world with several million contributors (Bonnet 2020b). Based on user demands, a set of tools and services following the FAIR (Findable, Accessible, Interoperable, Reusable) principles (Wilkinson 2016) were implemented to allow the development of e-floras. After a short description of the platform (see Joly (2015) and Joly (2016) for details), we present three complementary services dedicated to the customization of e-floras. The general workflow of Pl@ntNet can be divided into three main components. First, the Pl@ntNet mobile Android and iOS apps, dedicated to plant identification, allow both anonymous and authenticated users to take a picture of a plant, and to send it to a server for recognition at the species level. This recognition, which is performed by a convolutional neural network (Affouard 2017), allows the user to get a list of candidate species, each associated with a confidence score. The second component of Pl@ntNet's workflow is the enrichment of the database to improve the observation quality of the data collected. Because of the huge volume of data (over four million observations shared in 2020, Pl@nt Net statistics), this enrichment cannot be done solely by expert botanists. This is why various crowdsourcing mechanisms have been put in place allowing the collaborative curation of the data. The third part of the Pl@ntNet workflow is the exploitation of the database, which is made available for the needs of stakeholders such as researchers. For example, the recent sharing of Pl@ntNet data on GBIF's (Global Biodiversity Information Facility) platform, through two complementary datasets, has increased the benefit to the scientific community (examples are available here). To support the development of customized e-floras, three complementary concepts have been developed: micro-projects, groups, and monitoring work spaces, whose services are detailed below: (a) Micro-projects allow full adaptation of all the interfaces of the Pl@ntNet apps to a species list of interest. This e-flora can be linked to a specific geographical area, which allows it to be automatically selected according to the user's location. When several specific geographical areas overlap for a given geolocation, the smallest one is automatically selected. This adaptation increases the accuracy of the identification, as the number of potential species for a given identification request is reduced to the checklist of the micro-project (e.g., Bonnet (2020a)). Up to now, 12 micro-projects adapted to European, African or Asian contexts are running on the platform. A full description of this service is provided in Bonnet (2020c). The API (Application Programming Interface) for the identification service of each e-flora is available on My-Pl@ntNet. (b) Groups allow any user to create a private or public space on the platform (https://identify.plantnet.org/groups), to permit everyone to aggregate a part or all of their observations in the group. A group is "observation-centered" as opposed to a micro-project, which is species-centered. If the group is public, any authenticated user can join and contribute to it; if it is private, only users validated by the group's moderators can become members. When a group is restricted to a specific geographical zone (such as a school, city, or natural area), only observations found in that area are displayed in the group, contrary the micro-project. As all the group's observations can be downloaded (as tabbed or comma separated values) by any of the group's members, group features can be used to conduct statistical analyses on the data in order to study plant plots, plant phenology or user profiles. These groups are used by people who want to structure the activity of a group of people interested in monitoring the biodiversity of a given area, a taxonomical group, or a type of plant habitat. Over 260 groups have already been created by e.g., professional land managers, educators, and plant enthusiasts. (c) Monitoring work spaces allows a given stakeholder to access all the observations and identification requests of a given species list in a particular area. Micro-projects and Groups only allow exploration of plant observations explicitly shared by the authenticated users. However, Pl@ntNet's database contains hundreds of millions of plant identification requests submitted by anonymous users. Monitoring work spaces was set up to allow access by land managers to this rich and important material. These work spaces provide the maps, the list of plant observations, and identification requests (with a very high confidence score on the species identification), for all the species of interest to a given partner. For example, this service has been mobilized to follow the recent development of an invasive species (i.e., Hakea sericea Schrad. & J.C. Wendl.) in and around a natural reserve on the Mediterranean coast. All of these on-demand e-floras and monitoring services accelerate the use of dailyproduced data, and inform land managers and scientists of the changes in the floristic composition of monitoring areas. [ABSTRACT FROM AUTHOR]

Automated plant identification has recently improved significantly due to advances in deep learning and the availability of large amounts of field photos. As an illustration, the classification accuracy of 10K species measured in the LifeCLEF challenge (Goëau et al. 2018) reached 90%, very close to that of human experts. However, the profusion of field images only concerns a few tens of thousands of species, mainly located in North America and Western Europe. Conversely, the richest regions in terms of biodiversity, such as tropical countries, suffer from a shortage of training data (Pitman 2021). Consequently, the identification performance of the most advanced models on the flora of these regions is much lower (Goëau et al. 2019). Nevertheless, for several centuries, botanists have systematically collected, catalogued, and stored plant specimens in herbaria. Considerable recent efforts by the biodiversity informatics community, such as DiSSCo (Addink et al. 2018) and iDigBio (Matsunaga et al. 2013), have made millions of digitized specimens from these collections available online. A key question is therefore whether these digitized specimens could be used to improve the identification performance of species for which we have very few (if any) photos. However, this is a very difficult problem from a machine learning point of view. The visual appearance of a herbarium specimen is actually very different from a field photograph because the specimens are dried and crushed on a herbarium sheet before being digitized (Fig. 1). To advance research on this topic, we built a large dataset that we shared as one of the challenges of the LifeCLEF 2020 (Goëau et al. 2020) and 2021 evaluation campaigns (Goëau et al. 2021). It includes more than 320K herbarium specimens collected mostly from the Guiana Shield and the Northern Amazon Rainforest, focusing on about 1K plant species of the French Guiana flora. A valuable asset of this collection is that some of the specimens are accompanied by a few photos of the same specimen, allowing for more precise machine learning. In addition to this training data, we also built a test set for model evaluation, composed of 3,186 field photos collected by two of the best experts on Guyanese flora. Based on this dataset, about ten research teams have developed deep learning methods to address the challenge (including the authors of this abstract as the organizing team). A detailed description of these methods can be found in the technical notes written by the participating teams (Goëau et al. 2020, Goëau et al. 2021). The methods can be divided into two categories: • those based on classical convolutional neural networks (CNN) trained simply by mixing digitized specimens and photos and • those based on advanced domain adaptation techniques with the objective of learning a joint representation space between field and herbarium representations. The domain adaptation methods themselves were of two types, those based on 1. adversarial regularization (Motiian et al. 2017) to force herbarium specimens and photos to have the same representations, 2. metric learning to maximize inter-species distances and minimize intra-species distances in the representation space In Table 1, we report the results achieved by the different methods evaluated during the 2020 edition of the challenge. The evaluation metric used is the mean reciprocal rank (MRR), i.e., the average of the inverse of the rank of the correct species in the list of the predicted species. In addition to this main score, a second MRR score is computed on a subset of the test set composed of the most difficult species, i.e., the ones that are the least frequently photographed in the field. The main outcomes we can derive from these results are the following:... Classical deep learning models fail to identify plant photos from digitized herbarium specimens. The best classical CNN trained on the provided data resulted in a very low MRR score (0.011). Even with the of use additional training data (e.g. photos and digitized herbarium from GBIF) the MRR score remains very low (0.039). Domain adaptation methods provide significant improvement but the task remains challenging. The best MRR score (0.180) was achieved by using adversarial regularization (FSDA Motiian et al. 2017). This is much better than the classical CNN models but there is still a lot of progress to be made to reach the performance of a truly functional identification system (the MRR score on classical plant identification tasks can be up to 0.9). No method fits all. As shown in Table 1, the metric learning method has a significantly better MRR score on the most difficult species (0.107). However, the performance of this method on the species with more photos is much lower than the adversarial technique. In 2021, the challenge was run again but with additional information provided to train the models, i.e., species traits (plant life form, woodiness and plant growth form). The use of the species traits allowed slight performance improvement of the best adversarial adaptation method (with a MRR equal to 0.198). In conclusion, the results of the experiments conducted are promising and demonstrate the potential interest of digitized herbarium data for automated plant identification. However, progress is still needed before integrating this type of approach into production applications. [ABSTRACT FROM AUTHOR]

Millions of herbarium records provide an invaluable legacy and knowledge of the spatial and temporal distributions of plants over centuries across all continents (Soltis et al. 2018). Due to recent efforts to digitize and to make publicly accessible most major natural collections, investigations of ecological and evolutionary patterns at unprecedented geographic scales are now possible (Carranza-Rojas et al. 2017, Lorieul et al. 2019). Nevertheless, biologists are now facing the problem of extracting from a huge number of herbarium sheets basic information such as textual descriptions, the numbers of organs, and measurements of various morphological traits. Deep learning technologies can dramatically accelerate the extraction of such basic information by automating the routines of organ identification, counts and measurements, thereby allowing biologists to spend more time on investigations such as phenological or geographic distribution studies. Recent progress on instance segmentation demonstrated by the Mask-RCNN method is very promising in the context of herbarium sheets, in particular for detecting with high precision different organs of interest on each specimen, including leaves, flowers, and fruits. However, like any deep learning approach, this method requires a significant number of labeled examples with fairly detailed outlines of individual organs. Creating such a training dataset can be very time-consuming and may be discouraging for researchers. We propose in this work to integrate the Mask-RCNN approach within a global system enabling an active learning mechanism (Sener and Savarese 2018) in order to minimize the number of outlines of organs that researchers must manually annotate. The principle is to alternate cycles of manual annotations and training updates of the deep learning model and predictions on the entire collection to process. Then, the challenge of the active learning mechanism is to estimate automatically at each cycle which are the most useful objects that must be manually extracted in the next manual annotation cycle in order to learn, in as few cycles as possible, an accurate model. We discuss experiments addressing the effectiveness, the limits and the time required of our approach for annotation, in the context of a phenological study of more than 10,000 reproductive organs (buds, flowers, fruits and immature fruits) of Streptanthus tortuosus, a species known to be highly variable in appearance and therefore very difficult to be processed by an instance segmentation deep learning model. [ABSTRACT FROM AUTHOR]

Automated identification of plants and animals has improved considerably in the last few years, in particular thanks to the recent advances in deep learning. In order to evaluate the performance of automated plant identification technologies in a sustainable and repeatable way, a dedicated system-oriented benchmark was setup in 2011 in the context of ImageCLEF (Goëau et al. 2011). Each year, since that time, several research groups participated in this large collaborative evaluation by benchmarking their image-based plant identification systems. In 2014, the LifeCLEF research platform (Joly et al. 2014) was created in the continuity of this effort so as to enlarge the evaluated challenges by considering birds and fishes in addition to plants, and audio and video contents in addition to images. The 2017-th edition of the LifeCLEF plant identification challenge (Joly et al. 2017) is an important milestone towards automated plant identification systems working at the scale of continental floras with 10.000 plant species living mainly in Europe and North America illustrated by a total of 1.1M images. Nowadays, such ambitious systems are enabled thanks to the conjunction of the dazzling recent progress in image classification with deep learning and several outstanding international initiatives, aggregating the visual knowledge on plant species coming from the main national botanical institutes. The PlantCLEF plant challenge that we propose to present at this workshop aimed at evaluating to what extent a large noisy training dataset collected through the web (then containing a lot of labelling errors) can compete with a smaller but trusted training dataset checked by experts. To fairly compare both training strategies, the test dataset was created from a third data source, the Pl@ntNet (Joly et al. 2015) mobile application that collects millions of plant image queries all over the world. Due to the good results obtained at the 2017-th edition of the LifeCLEF plant identification challenge, the next big question is how far such automated systems are from the human expertise. Indeed, even the best experts are sometimes confused and/or disagree with each other when validating images of living organism. A multimedia data actually contains only partial information that is usually not sufficient to determine the right species with certainty. Quantifying this uncertainty and comparing it to the performance of automated systems is of high interest for both computer scientists and expert naturalists. This work reports an experimental study following this idea in the plant domain. In total, 9 deeplearning systems implemented by 3 different research teams were evaluated with regard to 9 expert botanists of the French flora. The main outcome of this work is that the performance of state-of-the-art deep learning models is now close to the most advanced human expertise. This shows that automated plant identification systems are now mature enough for several routine tasks, and can offer very promising tools for autonomous ecological surveillance systems. [ABSTRACT FROM AUTHOR]

Hundreds of herbarium collections have accumulated a valuable heritage and knowledge of plants over several centuries (Page et al. 2015). Recent initiatives, such as iDigBio (https:// www.idigbio.org), aggregate data from and images of vouchered herbarium sheets (and other biocollections) and make this information available to botanists and the general public worldwide through web portals. These ambitious plans to transform and preserve these historical biodiversity data into digital format are supported by the United States National Science Foundation (NSF) Advancing the Digitization of Natural History Collections (ADBC) and the digitization is done by the Thematic Collections Networks (TCNs) funded under the ADBC program. However, thousands of herbarium sheets are still unidentified at the species level while numerous sheets should be reviewed and updated following more recent taxonomic knowledge. These annotations and revisions require an unrealistic amount of work for botanists to carry out in a reasonable time (Bebber et al. 2010). Computer vision and machine learning approaches applied to herbarium sheets are promising (Wijesingha and Marikar 2012) but are still not well studied compared to automated species identification from leaf scans or pictures of plants taken in the field. In a recent study, we evaluate the accuracy with which herbarium images can be potentially exploited for species identification with deep learning technology (Carranza-Rojas et al. 2017), particularly Convolutional Neural Networks (CNN) (Szegedy et al. 2015). This type of network allows automatic learning of the most prominent visual patterns in the images since they are trainable end-to-end (thus, differentiable), as opposed to previous approaches that use custom, hand-made feature extractors. A first challenge is to use herbarium sheet images alone to automatically identify the species of plants mounted on herbarium sheets. Secondly, we propose studying if the combination of herbarium sheet images with photos of plants in the field (Joly et al. 2015, Carranza-Rojas and Mata- Montero 2016) is a viable idea to train models that provide accurate results during identification. Finally, we explore if herbarium images from one region with a specific flora can be used in transfer learning (a technique in deep learning that first allows training a model with a dataset and then once trained, uses the weighted results to train another model with that knowledge as the baseline) to another region with other species; for example, in a region under-represented in terms of collected data. Our evaluation shows that the accuracy for species identification with deep learning technology, based on herbarium images, reaches 90.3% on a dataset of more than 1200 European plant species. This could potentially lead to the creation of a semi-, or even fully automated system to help taxonomists and experts with their annotation, classification, and revision works. In this paper, we take a closer look at the accuracy levels achieved with respect to the first two challenges. We evaluate the accuracy levels for each species included in the dataset, which encompasses 253,733 images, 1,204 species. [ABSTRACT FROM AUTHOR]

Pl@ntNet is an international initiative which was the first one attempting to combine the force of citizens networks with automated identification tools based on machine learning technologies (Joly et al. 2014). Launched in 2009 by a consortium involving research institutes in computer sciences, ecology and agriculture, it was the starting point of several scientific and technological productions (Goëau et al. 2012) which finally led to the first release of the Pl@ntNet app (iOS in February 2013 (Goëau et al. 2013) and Android (Goëau et al. 2014) the following year). Initially based on 800 plant species, the app was progressively enlarged to thousands of species of the European, North American and tropical regions. Nowadays, the app covers more than 15 000 species and is adapted to 22 regional and thematic contexts, such as the Andean plant species, the wild salads of southern Europe, the indigenous trees species of South Africa, the flora of the Indian Ocean Islands, the New Caledonian Flora, etc. The app is translated in 11 languages and is being used by more than 3 millions of end-users all over the world, mostly in Europe and the US. The analysis of the data collected by Pl@ntnet users, which represent more than 24 millions of observations up to now, has a high potential for different ecological and management questions. A recent work (Botella et al. 2018), in particular, did show that the stream of Pl@ntNet observations could allow a fine-grained and regular monitoring of some species of interest such as invasive ones. However, this requires cautious considerations about the contexts in which the application is used. In this talk, we will synthesize the results of this study and present another one related to phenology. Indeed, as the phenological stage of the observed plants is also recorded, these data offer a rich and unique material for phenological studies at large geographical or taxonomical scale. We will share preliminary results obtained on some important pantropical species (such as the Melia azedarach L., and the Lantana camara L.), for which we have detected significant intercontinental phenological patterns, among the project data. [ABSTRACT FROM AUTHOR]