Search

Natural language descriptions of organismal phenotypes, a principal object of study in biology, are abundant in the biological literature. Expressing these phenotypes as logical statements using ontologies would enable large-scale analysis on phenotypic information from diverse systems. However, considerable human effort is required to make these phenotype descriptions amenable to machine reasoning. Natural language processing tools have been developed to facilitate this task, and the training and evaluation of these tools depend on the availability of high quality, manually annotated gold standard data sets. We describe the development of an expert-curated gold standard data set of annotated phenotypes for evolutionary biology. The gold standard was developed for the curation of complex comparative phenotypes for the Phenoscape project. It was created by consensus among three curators and consists of entity-quality expressions of varying complexity. We use the gold standard to evaluate annotations created by human curators and those generated by the Semantic CharaParser tool. Using four annotation accuracy metrics that can account for any level of relationship between terms from two phenotype annotations, we found that machine-human consistency, or similarity, was significantly lower than inter-curator (human-human) consistency. Surprisingly, allowing curatorsaccess to external information did not significantly increase the similarity of their annotations to the gold standard or have a significant effect on inter-curator consistency. We found that the similarity of machine annotations to the gold standard increased after new relevant ontology terms had been added. Evaluation by the original authors of the character descriptions indicated that the gold standard annotations came closer to representing their intended meaning than did either the curator or machine annotations. These findings point toward ways to better design software to augment human curators and the use of the gold standard corpus will allow training and assessment of new tools to improve phenotype annotation accuracy at scale.

Dinosaurs and other terrestrial vertebrates are poorly documented in the Mesozoic of the Canadian polar region. Here, we provide a complete review of the Upper Cretaceous (Maastrichtian) vertebrate fauna of the Bonnet Plume Formation in the northeastern Yukon Territory, Canada, which includes the description of the first newly collected dinosaur bones from this unit in almost half a century. Previously reported fragmentary dinosaur remains collected in the early 1960's pertain to an indeterminate hadrosaurid. New material includes a poorly preserved forelimb bone and a pedal phalanx. These new remains pertain to at least one species of non-hadrosaurid ornithischian dinosaur, and the humerus is tentatively referred to a small-bodied basal ornithopod. The new vertebrate fossils from the Bonnet Plume Formation provide further evidence of vertebrates from this unit. However, directed field surveys in 2008 and 2009 suggest that vertebrate fossils are not abundant. A review of the known localities of terrestrial Mesozoic vertebrates from the Canadian Arctic indicate that it had a relatively diverse community of terrestrial vertebrates, including dinosaurs, during the Late Cretaceous, but emphasizes our limited knowledge of the Mesozoic Arctic and considerable potential for future exploration and discovery. Peu de dinosaures et d'autres vertebres terrestres sont documentes dans le Mesozoique des regions polaires canadiennes. La faune vertebree du Cretace superieur (Maastrichtien) de la Formation de Bonnet Plume, dans le nord-est du Yukon (Canada), fait l'objet d'un examen exhaustif, et les premiers nouveaux ossements de dinosaures preleves de cette unite apres pres d'un demi-siecle sont egalement decrits. Des restes fragmentaires de dinosaures recueillis au debut des annees 1960 et deja signales se rapportent a un hadrosauride indetermine. Le nouveau materiel comprend un os de membre anterieur mal preserve et une phalange pedieuse. Ces nouveaux restes se rapportent a au moins une espece de dinosaures ornithischien non hadrosauride, et l'humerus est provisoirement affecte a un ornithopode basal a petit corps. Les nouveaux fossiles de vertebres de la Formation de Bonnet Plume constituent de nouvelles preuves de la presence de vertebres dans cette unite. Des travaux de terrain diriges menes en 2008 et 2009 portent toutefois a croire que les fossiles de vertebres n'y sont pas abondants. Un examen des localites connues de vertebres terrestres mesozoiques de l'Arctique canadien indique que cette region presentait une assez grande diversite de vertebres terrestres, dont des dinosaures, au Cretace tardif, mais souligne le peu de connaissances disponibles sur l'Arctique au Mesozoique et l'important potentiel que presente cette region sur le plan de l'exploration et de decouvertes futures. [Traduit par la Redaction], Introduction Late Cretaceous and Early Paleogene high latitude terrestrial vertebrate faunas are poorly known, despite significant amounts of Late Cretaceous outcrop in the Canadian Arctic (Embry 1991; Nichols and Sweet [...]

Natural language descriptions of organismal phenotypes, a principal object of study in biology, are abundant in the biological literature. Expressing these phenotypes as logical statements using ontologies would enable large-scale analysis on phenotypic information from diverse systems. However, considerable human effort is required to make these phenotype descriptions amenable to machine reasoning. Natural language processing tools have been developed to facilitate this task, and the training and evaluation of these tools depend on the availability of high quality, manually annotated gold standard data sets. We describe the development of an expert-curated gold standard data set of annotated phenotypes for evolutionary biology. The gold standard was developed for the curation of complex comparative phenotypes for the Phenoscape project. It was created by consensus among three curators and consists of entity-quality expressions of varying complexity. We use the gold standard to evaluate annotations created by human curators and those generated by the Semantic CharaParser tool. Using four annotation accuracy metrics that can account for any level of relationship between terms from two phenotype annotations, we found that machine-human consistency, or similarity, was significantly lower than inter-curator (human-human) consistency. Surprisingly, allowing curatorsaccess to external information did not significantly increase the similarity of their annotations to the gold standard or have a significant effect on inter-curator consistency. We found that the similarity of machine annotations to the gold standard increased after new relevant ontology terms had been added. Evaluation by the original authors of the character descriptions indicated that the gold standard annotations came closer to representing their intended meaning than did either the curator or machine annotations. These findings point toward ways to better design software to augment human curators and the use of the gold standard corpus will allow training and assessment of new tools to improve phenotype annotation accuracy at scale.

Background: Phenex (http://phenex.phenoscape.org/) is a desktop application for semantically annotating the phenotypic character matrix datasets common in evolutionary biology. Since its initial publication, we have added new features that address several major bottlenecks in the efficiency of the phenotype curation process: allowing curators during the data curation phase to provisionally request terms that are not yet available from a relevant ontology; supporting quality control against annotation guidelines to reduce later manual review and revision; and enabling the sharing of files for collaboration among curators., Results: We decoupled data annotation from ontology development by creating an Ontology Request Broker (ORB) within Phenex. Curators can use the ORB to request a provisional term for use in data annotation; the provisional term can be automatically replaced with a permanent identifier once the term is added to an ontology. We added a set of annotation consistency checks to prevent common curation errors, reducing the need for later correction. We facilitated collaborative editing by improving the reliability of Phenex when used with online folder sharing services, via file change monitoring and continual autosave., Conclusions: With the addition of these new features, and in particular the Ontology Request Broker, Phenex users have been able to focus more effectively on data annotation. Phenoscape curators using Phenex have reported a smoother annotation workflow, with much reduced interruptions from ontology maintenance and file management issues.