CharaParser+EQ: Performance evaluation without gold standard

To make phenotypic characters of organisms widely useful for computerized biology research, biocurators manually convert character descriptions to a structured format, for example the Entity-Quality (EQ) format. The manual approach is time consuming and affected by inter-curator variations. In this paper we report a software application, CharaParser+EQ, to our knowledge the first software that produces EQ statements from textual character descriptions. We report a recent experiment that evaluates the performance of the software against three experienced biocurators. While the software is still far from being able to compete with biocurators on this highly intellectual task, the results show (1) CharaParser+EQ's performance (precision and recall) is greatly improved compared to a previous version, (2) the completeness of the ontologies used in the process has significant impact both on the software's EQ generation performance and on the agreement among curators, and (3) unlimited access to external knowledge (published papers, books) by curators has no significant impact on inter-curator agreements. A detailed error analysis that compares machine and curator generated EQs is included.