1. Prevention of data duplication for high throughput sequencing repositories
- Author
-
J. Seth Strattan, Carrie A. Davis, Forrest Y. Tanaka, Benjamin C. Hitz, J. Michael Cherry, Keenan Graham, Jean M. Davidson, Jason A. Hilton, Idan Gabdank, Kathrina C. Onate, Stuart R. Miyasato, Otto Jolanki, Timothy R. Dreszer, Esther T. Chan, Aditi K. Narayanan, Ulugbek K. Baymuradov, and Cricket A. Sloan
- Subjects
0301 basic medicine ,Computer science ,business.industry ,Extramural ,MEDLINE ,Computational biology ,General Biochemistry, Genetics and Molecular Biology ,DNA sequencing ,03 medical and health sciences ,030104 developmental biology ,0302 clinical medicine ,Text mining ,Data deduplication ,Original Article ,Databases, Nucleic Acid ,General Agricultural and Biological Sciences ,business ,Data Curation ,030217 neurology & neurosurgery ,Information Systems - Abstract
Prevention of unintended duplication is one of the ongoing challenges many databases have to address. Working with high-throughput sequencing data, the complexity of that challenge increases with the complexity of the definition of a duplicate. In a computational data model, a data object represents a real entity like a reagent or a biosample. This representation is similar to how a card represents a book in a paper library catalog. Duplicated data objects not only waste storage, they can mislead users into assuming the model represents more than the single entity. Even if it is clear that two objects represent a single entity, data duplication opens the door to potential inconsistencies between the objects since the content of the duplicated objects can be updated independently, allowing divergence of the metadata associated with the objects. Analogously to a situation in which a catalog in a paper library would contain by mistake two cards for a single copy of a book. If these cards are listing simultaneously two different individuals as current book borrowers, it would be difficult to determine which borrower (out of the two listed) actually has the book. Unfortunately, in a large database with multiple submitters, unintended duplication is to be expected. In this article, we present three principal guidelines the Encyclopedia of DNA Elements (ENCODE) Portal follows in order to prevent unintended duplication of both actual files and data objects: definition of identifiable data objects (I), object uniqueness validation (II) and de-duplication mechanism (III). In addition to explaining our modus operandi, we elaborate on the methods used for identification of sequencing data files. Comparison of the approach taken by the ENCODE Portal vs other widely used biological data repositories is provided. Database URL: https://www.encodeproject.org/
- Published
- 2018
- Full Text
- View/download PDF