Search

Requirements and processes in translational research are expressed in natural language making them subject to human interpretation and ambiguities. Using templates in other domains such as software engineering have enabled better precision and translation of requirements to developed products [1]. To enable better reproducibility, the Utah Center for Clinical and Translational Science (CCTS) is developing, evaluating and deploying templates for performing translational research. Translational research requisition processes are presented as templates or in natural language. In the absence of a template, natural language requests and processes are cast into templates using state of the art pattern matching methods. These requirements and processes are automatically checked for conformance with existing templates [1,2]. Those meeting the conformance checks are passed on to the respective CCTS Foundations and/or service cores for their fulfillment. Those failing the conformance checks are sent back to the investigator for review, and then discussed with the CCTS Foundations/Cores. This template-based approach will be useful for various types of translational research services including: requesting and designing statistical analysis plans [3,4], research data and informatics support requests [5], trial recruitment services [6], biospecimen and DNA analysis services, and clinical study services. Templates have been shown to be effective in increasing precision of natural language requirements and reducing ambiguities [1]. When implemented we envision that this informatics-based process will enable efficiency, quality and reproducibility of translational research processes and results., This research is supported by the Utah Center for Clinical and Translational Science funded by NCATS award UL1TR001067., {"references":["1.\tC. Arora, M. Sabetzadeh, L. Briand and F. Zimmer, \"Automated Checking of Conformance to Requirements Templates Using Natural Language Processing,\" in IEEE Transactions on Software Engineering, vol. 41, no. 10, pp. 944-968, Oct. 1 2015. doi: 10.1109/TSE.2015.2428709","2.\tT. Molka, D. Redlich, M. Drobek, A. Caetano, X.-J. Zeng, and W. Gilani, \"Conformance Checking for BPMN-based Process Models,\" in Proceedings of the 29th Annual ACM Symposium on Applied Computing, New York, NY, USA, 2014, pp. 1406–1413.","3.\tR. Gouripeddi, M.R. Cummins, R. Madsen, B. LaSalle, A. Redd, X. Ye, A. Presson, T. Greene, J. Facelli, Streamlining Study Design and Statistical Analysis, , Research Reproducibility Conference, Monday, November 14, 2016, S.J. Quinney Law School, University of Utah, Salt Lake City. http://campusguides.lib.utah.edu/UtahRR16/abstracts, DOI: http://doi.org/10.5281/zenodo.180452","4.\tR. Gouripeddi, M. Cummins, R. Madsen, B. LaSalle, A. Redd, X. Ye, A. Presson, S. Harper, T. Greene, Julio Cesar Facelli, Streamlining Study Design and Statistical Analysis for Quality Improvement and Research Reproducibility, Translational Science 2017, April 19-21, 2017, Washington, DC. Abstract: ACTS SDBC Final.pdf, DOI: http://doi.org/10.5281/zenodo.555966","5.\tR. Butcher, R. Gouripeddi, R. Madsen, P. Mo, B. LaSalle, Research Data Service,Research Reproducibility Conference, Monday, November 14, 2016, S.J. Quinney Law School, University of Utah, Salt Lake City. http://campusguides.lib.utah.edu/UtahRR16/abstracts, http://library.med.utah.edu/e-channel/research-reproducibility-2016-posters/","6.\tCummins, M.R., Gouripeddi, R., Facelli, J. (2016). A Low-cost, Low-barrier Clinical Trials Registry to Support Effective Recruitment, Research Reproducibility Conference, Salt Lake City, UT. Poster, Presented, 10/2016."]}

The immune system defends hosts from external and internal pathogenic agents and responds with inflammatory processes throughout the healing process. The environment plays a much greater role than genes in the development, adaptation and modulation of the immune system factors1–3. These combinatorial interactions between the immune system, the environment and the genome lead to variations in disease phenotypes and responses to therapeutic interventions. Therefore, understanding disease mechanisms and the development of interventions would need to consider the complex interplay of the endogenous immunological process along with the effects of the modern environment, thus requiring generation of a complete picture of environmental exposures, and immune responses along with genetic factors. The immunome is defined as the complete set of all genes and their translating immunological proteins, - many of them resulting from various environmental stimuli4–6. The exposome encompasses the life-course of environmental exposures (including lifestyle factors) from prenatal periods and complements the genome by providing a comprehensive description of lifelong exposure history 7 . Immunome research requires the integration of diverse data types for supporting different research use-cases. While there exist gaps and sparseness in data points needed to generate sufficiently complete immunomes and exposomes, using available data with an understanding of their limitations could enable reproducibility research. In order to systematically support these needs, we are developing a scalable computation infrastructure, the Utah PRISMS Informatics Ecosystem (UPIE)8, that enables generation, integration and utilization of immunomes and exposomes for translational research. UPIE is a semantically consistent, metadata-driven, event-based big data infrastructure. It includes methods and processes for collection of sensor and person-generated data; selection and invocation of computational models for filling spatio-temporal gaps in exposomic data; data integration; and participant and research facing tools. In this presentation, we discuss how our metadata driven and uncertainty-characterizing approaches within UPIE enable the generation of immunomes and exposomes, and how their integration is essential for reproducibility of research. We explain the generalizability of this multi-scale and multi-omics platform for providing robust pipelines for reproducible research in conditions such as asthma and type 1 diabetes mellitus using publicly available immunome data from sources such Immport9., This research is partially supported by the Utah PRISMS Informatics Center through NIH/NIBIB U54EB021973, the Utah Center for Clinical and Translational Science funded by NCATS award UL1TR001067, the NIH Ruth L. Kirschstein NRS Award 5T32DK091317 from NIDDK, and the NLM Training grant T15LM007124., {"references":["Brodin, P. et al. Variation in the Human Immune System Is Largely Driven by Non-Heritable Influences. Cell 160, 37–47 (2015).","Björkstén, B. Environmental Influences on the Development of the Immune System: Consequences for Disease Outcome. Window Oppor. Pre-Pregnancy 24 Mon. Age 61, 243–254 (2008).","Morrot, A. et al. Metabolic Symbiosis and Immunomodulation: How Tumor Cell-Derived Lactate May Disturb Innate and Adaptive Immune Responses. Front. Oncol. 8, (2018).","Ortutay, C. & Vihinen, M. Immunome: A reference set of genes and proteins for systems biology of the human immune system. Cell. Immunol. 244, 87–89 (2006).","El-Chemaly, S. et al. The Immunome in Two Inherited Forms of Pulmonary Fibrosis. Front. Immunol. 9, (2018).","Biancotto, A. & McCoy, J. P. Studying the Human Immunome: The Complexity of Comprehensive Leukocyte Immunophenotyping. Curr. Top. Microbiol. Immunol. 377, 23–60 (2014).","Wild, C. P. Complementing the Genome with an \"Exposome\": The Outstanding Challenge of Environmental Exposure Measurement in Molecular Epidemiology. Cancer Epidemiol. Prev. Biomark. 14, 1847–1850 (2005).","Sward, K., Patwari, N., Gouripeddi, R. & Facelli, J. An Infrastructure for Generating Exposomes: Initial Lessons from the Utah PRISMS Platform. in the 27th Annual Meeting of the International Society for Exposure Science, Research Triangle, NC, USA, (2017).","Bhattacharya, S. et al. ImmPort, toward repurposing of open access immunological assay data for translational and clinical research. Sci. Data 5, (2018)."]}

Objectives/Goals: Opioid analgesic misuse poses a critical public health problem in the United States. The 2015 National Survey on Drug Use and Health (SAMHSA) 33,091 people died due to opioid overdosing and 12.5 million people misused prescription opioids in 2015. In the state of Utah alone, there are around 24 prescription opioid deaths every month constituting half of the poisoning deaths in the state. Yet, there is a lack of understanding of the role of the underlying treatment, prescribing and dispensing systems and processes in these areas. There are multiple factors involved in misuse including those related to the patient, physician and pharmacy; knowledge guiding opioid prescribing; and population-level factors such urban, rural and frontier regions of residence of patients, their socioeconomic status, unemployment, a culture of utilizing prescription pain medications, and the scarcity of illegal drugs such as heroin and cocaine. Discovering patterns in misuse trajectories therefore requires flexible approaches to integrate their heterogeneous data and support knowledge discovery. Methods/Study Population: In order to understand the role of multiple factors involved, we are developing a framework for graph-based methods for discovering opioid misuse trajectories. In this approach, we elicited key concepts relevant to understand opioid misuse from literature review, domain experts and existing data sources. We then used data modeling methods to represent the data in sharable formats. Results/Anticipated Results: At a high level, these concepts are categorized into three domains and are represented as three hypergraphs: (i) patient-provider-pharmacy, (ii) knowledge guiding opioid prescribing, and (iii) population level information related to opioid misuse. We developed a logical model for representing these opioid misuse related metadata. We then transform and store various Utah opioid misuse related data in graphical data stores to support flexible methods of data integration and knowledge discovery over geographic and temporal dimensions using Big Data methods such as graph clustering, probabilistic graphical models and other machine learning methods for learning spatio-temporal trajectories of opioid misuse. Discussion/Significance of Impact: Graph-based methods provide a holistic and flexible approach to integrate diverse heterogeneous data associated with opioid misuse and supports different methods of knowledge discovery. We will our approach will provide generalizable and reproducible methods for understanding trajectories of opioid misuse. This knowledge discovery framework could support the next generation of translational research of opioid misuse by providing personal risk scores for managing patients in the clinic, population level strategies for reducing misuse, stratifying poison control center resources, and hypotheses for clinical research., This work is partially supported by the Utah Center for Clinical and Translational Science funded by NCATS award UL1TR001067 and Utah PRISMS Informatics Center through NIH/NIBIB U54EB021973

Objectives/Goals: Issues with recruiting the targeted number of participants in a timely manner often results in underpowered studies, with more than 60% of clinical studies failing to complete or requiring extensions due to enrollment issues. The objective of this study is to understand temporal complexities within clinical trial descriptions in order to develop tractable computational approaches for automatic matching of trial descriptions with patient records. Methods/Study Population: In order to understand the temporal complexity present within clinical trial descriptions, we did the following: We randomly selected 100 cardiac study descriptions from Clinicaltrials.gov (as of September 2017), and had two experts in clinical vocabularies independently categorize the trial descriptions into those having and lacking temporality. The experts discussed discrepancies in their categorizations and came up with a consensus. We automatically mapped all the trial description available from ClinicalTrials.gov (as of September 2017) to Unified Medical Language System (UMLS) concepts using Metamap version 2016v2. We then extracts concepts of the semantic type ‘Temporal Concept (TMCO)’ and analyzed their description across the trial descriptions. In addition, we reconstructed the UMLS hierarchy for the concepts available within the trial descriptions in order to group them into semantic groups using the UMLS version 2017AB. We also analyzed these UMLS concepts with existing time related taxonomies such the Time ontology and Allen’s Interval Algebra. Results/Anticipated Results: The three experts categorized 86 and 90 percent of the clinical trial descriptions as having temporalities within them. Upon discussion, the experts arrived at a consensus result of 89% of the trial descriptions having temporality within them. 90% of the trial descriptions from ClinicalTrials.gov contained one or more UMLS concepts of the ‘Temporal Concept’ semantic type. We found 1019 distinct UMLS concepts of the within the trial descriptions from ClinicalTrials.gov. The top 20 concepts were present in at least 10% of the trial descriptions. 206 of these 1019 concepts were equivalent to or contained the thirteen elementary relations between time periods as described the Time ontology. The existing relationships between the extracted temporal concepts resulted in 46 trees and 427 unconnected concepts. Informed by Allen’s Interval Algebra and the Time ontology along with existing UMLS tree structures within these concepts, we manually assigned the concepts into 5 categories: Stage/Duration/Interval Time point/Instance/Event Time Frequency/Pattern Time Unit/Measure Temporal Relation We also found that many of the concepts belonged to more than one of these categories forming a complex network of temporal concepts and their use in trial descriptions was context dependent. Discussion/Significance of Impact: Our results from above two methods show that most clinical trial descriptions include a temporal dimension. The semantics of the temporality within these descriptions is also very complex and is context dependent. The temporality in clinical trial description could be unbounded; dense; and represented as instants or intervals including non-convex intervals, branching and circular times, and relative as well as absolute times. In addition, there might be differences in temporal granularities, and indeterminacy or veracity of time (incomplete knowledge of when considered event or fact happened), uncertainty regarding how long event or fact was true. Time is an essential component of most entries in a patient record. Key domains in this data such conditions, procedures, medications and laboratory as all documented as events in relation to patient visits and actual ages of patients. In addition, our analysis of notes within the MIMIC-III data using similar approaches as described above show the that patient notes have similar distributions of temporal concepts. Automated matching of clinical trial descriptions with patient records would there need to accommodate the temporal nature of both these document types. Our future work will include the development of temporal reasoning methods to optimize patient recruitment based on our current analysis findings. These methods will utilize fuzzy time-interval pattern mining over the electronic health record based patient trajectories to predict the most adequate requirement schedules based on the current and potential patient states., This work is supported by the Utah Center for Clinical and Translational Science funded by NCATS award UL1TR001067 and NLM Training grant T15LM007124.

Objectives/Goals: Translational research inclusive of observational, comparative effectiveness, clinical trials and population health studies is increasingly dependent on the secondary use of existing data including electronic health record (EHR), as a source for knowledge discovery. Researchers often provide natural language descriptions of their data needs. Research data teams use these descriptions to develop queries to run against EHR systems and provide result back to the researcher. Within this process, the data team and the researcher usually engage in complex written and verbal communication in order to mediate the details within the natural language description. The data team then abstracts the understood request to the complexities present with the EHR system. This is followed by the development of appropriate query scripts, extraction, transform and provisioning of query results and analysis. The data team and researcher usually iterate over these steps multiple times in order to develop the final data deliverable. In this study, we analyze the reproducibility of the current process of using natural language descriptions for acquiring research data from the EHR. Methods/Study Population: We provided a natural language description of an Upper Respiratory Tract Infection (URTI) study two data teams of three CTSA sites. The teams were blinded to the true nature of the study, which was to understand the reproducibility of the natural language description. Results and processes followed at each were analyzed. The following is a summary of the URTI data request: Patients eligible for enrollment between July 1, 2012 and September 30, 2015. Data will be reviewed up to six months pre-index to identify baseline characteristics and exclusion criteria. Subjects identified with an ICD-9-CM diagnosis code for an URTI during an outpatient patient encounter will be included. The index date is defined as the first ICD-9-CM documentation for an URTI. A 6-month pre-index period will be used to identify exclusion criteria, and to observe baseline characteristics. Patients will be examined for outcomes of interest within 24-hours of the index clinic date and time. In addition, we exclude patients with a positive rapid antigen detection test (RADT) for group A streptococcal pathogens at the initial visit (results available within 24-hours) as this is an instance when antibiotic prescribing is appropriate. Inclusion criteria: Age >18 years old Diagnosis (ICD – 9 code list provided) of a URTI in the outpatient setting Exclusion criteria: AIDS/HIV (ICD – 9 code list provided) COPD/Asthma (ICD – 9 code list provided) Cancer (ICD – 9 code list provided) Conditions for which antibiotic prescribing may be appropriate An URTI diagnosis within the 180 day pre-index period Additional infectious diseases (ICD – 9 code list provided) A positive rapid antigen detection test for group A streptococcal pathogen (LOINC code list provided). Results/Anticipated Results: Our results yielded 684,478, 460,159, and 412,942 individuals having outpatient visits between July 1, 2012 and September 30, 2015 at the three sites respectively. Of these, 18.7%, 1.7%, and 3.1% had URTI at each site. Of these, 17.5%, 34.2%, 39.9% were over 18 years of age at the three sites, respectively. After applying all exclusion criteria 6,797, 623 and 3,092 patients respectively, were obtained at the three sites. These patients would NOT be expected to receive antibiotics. Of those, 9.4%, 0.3% and 36.5%, and 11.2%, 0.3%, and 40.3% were prescribed antibiotics within the first 24 hours and first 8 days at each of the three sites, respectively. Discussion/Significance of Impact: Analysis of the results at each stage of the query building showed 10-fold to double differences across the sites. The site and magnitude of these differences varied at each step of inclusion/exclusion criteria. In order to ascertain possible reasons for these discrepancies, we asked the data teams at each site to describe their data query process and analyzed the query results with the undertaken processes. In our analysis we found that contextual, organizational and data analyst specific issues played a significant role in how the data team constructed their queries. In addition, differences in how data was transformed and stored from the EHR source systems, as well as presented to the data team played an important role. Data obtained from the EHR has great potential for use in translational research. However, the inability to successfully reproduce data requests across different sites should be an important consideration. Reproducing research data requires effective communication between the research and data teams. In addition, there is a need for (1) structured or semi-structured data requisition methods using templates, and (2) context-sensitive and metadata-driven workflows that supports the entire life cycle of research data requisition including the development of the natural language description of the research data, query mediation, data abstraction, data extraction and provisioning of results and analysis., This work is supported by the Utah Center for Clinical and Translational Science funded by NCATS award UL1TR001067.

Research Overview: This summarizes the current and future work done in streamlining the processes and methods involved with study design and statistical analyses in order to ensure quality of statistical methods and reproducibility of research. Objectives/Goals: Key factors causing irreproducibility of research include those related to inappropriate study design methodologies and statistical analysis. In modern statistical practice irreproducibility could arise due to statistical (false discoveries, p-hacking, overuse/misuse of p-values, low power, poor experimental design) and computational (data, code & software management) issues. These require understanding the processes and workflows practiced by an organization, and the development and use of metrics to quantify reproducibility. Methods/Study Population: Within the Foundation of Discovery - Population Health Research, Center for Clinical and Translational Science, University of Utah, we are undertaking a project to streamline the study design and statistical analysis workflows and processes. As a first step we met with key stakeholders to understand the current practices by eliciting example statistical projects, and then developed process information models for different types of statistical needs using Lucidchart. We then reviewed these with the Foundation’s leadership and the Standards Committee to come up with ideal workflows and model, and defined key measurement points (such as those around study design, analysis plan, final report, requirements for quality checks, and double coding) for assessing reproducibility. As next steps we are using our finding to embed analytical and infrastructural approaches within the statisticians’ workflows. This will include data and code dissemination platforms such as Box, Bitbucket and GitHub, documentation platforms such as Confluence, and workflow tracking platforms such as Jira. These tools will simplify and automate the capture of communications as a statistician work through a project. Data-intensive process will use process-workflow management platforms such as Activiti, Pegasus and Taverna. Results/Anticipated Results: These strategies for sharing and publishing study protocols, data, code and results across the spectrum, active collaboration with the research team, automation of key steps, along with decision support. Discussion/Significance of Impact: This analysis of the statistical methods and process and computational methods to automate them ensure quality of statistical methods and reproducibility of research.

Research Overview: This summarizes the work being done at the Center for Clinical and Translational Sciences, University of Utah in developing an informatics platform for enhancing research recruitment, Utah Utility for Research Recruitment (U2R2), and its evaluation for a back pain study. Objectives/Goals: Issues with recruiting the targeted number of participants in a timely manner often results in underpowered studies, with more than 60% of clinical studies failing to complete or requiring extensions due to enrollment issues. The objective of this study is to develop and implement a scalable, organization wide platform to enhance accrual into clinical research studies. Methods/Study Population: We are developing and evaluating an informatics platform called Utah Utility for Research Recruitment (U2R2). U2R2 consists of two components: (i) Semantic Matcher: an automated trial criterion to patient matching component that also reports uncertainty associated with the match, and (ii) Match Delivery: mechanisms to deliver the list of matched patients for different research and clinical settings. As a first step, we limited the Semantic Matcher to utilize only structured data elements from the patient record and trial criteria. We are now including distributional semantic methods to match complete patient records and trial criteria as documents. We evaluated the first phase of U2R2 based on a randomized trial with a target enrollment of 220 participants that compares two treatment strategies for managing back pain (physical therapy and usual care) for individuals consulting a non-surgical provider and symptomatic less than 90 days. Results/Anticipated Results: U2R2 identified 9370 patients from the University of Utah Hospitals and Clinics as potential matches. Of these 9370, 1145 responded to the Back Pain study research team’s email or phone communications, and were further screened by phone. 250 participants completed a screening visit, resulting in the current study enrollment of 130 participants. 43 of 1145 patients refused to participate, and 50 participants no-showed their screening visit.

Understanding the effects of the modern environment on pediatric asthma requires generation of a complete picture of the contributing environmental exposures and socio-economic factors. Such an exposome requires integration of data from wearable and stationary sensors, environmental monitors, physiology, medication use and other clinical data. In addition, such an integration would need to have a high spatial-temporal resolution for correlating times and location of exposures to occurrences of conditions and their severities. This would require filling any gaps in the measured data with modeled data along with characterization of any uncertainties. The Utah PRISMS Federated Integration Architecture is a comprehensive, standards-based, open-source informatics platform that allows sensor data and biomedical data to be integrated in a meaningful manner. We are developing the architecture, data models, processes, hardware and software to acquire, manage, process, and communicate high-resolution clinically relevant exposome information from environmental, physiological, and behavioral sensors and computational models. We envision this infrastructure to support different types of environmental biomedical studies of pediatric asthma and other chronic conditions, and potentially other research. In this presentation, we discuss two main components of the Utah PRISMS infrastructure: Sensor Common Data Model: Federating or integrating increasingly large, complex and multi-dimensional sensor data requires a thorough human understanding of them. These metadata specifications are designed to support the conduct of research utilizing personalized and environmental sensors. These includes sensors ranging from nano-sensors up to satellites. Sensor measurements may include physical, chemical, and biological species. In addition, these sensors include those that instantaneously (or with a transient storage) measure these species or those that collect physical samples with material transfer for later analysis. Sensors may be deployed in various environments, including personal (i.e. implanted & mobile), immediate (i.e. indoor), and general environment (i.e. external environmental protection agency monitors). The purpose of the data model is to establish a library of instruments; describe and document deployments of sensors; assess quality of data collected by different instruments within its deployment environments; support harmonization and integration of data collected from various sensors; and guide for structuring and storing sensor output data. Central Big Data Federation/Integration Platform: This is a standards-based, open-access infrastructure that integrates sensor data and mathematically modeled data with biomedical information along with characterizing uncertainties associated with using these data. The platform consists of components that: Discover, characterize, store and version metadata of different sensor data sources Standardize semantics across different sensor data sources using ontology services Store data in a temporal event based model to support different research uses-cases Methods to transform and present data for different use-cases of exposomic studies. In this symposium, we present initial lessons from the Utah PRISMS Platform that summarizes the interconnected work by diverse expertise including electrical, computer, chemical and industrial engineers, atmospheric and computer scientists, informaticists and pediatric researchers in developing an infrastructure for generating exposomes.

Current approaches to metadata discovery are dependent on time consuming manual curations. To realize the full potential of Big Data technologies in biomedicine, enhance research reproducibility and increase efficiency in translational sciences it is critical to develop automatic and/or semiautomatic metadata discovery methods and the corresponding infrastructure to deploy and maintain these tools and their outputs. Towards such a discovery infrastructure: We conceptually designed a process workflow for Metadata Discovery and Mapping Service, for automated metadata discovery. Based on steps taken by human experts in discovering and mapping metadata from various biomedical data, we designed a framework for automation. It consists of a 3-step process: (1) identification of data file source and format, (2) followed by detailed metadata characterization based on (1), and (3) characterization of the file in relation to other files to support harmonization of content as needed for data integration. The framework discovers and leverages administrative, structural, descriptive and semantic metadata, and consists of metadata and semantic mappers, along with uncertainty characterization and provision of expert review. As next steps we will develop and evaluate this framework using workflow platforms (e.g. Swift, Pegasus). In order to store discovered metadata about digital objects, we enhanced OpenFurther’s Metadata Repository (MDR). We configured the bioCADDIE metadata specifications (DatA Tag Suite (DATS) model) as assets in the MDR for harmonizing metadata of individual datasets (e.g. different protein files) for data integration. This method of metadata management provides a flexible data resource metadata storage system that supports versioning metadata (e.g. DATS 1.0 to 2.1) and data files mapped to different versions, enhance descriptors of resources (DATS) with descriptions of content within resources, and translations to other metadata specifications (e.g. schema.org). Also, this MDR stored metadata is available for various data services including data integration.

Current approaches to metadata discovery are dependent on manual curations which are time consuming processes and it is critical to develop automatic and/or semiautomatic metadata discovery methods to realize the full potential of Big Data technologies in biomedicine, enhance research reproducibility and increase efficiency in translational biomedical sciences. We are developing a two-step metadata discovery workflow: (1) Identification of data source and type using their intrinsic document structure, and (2) Discovery of detailed metadata within the data by associating specific metadata discovery tools based on the data’s source and type ascertained from (1). Here we discuss our initial results for (1) using machine learning. In this work we included biomedical data from various sources and in different file formats: 1) Human genetic variants - ClinVar (XML, tab-delimited), 2) Protein structure (PDB, mmCIF, XML) 3) Biomedical literature, and General English corpus. We tokenized the data files using the Natural Language Toolkit and considered various document structural features: normalized count of numerical tokens, negative numerical tokens, number of words, number of capitalized words, number of words with all upper letters, and median length of tokens. We developed a decision tree model with these structural features to classify these data and types, and evaluated the performance of the model using 10-fold cross-validation and a test set for the following metrics: precision, recall, and F1 score using scikit-learn. Our model was able to distinguish protein structure, genetic variant, scientific paper and general English files with an average F1 score of 0.997, 0.997, 0.886 and 0.919 when evaluated using cross-validation, and 1, 0.999, 0.980, and 0.935 when using independent test sets.. Our approach shows it is possible to automatically identify data sources and types using only document structural features and therefore reasonable to programmatically associate metadata extraction tools specific for each data source and type as next steps.

Shao, J., Gouripeddi, R., & Facelli, J.C. (2016). Improving Clinical Trial Cohort Definition Criteria and Enrollment with Distributional Semantic Matching (poster). Research Reproducibility 2016. Salt Lake City, UT, USA

Cummins, M.R., Gouripeddi, R., & Facelli, J. (2016). A Low-cost, Low-barrier Clinical Trials Registry to Support Effective Recruitment (poster). Research Reproducibility 2016. Salt Lake City, UT, USA

Key factors causing irreproducibility of research include those related to inappropriate study design methodologies and statistical analysis1. In modern statistical practice irreproducibility could arise due to statistical (false discoveries, p-hacking, overuse/misuse of p-values, low power, poor experimental design) and computational (data, code & software management) issues2. These require understanding the processes and workflows practiced by an organization, and the development and use of metrics to quantify reproducibility. Within the Foundation of Discovery - Population Health Research, Center for Clinical and Translational Science, University of Utah, we are undertaking a project to streamline the study design and statistical analysis workflows and processes. As a first step we met with key stakeholders to understand the current practices by eliciting example statistical projects, and then developed process information models for different types of statistical needs using Lucidchart. We then reviewed these with the Foundation’s leadership and Standards Committee to come up with ideal workflows and model, and defined key measurement points (such as those around study design, analysis plan, final report, requirements for quality checks, and double coding) for assessing reproducibility. As next steps we will use our finding to embed analytical and infrastructural approaches within the statisticians’ workflows. This will include data and code dissemination platforms such as Box, Bitbucket and GitHub, documentation platforms such as Confluence, and workflow tracking platforms such as Jira. These tools will simplify and automate the capture of communications as a statistician work through a project. Data-intensive process will use process-workflow management platforms such as Activiti3, Pegasus4 and Taverna6. These strategies for sharing and publishing study protocols, data, code and results across the spectrum7, active collaboration with the research team, automation of key steps, along with decision support will ensure quality of statistical methods and reproducibility of research. References 1. Reproducibility and reliability of biomedical research’, organised by the Academy of Medical Sciences, BBSRC, MRC and Wellcome Trust in April 2015. 2. National Academies of Sciences, Engineering, and Medicine. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press, 2016. doi:10.17226/21915. 3. Acivit, http://activiti.org 4. Pegasus, https://pegasus.isi.edu/ 5. Apache Taverna. https://taverna.incubator.apache.org/ 6. Peng, R. 2011. Reproducible research in computational science. Science 334(6060):1226-1227