Search

We show how two models of provenance can work together to answer basic questions about data provenance, such as “What computed variables were affected by values of variable X?” The W3C PROV data model is a standard for describing activities and persons that produce digital artifacts. PROV associates processes with inputs and outputs, but it does not have a way to describe how data are changed within the process. PROV has no language for program components, like mathematical expressions or joining data tables. Structured Data Transformation Language (SDTL) provides machine-actionable representations of data transformation commands in the five most widely-used statistical analysis applications. SDTL is a procedural language in which commands are executed sequentially. Thus, SDTL describes the inner workings of programs that are black boxes in PROV. However, SDTL is detailed and verbose, and simple queries can be very complicated in SDTL. Combining PROV and SDTL allows us to answer questions about data preparation and management at levels not available in PROV. Our bridge between PROV and SDTL rests on two pillars: ProvONE, an extension of PROV, and Structured Data Transformation History (SDTH), a simplified view of SDTL.

The presentation is in two parts: 1. Colectica 7.1 Colectica is happy to announce at EDDI the release of Colectica version 7.1. Colectica is software for creating, publishing, centralizing and managing DDI metadata within and across organizations. It is used by national statistical organizations, university research groups, and data collection agencies to provide well-documented data to researchers and the public. Colectica is built on open standards like DDI and GSIM, ensuring that information can be presented in numerous formats and shared among different organizations and tools. In this session we will give an overview of new features of Colectica 7.1, including: New enterprise authentication methods and features, including role mapping and OpenIDConnect PKCE Improved ElasticSearch indexer Difference viewer within Colectica Workflow Language and translation improvements on Colectica Portal Metadata input sheets A new CLI tool for Colectica Repository Nesstar and DDI 2 import improvements Parquet file support 2. Quality Reporting: Using DDI and Colectica to manage reference metadata in the European Statistical System Reference metadata provides high level information about datasets, data collection, and methods. The European Statistical System (ESS) maintains several reporting standards that describe the information that European statistical agencies must report. These standards include the Single Integrated Metadata Structure (SIMS), Euro SDMX Metadata Structure (ESMS), and the ESS Standard Quality Report Structure (ESQRS). DDI Lifecycle provides a metadata model to represent the information required by these reports. DDI Quality Standards can describe the structure of the reference metadata, and DDI Quality Statements hold the quality content for individual statistical products. Working with several European statistical agencies, Colectica created DDI Quality Standards and Concept Systems to represent the ESS reporting standards. Colectica now offers web-based editors for managing both Quality Standards and Quality Statements. The Colectica quality tools offer review and approval workflows, rollover and pre-fill functionality, and standardized report outputs in SDMX, Excel, PDF, and HTML formats. This presentation will include a demonstration of the new tools. We will also discuss how the tools can be configured and extended to record extra quality information, beyond what Eurostat requires.

A controlled vocabulary is an organized set of terms. Controlled vocabularies make it easier to make information consistent. They also improve indexing and searching for information. DDI Lifecycle supports using controlled vocabularies for many of its metadata fields. Colectica 6.2 adds support for controlled vocabularies in its desktop and Web tools. Administrators can configure which vocabulary to use for specific content fields. Users can choose terms from the appropriate vocabulary when editing content. Vocabularies are specified as DDI Lifecycle code lists. DDI code lists allow specifying the values and terms of a vocabulary in one or more languages. Organizations can register these code lists in a metadata repository for persistence and revision tracking. Colectica can import vocabularies from the Simple Knowledge Organization System (SKOS). The Consortium of European Social Science Data Archives (CESSDA) runs a vocabulary service, providing 28 vocabularies in more than 10 languages. These vocabularies are downloadable in SKOS format. This presentation will show how all these vocabularies can integrate with DDI Lifecycle and the Colectica software. Recorded version of this talk: https://youtu.be/2DGOooUQDjQ

Since DDI Lifecycle 3.2, the variable cascade has allowed data producers to define the concordance among variables in datasets. Since at least 2014, several longitudinal studies and national statistics organizations have used this metadata structure to document and publish information on variables within single studies or organizations. Separating data definitions into conceptual variables, represented variables, and instance variables allows precise documentation of the data and how it changes over time. The same basic metadata structure can be used to document concordance across studies. Since separate organizations often create conceptual variables with similar meanings as conceptual variables from other organizations, the challenge becomes declaring the comparability of conceptual variables. For the intellectual content work of deciding on comparability, this presents an easier task than individual concording a large number of instance variables. For the technical task of defining the concordance in a structured manner, DDI Lifecycle offers a standardized solution. This presentation will provide an analysis of the DDI Lifecycle metadata structure used to perform the cross study concordance; discuss the workflow used to harmonize hundreds of variables from several large, longitudinal studies; and demonstrate the software tools used to create, publish, and visualize the data concordance. Recorded version of the whole session: https://youtu.be/mBJhJPGM60g

1. “Metadata first” questionnaire design using DDI metadata 2. Transform DDI metadata into a data capture instrument 3. Field the survey and collect survey data 4. Export data and metadata 5. Integrate data with metadata 6. View results in a DDI 3 (Lifecycle) metadata portal

DDI supports documenting data with variable-level detail. Normally such information includes a variable’s name, label and data type, but DDI supports the inclusion of more details (e.g., the variable’s lineage). The data behind a variable may have originated from a survey, from administrative data or from some other dataset. The variable may have been copied directly from a source, or it may have been calculated using a formula or algorithm. For a researcher to properly understand and analyze the data, the lineage of each variable should be documented in as much detail as possible. Variable-level provenance information can be recorded by data management staff manually. However, in some cases it is also possible to extract the transformations that were used to create a variable based on statistical source code. The C2Metadata project, sponsored by the U.S. National Science Foundation, is building tools to accomplish this and to record the transformations in structured ways that can be included with data documentation. This presentation will show data documentation from real studies that are providing variable-level provenance information and will discuss the methods used to create and publish the documentation.

Colectica is software used to document and publish statistical data using open standards. The software is used by national statistical organizations and major longitudinal studies worldwide. Colectica provides several tools: Colectica Questionnaires for specifying surveys in a standard way Colectica Datasets for documenting SAS, SPSS, Stata, and other statistical datasets Colectica Designer for documenting the entire data lifecycle Colectica Portal for pushing richly-documented data on the Web, with full variable-level lineage and concordance over time This presentation will provide an overview of the tools, show how they are used in production at various statistical agencies and research projects, and will highlight new functionality available in 2018.

This workshop offers a hands-on, practical approach to creating and documenting both surveys and datasets with DDI and Colectica. Participants will build and field a DDI-driven survey using their own questions or samples provided in the workshop. They will then ingest, annotate, and publish DDI dataset descriptions using the collected survey data.

Accurate and complete metadata is essential for data sharing and for interoperability across different data types. However, the process of describing and documenting scientific data has remained a tedious, manual process even when data collection is fully automated. Researchers are often reluctant to share data even with close colleagues, because creating documentation takes so much time. This presentation will describe a project to greatly reduce the cost and increase the completeness of metadata by creating tools to capture data transformations from general purpose statistical analysis packages. Researchers in many fields use the main statistics packages (SPSS®, SAS® Stata® R) for data management as well as analysis, but these packages lack tools for documenting variable transformations in the manner of a workflow system or even a database. At best the operations performed by the statistical package are described in a script, which more often than not is unavailable to future data users. Our project is developing new tools that will work with common statistical packages to automate the capture of metadata at the granularity of individual data transformations. Software-independent data transformation descriptions will be added to metadata in two internationally accepted standards, the Data Documentation Initiative (DDI) and Ecological Markup Language (EML). These tools will create efficiencies and reduce the costs of data collection, preparation, and re-use. Our project targets research communities with strong metadata standards and heavy reliance on statistical analysis software (social and behavioral sciences and earth observation sciences), but it is generalizable to other domains, such as biomedical research.

The DDI 3.2 standard is called “DDI Lifecycle,” so named because it describes metadata at each stage of the research data lifecycle, i.e., “from cradle to grave.” While few projects actually employ DDI Lifecycle to drive these processes, demonstrating that DDI is an efficient framework for organizing typical survey tasks would prove that the DDI community can “eat your own dogfood” by using the standard to manage its own research. In 2015 the North American DDI (NADDI) conference introduced a DDI-based protocol to manage the feedback survey conducted with conference participants. Because similar evaluations had been performed at previous NADDI conferences, this project also demonstrated DDI 3.2’s facility in describing study series in a cross-sectional panel survey design. This presentation will elaborate the rationale for the project, describe the relationship between the two principal stakeholders (Colectica and Nooro), and describe how DDI informed and drove each step of the fielding process: designing the conceptual questionnaire; fielding the designed instrument; documenting response data and linking to questionnaire metadata; and displaying, harmonizing, and comparing results with prior years. Finally, this presentation demonstrates that feasibility of DDI-driven conference evaluations and proposes a similar project be conducted for EDDI.

This presentation discusses how to prepare data files for preservation with Colectica Datasets.

The DDI Lifecycle metadata standard enables creating, documenting, managing, distributing, and discovering data. Colectica is a software tool that is built on open metadata standards, and helps facilitate adopting DDI into the research data management process. This workshop starts with a high-level overview of the DDI content model, and then teaches how to create DDI XML, both manually and with Colectica. Finally, participants will learn how to publish DDI metadata. This workshop covers the following topics: Introduction to DDI 3.2Introduction to ColecticaDocumenting concepts and general study designDesigning and documenting data collection instruments and surveysDocumenting variables and creating linkagesIngesting existing resourcesPublishing resourcesHands-on: use Colectica and DDI to manage a sample study