Search

We show how two models of provenance can work together to answer basic questions about data provenance, such as “What computed variables were affected by values of variable X?” The W3C PROV data model is a standard for describing activities and persons that produce digital artifacts. PROV associates processes with inputs and outputs, but it does not have a way to describe how data are changed within the process. PROV has no language for program components, like mathematical expressions or joining data tables. Structured Data Transformation Language (SDTL) provides machine-actionable representations of data transformation commands in the five most widely-used statistical analysis applications. SDTL is a procedural language in which commands are executed sequentially. Thus, SDTL describes the inner workings of programs that are black boxes in PROV. However, SDTL is detailed and verbose, and simple queries can be very complicated in SDTL. Combining PROV and SDTL allows us to answer questions about data preparation and management at levels not available in PROV. Our bridge between PROV and SDTL rests on two pillars: ProvONE, an extension of PROV, and Structured Data Transformation History (SDTH), a simplified view of SDTL.

The presentation is in two parts: 1. Colectica 7.1 Colectica is happy to announce at EDDI the release of Colectica version 7.1. Colectica is software for creating, publishing, centralizing and managing DDI metadata within and across organizations. It is used by national statistical organizations, university research groups, and data collection agencies to provide well-documented data to researchers and the public. Colectica is built on open standards like DDI and GSIM, ensuring that information can be presented in numerous formats and shared among different organizations and tools. In this session we will give an overview of new features of Colectica 7.1, including: New enterprise authentication methods and features, including role mapping and OpenIDConnect PKCE Improved ElasticSearch indexer Difference viewer within Colectica Workflow Language and translation improvements on Colectica Portal Metadata input sheets A new CLI tool for Colectica Repository Nesstar and DDI 2 import improvements Parquet file support 2. Quality Reporting: Using DDI and Colectica to manage reference metadata in the European Statistical System Reference metadata provides high level information about datasets, data collection, and methods. The European Statistical System (ESS) maintains several reporting standards that describe the information that European statistical agencies must report. These standards include the Single Integrated Metadata Structure (SIMS), Euro SDMX Metadata Structure (ESMS), and the ESS Standard Quality Report Structure (ESQRS). DDI Lifecycle provides a metadata model to represent the information required by these reports. DDI Quality Standards can describe the structure of the reference metadata, and DDI Quality Statements hold the quality content for individual statistical products. Working with several European statistical agencies, Colectica created DDI Quality Standards and Concept Systems to represent the ESS reporting standards. Colectica now offers web-based editors for managing both Quality Standards and Quality Statements. The Colectica quality tools offer review and approval workflows, rollover and pre-fill functionality, and standardized report outputs in SDMX, Excel, PDF, and HTML formats. This presentation will include a demonstration of the new tools. We will also discuss how the tools can be configured and extended to record extra quality information, beyond what Eurostat requires.

A controlled vocabulary is an organized set of terms. Controlled vocabularies make it easier to make information consistent. They also improve indexing and searching for information. DDI Lifecycle supports using controlled vocabularies for many of its metadata fields. Colectica 6.2 adds support for controlled vocabularies in its desktop and Web tools. Administrators can configure which vocabulary to use for specific content fields. Users can choose terms from the appropriate vocabulary when editing content. Vocabularies are specified as DDI Lifecycle code lists. DDI code lists allow specifying the values and terms of a vocabulary in one or more languages. Organizations can register these code lists in a metadata repository for persistence and revision tracking. Colectica can import vocabularies from the Simple Knowledge Organization System (SKOS). The Consortium of European Social Science Data Archives (CESSDA) runs a vocabulary service, providing 28 vocabularies in more than 10 languages. These vocabularies are downloadable in SKOS format. This presentation will show how all these vocabularies can integrate with DDI Lifecycle and the Colectica software. Recorded version of this talk: https://youtu.be/2DGOooUQDjQ

Since DDI Lifecycle 3.2, the variable cascade has allowed data producers to define the concordance among variables in datasets. Since at least 2014, several longitudinal studies and national statistics organizations have used this metadata structure to document and publish information on variables within single studies or organizations. Separating data definitions into conceptual variables, represented variables, and instance variables allows precise documentation of the data and how it changes over time. The same basic metadata structure can be used to document concordance across studies. Since separate organizations often create conceptual variables with similar meanings as conceptual variables from other organizations, the challenge becomes declaring the comparability of conceptual variables. For the intellectual content work of deciding on comparability, this presents an easier task than individual concording a large number of instance variables. For the technical task of defining the concordance in a structured manner, DDI Lifecycle offers a standardized solution. This presentation will provide an analysis of the DDI Lifecycle metadata structure used to perform the cross study concordance; discuss the workflow used to harmonize hundreds of variables from several large, longitudinal studies; and demonstrate the software tools used to create, publish, and visualize the data concordance. Recorded version of the whole session: https://youtu.be/mBJhJPGM60g

1. “Metadata first” questionnaire design using DDI metadata 2. Transform DDI metadata into a data capture instrument 3. Field the survey and collect survey data 4. Export data and metadata 5. Integrate data with metadata 6. View results in a DDI 3 (Lifecycle) metadata portal

DDI supports documenting data with variable-level detail. Normally such information includes a variable’s name, label and data type, but DDI supports the inclusion of more details (e.g., the variable’s lineage). The data behind a variable may have originated from a survey, from administrative data or from some other dataset. The variable may have been copied directly from a source, or it may have been calculated using a formula or algorithm. For a researcher to properly understand and analyze the data, the lineage of each variable should be documented in as much detail as possible. Variable-level provenance information can be recorded by data management staff manually. However, in some cases it is also possible to extract the transformations that were used to create a variable based on statistical source code. The C2Metadata project, sponsored by the U.S. National Science Foundation, is building tools to accomplish this and to record the transformations in structured ways that can be included with data documentation. This presentation will show data documentation from real studies that are providing variable-level provenance information and will discuss the methods used to create and publish the documentation.

Colectica is software used to document and publish statistical data using open standards. The software is used by national statistical organizations and major longitudinal studies worldwide. Colectica provides several tools: Colectica Questionnaires for specifying surveys in a standard way Colectica Datasets for documenting SAS, SPSS, Stata, and other statistical datasets Colectica Designer for documenting the entire data lifecycle Colectica Portal for pushing richly-documented data on the Web, with full variable-level lineage and concordance over time This presentation will provide an overview of the tools, show how they are used in production at various statistical agencies and research projects, and will highlight new functionality available in 2018.

This workshop offers a hands-on, practical approach to creating and documenting both surveys and datasets with DDI and Colectica. Participants will build and field a DDI-driven survey using their own questions or samples provided in the workshop. They will then ingest, annotate, and publish DDI dataset descriptions using the collected survey data.

Accurate and complete metadata is essential for data sharing and for interoperability across different data types. However, the process of describing and documenting scientific data has remained a tedious, manual process even when data collection is fully automated. Researchers are often reluctant to share data even with close colleagues, because creating documentation takes so much time. This presentation will describe a project to greatly reduce the cost and increase the completeness of metadata by creating tools to capture data transformations from general purpose statistical analysis packages. Researchers in many fields use the main statistics packages (SPSS®, SAS® Stata® R) for data management as well as analysis, but these packages lack tools for documenting variable transformations in the manner of a workflow system or even a database. At best the operations performed by the statistical package are described in a script, which more often than not is unavailable to future data users. Our project is developing new tools that will work with common statistical packages to automate the capture of metadata at the granularity of individual data transformations. Software-independent data transformation descriptions will be added to metadata in two internationally accepted standards, the Data Documentation Initiative (DDI) and Ecological Markup Language (EML). These tools will create efficiencies and reduce the costs of data collection, preparation, and re-use. Our project targets research communities with strong metadata standards and heavy reliance on statistical analysis software (social and behavioral sciences and earth observation sciences), but it is generalizable to other domains, such as biomedical research.

The DDI 3.2 standard is called “DDI Lifecycle,” so named because it describes metadata at each stage of the research data lifecycle, i.e., “from cradle to grave.” While few projects actually employ DDI Lifecycle to drive these processes, demonstrating that DDI is an efficient framework for organizing typical survey tasks would prove that the DDI community can “eat your own dogfood” by using the standard to manage its own research. In 2015 the North American DDI (NADDI) conference introduced a DDI-based protocol to manage the feedback survey conducted with conference participants. Because similar evaluations had been performed at previous NADDI conferences, this project also demonstrated DDI 3.2’s facility in describing study series in a cross-sectional panel survey design. This presentation will elaborate the rationale for the project, describe the relationship between the two principal stakeholders (Colectica and Nooro), and describe how DDI informed and drove each step of the fielding process: designing the conceptual questionnaire; fielding the designed instrument; documenting response data and linking to questionnaire metadata; and displaying, harmonizing, and comparing results with prior years. Finally, this presentation demonstrates that feasibility of DDI-driven conference evaluations and proposes a similar project be conducted for EDDI.

This presentation discusses how to prepare data files for preservation with Colectica Datasets.

The DDI Lifecycle metadata standard enables creating, documenting, managing, distributing, and discovering data. Colectica is a software tool that is built on open metadata standards, and helps facilitate adopting DDI into the research data management process. This workshop starts with a high-level overview of the DDI content model, and then teaches how to create DDI XML, both manually and with Colectica. Finally, participants will learn how to publish DDI metadata. This workshop covers the following topics: Introduction to DDI 3.2Introduction to ColecticaDocumenting concepts and general study designDesigning and documenting data collection instruments and surveysDocumenting variables and creating linkagesIngesting existing resourcesPublishing resourcesHands-on: use Colectica and DDI to manage a sample study

Researchers wishing to use data from longitudinal studies or to replicate other's research must currently navigate thousands of variables across multiple waves and datasets to answer simple analysis questions. A tool that allows researchers to create documented and citable data extracts that are directly related to their queries would allow more time to be spent on public health research questions instead of data management. MIDUS (Midlife in the United States) is a national longitudinal study of approximately 10,000 Americans designed to study aging as an integrated biopsychosocial process. The study has a unique blend of social, health, and biomarker data collected over several decades. In late 2013, the the United States National Institutes of Health funded MIDUS to create a DDI-based, harmonized data extraction system. This tool will facilitate identification and harmonization of similar MIDUS variables, while enhancing the MIDUS online repository with a data extract function. This will accomplish something unprecedented: the ability to obtain customized cross-project downloads of harmonized MIDUS data that are DDI-compliant. Doing so will greatly enhance efficient and effective public use of the large longitudinal and multi-disciplinary datasets that comprise the MIDUS study. This session will discuss project background and demonstrate the current state of the software.

Colectica is a standards-based platform for creating, documenting, managing, distributing, and discovering data. Colectica aims to create publishable documentation as a by-product of the data management process. This booth will provide live demonstrations of the various components of the Colectica platform. Colectica for Excel is a new, free add-in for Microsoft Excel that allows you to document spreadsheet data using the DDI-Lifecycle standard. Colectica Repository is a centralized storage system for managing data resources, enabling team-based data management, and providing automatic version control. Colectica Designer interacts with Colectica Repository to provide advanced data management and documentation functionality. Colectica Designer can import data and metadata from a variety of formats, and can generate documentation and source code in a variety of formats. Colectica Portal is a web-based application, powered by Colectica Repository, which enables data and metadata publication and discovery. Colectica Portal integrates with several social networking technologies to provide enhanced collaboration and discovery.nbsp

Traditionally, data in spreadsheets and plain text formats do not contain rich documentation. Often, single-word column headers are the only hint given to data users, making it difficult to make sense of the data. Colectica for Microsoft Excel is a new, free tool to document your spreadsheet data using DDI, the open standard for data documentation. With this Excel add-in, users can add extensive information about each column of data. Variables, Code Lists, and the datasets can be globally identified and described in a standard format. This documentation is embedded with the spreadsheet, ensuring the information is available when data are shared. The add-in also adds support for SPSS and Stata formats to Excel. When opening an SPSS or Stata file in Excel, standard metadata is automatically created from the variable and value labels. Colectica for Excel can create print-ready reports based on the data documentation. The information can also be exported to the DDI standard, which can be ingested into other standards-based tools. This presentation will include a live demonstration of the Colectica for Excel tool, showing how to document the contents of a spreadsheet, publish the information, and use the documentation to access data in an informed way.

The Swedish National Data Service (SND) maintains metadata about its holdings in the Data Documentation Initiative's DDI-Lifecycle format. The total amount studies in the holdings amounts to over one thousand, both quantitative and qualitative. SND stores and indexes this metadata using eXist-db, an open source XML database. Colectica is another DDI 3-based tool, but by default it uses a different repository structure for storing metadata. In order to allow Colectica tools to interact with SND metadata, we implemented a set of Web Services on top of eXist-db that allow Colectica to store and load information using eXist-db. We will demonstrate functionality provided by the eXist-db system, discuss the steps we took to integrate with Colectica, and demonstrate the resulting functionality with the two systems working together. We will also present recommendations on how to interact between DDI repositories in general and DDI tools. Implementations on our approach could be done from other DDI repositories.

Colectica is a DDI 3-based platform for creating, documenting, managing, distributing, and discovering data. Colectica aims to create publishable documentation as a by-product of the data management process. This demonstration will focus on features that have been added to Colectica over the past year, including: - Metadata repository for multi-user collaboration - Workflow management - Automated metadata harmonization - Improved Web-based data discovery and dissemination.

This paper is part of a series that focuses on DDI usage and how the metadata specification should be applied in a variety of settings by a variety of organizations and individuals. Support for this working paper series was provided by the authors’ home institutions; by GESIS - Leibniz Institute for the Social Sciences; by Schloss Dagstuhl - Leibniz Center for Informatics; and by the DDI Alliance.

This demonstration will show Colectica, a set of fully supported, commercial tools specifically designed for questionnaire creation and data documentation. These tools can automatically create CAI source code, paper questionnaires, and statistical source code. They enable data and documentation to be published to the web and to paper documentation formats. An ISO 11179 based metadata repository, backed by DDI3, enables collaborative workflows for the entire research process. The entire data life cycle can be easily visualized using the free Colectica Express viewer.

As presented at IASSIST 2009 conference

As presented at IASSIST 2009 conference