Search

Statistical data tools like SAS, SPSS, and Stata and programming languages such as R and Python give data users powerful capabilities for data analysis. However, these tools have limited metadata capabilities. Colectica Datasets is a new tool that takes a metadata first approach to dataset creation, embedding DDI metadata into the data files themselves. Colectica Datasets can import SAS, SPSS, Stata, CSV, and Parquet files. Additional metadata about the dataset and its variables can be created by the user. Variable labels and value labels can be specified in multiple languages. The tool can then export the enhanced dataset to SPSS, Stata, CSV, Parquet, or Excel, and create summary statistics, weighted statistics, and codebook documentation. Users can run quality checks on their dataset which show alerts where additional metadata would be useful. In addition to embedding enriched metadata, Colectica Datasets can be used as a transfer tool between various statistical file formats. It automatically handles different types of missing values, format mapping, and data type detection. Colectica Datasets is free for personal use and available on Windows and macOS.

We show how two models of provenance can work together to answer basic questions about data provenance, such as “What computed variables were affected by values of variable X?” The W3C PROV data model is a standard for describing activities and persons that produce digital artifacts. PROV associates processes with inputs and outputs, but it does not have a way to describe how data are changed within the process. PROV has no language for program components, like mathematical expressions or joining data tables. Structured Data Transformation Language (SDTL) provides machine-actionable representations of data transformation commands in the five most widely-used statistical analysis applications. SDTL is a procedural language in which commands are executed sequentially. Thus, SDTL describes the inner workings of programs that are black boxes in PROV. However, SDTL is detailed and verbose, and simple queries can be very complicated in SDTL. Combining PROV and SDTL allows us to answer questions about data preparation and management at levels not available in PROV. Our bridge between PROV and SDTL rests on two pillars: ProvONE, an extension of PROV, and Structured Data Transformation History (SDTH), a simplified view of SDTL.

The presentation is in two parts: 1. Colectica 7.1 Colectica is happy to announce at EDDI the release of Colectica version 7.1. Colectica is software for creating, publishing, centralizing and managing DDI metadata within and across organizations. It is used by national statistical organizations, university research groups, and data collection agencies to provide well-documented data to researchers and the public. Colectica is built on open standards like DDI and GSIM, ensuring that information can be presented in numerous formats and shared among different organizations and tools. In this session we will give an overview of new features of Colectica 7.1, including: New enterprise authentication methods and features, including role mapping and OpenIDConnect PKCE Improved ElasticSearch indexer Difference viewer within Colectica Workflow Language and translation improvements on Colectica Portal Metadata input sheets A new CLI tool for Colectica Repository Nesstar and DDI 2 import improvements Parquet file support 2. Quality Reporting: Using DDI and Colectica to manage reference metadata in the European Statistical System Reference metadata provides high level information about datasets, data collection, and methods. The European Statistical System (ESS) maintains several reporting standards that describe the information that European statistical agencies must report. These standards include the Single Integrated Metadata Structure (SIMS), Euro SDMX Metadata Structure (ESMS), and the ESS Standard Quality Report Structure (ESQRS). DDI Lifecycle provides a metadata model to represent the information required by these reports. DDI Quality Standards can describe the structure of the reference metadata, and DDI Quality Statements hold the quality content for individual statistical products. Working with several European statistical agencies, Colectica created DDI Quality Standards and Concept Systems to represent the ESS reporting standards. Colectica now offers web-based editors for managing both Quality Standards and Quality Statements. The Colectica quality tools offer review and approval workflows, rollover and pre-fill functionality, and standardized report outputs in SDMX, Excel, PDF, and HTML formats. This presentation will include a demonstration of the new tools. We will also discuss how the tools can be configured and extended to record extra quality information, beyond what Eurostat requires.