Search

We present an overview of key aspects of the Atmospheric Radiation Measurement (ARM) Program Climate Research Facility (ACRF) data quality assurance program. Processes described include instrument deployment and calibration; instrument and facility maintenance; data collection and processing infrastructure; data stream inspection and assessment; problem reporting, review and resolution; data archival, display and distribution; data stream reprocessing; engineering and operations management; and the roles of value-added data processing and targeted field campaigns in specifying data quality and characterizing field measurements. The paper also includes a discussion of recent directions in ACRF data quality assurance. A comprehensive, end-to-end data quality assurance program is essential for producing a high-quality data set from measurements made by automated weather and climate networks. The processes developed during the ARM Program offer a possible framework for use by other instrumentation- and geographically-diverse data collection networks and highlight the myriad aspects that go into producing research-quality data.

The total quantity of digital information in the world is growing at an alarming rate. Scientists and engineers are contributing heavily to this data “tsunami” by gathering data using computing and instrumentation at incredible rates. As data volumes and complexity grow, it is increasingly arduous to extract valuable information from the data and derive knowledge from that data. Addressing these demands of ever-growing data volumes and complexity requires game-changing advances in software, hardware, and algorithms. Solution technologies also must scale to handle the increased data collection and processing rates and simultaneously accelerate timely and effective analysis results. This need for ever faster data processing and manipulation as well as algorithms that scale to high-volume data sets have given birth to a new paradigm or discipline known as “data-intensive computing.” In this chapter, we define data-intensive computing, identify the challenges of massive data, outline solutions for hardware, software, and analytics, and discuss a number of applications in the areas of biology, cyber security, and atmospheric research.

Global climate researchers rely upon many forms of sensor data and analytical methods to help profile subtle changes in climate conditions. The U.S. Department of Energy Atmospheric Radiation Measurement (ARM) program provides researchers with curated Value Added Products (VAPs) resulting from continuous instrumentation streams, data fusion, and analytical profiling. The ARM operational staff and software development teams (data producers) rely upon a number of techniques to ensure strict quality control (QC) and quality assurance (QA) standards are maintained. Climate researchers (data consumers) are highly interested in obtaining as much provenance evidence as possible to establish data trustworthiness. Currently all the evidence is not easily attainable or identifiable without significant efforts to extract and piece together information from configuration files, log files, codes, or status information on the ARM website. Our objective is to identify a provenance model that serves the needs of both the VAP producers and consumers. This paper shares our initial results – a comprehensive multi-tier provenance model. We describe how both ARM operations staff and the climate research community can greatly benefit from this approach to more effectively assess and quantify the data historical record.

This report documents key aspects of the Atmospheric Radiation Measurement (ARM) Climate Research Facility (ACRF) data quality assurance program as it existed in 2008. The performance of ACRF instruments, sites, and data systems is measured in terms of the availability, usability, and accessibility of the data to a user. First, the data must be available to users; that is, the data must be collected by instrument systems, processed, and delivered to a central repository in a timely manner. Second, the data must be usable; that is, the data must be inspected and deemed of sufficient quality for scientific research purposes, and data users must be able to readily tell where there are known problems in the data. Finally, the data must be accessible; that is, data users must be able to easily find, obtain, and work with the data from the central repository. The processes described in this report include instrument deployment and calibration; instrument and facility maintenance; data collection and processing infrastructure; data stream inspection and assessment; the roles of value-added data processing and field campaigns in specifying data quality and haracterizing the basic measurement; data archival, display, and distribution; data stream reprocessing; and engineering and operations management processes and procedures. Future directions in ACRF data quality assurance also are presented.