Search

The NSF Office of Advanced Cyberinfrastructure has recognized the emerging and evolving need for platforms that fully integrate data and computing workflows, and is calling for research to deliver systems that provide a full spectrum of data services and also offer a coherent coupling with computing software. The Digital Environment to Enable Data-driven Science (DEEDS) project has created a cross-domain, self-serve platform for data and computing that supports the entire end-to-end research investigation process. DEEDS offers interactive interfaces to 1) collect, manage, and explore data, 2) define and launch tools, 3) track computational workflows, 4) access toolkits for ad-hoc analytics, and 5) publish and share the investigations with broader research community. All interfaces are available from a single dashboard so that the workflow between data and tools is smooth and intuitive. This paper highlights the DEEDS platform capabilities for providing integrated data and computing services throughout the entire research investigation lifecycle and presents some examples from use cases that contributed to the platform development.

Scientific workflows have emerged as a model for representing the complex processes carried out by scientists throughout their investigations, encompassing research activities corresponding to data collection, data flow, computation, output analysis, and all the ways these are used together to produce results. Existing infrastructures support elements of the workflow, such as data repositories or computing services, but these are not integrated as interactive environments that provide full investigation lifecycle support. The digital environment for enabling data-driven sciences (DEEDS) project brought together domain scientists and computer scientists to create a platform that provides interactive end-to-end support for diverse scientific workflows. Key among requirements were preservation, provenance, coupling of data and computing, results traceability, collaborative sharing, exploration, and publication of the full products of research work. This paper highlights use cases that contributed to DEEDS development and concludes with lessons learned from a process that joined experiences and perspectives from diverse science domains.

DataCenterHub is a new solution for preserving, sharing, and discovering data produced by scientific research. Datasets are organized by experiments, with a simple common structure for metadata, file collections, and key parameters. Researchers associate annotations, reports, media, and measurements to each experiment, and interactive viewers interpret data by type and use so that they can be investigated before downloading. Parameters are extracted for discovery of key data otherwise hidden in files. DataCenterHub provides an alternative discipline-neutral solution, with the goal of helping researchers classify and share data for easy discovery and exploration.

We offer the Digital Environment for Enabling Data-Driven Science (DEEDS) to scientific and engineering communities everywhere, as a full-service platform that provides end-to-end support for your research investigations.DEEDS is a systematic, secure, reliable way to conduct your research, with powerful yet user-friendly interactive support for big data, high-performance computing and shared scientific workflows.The design and development of DEEDS is a collaborative effort, bringing together scientists and engineers from chemistry (molecular dynamics, quantum chemistry), agriculture (environmental science, ecotoxicoloy, forest biodiversity, global ecology), electrical engineering (solarPV efficiency diagnosis), civil engineering (bridge health monitoring, earthquake engineering), health and human services (nutrition science, clinical studies), biology (RNA sequencing) and computer science (infrastructure, big data, high performance computing). The platform we developed supports research activities throughout the research investigation and is effective across science domains. Why DEEDS?In research investigations, scientists and engineers carry out complex, long-running, team-shared activities involving the collection and integration of data, the execution of computational software, the analysis and visualization of data and software output, and the synthesis of results to produce conclusions. In most disciplines this process is done in an ad-hoc manner because existing IT platforms support only part of the investigative workflow. Many ongoing research projects do not use cyberinfrastructure to support any part of their shared investigations. This means that data, computing, analysis, and outcomes reside in disconnected environments. The lack of continuity in the research process hinders it in many ways: the process is difficult to share and difficult to validate. It compromises reproducibility of results and makes reuse and reinterpretation of data and algorithms more difficult.With DEEDS ... research data, computing, and scientific workflows come together in datasets that you build, use and share across your entire investigation

With DEEDS ... data, computing, and scientific workflows come together in datasets that research teams build, use and share across their entire investigation.Researchers build and share DEEDS datasets using an interactive dashboard. The dashboard provides full-featured services for the organization and structuring of research activities; the upload and classification of data; the extraction and assignment of metadata; research computing and statistical modeling (including HPC computing); the automatic capture of scientific workflows for data provenance and reproducibility; and the analysis and visualization of results.User-friendly interfaces help research teams create, annotate, and link file repositories, multi-dimensional, hierarchical data tables, computational software, statistical models, scientific workflows and analytics – offering interactive search, exploration, and visualization across all elements of the dataset. Datasets are FAIR-compliant and can be published for discovery and interactive exploration of data, computing tools, and workflows for reuse and reinterpretation.The DEEDS dashboard provides services forData: Upload, preserve, manage and explore your data. Assign metadata, follow rules for metadata standards (FAIR compliance), integrate scripts to automatically transform, validate, curate and check completeness of your uploaded data. Datasets provide data services for-- Files classified by type, format and use, including standard categories and user defined categories (e.g., sensor data, mass spectrometry data, geospatial data, protocols). Files can be imported from external repositories (e.g., DuraMat through CKAN API). DEEDS offers applications to search, explore and visualize data by type (e.g., geotiff tile generation for map overlays). Some DEEDS dataset repositories have more than 3M files that are indexed, linked and classified by DEEDS for fast, user-friendly navigation, search and visualization.-- Data Tables that represent hierarchical, multi-dimensional data models for measurements, properties, observations and other data. These can be customized, organized, re-organized, cloned, and annotated across the investigation lifecycle. Users can upload spreadsheets or interactively update (including bulk updates). Data tables can be viewed, browsed, searched, filtered, and downloaded. Data table operations are robust and user friendly. Some DEEDS datasets have more than 300 columns, some have more than 11M rows. Data table cells can represent a single data points, data arrays, and single/multiple linked tabular datasheets. DEEDS data table update, view, search, filter and exploration remain robust across all representations. Data tables are linked to map overlays for unified location data exploration, and data tables are fundamental analytics structures for visualization, computing and analysis.Computing tools: Define computing tools to your dataset, then launch and track execution workflows. Your dataset computing tools can be computational research codes, open source software packages, modeling scripts, licensed software, Jupyter notebooks, RShiny and other Hubzero tools. DEEDS tools have full access to your dataset repository. A tool launched from the DEEDS dashboard follows the owner’s tool definition, allowing users to specify tool arguments, choose input files, select execution resources (including HPC facilities), and determine how the computing workflow should be tracked and captured. Output data are returned by DEEDS to the dataset – they are annotated and linked to input, tool, resource and user. Captured workflows can be viewed and searched for data provenance and results traceability. Tools defined for DEEDS datasets are added to the DEEDS tool repository, where they can be imported into other datasets with permission of their owners.Analytics: Define R data frames based on dataset files and data tables. DEEDS R-based analytics supports data filtering, merging, statistical computing, algorithm specification/ computation, and visualization. Data across datasets as well as external data can be merged within analytics for comparison and analysis. Introduction of new analytics features is ongoing, based on priorities of DEEDS research groups.FAIR compliance: DEEDS guarantees adherence to the principles of Findable, Accessible, Interoperable, and Reusable data management and stewardship for your research. DEEDS offers fine-grained access control as needed for your data and computing tools.

We offer the Digital Environment for Enabling Data-Driven Science (DEEDS) to scientific and engineering communities everywhere, as a full-service platform that provides end-to-end support for your research investigations.DEEDS is a systematic, secure, reliable way to conduct your research, with powerful yet user-friendly interactive support for big data, high-performance computing and shared scientific workflows.The design and development of DEEDS is a collaborative effort, bringing together scientists and engineers from chemistry (molecular dynamics, quantum chemistry), agriculture (environmental science, ecotoxicoloy, forest biodiversity, global ecology), electrical engineering (solarPV efficiency diagnosis), civil engineering (bridge health monitoring, earthquake engineering), health and human services (nutrition science, clinical studies), biology (RNA sequencing) and computer science (infrastructure, big data, high performance computing). The platform we developed supports research activities throughout the research investigation and is effective across science domains. Why DEEDS?In research investigations, scientists and engineers carry out complex, long-running, team-shared activities involving the collection and integration of data, the execution of computational software, the analysis and visualization of data and software output, and the synthesis of results to produce conclusions. In most disciplines this process is done in an ad-hoc manner because existing IT platforms support only part of the investigative workflow. Many ongoing research projects do not use cyberinfrastructure to support any part of their shared investigations. This means that data, computing, analysis, and outcomes reside in disconnected environments. The lack of continuity in the research process hinders it in many ways: the process is difficult to share and difficult to validate. It compromises reproducibility of results and makes reuse and reinterpretation of data and algorithms more difficult.With DEEDS ... research data, computing, and scientific workflows come together in datasets that you build, use and share across your entire investigation

With DEEDS ... data, computing, and scientific workflows come together in datasets that research teams build, use and share across their entire investigation.Researchers build and share DEEDS datasets using an interactive dashboard. The dashboard provides full-featured services for the organization and structuring of research activities; the upload and classification of data; the extraction and assignment of metadata; research computing and statistical modeling (including HPC computing); the automatic capture of scientific workflows for data provenance and reproducibility; and the analysis and visualization of results.User-friendly interfaces help research teams create, annotate, and link file repositories, multi-dimensional, hierarchical data tables, computational software, statistical models, scientific workflows and analytics – offering interactive search, exploration, and visualization across all elements of the dataset. Datasets are FAIR-compliant and can be published for discovery and interactive exploration of data, computing tools, and workflows for reuse and reinterpretation.The DEEDS dashboard provides services forData: Upload, preserve, manage and explore your data. Assign metadata, follow rules for metadata standards (exceeding FAIR compliance), integrate scripts to automatically transform, validate, curate and check completeness of your uploaded data. Datasets consist of the following:Files classified by type, format and use, including standard categories and user defined categories (e.g., sensor data, mass spectrometry data, geospatial data, protocols). Files can be imported from external repositories (e.g., DuraMat through CKAN API). DEEDS offers applications to search, explore and visualize data by type (e.g., geotiff tile generation for map overlays). Some DEEDS dataset repositories have more than 3M files that are indexed, linked and classified by DEEDS for fast, user-friendly navigation, search and visualization.Data Tables that represent hierarchical, multi-dimensional data models for measurements, properties, observations and other data. These can be customized, organized, re-organized, cloned, and annotated across the investigation lifecycle. Users can upload spreadsheets or interactively update (including bulk updates). Data tables can be viewed, browsed, searched, filtered, and downloaded. Data table operations are robust and user friendly. Some DEEDS datasets have more than 300 columns, some have more than 11M rows. Data table cells can represent a single data points, data arrays, and single/multiple linked tabular datasheets. DEEDS data table update, view, search, filter and exploration remain robust across all representations. Data tables are linked to map overlays for unified location data exploration, and data tables are fundamental analytics structures for visualization, computing and analysis.Computing tools: Define computing tools to your dataset, then launch and track execution workflows. Your dataset computing tools can be computational research codes, open source software packages, modeling scripts, licensed software, Jupyter notebooks, RShiny and other Hubzero tools. DEEDS tools have full access to your dataset repository. A tool launched from the DEEDS dashboard follows the owner’s tool definition, allowing users to specify tool arguments, choose input files, select execution resources (including HPC facilities), and determine how the computing workflow should be tracked and captured. Output data are returned by DEEDS to the dataset – they are annotated and linked to input, tool, resource and user. Captured workflows can be viewed and searched for data provenance and results traceability. Tools defined for DEEDS datasets are added to the DEEDS tool repository, where they can be imported into other datasets with permission of their owners.Analytics: Define R data frames based on dataset files and data tables. DEEDS R-based analytics supports data filtering, merging, statistical computing, algorithm specification/ computation, and visualization. Data across datasets as well as external data can be merged within analytics for comparison and analysis. Introduction of new analytics features is ongoing, based on priorities of DEEDS research groups.FAIR compliance: DEEDS guarantees adherence to the principles of Findable, Accessible, Interoperable, and Reusable data management and stewardship for your research. DEEDS offers fine-grained access control as needed for your data and computing tools.

We offer the Digital Environment for Enabling Data-Driven Science (DEEDS) to scientific and engineering communities everywhere, as a full-service platform that provides end-to-end support for your research investigations.DEEDS is a systematic, secure, reliable way to conduct your research, with powerful yet user-friendly interactive support for big data, high-performance computing and shared scientific workflows.The design and development of DEEDS is a collaborative effort, bringing together scientists and engineers from chemistry (molecular dynamics, quantum chemistry), agriculture (environmental science, ecotoxicoloy, forest biodiversity, global ecology), electrical engineering (solarPV efficiency diagnosis), civil engineering (bridge health monitoring, earthquake engineering), health and human services (nutrition science, clinical studies), biology (RNA sequencing) and computer science (infrastructure, big data, high performance computing). The platform we developed supports research activities throughout the research investigation and is effective across science domains. Why DEEDS?In research investigations, scientists and engineers carry out complex, long-running, team-shared activities involving the collection and integration of data, the execution of computational software, the analysis and visualization of data and software output, and the synthesis of results to produce conclusions. In most disciplines this process is done in an ad-hoc manner because existing IT platforms support only part of the investigative workflow. Many ongoing research projects do not use cyberinfrastructure to support any part of their shared investigations. This means that data, computing, analysis, and outcomes reside in disconnected environments. The lack of continuity in the research process hinders it in many ways: the process is difficult to share and difficult to validate. It compromises reproducibility of results and makes reuse and reinterpretation of data and algorithms more difficult.With DEEDS ... research data, computing, and scientific workflows come together in datasets that you build, use and share across your entire investigation

With DEEDS ... data, computing, and scientific workflows come together in datasets that research teams build, use and share across their entire investigation.Researchers build and share DEEDS datasets using an interactive dashboard. The dashboard provides full-featured services for the organization and structuring of research activities; the upload and classification of data; the extraction and assignment of metadata; research computing and statistical modeling (including HPC computing); the automatic capture of scientific workflows for data provenance and reproducibility; and the analysis and visualization of results.User-friendly interfaces help research teams create, annotate, and link file repositories, multi-dimensional, hierarchical data tables, computational software, statistical models, scientific workflows and analytics – offering interactive search, exploration, and visualization across all elements of the dataset. Datasets are FAIR-compliant and can be published for discovery and interactive exploration of data, computing tools, and workflows for reuse and reinterpretation.The DEEDS dashboard provides services forData: Upload, preserve, manage and explore your data. Assign metadata, follow rules for metadata standards (exceeding FAIR compliance), integrate scripts to automatically transform, validate, curate and check completeness of your uploaded data. Datasets consist of the following:Files classified by type, format and use, including standard categories and user defined categories (e.g., sensor data, mass spectrometry data, geospatial data, protocols). Files can be imported from external repositories (e.g., DuraMat through CKAN API). DEEDS offers applications to search, explore and visualize data by type (e.g., geotiff tile generation for map overlays). Some DEEDS dataset repositories have more than 3M files that are indexed, linked and classified by DEEDS for fast, user-friendly navigation, search and visualization.Data Tables that represent hierarchical, multi-dimensional data models for measurements, properties, observations and other data. These can be customized, organized, re-organized, cloned, and annotated across the investigation lifecycle. Users can upload spreadsheets or interactively update (including bulk updates). Data tables can be viewed, browsed, searched, filtered, and downloaded. Data table operations are robust and user friendly. Some DEEDS datasets have more than 300 columns, some have more than 11M rows. Data table cells can represent a single data points, data arrays, and single/multiple linked tabular datasheets. DEEDS data table update, view, search, filter and exploration remain robust across all representations. Data tables are linked to map overlays for unified location data exploration, and data tables are fundamental analytics structures for visualization, computing and analysis.Computing tools: Define computing tools to your dataset, then launch and track execution workflows. Your dataset computing tools can be computational research codes, open source software packages, modeling scripts, licensed software, Jupyter notebooks, RShiny and other Hubzero tools. DEEDS tools have full access to your dataset repository. A tool launched from the DEEDS dashboard follows the owner’s tool definition, allowing users to specify tool arguments, choose input files, select execution resources (including HPC facilities), and determine how the computing workflow should be tracked and captured. Output data are returned by DEEDS to the dataset – they are annotated and linked to input, tool, resource and user. Captured workflows can be viewed and searched for data provenance and results traceability. Tools defined for DEEDS datasets are added to the DEEDS tool repository, where they can be imported into other datasets with permission of their owners.Analytics: Define R data frames based on dataset files and data tables. DEEDS R-based analytics supports data filtering, merging, statistical computing, algorithm specification/ computation, and visualization. Data across datasets as well as external data can be merged within analytics for comparison and analysis. Introduction of new analytics features is ongoing, based on priorities of DEEDS research groups.FAIR compliance: DEEDS guarantees adherence to the principles of Findable, Accessible, Interoperable, and Reusable data management and stewardship for your research. DEEDS offers fine-grained access control as needed for your data and computing tools.

Research on recommendation systems has gained a considerable amount of attention over the past decade as the number of online users and online contents continue to grow at an exponential rate. With the evolution of the social web, people generate and consume data in real time using online services such as Twitter, Facebook, and web news portals. With the rapidly growing online community, web-based retail systems and social media sites have to process several millions of user requests per day. Generating quality recommendations using this vast amount of data is itself a very challenging task. Nevertheless, opposed to the web-based retailers such as Amazon and Netflix, the above-mentioned social networking sites have to face an additional challenge when generating recommendations as their contents are very rapidly changing. Therefore, providing fresh information in the least amount of time is a major objective of such recommender systems. Although collaborative filtering is a widely used technique in recommendation systems, generating the recommendation model using this approach is a costly task, and often done offline. Hence, it is difficult to use collaborative filtering in the presence of dynamically changing contents, as such systems require frequent updates to the recommendation model to maintain the accuracy and the freshness of the recommendations. Parallel processing power of graphic processing units (GPUs) can be used to process large volumes of data with dynamically changing contents in real time, and accelerate the recommendation process for social media data streams. In this paper, we address the issue of rapidly changing contents, and propose a parallel on-the-fly collaborative filtering algorithm using GPUs to facilitate frequent updates to the recommendations model. We use a hybrid similarity calculation method by combining the item–item collaborative filtering with item category information and temporal information. The experimental results on real-world datasets show that the proposed algorithm outperformed several existing online CF algorithms in terms of accuracy, memory consumption, and runtime. It was also observed that the proposed algorithm scaled well with the data rate and the data volume, and generated recommendations in a timely manner.

The NSF Office of Advanced Cyberinfrastructure has recognized the emerging and evolving need for platforms that fully integrate data and computing workflows, and is calling for research to deliver systems that provide a full spectrum of data services and also offer a coherent coupling with computing software. The Digital Environment to Enable Data-driven Science (DEEDS) project has created a cross-domain, self-serve platform for data and computing that supports the entire end-to-end research investigation process. DEEDS offers interactive interfaces to 1) collect, manage, and explore data, 2) define and launch tools, 3) track computational workflows, and 4) access toolkits for ad hoc analytics. All interfaces are available from a single dashboard so that the workflow between data and tools is smooth and intuitive. In this paper, we describe DEEDS innovations for handling data and computational workflows, and we present the use cases from four science domains that defined features, services, and usability requirements for DEEDS.

Due to the continuous, unbounded, and dynamic characteristics of the streaming data, mining data streams becomes a very challenging task. When analyzing online data streams, it is necessary to produce accurate results in a very short amount of time. The parallel processing power of Graphics Processing Units (GPUs) can be used to accelerate the processing and produce results in a timely manner. In this paper, we present GStreamMiner, a GPU-accelerated data stream mining framework and demonstrate its application using outlier detection over continuous streaming data as a case study. The demo software provides a visual interface which is continuously get updated with new results as the data stream progresses. It also facilitates the users to compare the performance of the GPU and CPU versions of the outlier detection algorithm.

Outlier detection or anomaly detection is applied in numerous applications, such as fraud detection, network intrusion detection, manufacturing, and environmental monitoring. Due to the continuous and dynamic characteristics of streaming data, outlier detection over data streams becomes a very challenging task. When analyzing real-time data streams, it is typically impossible to store the entire set of data due to space limitations. Also due to the high data rates, it is necessary to produce the results in a limited amount of time. Parallel processing power of Graphics Processing Units (GPUs) can be used to accelerate the outlier detection process, and thus address the challenges of outlier detection over data streams. This paper proposes a GPU-accelerated outlier detection algorithm for continuous data streams using kernel density estimation approach. Experiments show that the proposed SOD_GPU algorithm is efficient for detecting outliers in high-dimensional, high-speed data streams, and produces results in a timely manner without compromising the outlier detection accuracy. The proposed method achieved up to 20X speedup compared to a respective multi-core CPU implementation, and the speedup increases with the number of data attributes and the input data rate.

Data uncertainty is inherent in many real-world applications such as sensor monitoring systems, location-based services, and medical diagnostic systems. Moreover, many real-world applications are now capable of producing continuous, unbounded data streams. During the recent years, new methods have been developed to find frequent patterns in uncertain databases; nevertheless, very limited work has been done in discovering frequent patterns in uncertain data streams. The current solutions for frequent pattern mining in uncertain streams take a FP-tree-based approach; however, recent studies have shown that FP-tree-based algorithms do not perform well in the presence of data uncertainty. In this paper, we propose two hyper-structure-based false-positive-oriented algorithms to efficiently mine frequent itemsets from streams of uncertain data. The first algorithm, UHS-Stream, is designed to find all frequent itemsets up to the current moment. The second algorithm, TFUHS-Stream, is designed to find frequent itemsets in an uncertain data stream in a time-fading manner. Experimental results show that the proposed hyper-structure-based algorithms outperform the existing tree-based algorithms in terms of accuracy, runtime, and memory usage.