Back to Search
Start Over
Extending science gateway frameworks to support Big Data applications in the cloud
- Source :
- Journal of Grid Computing, 2016, 14(4), 589-601, UCrea Repositorio Abierto de la Universidad de Cantabria, Universidad de Cantabria (UC)
- Publication Year :
- 2016
- Publisher :
- Springer Nature, 2016.
-
Abstract
- Cloud computing offers massive scalability and elasticity required by many scientific and commercial applications. Combining the computational and data handling capabilities of clouds with parallel processing also has the potential to tackle Big Data problems efficiently. Science gateway frameworks and workflow systems enable application developers to implement complex applications and make these available for end-users via simple graphical user interfaces. The integration of such frameworks with Big Data processing tools on the cloud opens new opportunities for application developers. This paper investigates how workflow systems and science gateways can be extended with Big Data processing capabilities. A generic approach based on infrastructure aware workflows is suggested and a proof of concept is implemented based on the WS-PGRADE/gUSE science gateway framework and its integration with the Hadoop parallel data processing solution based on the MapReduce paradigm in the cloud. The provided analysis demonstrates that the methods described to integrate Big Data processing with workflows and science gateways work well in different cloud infrastructures and application scenarios, and can be used to create massively parallel applications for scientific analysis of Big Data. This work is partially funded by the CloudSME Cloud-Based Simulation platform for Manufacturing and Engineering Project No. 608886 (FP7-2013-NMPICT-FOF). Financial support from Programa de Personal Investigador en Formacion Predoctoral from Universidad de ´ Cantabria, co-funded by the regional government of Cantabria, has also been utilized.
- Subjects :
- Group method of data handling
Computer science
Computer Networks and Communications
Big data
Science gateway
Cloud computing
02 engineering and technology
computer.software_genre
WS-PGRADE
Workflow
020204 information systems
0202 electrical engineering, electronic engineering, information engineering
MapReduce
Massively parallel
Graphical user interface
Database
9. Industry and infrastructure
business.industry
Elasticity (cloud computing)
Hardware and Architecture
Hadoop
Scalability
020201 artificial intelligence & image processing
business
Software engineering
computer
Software
Information Systems
Subjects
Details
- Database :
- OpenAIRE
- Journal :
- Journal of Grid Computing, 2016, 14(4), 589-601, UCrea Repositorio Abierto de la Universidad de Cantabria, Universidad de Cantabria (UC)
- Accession number :
- edsair.doi.dedup.....5e3f9d90e988c3516b845456a068a64a