NetCDF4 data dissemination in the context of the Copernicus Marine Service

Marine Information systems like the operational Copernicus Marine Service (CMEMS) are widely used to enable access to sea data by a significant worldwide user community of nearly 5000 registered users from intergovernmental bodies, European agencies, regional and national service provides and the private sector. The central information system (CIS) beneath this service is the result of more than 6 years of intensive research and improvements carried out in the course of the MyOcean projects (2009-2013) under FP7 and H2020. The Central Information System covers all the service needs from data discovery, download with authentication, viewing and catalogue management. This abstract focuses on the evolutions being performed on the new version of the distributed software component named MOTU for data dissemination, present in all the oceanographic data production units spread through Europe. The MOTU open-source servlet provides REST (Representational State Transfer) services to the registered users willing to download data with or without subset through a queue system and with authentication relaying on a Central Authentication System (CAS). MOTU 2.0 presented strong limitations for the management of NetCDF4 data and intensive memory usage. In the scope of CMEMS, MOTU 3.0 will include several new features, being the compatibility with compressed NetCDF4 data the most attractive one for fast download and increased disk management efficiency. Other improvements such as the OGC protocol WCS 2.0 and better log management to gather download service statistics will be also conducted and included in this upgraded version here presented. For the design of the MOTU 3.0 subset service, TDS (Thredds Data Server) has been considered due to its overall performance and full compatibility with NetCDF4. However, TDS alone may present operational limitations, particularly in the handling of large volumes and number of concurrent users. Hence, MOTU 3.0 will be based on the new NetCDF subset service (NCSS) in conjunction with the latest NetCDF Java library in order to provide geographic, temporal and depth subset of the NetCDF files. Moreover, the use of the MOTU servlet as a proxy of the subset service will add robustness and performance through the queue management system and solve the around the world subset and the depth selection limitations of the NCSS implementation. In conclusion, the upgraded MOTU software envisaged under the Copernicus Marine Service in 2016 will allow users to benefit from the compression of the files and the new OGC standard WCS with a system that will increase in robustness and performance.