ESiWACE is developing and maintaining software to help produce and manage data at scale, which will be
- Supporting data reduction in ensembles by providing tools to carry out ensemble statistics “in-flight” and compress ensemble members,
- Hiding the complexity of multiple-storage tiers (middleware between NetCDF and storage) with industrial prototype backends, and
- Delivering portable workflow support for manual migration of semantically important content between disk, tape, and object stores.
The ensemble data management tools are being tested and deployed using the Met Office Unified Model, the complexity of storage tiers is being addressed using the Earth System Data Middleware, and prototype workflow tools currently include S3NetCDF and the Joint Data Migration App, JDMA.
