The efficient, convenient, and robust execution of data-driven workflows are key for productivity in scientific computing. However, managing IO workflows efficiently in data centers is challenging for users. In order to achieve best IO performance, a user must consider the characteristics of the various available storage systems and file systems and map the IO of their workflows manually to the storage systems.
Within ESiWACE, we aim to integrate capabilities into the software stack that increase the abstraction level of IO within workflows and enable the software stack to optimize data placement and handle the information lifecycle automatically. This talk firstly introduces our general vision and presents the challenges and potential the abstraction bears to the typical IO stack in the domain of climate and weather. We then show our design for the ESiWACE prototype that utilizes software components such as Cylc, SLURM, ESDM, and XIOS. Finally, we discuss how such an approach could be integrated into existing workflows.
