We present here the results of experiments on DataDirect Networks (DDN) Infinite Memory Engine (IME), using the weather forecasting simulation code HARMONIE, performed by ICHEC staff. The point of the experiment was to run the complete weather forecasting tool chain so that all file reading and writing used the IME memory burst buffer, rather than a conventional parallel file system, in order to evaluate the performance gain this new environment could offer in the context of "real-world" operational forecasts.
These production runs are time-critical in the sense that they must complete within a narrow time-range (typically 60-90 minutes) between the collection of initial conditions and dissemination of the forecast output. Furthermore, a shorter turnaround time opens up possibilities for more computationally intensive and accurate methods which wouldn't have fit in the allowed time window otherwise. Therefore, every feature that reduces the run-time time of the full tool chain or improves the accuracy of the forecast contributes to a more valuable product. Individual file sizes generated by large contemporary forecast models, are typically in the 50-100GB range (uncompressed). Such a file (containing output of each forecast hour) may be generated every couple of minutes as the forecast runs, requiring a mean storage rate between 1-2 GB/s. However, such output is "bursty" rather than steady, and so any technique, such as the burst-buffer of IME, that enables the forecast calculations to proceed with minimal interruption for IO, helps overall performance.
The overall work-flow also includes a lot of short dependent pre- and post-processing stages, many of which are concerned mainly with file manipulation, and which produce and/or consume large numbers of temporary files. By speeding-up the read and write accesses to these many intermediate files, we expect a good speed-up of the entire work-flow. Measuring the exact impact of this IO performance improvement, as well as the complexity it entails from the standpoint of the forecasters in charge of running the code, is the main goal of the experiment.
We report here our findings by comparing the performance and ease of use of the well-known Lustre parallel file system and the new DDN IME burst buffer. We present the details involved in using IME on a standard cluster environment, and the benefits it brings. We also assess the specific benefit IME brings when running in degraded mode - with the IO sub-system experiencing partial (recoverable) failures - compared to the same situation on a conventional file system.