Depending on context and focus, the understanding of what sums up to the ESM workflow can vary tremendously. Thus as a startingpoint for the discussion on workflow solutions a common understanding of what is meant by the ESM workflow is needed.
In the following graph (developed at the "ISENES2 Workshop on Workflow Solutions in Earth System Modelling") the complete ESM end-to-end workflow is represented as a ring; starting with the "model workflow" (model configuration, data preprocessing, model run, monitoring) on the lefthand side and ending with the "data workflow" (post-processing, storing, archiving) on the righthand side.
Figure 1: Ring-representation of the ESM workflow (click for enlargement)
What is needed: Towards workflow solutions
The above workflow representation illustrates the manifoldness of different steps, which are – or could be – expressed via scripts. Such scripts are invoked repeatedly, and not necessarily linearily in the same order, as there is a tendency of increasing interaction among the tasks of the model- and the data-workflow. In particular, the preprocessing depends on post-processed data, whilst archiving utilizes provenance data, which needs to be processed and harvested during the model run.
The resulting complexity calls for a meta-scheduler being able to orchestrate a number of individual scripts or tools (e.g. post-processing tools, configuration management tools, software build tools, meta data capture tools, models, couplers). Moreover, experiment repeatability and result reproducibility will become evermore challenging for comprehensive Model Intercomparison Projects (like CMIP6) running on future exascale HPC systems. This calls for a change away from traditional manual shell methods towards professional tools.
What is there: Existing Tools
There is a tendency in the community to strive for common solutions. Cylc, e.g. is increasingly used and promoted as meta-scheduler, CDOs are widely used for pre- and postprocessing and Oasis, the coupler, is currently used by approximately 35 climate modelling groups around the world.
Yet, so far only small parts of the above workflow are covered by professional tools. Still individual and often even manual solutions revail. In order to develop, maintain and establish tools as community tools, communication and commitment of the community is required. ENES organizes workshops and community interaction to foster the commitment to common solutions. A collection of potenital community tools can be found here.