Today’s scientific applications can run on hundreds of thousands of processors and produce massive amounts of data. To meet such requirements, high-performance computing (HPC) systems have grown in size and complexity, incorporating data analysis in addition to computational modeling. In this context, an important key to high performance is data management, which includes not only I/O but also analysis of data in situ in order to obtain scientific insight.
In this talk, we present and discuss several strategies toward this goal. First, we focus on the I/O interference problem which can be a major performance bottleneck for HPC applications. Then, we present an I/O management scheme that can enable efficient big data processing on HPC systems. Lastly, we present our efforts in extending in situ workflows with new capabilities such as different programming models (e.g., bag-of-tasks, looping) and dynamic features.
Speaker: Orcun Yildiz is a postdoctoral researcher in the Mathematics and Computer Science Division at Argonne. He received his Ph.D. in computer science from Ecole Normale Superieure de Rennes. His research interests include scientific workflows, I/O management, big data processing, and HPC.