Home:
-
Related: DataLife, DaYu, FlowForecaster
About (Technical): DataLife is a measurement and analysis toolset for distributed scientific workflows that use I/O and storage for task composition. DataLife performs data flow lifecycle (DFL) analysis to guide decisions regarding coordinating tasks and data flows on distributed resources. DataLife provides measurement, analysis, visualization, and opportunity identification for data flow lifecycles (DFLs). With the aid of the DaYu module, it analyzes semantic relationships between logical datasets and file addresses, how dataset operations translate into I/O, and combinations of the two across entire workflows.
DataLife analysis involves a profiling and post-mortem analysis phase. The profiling is distributed and scalable; and supports I/O through HDF5, POSIX, C I/O. The post-mortem analysis provides several helpful analyses and visualizations in order to extract workflow data patterns, develop insights into the behavior of data flows, and identify opportunities for both users and I/O libraries to optimize improving task placement and data placement and resource assignment.
About (General):
New materials have the potential for improving solar generation, creating new batteries, developing new health care treatments, and enabling new techniques in computing. The problem is that new materials with just the right properties are extremely hard to find. The key to accelerating this discovery is automation of the complex theory-experiment cycle that consists of guidance and explanation from theory and experimental measurement and validation of experimentalists. In other words, new computational techniques are needed for the workflows that coordination large computational models, hypothesis generation, instrument control, and experimental interpretation and feedback.
Distributed scientific workflows pass information -- often large volumes -- along chains of different computational tasks [in the experiment-instrument-theory cycle], causing data flow bottlenecks in storage and networks. We have developed DataLife, a measurement and analysis toolset for these workflows. DataLife performs data flow lifecycle (DFL) analysis to guide decisions regarding coordinating task and data flows on distributed resources. DataLife provides tools for measuring, analyzing, visualizing, and estimating the severity of flow bottlenecks. DataLife's measurement introduces techniques that deliver high precision while also imposing minimal overhead. The bottleneck estimator provides several analyses and visualizations to identify and rank opportunities for improving task and data placement and resource assignment.
Contacts: (firstname.lastname@pnnl.gov)
- Nathan R. Tallent (www), (www)
- Lenny Guo (www)
- Jesun Firoz (www)
- Meng Tang (Illinois Institute of Technology) (www)
- Hyungro Lee (www), (www)
Contributors:
- Meng Tang (Illinois Institute of Technology) (www)
- Hyungro Lee (PNNL) (www), (www)
- Lenny Guo (www)
- Jesun Firoz (www)
- Nathan R. Tallent (PNNL) (www), (www)
-
H. Lee, L. Guo, M. Tang, J. Firoz, N. Tallent, A. Kougkas, and X.-H. Sun, “Data flow lifecycles for optimizing workflow coordination,” in Proc. of the Intl. Conf. for High Performance Computing, Networking, Storage and Analysis (SuperComputing), SC ’23, (New York, NY, USA), Association for Computing Machinery, November 2023. (doi)
-
M. Tang, J. Cernuda, J. Ye, L. Guo, N. R. Tallent, A. Kougkas, and X.-H. Sun, “DaYu: Optimizing distributed scientific workflows by decoding dataflow semantics and dynamics,” in Proc. of the 2024 IEEE Conf. on Cluster Computing, pp. 357–369, IEEE, September 2024. (doi)
-
L. Guo, H. Lee, J. Firoz, M. Tang, and N. R. Tallent, “Improving I/O-aware workflow scheduling via data flow characterization and trade-off analysis,” in Seventh IEEE Intl. Workshop on Benchmarking, Performance Tuning and Optimization for Big Data Applications (Proc. of the IEEE Intl. Conf. on Big Data), IEEE Computer Society, December 2024. (doi)
-
H. Lee, J. Firoz, N. R. Tallent, L. Guo, and M. Halappanavar, “FlowForecaster: Automatically inferring detailed & interpretable workflow scaling models for better scheduling,” in Proc. of the 39th IEEE Intl. Parallel and Distributed Processing Symp., IEEE Computer Society, June 2025.
This work was supported by the U.S. Department of Energy's Office of Advanced Scientific Computing Research:
- Orchestration for Distributed & Data-Intensive Scientific Exploration