- By:
- Banerjee, Tania; Lee, Jaemoon; Choi, Jong Youl ; Gong, Qian ; Chen, Jieyang; Klasky, Scott A; Rangarajan, Anand; Ranka, Sanjay
- Page Number:
- 143-152
- Book Title:
- 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
- Publication Date:
- March 12, 2026
- Publisher Location:
- IEEE, New Jersey, United States of America
- Conference Name:
- 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
- Conference Location:
- Goa, India
- Conference Sponsor:
- IEEE Computer Society
- View DOI Listing:
- https://doi.org/10.1109/HiPC58850.2023.00030
Abstract
Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.
