- By:
- Liu, Zhengchun; Kettimuthu, R.; Foster, Ian; Rao, Nageswara S
- Page Number:
- 267-278
- Book Title:
- HPDC '18: Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing
- Publication Date:
- March 12, 2026
- Conference Name:
- 27th International Symposium on High-Performance Parallel and Distributed Computing
- Conference Location:
- Tempe, Arizona, United States of America
- Conference Sponsor:
- ACM
- View DOI Listing:
- https://doi.org/10.1145/3208040.3208053
Abstract
Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.
