- By:
- Rogers, David M; Mariani, Valerio; Wang, Cong; Coffee, Ryan; Kroeger, Wilko; Shankar, Murali; Schwander, Hans; Beck, Thomas L; Poitevin, Frédéric; Thayer, Jana
- Page Number:
- 433-442
- Book Title:
- SCA/HPCAsiaWS '26: Proceedings of the Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region Workshops
- Publication Date:
- September 11, 2026
- Publisher Location:
- Association for Computing Machinery, New York, New York, United States of America
- Conference Name:
- SCA/HPCAsiaWS 2026: SCA/HPCAsia 2026 Workshops: Supercomputing Asia and International Conference on High Performance Computing in Asia Pacific Region Workshops
- Conference Location:
- Osaka, Japan
- Conference Sponsor:
- ACM and IEEE
- View DOI Listing:
- https://doi.org/10.1145/3784828.3786261
Abstract
We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications – AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments.
