When Faster Kernels Do Not Mean Faster Applications in Exascale GPU Systems

Abstract

Power management methods such as power capping or dynamic frequency capping have long been considered primary mechanisms for addressing the power constraints of exascale computing and beyond. However, their effectiveness in GPU-accelerated HPC applications is not straightforward, as they are inherently multi-kernel and exhibit heterogeneous computational and memory behavior across execution phases. As a result, improvements observed at the kernel level may not translate into better application-level performance. In this work, we investigate how device-level power capping and frequency capping affect the performance–energy behavior of GPU applications on the Frontier exascale supercomputer. We evaluate multiple power and frequency configurations across three representative applications (Cholla, Pennant, and HACC) to identify those that deliver the best performance–energy trade-offs. Using these configurations, we perform an intra-kernel analysis combining GPU hardware counters with runtime profiling to examine how power management policies influence kernel execution and distributed runtime behavior. Our results show that faster kernels do not necessarily lead to faster applications. For example, in Pennant running across 32 nodes, the dominant kernel executes up to 20% faster under power capping, yet these gains are partially offset by higher communication costs and load imbalance across MPI calls. Similarly, in Cholla, synchronization imbalance increases from 30.6% to 39.8% and communication imbalance from 5.8% to 16.6%, leading to longer MPI wait times despite faster kernel execution.