Machine learning training has emerged as one of the most prominent workloads in modern data centers. These training jobs are large-scale, long-lasting, and tightly coupled, and are often disrupted by various events in the cluster such as failures, maintenance, and job scheduling. To handle these events, we rely on cold migration, where we first checkpoint the entire cluster, replace the related machines, and then restart the training. This approach leads to disruptions to the training jobs, resulting in significant downtime.

In this paper, we present TrainMover, a live migration system that enables machine replacement during machine learning training. TrainMover minimizes downtime by leveraging member replacement of collective communication groups and sandbox lazy initialization.

Our evaluation demonstrates that TrainMover achieves 16X less downtime compared to all baselines, effectively handling data center events like straggler rebalancing, maintenance, and unexpected failures.

Chon Lam Lao (刘俊林) is a fourth-year Ph.D. candidate at Harvard University, advised by Professor Minlan Yu. Before Harvard, he obtained his master’s degree at IIIS, Tsinghua University, advised by Professor Wenfei Wu. His previous research received the Best Paper Award at USENIX NSDI 2021 and the Distinguished Paper Award at ASPLOS 2023. His work focuses on designing efficient machine learning training and inference systems with enhanced networking support and reliability.