PyTorch RFC seeks peer-HBM activation offload for pipeline ranks
A proposal would let memory-heavy pipeline stages park saved activations on spare peer GPU memory without changing the compute schedule.
Sanket Purandare has asked the PyTorch project to consider cross-rank activation offloading and prefetching in its distributed pipelining layer, so memory-heavy pipeline stages can borrow spare GPU memory from other ranks instead of forcing a schedule change.
In interleaved schedules such as Interleaved1F1B, earlier ranks typically hold more in-flight activations than later ones, and rank 0 is often the activation-memory bottleneck. For uniform stage workloads, peak live-activation counts are mirrored: the sum for ranks i and P minus 1 minus i is constant, where P is the number of pipeline ranks. Purandare argues that this count symmetry does not by itself mean equal HBM use, so any planner would have to use actual activation bytes and each rank's full memory budget.
The requested feature would let selected saved-forward activations sit temporarily in a storage peer's HBM and return to the owning rank before backward needs them. Forward and backward compute, and parameter ownership, would stay on the original rank. The pitch uses PyTorch's one-sided distributed transport APIs and is framed as a different tradeoff from changing the pipeline schedule, activation checkpointing, or offloading to local CPU memory.
Purandare is explicit that moving activations does not shrink the job's total saved bytes and adds network traffic. "Neither a memory saving nor a throughput benefit should be assumed without accounting for staging and prefetch residency." The request seeks agreement on the feature and its ownership contract, not on an implementation plan.