freenode
AI & ML

PyTorch proposal would cache shared pipeline activations per rank

A rank-local cache would serve reused residuals and return one summed gradient, replacing custom stage subclasses.

PyTorch developers have proposed teaching torch.distributed.pipelining to own activations that several pipeline stages reuse, including stages that share a physical rank. Models such as Kimi K3 keep block residuals as immutable values that later stages read again. Today that contract lives in application code: a custom pipeline stage caches copies and deposits gradients through private bookkeeping.

The request is for the runtime to send each shared value once per consuming rank and microbatch, then serve every local consumer from a rank-owned cache. Each stage would still see its own autograd leaf over that read-only storage. The rank would sum those leaves' gradients and return a single complete gradient to the producer, which runs backward once. Storage would stay live until every reader and every communication has finished, so a differentiated use contributes exactly once.

Ordinary single-consumer values would use the same lookup and lifetime rules. The proposal extends earlier work on skip connections and stage endpoints, which could select send and receive points but did not share one activation among several stages on a rank and return their gradient sum. Stage outputs would be identified by logical producer and output slot, with physical ranks bound later from the schedule's existing stage-to-rank map.