Dynamic Latency-Throughput Balancing in Distributed Large Model Inference with Interleaved Parallelism

Overview

Distributed large model inference requires balancing latency and throughput objectives. Tensor parallelism reduces latency through intensive inter-GPU communication, while pipeline parallelism increases throughput at the cost of per-request latency. Liger+ proposes interleaved parallelism to dynamically balance these competing metrics by interleaving computation and communication across multiple requests on multi-GPU architectures.

Methods and approach

Liger+ comprises two primary components: a task-aware batch management module and a distributed runtime module. The batch management layer organizes requests according to task characteristics, distinguishing between discriminative and generative workloads. The runtime module executes strategic scheduling of computation and communication kernels across multiple GPU streams through three mechanisms: precise kernel execution control via CPU-GPU and inter-stream synchronization, fine-grained resource mapping with contention factor anticipation, and kernel decomposition into smaller units to maximize overlap between computation and communication operations across requests.

Key Findings

On a 4-device discriminative task, Liger+ achieves 43.8% reduction in P90 latency while maintaining throughput parity with pipeline parallelism, and 1.53× throughput improvement with improved P90 latency relative to tensor parallelism. For generative tasks on 4 devices, the system demonstrates 1.15× average throughput improvement and 26.2% P90 latency reduction compared to tensor parallelism. The dynamic scheduling strategy enables simultaneous optimization across latency and throughput metrics in most evaluated scenarios.

Implications

Interleaved parallelism addresses a fundamental constraint in distributed inference systems where static parallelism strategy selection forces fixed trade-offs between latency and throughput. By enabling dynamic adaptation across requests, Liger+ expands the operational feasible region for inference deployments that must satisfy heterogeneous performance constraints. The fine-grained kernel scheduling and resource contention modeling represent transferable techniques for optimizing multi-GPU execution.