FluidPD: Real-time resource reassignment to balance prefill and decode in LLM serving

The imbalance between prefill and decode

The architecture for serving large language models (LLMs) has evolved toward separating the prefill and decode phases. This dissociation, known as prefill-decode disaggregation, allows each phase to be optimized independently, as they exhibit very distinct execution patterns and service level objective (SLO) targets.

However, real-world workloads are not static. Short bursts and sustained shifts in the demand ratio between prefill and decode can quickly unbalance a configuration that was initially well-provisioned. This mismatch causes latency SLO violations, even when idle capacity exists elsewhere in the system.

Separating prefill from decode gives each phase room to be tuned for its own execution pattern and service objectives, but it also means that the two sides can face different levels of pressure at the same time. A configuration that suits one demand ratio may no longer match the workload when that ratio changes, so aggregate idle capacity does not necessarily prevent a latency problem: it may be available in the phase that is not currently under pressure.

The distinction between short-lived bursts and sustained shifts matters because they call for different responses. A brief mismatch may ease before a conventional scaling action takes effect, while a longer change can leave the original allocation unsuitable for an extended period. In either case, the central issue is not simply the total number of workers, but whether capacity is available in the phase that needs it when it needs it.

FluidPD's in-place elasticity mechanisms

FluidPD is a P/D-disaggregated serving system that introduces two complementary mechanisms to manage this imbalance. The first, FluidToken, addresses transient imbalance by offloading a bounded portion of prefill computation to decode workers when they have available slack capacity.

The second mechanism, FluidRole, manages sustained imbalance by reassigning running workers between prefill and decode roles. This reassignment is performed in-place, avoiding model reload and inference engine restart, which allows for much faster adaptation to load changes.

FluidToken and FluidRole address different durations of imbalance rather than serving as interchangeable scaling actions. FluidToken uses available slack on decode workers to take on a bounded share of prefill computation; its stated purpose is to respond to a transient mismatch without adding reserve GPUs. The bound on the offloaded portion is relevant: the mechanism is described as shifting only part of the work, not as moving all prefill computation to decode workers.

FluidRole addresses a different situation: a sustained change in phase demand can justify changing the roles of workers already running. Because the role change occurs in place, the stated design avoids reloading the model and restarting the inference engine. That is the basis for the claim of faster adaptation; it does not, by itself, establish a particular reassignment time or guarantee that every change in workload can be handled without delay.

The comparison with traditional autoscaling therefore turns on how capacity is made available. FluidPD's mechanisms use offloading or role reassignment, whereas the table describes traditional autoscaling as adding or removing GPUs and requiring reserve GPUs. These are the mechanisms as presented in the comparison; the material does not establish that all autoscaling systems behave identically or that FluidPD can meet every workload demand without additional capacity.

Metrics and evaluation conditions

The authors of the preprint report that FluidPD improves overall SLO attainment by up to 94.6 percentage points compared to static SGLang. This improvement is evaluated using production Azure trace workloads, suggesting practical relevance for large-scale environments.

It is important to note that these metrics come from the preprint's abstract and have not been independently verified. The evaluation focuses on the system's ability to maintain SLOs without provisioning additional workers, a key goal for operational efficiency.

The reported figure is a difference in SLO attainment measured in percentage points, not a percentage increase in performance. It describes the largest improvement reported against the static SGLang configuration in the evaluation, rather than a result that should be assumed for every workload or deployment. The available description ties the evaluation to production Azure traces, but does not provide independent confirmation of the headline value.

The result should be read with the stated evidence limits in mind. The figure comes from the preprint abstract and has not been independently verified, and the work has not undergone independent peer review. The reported comparison can therefore indicate what the authors observed in their evaluation, but it is not, on its own, a guarantee of the same SLO attainment under different conditions.

The evaluation's stated emphasis on maintaining SLOs without provisioning additional workers connects the result to operational efficiency, but it does not settle questions about how the system behaves outside the tested conditions. In particular, the supplied information does not confirm the detailed serving architecture or platform configuration used, so the figure should not be generalized beyond the evaluation as described.

Operational costs and dependencies

FluidPD's strategy relies on the ability of decode workers to absorb part of the prefill load (FluidToken) and the possibility of reassigning roles without restarting the engine (FluidRole). This implies that the inference system architecture must natively support these operations.

Although the system avoids the need for reserve GPUs, the speed of reassignment and the efficiency of computation offloading are critical factors. If reassignment is not fast enough, the imbalance could persist and cause SLO violations before the system stabilizes.

Both mechanisms depend on the serving architecture being able to carry out their respective actions: transferring a bounded share of prefill computation to decode workers, and changing a running worker's role without restarting the engine. The material presents this support as a dependency, not as a capability that can be assumed in every inference system. Compatibility with an existing setup therefore remains something to establish rather than a confirmed property.

The absence of a need for reserve GPUs in the comparison does not remove the operational importance of timing. FluidRole must complete reassignment quickly enough to respond to sustained imbalance, while FluidToken must make effective use of decode capacity that is actually available. The description gives no quantified limits for either mechanism, so it does not establish how much imbalance they can absorb or how quickly they can stabilize a workload.

The simplified reassignment flow identifies the stages involved: detect sustained imbalance through pressure indices, find candidate workers, change their roles in place, and rebalance work among the reassigned workers. This outlines the intended sequence, but it does not specify thresholds, timing, or other implementation details. Those aspects would need to be understood before treating the flow as evidence of a particular operational response time.

External testing and production deployment

Before recommending the deployment of FluidPD in production, it would be necessary to evaluate its performance with real and diverse workloads, not just Azure traces. This would include testing with different language models, batch sizes, and traffic patterns.

Additionally, the availability of code and configurations to reproduce the results, as well as the system's elasticity limits, must be verified. The lack of independent peer review and the preprint nature of the work require additional caution when interpreting its performance claims.

Testing beyond the Azure traces described for the evaluation would help establish whether the reported behavior carries over to other real workloads. The proposed checks vary language models, batch sizes, and traffic patterns; these are evaluation dimensions to examine, not evidence that FluidPD has already been tested across them. Without such testing, the reported outcome should remain tied to the conditions that the preprint describes.

Reproducibility is a separate consideration from reported performance. The availability of code and configurations has not been confirmed in the supplied information, nor have the system's elasticity limits. Until those details are verified, it is not possible to establish from the stated results alone how another team could reproduce the evaluation or what range of imbalance the system can handle.

Taken together, these caveats support treating the result as promising but provisional rather than as a deployment guarantee. Compatibility with existing infrastructure, reproducibility, and the system's operating limits remain open checks, while the preprint's status and the lack of independent verification warrant caution. The material does not provide enough evidence to resolve those questions, so they should remain explicit considerations before production adoption.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.