First Token Latency is reduced by 32.4%! Shishi Technology has launched a game-changing cluster scheduling solution, further boosting the efficiency of domestic AI inference.

In a GPU cluster, request queuing and resource idling can occur simultaneously. During the 2026 CCF National High Performance Computing Academic Conference, METASTONE released its new product TokenFleet distributed scheduling framework, which focuses optimization on request routing, cross-data-center capacity collaboration and mixed load governance, aiming to improve the effective output of existing computing power.

Regard the inference service as a domestic Token optimization factory, GPUs as production equipment, inference instances as workstations, and business requests as individual orders. TokenFleet is responsible for matching orders with production capacity. For the same batch of equipment, different tasks and different allocation methods will lead to different final waiting time and compliant output. Large-scale services require all these production arrangements to be considered together.

TokenFleet Focuses on the Effective Output of Clusters

After the model deployment is completed, the cluster still needs to handle multiple types of businesses at the same time. Multiple models share computing power, and different tenants have their own service level requirements. Short conversations need timely response, long document processing requires more computing and video memory, and offline batch tasks have their own deadlines. When services are distributed in multiple data centers, it is also necessary to consider whether the capacity of each location can be coordinated.

Simple offloading may cause long queues for some instances, while the capacity at other locations is not utilized in time. Mutual preemption between long requests and batch tasks will slow down real-time interaction; timeout, cancellation and repeated calculation will consume resources without generating qualified deliverables.

Under the conditions of consistent model quality, service requirements and computing power budget, TokenFleet improves the effective Token output and reduces the cost per unit of effective output. TokenPilot selects instances in the same model pool, FleetMind is oriented to cross-data-center capacity collaboration, and SLOMax is responsible for mixed load admission, queue management and capacity allocation. The three core technologies have their respective divisions of labor.

Who the request is assigned to, when the task enters the queue, and how the data center capacity is coordinated will eventually return to the same indicator: the effective Token output that is successfully completed and meets the service requirements.

Figure 1 Overview of the three core scheduling technologies of TokenFleet

TokenPilot Calculates Both Cache Benefits and Queuing Costs

First look at request point selection. For one request, which instance should it be sent to? A short question and answer and a long document, even if counted as one request, may consume vastly different amounts of computing and video memory. It is difficult to balance the actual workload of each instance only by shunting according to the number of requests.

Caching adds an extra trade-off to this matter. If requests with the same long prefix are sent to instances that retain the relevant KV cache, they can reuse previous calculation results and reduce Prefill, that is, repeated calculation when processing input content.

However, instances with high cache hit rates may also have long queues. If all requests flock to the same place, the saved calculation time may be consumed in waiting.

When selecting points, TokenPilot estimates both the cache reuse benefit and the queuing cost. It estimates the workload by combining the input scale, output budget and reusable prefix, then filters and scores candidate instances by integrating the queue, in-transit Token, KV cache pressure and predicted latency. The output budget is a constraint condition and does not represent the actual output length.

The scheduler compares the estimated processing cost of different instances accordingly, and judges whether the calculation saved by reusing the cache can offset the extra waiting.

Figure 2 TokenPilot Intelligent Inference Routing Framework

The GLM5.3 tuning data provided by METASTONE shows that under the tested graphics card cluster and load conditions, after adopting the predictive latency routing, the P95 first Token latency is reduced by 32.4%, and the P50 first Token latency is reduced by 71.3%.

Table 1 TokenPilot First Token Latency Tuning Results (Data source: METASTONE conference presentation materials. The improvement rate is calculated as (baseline value - optimized value) ÷ baseline value; latency reduction does not directly equal cost reduction.)

P50 reflects the median level, and P95 reflects the waiting situation of tail requests. This set of data shows the response improvement under the tested conditions; the cost benefit still needs to be evaluated in combination with throughput and resource occupancy.

FleetMind Clarifies Service Capacity Even Across Data Centers

When the perspective is expanded from one model pool to multiple data centers, scheduling adds an extra layer of capacity judgment. Having idle GPUs does not necessarily mean that orders can be accepted immediately. Whether the target model is deployed, whether the version matches, whether the service endpoint is ready, and how much time the network transmission takes will all affect the shunting effect.

FleetMind focuses on the actual available model service capacity. In the design of TokenFleet, it aggregates the model availability, real-time capacity and network status of each data center, selects the execution data center first, and then the local decision engine of that data center selects the specific inference instance.

When the hotspot data center starts queuing, the system will evaluate whether other data centers have ready endpoints of the same version of the model and whether there is remaining service capacity, then judge whether it can accept new requests that allow cross-regional execution.

Whether shunting is allowed must first meet the permission and data geographic constraints. Whether shunting is worthwhile requires a comprehensive comparison of back-to-source latency, queuing, calculation, cache and transmission costs.

Capacity collaboration also has different time scales: adjust the destination of new requests in a short cycle, adjust model replicas and preheating positions in a long cycle, and combine capacity reservation, load upper limit perception and automatic fallback to reduce capacity mismatch.

Figure 3 FleetMind Cross-Data-Center Execution Framework (Note: Region A is the business entrance and is not included in the GPU capacity pool; Region B is the existing cluster, and Regions C and D are extended data centers.)

TokenPilot and SLOMax are oriented to the existing cluster, while FleetMind is oriented to the expansion of new computing power data centers in the future. The goal of cross-data-center collaboration is to let dispersed capacity that meets business constraints undertake hotspot demands, reducing capacity mismatch and repeated reservation.

SLOMax Arranges Production According to Service Requirements

When requests enter the same cluster, their delivery requirements are different. Real-time conversations focus on the first Token response and continuous generation speed, long-context tasks rely more on input processing capability and KV capacity, while offline batch tasks pay attention to deadlines and throughput. Putting them into the same queue will easily cause resource contention.

Long Prefill, KV capacity pressure and batch load may all squeeze interactive requests. Timeout, cancellation and repeated execution will also make part of the calculation unable to be converted into effective deliverables.

SLOMax identifies the workload by combining input length, output budget, task type and tenant service objectives, then performs grade-based admission, queuing in separate pools and capacity coordination. It needs to guarantee interactive capacity and control the concurrency and mutual interference of long and short requests.

Figure 4 SLOMax Classification Queue and Capacity Feedback Framework (Note: Effect feedback is used to adjust capacity and admission, and the framework does not preset the preemption capability of running tasks.)

To measure the effect of this set of scheduling, TokenFleet takes compliant output as the optimization goal. Goodput, namely compliant output throughput, is calculated as: the total amount of output Tokens generated by requests that are successfully completed and meet the agreed service requirements during the complete test period, divided by the test duration. Different task categories need to define the compliance threshold separately and report the results.

The system feeds back the compliant throughput, failure, cancellation and repeated execution status to the scheduling strategy, and continuously adjusts the admission and capacity ratio. How busy the GPU is and how high the total throughput is, all need to be considered together with the final qualified deliverables.

In the GLM5.3 benefit calculation example provided by METASTONE, after adopting the flow control collaborative scheduling, the total output throughput increases by about 1.5%, the compliant output throughput increases by about 29.4%, and the compliance rate rises from about 67% to about 86%.

Table 2 SLOMax Effective Token Throughput Benefit Calculation Example

This set of values belongs to benefit calculation, showing the output structure that scheduling aims to improve. The actual benefit still needs to be verified in combination with specific load and service thresholds. For the same scale of total output, if timeout, cancellation and repeated calculation can be reduced, there is an opportunity to deliver more compliant Tokens. SLOMax incorporates these results into feedback to adjust subsequent resource allocation.

The Efficiency of the Domestic Token Optimization Factory Also Depends on Scheduling

TokenFleet pushes the optimization of inference services to the cluster level. How instance performance is allocated to different requests, and how the capacity of multiple data centers participates in services, all require scheduling to connect resources with business requirements.

METASTONE is an independent third-party full-stack computing power operator, not bound to any specific large model manufacturer, providing customers with one-stop full-stack computing power delivery services covering hardware, networking, scheduling, optimization and operation.

With rich experience derived from the national-level computing center, the SLA of its computing power cluster operation reaches over 99.95%. At present, METASTONE has managed more than 20,000P+ computing power resources, and operates more than 10 super-intelligent computing centers, including 2 national supercomputing centers; team members have won 3 Gordon Bell Prizes, and accumulated full-stack technical service experience for super-large-scale clusters.

Around the domestic Token optimization factory, METASTONE continuously polishes its software capabilities through self-research and self-use, hoping to make general and high-frequency inference services more economical, and give full play to the capabilities of domestic computing power.

The previously released Meta-Infer high-efficiency inference engine focuses on improving the execution efficiency of a single inference instance. This time, TokenFleet focuses on cluster scheduling, and organizes large-scale services through request allocation, capacity collaboration and mixed load governance.

To make more businesses afford and make good use of Token, not only hardware is needed to provide computing power, but also software to organize production capacity. TokenFleet takes the direction of further optimization on how each request is allocated and how each capacity is used.