At the level of 10 million tokens, it is 22 times faster than DeepSeek-style sparse attention.
Nowadays, the million-token context has evolved from a selling point to a standard configuration. In April, DeepSeek-V4 set the 1M context as the default specification for all its official services; in August, Zhipu AI released GLM-5.3-Flash, which also supports 1M context, and for the first time introduced a hybrid architecture of sparse attention and linear attention in the GLM series. There is a common choice behind these models: let each query no longer go through all historical tokens, but only select a small number of the most relevant ones to "look at".
This technical route has been repeatedly verified at the 1M-token scale. However, when the context continues to expand to 10 million tokens, a previously negligible overhead starts to get out of control. Taking a global attention layer of GLM-5.3-Flash as an example, when performing prefill under 10M tokens, 98% of the time is not spent on attention calculation, but on the indexer that determines "which tokens to look at".
The nature of the problem has therefore changed. Sparse attention has long proven that "looking at only a small part" is sufficient to maintain accuracy, and what really bottlenecks the ultra-long context is that "finding that small part" still grows quadratically with the length of the context.
Now, London-based startup Prefrontal AI and ETH Zurich have jointly published a paper, proposing a new method called Latent Sparse Attention (LSA).
It accelerates the prefill of a single attention layer of GLM-5.3-Flash by 22.7 times under 10M tokens, and accelerates Qwen3.8-27B by 72.4 times, while maintaining accuracy close to that of the original model on a series of tasks.
- Paper Title: Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
- Paper Link: https://arxiv.org/abs/2610.42147
- Code Link: https://github.com/Prefrontal-AI/latent-sparse-attention
The "Hidden Bill" of Sparse Attention
Let's first review the cost structure of attention. In standard full attention, each query needs to calculate the correlation with all previous keys. When the context length is T, the computation volume of prefill is O(T²), and the computation volume for decoding each token is O(T). When the context grows from 1 million to 10 million tokens, the attention computation volume of prefill will increase by 100 times.
The idea of sparse attention is division of labor: first, a lightweight "indexer" selects k most relevant keys for each query, then the real attention is only calculated on these k keys. In this way, the overhead of attention itself is reduced to O(Tk).
There is a premise for this idea to hold: the model really only needs very few keys. The appendix of the paper specifically conducts verification: under 128k tokens, when each query of Qwen3.8-27B is only allowed to look at the 512 keys that are "truly" most relevant to it (accounting for about 0.8% of all keys), the model can still retain 98% of the accuracy of full attention on the RULER long context benchmark.
However, there is a paradox hidden here. For the indexer to know which 512 keys are the most relevant, it has to score every key first. DeepSeek's DSA, MiniMax's sparse attention, and the block sparse schemes adopted by models such as GLM-5.3-Flash and Qwen all use methods such as reducing precision, reducing dimensions, and compressing 4 keys into 1 to "reduce the burden" on the indexer. But these improvements only divide the constant term by several times, the workload of the indexer still grows linearly with the context, and prefill is still quadratic.
For example: you want to borrow 4096 books from a library with tens of millions of books. The act of borrowing itself is very fast. The trouble is that in order to determine which 4096 books to borrow, you have to look at the spine of every book. The more books there are, the more time-consuming it is to check the book spines. The paper calculates that under 10M tokens, if the overhead of the indexer can be reduced to zero, the prefill of sparse attention can be 58 times faster.
Time composition of traditional sparse attention under ultra-long context (single global attention layer of GLM-5.3-Flash, single H200 GPU). At 10M tokens, the indexer accounts for 98% of the total prefill time.
Arrange by Semantics, Not by Chronology
The solution of LSA is to let the indexer no longer check keys one by one, but check "groups of keys".
Overview of D-LSA and N-LSA. Top: In one attention layer, keys are first pooled and clustered into buckets, the query only scores the bucket centers, and then only looks at the selected remote keys (plus sink tokens and sliding window); Bottom: Accuracy and acceleration of the two variants.
The key lies in how to group. Most existing block sparse methods group by position, and combine several adjacent keys into one block. This is equivalent to the library packing books by warehousing time, with a summary attached to each box. The problem is that as long as there is one key book in a box, its summary may be "submerged" by the summaries of several other irrelevant books.
LSA chooses to group by content. It first projects keys into a learned low-dimensional latent space, then uses hierarchical clustering (Ward's method) in this space to assign semantically similar keys to the same "bucket", and each bucket only retains one center vector and an index table recording the positions of the keys in the bucket. When querying, the query only needs to score the center vectors of the buckets, select the most relevant buckets, then "expand" the buckets into specific keys, gather k of them and send them to the attention module.
Back to the library metaphor: books are no longer packed by warehousing date, but shelved by theme. To find books about quantum error correction, you can go directly to that row of bookshelves. Books of the same theme may have been warehousing over ten years, but they are placed next to each other.
The benefit of this step is at the constant level, which the paper calls Flat search. What really changes the complexity is the Tree variant: the buckets are further aggregated into larger "node buckets", forming a hierarchical tree layer by layer. The query starts from the top of the tree, and only keeps the top-scoring branches to go down at each layer (beam search). It is like selecting the floor first, then the area, then the bookshelf. Each query only needs to check O(log T) center vectors, the prefill of the indexer is reduced from quadratic to O(T log T), which is the origin of the term "subquadratic" in the paper title.
D-LSA design. Left: Flat search: keys are projected, pooled, clustered and stored in the bucket cache, the query scores the bucket centers and expands to top-k indices; Right: Tree search: buckets are organized into a multi-layer hierarchical structure, the query uses beam search to go down layer by layer.
Two Usage Modes: Graft to Existing Models, or Grow from Scratch
LSA provides two implementation methods.
The first one is D-LSA (Distilled-LSA), for already trained models. It freezes all parameters of the original model, only adds two small projection matrices in each layer to project keys and queries to the 64-dimensional index space, and then makes the scoring of LSA approximate the attention distribution of the original model through distillation. The entire distillation only uses 352 million tokens. For reference, DeepSeek DSA uses 2.1 billion tokens only in the dense warm-up phase of the indexer. The paper also provides a completely training-free version, which directly uses the principal component direction of the key as the projection, and its accuracy on Qwen3.8-27B is close to that of the trained version.
The second one is N-LSA (Native-LSA), which natively uses LSA during pre-training, and is only trained by the standard next-token prediction loss. It directly clusters the keys of its own attention, does not require additional index projection, teacher model, or auxiliary loss, and the number of model parameters does not increase at all.
N-LSA design. The attention layer directly clusters its own keys, and can be trained from scratch only by the next-token prediction loss.
At the engineering level, the team wrote GPU kernels for clustering, scoring, selection and expansion for H200 and B300 (there is also a Triton version that can run on any GPU), and integrated LSA into SGLang, supporting tensor parallelism, expert parallelism, pipeline parallelism, and context parallelism in prefill and decode phases.
Performance Report Under 10 Million Tokens
First look at accuracy. The team tested D-LSA on two models, Qwen3.8-27B (its attention layer is full attention) and GLM-5.3-Flash (with built-in sparse attention), and pushed the context to 10M tokens, far exceeding the official 1M upper limit of both. On knowledge, reasoning and long context tasks such as MMLU-Pro, GPQA-Diamond, AIME25, MATH500, and LongBench v2, D-LSA falls within the 95% confidence interval of the baseline, and does not significantly change the inference length of the model.
On RULER with 4k to 10M tokens, the gap between Qwen3.8-27B with D-LSA and full attention is no more than 5 points within 1M, and it even surpasses full attention with 63.9 vs 59.9 at 10M. The gap between GLM-5.3-Flash with D-LSA and native sparse attention is within 3 points within 1M, up to 8 points behind in the 2M to 5M range, and the gap returns to within 4 points at 10M. It should be noted that the baseline sparse attention of GLM is trained together with the model, while D-LSA is only distilled on the frozen model, so this comparison itself is favorable to the baseline.
In order to test the ultra-long context capability in more real scenarios, the team also built a new benchmark PaperTrail.
It embeds fictional "research clues" into a large number of real 2026 arXiv papers: a fictional paper proposes a certain result, paragraphs citing it are inserted into several real papers, a few weeks later another fictional paper revises the value and renames the method, while some subsequent papers still cite the old value, just like real academic literature. The model needs to find and connect these facts like doing a literature review. The 10M-scale sample contains nearly 400 to 465 papers. On this benchmark, Qwen3.8-27B with D-LSA scores 29.1 under 10M tokens, while full attention scores 19.8.
The accuracy of D-LSA on knowledge, reasoning and long context tasks (up to 10M tokens) is comparable to that of full attention and sparse attention baselines.
Then look at speed, which is also the most prominent part of this work. Under 10M tokens, for the prefill of a single attention layer, D-LSA is 22.7 times faster than the native sparse attention of GLM-5.3-Flash, and 72.4 times faster than the full attention of Qwen3.8-27B; in the decode phase, it is 4.6 times and 42.5 times faster respectively. This speedup ratio expands as the context grows. At 1M tokens, the prefill speedup for GLM-5.3-Flash is 3.5 times, and only 2.1 times at 512k, because block-by-block scanning is not expensive when the context is short. Extrapolating according to the trend, the Tree variant is expected to be more than 90 times faster than the baseline sparse attention at 100 million tokens.
The speedup of a single layer will be diluted when transmitted to the entire model, because there are a large number of other unaffected layers in the model. In the actual deployment of SGLang, under 10M tokens, D-LSA reduces the time to first token (TTFT) of GLM-5.3-Flash by 2.2 times (context parallelism), from about 7.5 minutes to about 3.4 minutes; it reduces the TTFT of Qwen3.8-27B by 9.2 times, from about 76 minutes to about 8 minutes.
The memory footprint of the index itself is also smaller. Under 10M tokens, the index of D-LSA per layer occupies 246 MiB, the native scheme of GLM-5.3-Flash is 331 MiB, and the Qwen-style block sparse scheme is 642 MiB.