New Meta Research: ByteDance's model breaks the ceiling after distillation

Tokens have replaced bytes, but that does not mean all large models have to be token-based.

Byte-level models, especially after distillation, have brought up a set of new, interesting perspectives.

Recently, a paper from Meta FAIR and the University of Washington titled *Breaking the Token Ceiling: Distill a Smaller, Stronger Byte-Level Model* puts forward a very intriguing idea —

During the distillation process, can we replace the learning unit from Token to byte, so that the student model can learn directly from the byte-level probability distribution?

It turns out that this approach works perfectly.

The team adds an end marker to each Token, to fully convert the teacher's Token probability distribution to the byte level. In the distillation experiment using Llama 3-8B as the teacher model, the team predicted based on the scaling law that:

As the amount of training computation increases, byte-level distillation will outperform token-level distillation, with the upper limit of average downstream accuracy 4 percentage points higher.

Traditional distillation allows the student to learn the teacher's probability distribution over the huge Token vocabulary. In contrast, bytes only have 256 basic values, the prediction space for a single position is smaller, and the basic values do not change with the tokenizer.

In other words, the basic unit of learning is smaller, but the upper limit of capabilities the model can finally learn is higher.

How is this achieved?

Byte-Level Distillation

In fact, distillation is no longer an unfamiliar technology today. We all know that larger models have stronger capabilities, but when deploying them to practical applications, memory, computation and response speed all bring non-negligible costs.

Therefore, a common practice is to let a large model act as a teacher and distill its capabilities into a smaller student model.

Different from ordinary supervised training which only tells the model "what the next correct Token is", distillation also allows the student to learn the teacher's probability judgment for each candidate Token.

But the problem also arises here: the candidate space for Tokens is extremely large. Take Llama 3-8B as an example, its vocabulary contains 128256 Tokens, which means that each prediction position corresponds to hundreds of thousands of candidate probabilities.

If you perform offline distillation and save the entire probability distribution, the storage cost will be very high. Therefore, in actual operation, usually only the part with the highest probability is retained, that is, top-k truncation.

The byte-level model just reduces this candidate space significantly.

A byte consists of 8 bits, and there are 256 possible values in total. Even if a small number of special symbols are added, only more than 200 probabilities need to be saved for each position, and the complete distribution is naturally easier to retain.

However, the teacher predicts the entire Token, while the student predicts a single byte. To align the two, you cannot just split the text, you also need to convert the teacher's probability distribution accordingly.

This is exactly the first problem the paper solves.

Specifically, the team proposes two schemes, named Marginalize-It and End-Of-Token respectively.

The basic idea of the two methods is roughly the same: gradually decompose the teacher's probability for different Tokens into the corresponding byte positions.

Take the word Tiramisu in the paper as an example, it may be split by the tokenizer into several Tokens: T, iram, isu.

Assume that when the teacher predicts the next Token, the candidates include isu, isk and is at the same time.

All three candidates start with the byte i, so when predicting the first byte, you can sum the probabilities of all candidates starting with i.

When the real prefix becomes i, only the candidates that match the prefix are retained, and the probability of the next byte is calculated continuously.

By repeating this process along the real byte sequence, the original Token distribution can be gradually decomposed into byte-by-byte training targets.

The first method, Marginalize-It, directly aggregates and reassigns probabilities.

However, problems arise when different Tokens have different lengths.

For example, when the prefix is already is, both isu and isk have subsequent bytes, but the Token is has already ended.

At this time, if you want to know what comes after is, you need to continue to query the teacher's prediction after the end of this Token; different candidate paths may correspond to different subsequent distributions, and the amount of computation will increase rapidly.

The processing of Marginalize-It is very straightforward:

Candidates that have already ended will not be traced further, and the probabilities of the remaining candidates are renormalized.

This method only requires one forward pass of the teacher model, which is very efficient, but the cost is that part of the probability information of the already ended Tokens will be lost, so it is essentially an approximation.

The second method, End-Of-Token, reserves a position for "Token end".

End-Of-Token adopts a different idea: since Tokens will end, simply give a clear prediction position for the "end" itself.

Specifically, the team adds a special symbol at the end of each Token. So the original three candidates become isu, isk and is with their own end marker.

In this way, when the prefix is already is, the student can next predict u, k, or the special end marker, which indicates that the current Token ends here.

The part of the probability that was originally "suspended" due to different lengths has a clear destination.

Therefore, End-Of-Token can map the teacher's distribution to the byte space more completely while retaining the Token boundary.

Moreover, both methods only require the teacher to perform one forward calculation to complete the conversion from the Token distribution to the byte distribution, and there is no need to repeatedly call the teacher model for different tokenization paths.

At this point, the knowledge of the teacher model can finally be truly passed to the byte-level student model.

The next step is to see how much it can actually learn.

Experimental Verification

The team tested three representations: Token, ordinary byte, and byte with end marker respectively, each of which is divided into supervised training and distillation training, for a total of six groups.

This setup can not only compare whether distillation is helpful, but also observe how the training performance of the model changes after changing the learning unit.

The teacher model uniformly uses Llama 3-8B. The Transformer layers of the three types of student models have the same parameter count, which is about 1.28 billion.

After including the vocabulary-related parameters, the total number of parameters is different: the Token student model has about 1.81 billion parameters, and the byte student model has about 1.28 billion parameters.

The research gradually increases the training data and computation volume, the training of the byte model covers a scale of about trillions of bytes, and then completes eight evaluations in three types of tasks: multiple-choice question answering, language generation and machine translation.

First look at the training process.

The Token model learns faster when the computation volume is small, but soon begins to approach the plateau phase; the byte model starts slower, but continues to improve as the computation volume increases.

In other words, the Token model wins in the early stage efficiency, while the byte model shows better scaling potential.

However, if you only look at BPB, that is, the prediction loss per byte, several byte models have already surpassed the Token model very early, but when they are actually used to complete tasks, their performance is still lower.

The reason for this is that BPB measures how accurately the model predicts text, while downstream tasks focus on whether the final answer can be selected correctly, and the two are not the same thing.

Therefore, the team did not directly use BPB to judge which one is stronger, but further fitted the relationship: training computation → BPB → downstream task performance, and used this relationship to predict the upper limit of capabilities of different schemes after further scaling.

The results show that the predicted upper limits of average accuracy of the three distillation schemes are respectively:

48.4% for Token distillation, 50.5% for Marginalize-It distillation, and 52.4% for End-Of-Token distillation.

In other words, End-Of-Token is finally 4 percentage points higher than traditional Token distillation, and has the highest predicted upper limit among the six schemes.

However, the advantage of distillation does not hold in all predictions.

The asymptotic accuracy prediction of the ordinary byte supervised model is 51.2%, which is higher than the 50.5% of Marginalize-It distillation.

The paper believes that the reason here is most likely the one mentioned earlier: Marginalize-It loses part of the probability information after the Token ends in order to save computation.

End-Of-Token, which can retain the teacher's distribution more completely, outperforms both its own supervised version and all other schemes.

Further, extrapolating along the scaling curve, End-Of-Token can catch up with the 48.4% predicted upper limit of Token distillation at about 6.33×10²² FLOPs.

In the end, the experiment also shows that byte-level distillation can significantly reduce the storage pressure of data and teacher distribution.

The team also compared these asymptotic predictions with existing open-weight models.

According to the predicted computing budget in the paper, the actual amount of text processed by End-Of-Token is about one sixth of that of the Token scheme;

Under the setting that Token only saves top-600 and bytes save the complete distribution, the storage volume is only about one fifth.

However, less data does not mean less computation.

In particular, End-Of-Token adds an end marker after each Token, so when processing the same amount of text, the training computation is about 30.94% higher than that of the ordinary byte model.

At the same time, inference cost is also a problem, and the paper has not yet completed a fair comparison under the same inference cost.

So in general, this work demonstrates a new path for small model training: convert the teacher's knowledge to the byte level, so that the student can use a smaller prediction space and less training text to pursue a higher upper limit of capabilities.

End-Of-Token performs best in the extrapolation of this set of experiments, but to realize this potential, further investment in computation is still needed.

Author Introduction

Finally, let's introduce the authors of the paper.

The first author of the paper, Kalyani Marathe, is a PhD student at the University of Washington, supervised by Luke Zettlemoyer. Her research interests mainly focus on large model training, model architecture and scaling.

She previously interned at Meta AI Research and also worked on the Foundation Models team at Amazon Robotics.