Meta Unveils Its Game-Changing Speech Transcription Tool: It Accurately Recognizes Mixed Chinese and English Content, and Supports 1-hour Long Conversations With More Than 20 Participants.

Reported by Zhidx on September 2, today, Meta launched the first real-time audio perception model Muse Voice Transcribe developed by Meta Superintelligence Labs.

This model provides real-time streaming Automatic Speech Recognition (ASR), supports voice segmentation for more than 20 speakers and endpoint control functions, supports multilingualism, can process long audio of over 1 hour, achieves seamless code-switching, and improves recognition accuracy through language, keyword and context preferences.

Judging from the examples, whether it is speaking Chinese, Chinese-accented English, or mixed Chinese and English expressions, the transcription speed and effect of this model are quite good.

Muse Voice Transcribe ranks first in the Artificial Analysis streaming speech-to-text and public speech segmentation benchmarks. (Model inclusion and ranking as of September 1, 2026.)

This model can be accessed via the Meta Model API, Meta AI for Mac, and Muse Code.

Its API is priced at $3 per 1000 minutes (about 20 yuan), $0.18 per hour (about 1.21 yuan).

01. Capable of recognizing long audio over 1 hour, mixed Chinese and English, and simultaneous speech from more than 20 people

Muse Voice Transcribe has been trained with over 70 languages, 25 of which have been fully validated.

In the initial release, Meta recommends users try the 25 validated languages, while support for additional languages is also available.

In terms of language switching, Muse Voice Transcribe natively supports arbitrary code-switching, whether within or between sentences, and further improves recognition accuracy by leveraging contextual information.

The model natively supports long audio input of over 1 hour and more than 20 speakers without post-processing.

With just one click, the voice dictation feature of Meta AI and Muse Code can be powered by Muse Voice Transcribe.

It works with any application, and can be used with Meta AI and any window on the screen to complete various tasks - just hold down the "Fn" key to try it out.

02. Based on streaming automatic speech recognition, dynamically adjust latency to balance speed and accuracy

Muse Voice Transcribe is an autoregressive multimodal model in the Muse Spark series.

Audio is processed in units of 80ms (12.5Hz), and each audio segment is converted into one soft token. For each audio segment, the model decides whether to continue listening to the next audio segment or output a text token.

When the model decides to keep listening, it predicts a special <|next_audio|> token, and replaces <|next_audio|> with the actual audio segment of the next input.

When the audio stream stops, Meta inserts a special <|empty_audio|> token to inform the model that there are no more audio segments. After receiving <|empty_audio|>, the signal that "there is no subsequent audio input", the model will directly output all remaining text tokens without generating any more <|next_audio|> tokens.

Since the model has full control over when to listen, it decides how much audio context information (referred to as "latency") it needs before transcribing a word. There is a trade-off between accuracy and latency.

The longer the model waits to predict, the more accurate the transcription result will be, but the higher the latency will also be.

Muse Voice Transcribe features an "adaptive latency" function that dynamically adjusts latency based on the difficulty of each word. This is achieved through Reinforcement Learning (RL), where Word Error Rate (WER) rewards and latency rewards are combined in a multiplicative manner.

With adaptive latency, this model achieves the Pareto frontier in the speed-accuracy trade-off, measured by the final transcription time.

03. Build diarization and endpoint functions on the basis of ASR

Based on streaming ASR, other audio perception tasks can be easily supported by introducing additional special tokens.

To achieve speaker diarization, Meta introduces a <|start_of_turn|> token to indicate a potential speaker switch, and a <|speaker_{A-Z}|> tag to distinguish different speakers.

When the speaker switches, <|start_of_turn|> is predicted immediately, while the prediction of the speaker tag is delayed until the end of the audio chunk.

Audio from the same speaker can also be split into multiple segments by <|start_of_turn|> in this way, and all these split segments will have the same speaker tag.

The following is an example token sequence for speaker diarization (<|next_audio|> is omitted for simplified demonstration):

<|start_of_turn|>Hello, how are you doing?<|speaker_A|><|start_of_turn|> Did anything fun over the weekend?<|speaker_A|><|start_of_turn|> Hey I'm good!<|speaker_B|>

For voice endpoint detection, Meta introduces a <|speech_onset|> token to mark the start of speech, and a <|speech_endpoint|> token to mark the end of the user's utterance.

The following is an example token sequence for voice endpoint detection (<|next_audio|> is omitted for simplified demonstration):

<|speech_onset|>Hey Meta, what's the weather in Menlo Park?<|speech_endpoint|> [silence] <|speech_onset|>What should I wear today?<|speech_endpoint|>

Meta trains both tasks simultaneously using streaming ASR, and adds additional rewards for the speaker diarization and voice endpoint detection tasks respectively on the basis of the ASR reward signal.

04. Conclusion: Speech recognition is evolving toward a continuously online "ear"

In real scenarios, people's communication situations are very complex, such as multilingual mixing, people interjecting from time to time, overlapping voices of multiple people, etc., and people may chat continuously for dozens of minutes or even several hours. The capabilities that Muse Voice Transcribe highlights this time are very helpful for optimizing the transcription quality of these real-world scenarios.

Speech transcription is evolving from an independent "speech-to-text" tool to a real-time perception layer of AI systems. The core of the competition in the next stage will focus on low latency, long context, multiple speakers, multilingualism and context understanding at the same time, and eventually integrate with the reasoning, memory and tool calling of large models. At that time, "hearing" is only the entry point, and the system also needs to know who is speaking, when the speech ends, what the relationship between this sentence and the previous content is, and what to do next.

This article is from the WeChat Official Account "Zhidongxi" (ID: zhidxc), author: ZeR0, editor: Mo Ying, published with authorization by 36Kr.