Fudan University releases the human-centric audio-visual tracking benchmark for complex scenarios.

Imagine this scenario: in a multi-person online meeting, the frame automatically focuses on the speaker as they talk; for a variety show footage, the post-production system can automatically recognize who is speaking and generate corresponding subtitles; for a surveillance video, the system can continuously track the person who is speaking among the crowd.

Behind these functions lies the same core technology — Audio-Visual Speaker Tracking. It enables machines to locate and continuously track the person making sounds by leveraging both the audio signals it hears and the visual frames it captures.

In the past few years, tasks including Audio-Visual Segmentation (AVS) and Audio-Visual Instance Segmentation (AVIS) have drawn widespread attention and witnessed remarkable progress. Nevertheless, a fundamental question has not been fully addressed: when the scenario truly becomes dynamic, crowded and discontinuous, what exactly is the capability of existing models in terms of audio-visual correspondence and long-term identity association?

To this end, the CVL Laboratory of Fudan University proposes AVTrack, an audio-visual tracking benchmark oriented towards complex human-centric scenarios. The corresponding paper has been accepted by ICML 2026, and the project homepage, paper, code and dataset are now publicly available.

Project Homepage: https://fudancvl.github.io/AVTrack/

Paper: https://arxiv.org/abs/2606.02724

Code: https://github.com/FudanCVL/AVTrack

Dataset: https://huggingface.co/datasets/FudanCVL/AVTrack

More Than Simply "Locating the Speaker"

The goal of audio-visual tracking is intuitive: given a video with audio, the model should not only determine who is speaking at the moment, but also maintain stable instance identity across the continuous changes of the video and multiple sounding segments.

This may not sound difficult — after all, video conferencing software used in daily life can already complete basic speaker detection. However, once applied to real and complex video scenarios, the problem quickly becomes intractable: people may occlude each other, move frequently and take turns to speak, the camera may switch constantly, and there may even be audio-visual inconsistency such as off-screen voices.

AVTrack is exactly an audio-visual instance segmentation benchmark proposed for such complex scenarios. It requires the model to jointly understand audio and visual content, generate pixel-level masks for each sounding person, and maintain stable instance identity during the continuous change of the video. The model should not only know "who the voice comes from", but also always know "who this person is and where they are".

In this task, the model needs to answer three questions simultaneously:

Who is making the sound? That is, to determine the currently active speaker from the audio;

Where is the person? That is, to segment the corresponding person with pixel-level accuracy in each frame;

Is it still the same person across frames? Even if the person is occluded, the camera switches or the positions are swapped, the cross-frame identity consistency should be maintained.

The third question is particularly critical. Relying only on the appearance or lip movement in a single frame, the model may find the speaker temporarily; but to maintain correct identity continuously in long videos, it must also deal with situations such as visual occlusion, multiple people taking turns to speak, and audio-visual inconsistency.

What Is Missing From Existing Evaluation Benchmarks?

Most of the early audio-visual speaker tracking datasets are collected in controlled laboratories, with a small number of people and relatively fixed camera and microphone positions. In these simple scenarios, models do not even need to truly understand temporal dynamics and cross-modal reasoning, and can achieve good performance only through static association such as the simultaneous occurrence of sound and visual frames. But this does not mean that the model can also work well in complex dynamic scenarios. Although subsequent audio-visual segmentation benchmarks have introduced pixel-level annotations and more natural video sources, many samples are still short clips of 5-10 seconds, which is difficult to fully test long-term temporal modeling.

The AVISeg dataset extends the average video length to about 60 seconds, and provides instance-level segmentation and tracking annotations. However, the coverage of camera motion, occlusion, background switching and relative position change is still limited.

This leads to an evaluation blind spot: the model may achieve good results in relatively static videos, but still relies on short-term audio-visual co-occurrence relationships. Once a person disappears and reappears, the camera switches, or the voice is inconsistent with the most prominent person in the frame, it is difficult for existing evaluation benchmarks to fully answer whether the model truly maintains the cross-modal correspondence.

Constructing a Systematic Stress Test With 8 Types of Challenges

AVTrack contains 871 videos with an average duration of 54.0 seconds, providing a total of 3,120 pixel-level instance trajectories with cross-frame identities. The data covers 6 types of sources: TV series, vlogs, animations, reality shows, interviews and stage performances. Different from conventional training sets, AVTrack is designed as a pure test benchmark: as an independent test set, AVTrack verifies whether the model can be transferred to complex human-centric scenarios.

To systematically analyze the key difficulties in complex scenarios, AVTrack further divides them into 8 diagnosable challenges: Visual Occlusion, Relative Position Change, Background Switch, Camera Motion Change, Multiple Instances, Multi-turn Sounding, Audio-Visual Inconsistency, and Instance Scale Dynamics.

① Visual Occlusion: The speaker is partially occluded by other people or objects, resulting in instance overlap and blurred visual boundaries, which increases the difficulty of continuous localization and identity maintenance. This attribute covers 80.9% of the samples in AVTrack, while it is only 8.6% in AVISeg.

② Relative Position Change: The relative spatial relationship between multiple instances changes over time, for example, the speaker moves from the left side of another person to the right side. Such scenarios require the model to continuously track instance identities instead of relying on static spatial positions. This attribute accounts for 70.7% in AVTrack and only 9.7% in AVISeg.

③ Background Switch: Significant scene or background changes occur in the video, for example, switching from indoor to outdoor, which weakens the temporal continuity based on background and visual context. This attribute accounts for 60.5% and 5.9% in AVTrack and AVISeg respectively.

④ Camera Motion Change: The camera perspective changes significantly, including zooming, panning, perspective transformation or shot switching, which continuously changes the scale, perspective and visible area of the target. This is the most common challenge in AVTrack, covering 90.5% of the samples, while it is only 7.1% in AVISeg.

⑤ Multiple Instances: Multiple candidate people exist in the frame at the same time, but only some of the instances are sounding, which requires the model to establish correct audio-visual correspondence between multiple visual instances. This attribute accounts for 64.9% in AVTrack and only 13.6% in AVISeg.

⑥ Multi-turn Sounding: Different people take turns to speak over time, so that the active speaker and its audio-visual correspondence are constantly switched, requiring the model to dynamically update the target identity. This attribute covers 56.8% of the samples in AVTrack, while it is 16.0% in AVISeg.

⑦ Audio-Visual Inconsistency: There is no direct one-to-one correspondence between the audio and the currently visible person, such as narration, off-screen voice, or the speaker is temporarily outside the frame. About 9.2% of the samples in AVTrack contain such situations.

⑧ Instance Scale Dynamics: The target instance presents significant scale differences in the video or undergoes large-scale changes over time, which puts forward higher requirements for small target perception and cross-scale continuous tracking. This attribute accounts for 56.9% in AVTrack and 35.3% in AVISeg.

These attributes not only characterize the scene complexity of the dataset, but can also be further used to analyze the specific failure modes of the model. AVTrack is significantly more challenging in terms of dynamic visual changes: 90.5% of the videos contain camera motion changes, 80.9% contain visual occlusion, and 70.7% contain relative position changes, which are much higher than the corresponding 7.1%, 8.6% and 9.7% of AVISeg. At the same time, AVTrack explicitly covers scenarios that are rarely involved in traditional audio-visual segmentation data, such as audio-visual inconsistency, so that the evaluation can further test the robustness of the model under complex temporal changes and non-ideal audio-visual correspondence conditions.

To obtain reliable pixel-level and cross-frame annotations, annotators first screened candidate samples from about 1,300 public videos, used Grounded-SAM for automatic pre-annotation, and then annotators corrected and checked the temporal identity frame by frame. The entire data collection, annotation and verification process lasted for nearly three months, with 15 professional annotators participating.

Performance of Existing Methods in Complex Scenarios

The research team evaluated representative Video Instance Segmentation (VIS) and audio-visual instance segmentation methods under a unified setting. The VIS methods are pre-trained on YouTube-VIS and then fine-tuned on AVISeg; the AVIS methods are trained on AVISeg, and all are directly tested on AVTrack finally.

The results show that the HOTA test metrics of VITA, LBVQ and CAVIS that only use visual information on AVTrack are all lower than 12.

After adding audio information for assistance, AVISM and ACVIS reach 20.84 and 20.60 respectively, indicating that audio does bring obvious benefits, but long-term identity association is still difficult.

The research team also provides two complementary baseline methods: one is the end-to-end AVTrackFormer, which achieves a HOTA test metric of 21.47; the other is the training-free modular model AVTracker, which achieves a HOTA test metric of 29.08. Compared with AVISM, the best-performing existing AVIS method, AVTracker improves by 8.24 (29.08 vs. 20.84); compared with the stronger end-to-end baseline model AVTrackFormer, AVTracker improves by 7.61 (29.08 vs. 21.47).

The research team also tested the zero-shot performance of Gemini 2.5 Pro: after directly inputting 1 FPS video frames and original audio, its HOTA metric is 14.4. This result shows that the extensive audio-visual understanding capability of general Omni-LLM cannot be directly transformed into stable pixel-level speaker tracking capability in complex scenarios.

AVTracker First Performs Local Correspondence, Then Restores Global Identity

The core idea of AVTracker is to split the complex association problem in long videos into three stages.

First, the system uses Whisper to obtain speech segments with timestamps, and merges adjacent segments belonging to the same speaker based on ECAPA-TDNN speaker embedding to form more compact speaker segments.

Then, within each local time window, SAM3 provides candidate person masks, and the Local Reasoner jointly uses speech transcription and visual observations to map the current speech segment to the visible person and form a local short trajectory.

Finally, the Global Reasoner aggregates the key frames of each local trajectory, completes identity grouping on the entire video, and restores the continuous speaker trajectory.

Ablation experiments further verify the effectiveness of this local-to-global design. Under the same model settings, after removing the local segment compression, HOTA drops from 24.01 to 16.88; after turning off the dynamic window mechanism on the full model, HOTA also drops from 28.85 to 27.45. Speech separation does not always bring performance improvement:

When using MossFormer2, HOTA increases from 28.85 to 29.08, while when using SepFormer, it drops to 28.41, indicating that the error generated by speech separation may further propagate to the subsequent audio-visual alignment process.

Higher Accuracy Does Not Mean the Problem Has Been Solved

From the perspective of each sub-category, audio-visual inconsistency is still the most difficult scenario for all methods, and the HOTA of AVTracker is only 18.5.

In addition, in visual occlusion and multi-instance scenarios, when speakers who are close to each other are occluded and take turns to speak frequently, the global reasoner may still mistakenly merge different identities, or split the same identity into multiple trajectories. These failure modes indicate that long-term memory, cross-window evidence accumulation and correctable identity reasoning mechanisms are still worthy of further research.