Another major domestic model has been officially open-sourced, ranking first globally in audio and video editing, with 16 chip manufacturers and platforms completing the adaptation on its first release day.
All-in-one generation for text, image, audio and video content.
Reported by Zhidx on August 3, MiniMax officially open-sourced its new-generation general video model MiniMax H3.
MiniMax H3 is a universal full-modal generation system that can uniformly understand the multi-modal context composed of text, images, videos and audios, and generate videos with a maximum duration of 15 seconds, up to 2K resolution and native stereo audio.
Earlier on July 31, MiniMax H3 was released. Currently, in the Artificial Analysis audio-enabled video editing ranking, MiniMax H3 ranks first with an Elo score of 1130, leading domestic and overseas video models including Gemini Omni Flash, HappyHorse-1.0, Wan 2.7, etc.
MiniMax H3 tops the audio-enabled video editing ranking list
The H3 system consists of three modules: H3-Context-IR, H3-Base, H3-Regenerate-2K. Developers can directly download the MiniMax H3 model from Hugging Face, and H3-Base currently supports deployment through inference frameworks and workflows such as SGLang, vLLM, diffusers and ComfyUI.
Along with the model open-sourcing, a total of 16 ecological partners have completed relevant adaptation and support, including domestic and overseas chip manufacturers such as Huawei Ascend, Moore Threads, MetaX, Hygon Information, Kunlunxin, Iluvatar CoreX, Biren Technology, AMD, and Intel, as well as development communities and cloud inference platforms such as Hugging Face, ModelScope, ComfyUI, RunningHub, and fal, plus inference frameworks such as vLLM-Omni and SGLang, forming a total of 16 ecological partners.
Open source address:
huggingface.co/MiniMaxAI/MiniMax-H3
Model experience addresses:
H3-2K direct output:
platform.minimaxi.com/docs/api-reference/video-generation-v2-create
H3-Context-IR:
platform.minimaxi.com/docs/api-reference/video-generation-v2-h3-context-ir
H3-Regenerate-2K:
platform.minimaxi.com/docs/api-reference/video-generation-v2-regeneration
MiniMax Hub: hub.minimaxi.com
Hailuo AI: hailuoai.com
Today, Zhidx conducted an actual test on MiniMax H3. MiniMax H3 is already able to complete different types of video generation tasks, and performs well in picture naturalness, shot organization and overall style consistency. The model still has room for improvement in terms of picture realism and shot arrangement.
For example, we asked the model to generate a 15-second aerial landscape photography video, in which the natural landscapes such as snow-capped mountains and grasslands in the finished video are relatively realistic, the colors, light and shadow, and the movement of aerial shots are also relatively natural, and the long shot maintains good visual coherence as a whole. However, the temple building in the picture still has obvious traces of AI generation.
After that, we asked MiniMax H3 to generate an advertising promotional video for the collaboration between milk tea and game IP. The video generated by MiniMax contains 6 sub-shots, the overall narrative sequence is relatively clear, and the picture style and advertising atmosphere are basically unified. However, in terms of multi-shot arrangement, there is an obvious repetition in the model: the action of the shop assistant handing the milk tea to the customer is presented twice in a row.
01. Generates up to 15-second 2K videos with synchronized output of visuals and stereo audio
MiniMax H3 is a universal full-modal generation system. In terms of output specifications, H3 can generate videos ranging from 4 seconds to 15 seconds, supports multiple aspect ratios such as 21:9, 16:9, 4:3, 1:1, 3:4, 9:16, with an output frame rate of 24FPS, and can simultaneously generate 32kHz stereo audio.
The model generates videos with a short edge of 768 pixels by default, and can further generate 2K version through H3-Regenerate-2K. H3 stably supports 11 languages including Chinese, English, Japanese, Korean, French, German, etc., and also provides different degrees of support for other languages.
H3 includes two versions, FL2VA and Ref2VA, for first-last frame generation and full-modal reference generation respectively.
H3-Base-FL2VA is the first-last frame mode, which can input up to two images. When no image is input, the model executes text-to-video tasks; when one first frame or last frame image is input, it can generate the corresponding video starting from the first frame or video ending at the last frame; when two images are input at the same time, it can generate the intermediate video content according to the specified first and last frames.
H3-Base-Ref2VA supports the full-modal reference mode, which can input up to 9 images; it can input up to 3 videos, each with a duration of 2 to 15 seconds, and the total duration does not exceed 15 seconds; it can input up to 3 audios, each with a duration of 2 to 15 seconds, the total duration does not exceed 15 seconds, and the audio must be input together with images or videos, and cannot be used as the only input. The total number of images, videos and audio files that can be input is up to 12.
02. Three modules work collaboratively to generate audio and video, 2K visuals adopt context regeneration
The H3 system consists of three modules: H3-Context-IR responsible for understanding and sorting multi-modal instructions, H3-Base responsible for generating audio and video, and H3-Regenerate-2K responsible for generating 2K visuals.
Overview of MiniMax H3 System (Source: MiniMax)
H3-Context-IR is a hosted preprocessing and orchestration system that can understand the relationship between text, images, audios and reference videos, as well as the relationship between these materials and the expected generation results. Its internal workflow includes instruction parsing, cross-modal association, temporal understanding and complex logical reasoning.
H3-Context-IR relies on a multi-stage workflow, as well as multiple hosted models and services, and this module is not open-sourced for the time being. MiniMax currently provides corresponding APIs and prompt guidelines, based on which developers can build their own multi-modal preprocessing systems.
Video prompt writing guide:
huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md
Full reference mode output guide:
huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
H3-Base is responsible for the actual audio and video generation. The text is encoded by H3-Encoder, the visual materials are processed by both H3-Encoder and H3-VisualVAE, and the audio is encoded by H3-AudioVAE. The information of different modalities is then organized into a unified sequence and input into H3-Omni-Transformer.
Overview of MiniMax H3-Base Architecture (Source: MiniMax)
For the 2K resolution output of H3, H3 does not adopt the traditional dedicated super-resolution module. Instead, through H3-Regenerate-2K, the base model combines the original text and multi-modal reference materials to regenerate the already generated 768p video. This method can reuse the original context, which helps restore information that is difficult to complete only with low-resolution visuals, such as small texts and fine textures. This module is also not open-sourced at present, and developers can reproduce the complete 2K generation process through the official API.
03. Local deployment can generate 768p audio and video, the complete 2K workflow requires API calls
MiniMax provides developers with two verification methods: H3-Base local deployment and complete 2K workflow. H3-Base is released as two independent model repositories, FL2VA and Ref2VA, both containing inference components such as processor, tokenizer, text encoder, Omni Transformer, Visual VAE and Audio VAE, which can be deployed through frameworks or workflows such as SGLang, vLLM, diffusers and ComfyUI, and support multi-GPU parallel inference.
When only deploying H3-Base locally, developers can generate audio and video with a short edge of 768 pixels. Among them, the FL2VA version supports text-to-audio-video and first-last frame generation, and the Ref2VA version supports jointly referencing images, videos and audios to complete tasks such as character and scene retention, action and lip sync editing, timbre reference, etc.
The complete 2K workflow needs to combine the local model and the official API: first, H3-Context-IR understands and expands the user input, then the locally deployed H3-Base generates 768p audio and video, and finally H3-Regenerate-2K combines the original context to regenerate the 2K video. MiniMax also provides reproducible cases such as text-to-audio-video, first-frame-to-audio-video and full-modal reference generation, and relevant request parameters, sample codes and reference results can be viewed on the Hugging Face model page.
04. Conclusion: Full-modal video generation is accelerating to move towards an open ecosystem
From the perspective of actual effect, MiniMax H3 is already able to process different types of generation tasks such as natural scenery and commercial advertisements, showing a high degree of completion in terms of picture naturalness, shot organization and style consistency.
With multiple chip manufacturers, development communities, cloud inference platforms and inference frameworks completing the adaptation synchronously, this open-sourcing of MiniMax H3 not only opens up the model capabilities, but also initially connects the ecological link from model download, local deployment to application development. The competition between full-modal video models is further extending from a single picture effect to sound generation, complex instruction understanding and industrial ecological construction.
This article is from the WeChat Official Account "Zhidongxi" (ID: zhidxc), author: Yang Jingli, editor: Li Shuiqing, authorized to release by 36Kr.