# Tencent WeMM-Video: How to Keep the Soundtrack in Video Analysis

Start with the 2B model, pass audio explicitly, and test whether the soundtrack improves your answers without obscuring what the camera actually shows.

By Paras Madan | 2026-11-21


Tencent's WeChat Vision team released [WeMM-Video weights and serving instructions](https://github.com/Tencent/WeMM-Video) on November 20, following its [technical report](https://arxiv.org/abs/2611.26409). The family takes text, images, audio and video and returns text. One detail matters for anyone building a video assistant: sending a clip is not the same as sending the clip with its original soundtrack.

A product demo makes the distinction useful. The narrator might say an export failed while the screen shows only a progress bar. Reviewing frames alone loses that explanation. Reviewing sound alone loses the UI state. Start by proving your request includes the evidence each question needs.

## Choose a runtime before downloading weights

The release includes 2B, 4B and 9B dense models and a 35B-A3B mixture-of-experts model. Start with 2B to validate the media path. Move up a size only after you can identify a quality problem rather than a missing audio track, a sparse frame sample or an error in the prompt.

Tencent's direct Transformers example uses the 2B BF16 checkpoint on an NVIDIA environment with CUDA 12.9-compatible drivers and FFmpeg. It requires custom model code. Review that code and pin the checkpoint revision before running it, using public clips in an isolated environment for the first trial.

For an HTTP service, use the [runtime matrix](https://github.com/Tencent/WeMM-Video/blob/main/docs/serving.md) instead of assuming a stock vLLM or SGLang install works. The documented profiles are patched. FP8 and NVFP4 are different serving paths, not just interchangeable model IDs. Tencent also documents a [native GGUF route](https://github.com/Tencent/WeMM-Video/blob/main/native/README.md) for Linux CPU or CUDA; its bundles include separate language, visual-projector and audio files. Keep all three on the same revision.

## Ask for audio explicitly

In the Transformers processor, use the `video_audio` item type to include the soundtrack. In Tencent's patched HTTP backends, keep the item type as `video_url` and put `use_audio: true` inside that object. Use `false` for a frame-only control.

Here is an illustrative content item for the documented Chat Completions route, not a complete request:

```json
{
  "type": "video_url",
  "video_url": {
    "url": "https://example.com/demo.mp4",
    "use_audio": true,
    "fps": 1.0,
    "max_frames": 16
  }
}
```

Verify that the server can reach the actual asset and that the file contains an audio stream. Start with an audible phrase that never appears on screen and an on-screen label that nobody says. If both are recovered in one response, you have a better basis for testing joint reasoning.

## Treat frame sampling as part of the result

The documented request puts `fps` and `max_frames` inside the video object. They are two separate controls. Sampling roughly once per second can still miss a message that appears and disappears between sampled frames. A frame cap also constrains coverage on a long video.

Keep the model and prompt fixed, then compare a short segment at different frame settings. Record whether a brief screen change survives and what happens to first-token latency and peak memory. If short clips work but longer clips fail, try chunking the source before raising several limits at once.

For a support workflow, a good first prompt is: “What did the speaker say went wrong, and what does the screen independently show?” Grade the two parts separately. A correct summary of the narration should not count as proof that the model read the screen.

## Make soundtrack use an explicit test

Run a small set of known clips twice: once with audio enabled and once without it. Include a silent control, a case where audio resolves ambiguity, and a case where the narrator and screen disagree. Check that the response reports the disagreement instead of merging both into one invented account.

Use the same sampling settings and generation mode for both runs. Then compare thinking mode separately if the workload genuinely needs it. Keep test clips short enough that a missing answer is easy to diagnose.

Tencent's serving guide reports loopback request checks and a native CPU text smoke test, but lists GPU generation after downloading Hub checkpoints as pending in its validation notes. Treat the recipes as a starting point, not a performance guarantee. Benchmark your exact model, runtime and media settings before choosing a production profile.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams turn model launches and deployment choices into technical writing developers can use.

[Book a 30-minute call](https://cal.com/paras-madan/30min)
