TL;DR — Key Takeaways
- Microsoft says MLVC can slash video bandwidth requirements to about one-eighth of H.264 while maintaining comparable quality.
- NPUs are central to the codec’s efficiency, handling neural video processing with far less power than GPUs.
- Microsoft has open sourced MLVC to accelerate support for more hardware platforms and improve real-time performance and resilience.
Looking for outside help, Microsoft has open sourced a neural network-based video codec that promises to cut storage and bandwidth costs to one-eighth of what can be achieved by hardwired codecs, as shown in early tests against rate-distortion objectives. The technology is built on a new type of coprocessor called a Neural Processing Unit (NPU).
“Neural video codecs have surpassed classical codecs in coding efficiency but remain impractical for deployment due to cross-platform incompatibility and high computational cost,” the researchers write in an arXiv paper.
The company’s ML Video Codec (MLVC) aims to productize the neural video codec for everyday use, so it can be deployed in streaming video, teleconferencing, cloud gaming, surveillance, augmented and virtual reality and many other areas.
In the best practice of drinking your own champagne, Microsoft is already using the technology to power video calls in its Microsoft Teams enterprise workspace software, with the traditional codecs serving as backup.
Codecs are Long-Term Projects
Building a new generation of video codecs is difficult work. Videos are huge files: A codec compresses a bandwidth-heavy video at the point of origin and then decodes it at the destination for the user.
Each new generation of codecs (such as H.264/AVC and H.265/HEVC) tends to be used for a long time, given the immense amount of engineering that goes into the individual elements such as motion estimation, transforms and entropy modeling.
So in the age of AI, it would be natural to do the job not with fixed algorithms, but rather with end-to-end learned neural compression.
MLVC builds on a whole family of ML codecs, DCVC (Deep Contextual Video Compression) family of NVC (Neural Video Codec), which the company has been working on since 2021.
The video is passed through a transformer layer, values are trained against rate-distortion objectives, then unpacked at the receiver end using NPUs.
NPUs Do the Heavy Lifting
GPUs can do the work themselves, though they require a lot more power for the task. Intel, Qualcomm and Apple all have NPUs alongside their CPU cores. MLVC uses no more than 50% of these chips’ native capability to leave room for other tasks.
The results look promising: MLVC-encoded video requires only about 12% of the bandwidth that H.264-encoded video requires. For a 360p video at 30 fps, H.264 chews up about 1 Mbps, whereas MLVC requires roughly 122 kbps for the same quality.
For these benchmarks, Microsoft used the P.910 subjective test, with the recently-open sourced Video Conferencing Dataset (VCD) (Demo here).
Open Source Leads the Way
The company has open sourced the work, hoping to get third-party contributions. The team needs help across the board, but particularly seeks more NPU ports (the AMD NPU, for instance, is not yet supported) and work that pushes the frontier of rate distortion.
The MLVC repository, shared under the MIT License, includes the model source code, trained model weights, training scripts and platform conversion scripts to target different NPUs and runtimes. The code is mostly written in Python, with some C++ at the edges.
The group’s next goals are to stabilize 540p real-time performance, continue expanding hardware coverage, and improve loss resilience.
