Quant-VideoGen on RTX 5090

Based on Quant-VideoGen:Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization by Xi et al. (2026).

ArXiv Link: https://arxiv.org/abs/2602.02958v1

GitHub: svg-project/Quant-VideoGen

Why Quant VideoGen?

Autoregressive video generation models face a critical memory bottleneck that severely limits long video synthesis.

Picture this: generating just 5 seconds of 480p video demands approximately 34GB of memory for KV cache storage alone—already exceeding what a single RTX 5090 can handle.

Why is this such a problem? These models generate frames sequentially, and here's the catch: each new frame must maintain complete references to all previously generated frames. This means memory consumption grows linearly with every additional frame added, quickly overwhelming even the most powerful consumer hardware.

However,there is an improvement we can apply--quantization. While quantization for videos are very different from textual ones.

Quantization in Text Models (Why It Works):

Traditional quantization works straightforwardly. In text models like BERT, KV cache values are distributed uniformly. For example:

Sentence: "The cat is on the mat"

Token 1 (The):  [-0.5, 0.3, 1.2, -0.8, 0.6, ...]
Token 2 (cat):  [-0.4, 0.2, 1.1, -0.7, 0.5, ...]
Token 3 (is):   [-0.6, 0.4, 1.3, -0.9, 0.7, ...]
...

Observation: All values stay within [-1, 1.5]—nicely uniform!

Why does it work for text?

Because the value range is fixed and uniform across all tokens—no extreme outliers, all tokens treated equally.

Quantization in Video Models (Why It Fails):

Video KV cache exhibits wildly non-uniform value distributions. Within a single frame:

Static background (sky):        [0.1, 0.05, -0.02, 0.08, ...]
High-contrast edges (objects):  [50.2, 48.9, 52.1, 49.5, ...]
Fast-moving regions:            [1200.5, 1150.2, 1300.1, ...]

Result: Value range spans from 0.05 to 1300—a 26,000× difference!

Applying traditional text quantization causes catastrophic failure:

1. min = 0.05, max = 1300
2. Scale factor = (1300 - 0.05) / 255 ≈ 5.09

Quantizing background value 0.1:
  (0.1 - 0.05) / 5.09 ≈ 0.01 → rounds to 0 (8-bit integer)
  Dequantization: 0 * 5.09 + 0.05 = 0.05
  Original was 0.1, now we get 0.05 → 50% precision loss!

Quantizing motion value 1200:
  (1200 - 0.05) / 5.09 ≈ 236 (acceptable)
  Dequantization: 236 * 5.09 + 0.05 ≈ 1201 (reasonable)

The root cause: Because of the extreme outlier (1300), the entire quantization range stretches to accommodate it. This forces most normal values (0.1-50 range) into coarse bins, destroying precision.

Text Model KV Cache Distribution (Uniform):
 ━━━━━━━━━━━━━━━━━━
  -1      0     1.5
   ▁▂▃▄▅▆▇█▇▆▅▄▃▂▁
   → 8-bit quantization allocates resolution fairly to all values

Video Model KV Cache Distribution (Extremely Non-Uniform):
━┃━━━━━━━━━━━━━━━━━┃━━━━━━━━━━━━━━━
0.05              50              1300
▂▁▁▁▁░░░░░░░░░░░░░░░░░░░░░░░░░▂▃▄▅▆▇█
↑ Massive concentration here    ↑ Single outlier
→ 8-bit quantization wastes resolution on sparse outliers,
  leaving dense regions with almost no precision!

To address this challenge, researchers from UC Berkeley, MIT, NVIDIA, Amazon, and University of Texas at Austin introduce Quant VideoGen, an innovative framework that exploits the inherent spatial-temporal redundancy in video content. The key insight is that adjacent video frames and spatially nearby regions exhibit high similarity, enabling aggressive compression.

The solution comprises two main components:

(1) Semantic-Aware Smoothing groups similar tokens using k-means clustering, quantizing residuals relative to cluster centroids rather than raw values. This reduces key cache quantization error by 6.9× and value cache error by 2.6×.

(2) Progressive Residual Quantization applies the smoothing process iteratively across multiple stages, capturing semantic information at different granularities while maintaining uniform value distributions suitable for low-precision quantization.

Experimental results demonstrate exceptional performance across multiple state-of-the-art models (LongCat-Video, HY-WorldPlay, Self-Forcing). QVG achieves 6.94-7.05× compression ratios while maintaining near-lossless visual quality (PSNR >28.7). Crucially, end-to-end latency overhead remains below 4%, making the approach practical for real applications.

This breakthrough dramatically lowers hardware barriers—enabling long video generation on consumer-grade GPUs that previously required enterprise-level hardware, while maintaining sub-4% performance overhead.

How to run Quant-VideoGen on a single RTX 5090

Prerequisites

A Yotta Labs account with sufficient credits
A Hugging Face account and access token (HF_TOKEN)
Familiarity with basic Python and terminal commands
🐱 LongCat-Video — best starting point, smallest VRAM footprint
⚡ Self-Forcing — streaming video generation
🌍 HY-WorldPlay — world model video generation

Model

BF16 Total KV

INT2 Total KV

Compression

LongCat-Video

22,272 MB

3,231 MB

6.9×

Self-Forcing

46,073 MB

6,614 MB

7.0×

HY-WorldPlay

29,700 MB

4,235 MB

7.0×

Why RTX 5090? Self-Forcing's BF16 KV cache alone requires ~46 GB — impossible on a single 32 GB GPU. QVG brings it down to ~6.6 GB, making long video generation fully feasible on RTX 5090.

Resources: GitHub · Project Page · arXiv

Prerequisites

A Yotta Labs account with sufficient credits
A Hugging Face account and access token with read permissions
Basic familiarity with Python and the terminal

GPU Requirements

GPU

VRAM

Notes

RTX 5090

32 GB

✅ Recommended on Yotta Labs

RTX 4090

24 GB

Feasible for LongCat-Video only

H100 / A100

80 GB

Can run BF16 baseline without QVG

Step 1 — Deploy a Pod

Log in to the Yotta Labs Console.
Navigate to Compute → Pods and click Deploy.
Select RTX 5090 as the GPU.
Under Pod Template, choose pytorch.
Set System Volume to at least 150 GB (checkpoints vary: LongCat ~15 GB, Self-Forcing ~30 GB, HY-WorldPlay ~25 GB).
Click Deploy and wait for the Pod to reach Running state.

Step 2 — Open JupyterLab

Click Connect on the Pod card.
Click the link for port 8888 to open JupyterLab.
Create a new notebook: File → New → Notebook, select the default kernel.
For steps marked [Terminal], open a terminal via File → New → Terminal.

Steps below are clearly marked as either [Terminal] or [Notebook Cell].

Step 3 — Set Up the Environment

3.1 — Clone the Repository

[Terminal]

cd ~
git clone https://github.com/svg-project/Quant-VideoGen.git

3.2 — Create a Conda Environment

#!/bin/bash
set -e  

echo "📦 Initializing conda..."
conda init bash
source ~/.bashrc
conda create -n qvg python=3.12.9 -y
conda activate qvg

3.3 — Install QVG and Dependencies

[Terminal] — Run inside the qvg environment.

cd ~/Quant-VideoGen

pip install uv
pip install huggingface_hub        # provides the huggingface-cli command

# Install QVG with all three backend extras
uv pip install -e ".[all]"

If you only need one backend, install just that extra — e.g. uv pip install -e ".[longcat]".

3.4 — Install Flash Attention

uv pip install \
  https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp312-cp312-linux_x86_64.whl

If this wheel doesn't match your environment, check your versions first:
python -c "import torch; print(torch.__version__, torch.version.cuda)"
Then find the matching wheel at the flash-attention releases page.

Step 4 — Download Model Checkpoints

When downloading check points, the official repository provides three choices: LongCat-video, Self- forcing and HY-WorldPlay.

Quant-VideoGen supports three backends with very different hardware requirements.

Self-Forcing is the most lightweight option, built on Wan2.1-T2V-1.3B (~5 GB of weights), and runs comfortably on a single RTX 5090 (32 GB VRAM).
HY-WorldPlay uses a mid-sized model (~25 GB of weights) and fits within a single RTX 5090 with QVG INT2 quantization applied to the KV cache.
LongCat-Video is the most memory-intensive backend — it uses Mistral-Small-24B as its text encoder (22 GB) paired with a large DiT (51 GB), totaling ~83 GB of model weights. Since QVG only quantizes the inference-time KV cache and not the model weights themselves, LongCat-Video cannot be loaded onto a single 32 GB GPU regardless of quantization settings. It requires at least 2×H200 (160 GB total VRAM) to run.

We start from self-forcing video generation with Wan2.1-T2V-1.3B on a single RTX5090 GPU.

Backend

Model

Weights

Min GPU

Self-Forcing

Wan2.1-T2V-1.3B + DMD

~5 GB

1× RTX 5090

HY-WorldPlay

—

~25 GB

1× RTX 5090

LongCat-Video

Mistral-Small-24B + DiT

~83 GB

2× H200

cd ~/Quant-VideoGen
export HF_TOKEN="your_token_here"   # <-- replace with your HF token

bash scripts/Self-Forcing/download_models.sh

Verify the downloaded files:

du -sh ~/Quant-VideoGen/ckpts/Self-Forcing/*

Expected output :

17G     /home/user/Quant-VideoGen/ckpts/Self-Forcing/Wan2.1-T2V-1.3B
5.3G    /home/user/Quant-VideoGen/ckpts/Self-Forcing/self_forcing_dmd.pt

Step 5 — Run Video Generation

cd ~/Quant-VideoGen
bash scripts/Self-Forcing/run_qvg.sh

Output videos are saved to ~/Quant-VideoGen/results/.

Step 6 — View Output Videos

[ In Notebook cell ]

import glob, os
from IPython.display import Video, display

results_dir = os.path.expanduser("~/Quant-VideoGen/results")
videos = sorted(glob.glob(f"{results_dir}/**/*.mp4", recursive=True))

if not videos:
    print("No output videos found yet. Run a generation cell above first.")
else:
    for v in videos:
        print(f"📹 {v}")
        display(Video(v, embed=True, width=640))

Step 7 — Want to Customize Quantization Settings?

Quantization options are defined inside each run_qvg.sh script. Key parameters:

Parameter

Default

Description

quant_method

triton-nstages-kmeans-int2

Quantization algorithm. INT2 gives ~7× compression.

block_size

64

Token block size. Smaller = finer-grained but slower.

num_centroids

256

K-means centroids. More = better quality, slightly more memory.

Larger model & Further explorations

GPU Requirements
LongCat-Video — requires 2× H200 (model weights total ~83 GB: Mistral-Small-24B text encoder + large DiT)
HY-WorldPlay — runs on a single RTX 5090 (32 GB); uses --offload_text_encoder to stay within VRAM budget

LongCat-Video (2× H200)

LongCat-Video generates long videos auto-regressively by extending from an initial base clip. The workflow has three sequential steps: generate a base clip → extend with BF16 → extend with QVG INT2 quantization.

Step 1 — Download Checkpoints

cd ~/Quant-VideoGen
huggingface-cli download meituan-longcat/LongCat-Video \
  --local-dir ckpts/LongCat-Video \
  --token $HF_TOKEN

Verify:

du -sh ~/Quant-VideoGen/ckpts/LongCat-Video/*

Step 2 — Generate Base Clip

The base clip (results/longcat/base/1-0.mp4) is required by both run_bf16.sh and run_qvg.sh as the starting frame. Always run this first.

cd ~/Quant-VideoGen
bash scripts/LongCat/base.sh

Confirm the output exists before proceeding:

ls results/longcat/base/

Step 3 — Extend with BF16 (quality reference)

bash scripts/LongCat/run_bf16.sh

Output: results/longcat/bf16/

Step 4 — Extend with QVG INT2 (recommended)

QVG reduces the KV cache by ~7× via 2-bit quantization, significantly cutting memory during the long auto-regressive extension.

bash scripts/LongCat/run_qvg.sh

Output: results/longcat/triton-nstages-kmeans-int2_64/.../

View Output in JupyterLab Notebook

Open a new notebook cell and run:

import glob, os
from IPython.display import Video, display

#switch to the right directoty
import os
os.chdir('/home/user')
print(os.getcwd())
results_dir = os.path.expanduser("Quant-VideoGen/results/longcat")
videos = sorted(glob.glob(f"{results_dir}/**/*.mp4", recursive=True))

if not videos:
    print("No output videos found.")
else:
    for v in videos:
        print(f"📹 {v}")
        display(Video(v, embed=True, width=640))

HY-WorldPlay (1× RTX 5090)

HY-WorldPlay is an image-to-video world model — it takes a single input image and a camera movement sequence (--pose) to generate a navigable video. It uses --offload_text_encoder to keep the text encoder on CPU and fit within 32 GB VRAM.

Step 1 — Download Checkpoints

cd ~/Quant-VideoGen
bash scripts/HY-WorldPlay/download_models.sh

Verify:

du -sh ~/Quant-VideoGen/ckpts/HY-WorldPlay/*

Step 2 — Run with QVG INT2 (recommended)

cd ~/Quant-VideoGen
bash scripts/HY-WorldPlay/run_qvg.sh

Output: results/hyworldplay/triton-nstages-kmeans-int2_64/.../

The default input image is assets/hyworld.png and the default prompt describes a stone arch bridge scene. To use your own image and prompt, edit the script directly:

nano scripts/HY-WorldPlay/run_qvg.sh

Change these two lines:

PROMPT='your prompt here'
IMAGE_PATH=path/to/your/image.png

You can also customize the camera trajectory via --pose. The default is:

w-8,s-8,a-8,d-8,up-8,down-8

Each entry is a direction followed by number of frames: w = forward, s = backward, a = left, d = right, up = tilt up, down = tilt down.

Step 3 — Run BF16 baseline (optional, quality comparison)

bash

bash scripts/HY-WorldPlay/run_bf16.sh

Unlike LongCat, HY-WorldPlay does not require a base clip first — run_bf16.sh and run_qvg.sh are both standalone.

View Output in JupyterLab Notebook

import glob, os
from IPython.display import Video, display

#switch to the right directoty
import os
os.chdir('/home/user')
print(os.getcwd())
results_dir = os.path.expanduser("Quant-VideoGen/results/hyworldplay")
videos = sorted(glob.glob(f"{results_dir}/**/*.mp4", recursive=True))

if not videos:
    print("No output videos found.")
else:
    for v in videos:
        print(f"📹 {v}")
        display(Video(v, embed=True, width=640))

Check one of our ouput files segment_2

3MB

segment_2.mp4

Open

Troubleshooting

LongCat: IndexError: list index out of range The base clip is missing. Run bash scripts/LongCat/base.sh first before running run_bf16.sh or run_qvg.sh.
LongCat: CUDA out of memory on H200 Make sure you have 2× H200. The model weights alone total ~83 GB and cannot be split across GPUs with the current single-process setup.
HY-WorldPlay: CUDA out of memory on RTX 5090 Confirm --offload_text_encoder is present in the script. Check with:

Last updated 8 days ago

Was this helpful?

hashtagWhy Quant VideoGen?

hashtagHow to run Quant-VideoGen on a single RTX 5090

hashtagPrerequisites

hashtagPrerequisites

hashtagGPU Requirements

hashtagStep 1 — Deploy a Pod

hashtagStep 2 — Open JupyterLab

hashtagStep 3 — Set Up the Environment

hashtag3.1 — Clone the Repository

hashtag3.2 — Create a Conda Environment

hashtag3.3 — Install QVG and Dependencies

hashtag3.4 — Install Flash Attention

hashtagStep 4 — Download Model Checkpoints

hashtagStep 5 — Run Video Generation

hashtagStep 6 — View Output Videos

hashtagStep 7 — Want to Customize Quantization Settings?

hashtagLarger model & Further explorations

hashtag LongCat-Video (2× H200)

hashtagStep 1 — Download Checkpoints

hashtagStep 2 — Generate Base Clip

hashtagStep 3 — Extend with BF16 (quality reference)

hashtagStep 4 — Extend with QVG INT2 (recommended)

hashtagView Output in JupyterLab Notebook

hashtag HY-WorldPlay (1× RTX 5090)

hashtagStep 1 — Download Checkpoints

hashtagStep 2 — Run with QVG INT2 (recommended)

hashtagStep 3 — Run BF16 baseline (optional, quality comparison)

hashtagView Output in JupyterLab Notebook

hashtagTroubleshooting

Why Quant VideoGen?

How to run Quant-VideoGen on a single RTX 5090

Prerequisites

Prerequisites

GPU Requirements

Step 1 — Deploy a Pod

Step 2 — Open JupyterLab

Step 3 — Set Up the Environment

3.1 — Clone the Repository

3.2 — Create a Conda Environment

3.3 — Install QVG and Dependencies

3.4 — Install Flash Attention

Step 4 — Download Model Checkpoints

Step 5 — Run Video Generation

Step 6 — View Output Videos

Step 7 — Want to Customize Quantization Settings?

Larger model & Further explorations

LongCat-Video (2× H200)

Step 1 — Download Checkpoints

Step 2 — Generate Base Clip

Step 3 — Extend with BF16 (quality reference)

Step 4 — Extend with QVG INT2 (recommended)

View Output in JupyterLab Notebook

HY-WorldPlay (1× RTX 5090)

Step 1 — Download Checkpoints

Step 2 — Run with QVG INT2 (recommended)

Step 3 — Run BF16 baseline (optional, quality comparison)

View Output in JupyterLab Notebook

Troubleshooting