Scaling Video Matting via a Learned Quality Evaluator

Peiqing Yang¹ Shangchen Zhou^1† Kai Hao¹ Qingyi Tao²

¹S-Lab, Nanyang Technological University ²SenseTime Research, Singapore
^†Project lead

MatAnyone 2 is a practical human video matting framework that preserves fine details by avoiding segmentation-like boundaries, while also shows enhanced robustness under challenging real-world conditions.

:movie_camera: For more visual results, go checkout our project page ---

## 📮 Update - [2026.03] Add uv and huggingface support for easy installation and usage. - [2026.03] Release inference codes, evaluation codes, and gradio demo. - [2025.12] This repo is created. ## 🏄🏻‍♀️ TODO - [x] Release inference codes and gradio demo. - [x] Release evaluation codes. - [ ] Release training codes for video matting model. - [ ] Release checkpoint and training codes for quality evaluator model. - [ ] Release real-world video matting dataset **VMReal**. ## 🔎 Overview ![overall_structure](assets/matanyone1vs2.jpg) ## 🔧 Installation ### From PyPI / Source ```bash # install from repo pip install git+https://github.com/pq-yang/MatAnyone2.git#egg=matanyone2 # or install optional extras pip install git+https://github.com/pq-yang/MatAnyone2.git#egg=matanyone2[gui] # Gradio demo + PySide6 pip install git+https://github.com/pq-yang/MatAnyone2.git#egg=matanyone2[dev] # development / evaluation tools pip install git+https://github.com/pq-yang/MatAnyone2.git#egg=matanyone2[all] # everything ``` ### Conda ```bash conda create -n matanyone2 python=3.10 -y conda activate matanyone2 pip install git+https://github.com/pq-yang/MatAnyone2.git#egg=matanyone2[all] ``` ### uv (recommended) ```bash # create a new project and add matanyone2 uv init my-matting-project && cd my-matting-project uv add matanyone2@git+https://github.com/pq-yang/MatAnyone2.git # or with optional extras uv add matanyone2[gui]@git+https://github.com/pq-yang/MatAnyone2.git uv add matanyone2[all]@git+https://github.com/pq-yang/MatAnyone2.git ``` ## 🔥 Inference ### Download Model Download our pretrained model from [MatAnyone 2](https://github.com/pq-yang/MatAnyone2/releases/download/v1.0.0/matanyone2.pth) to the `pretrained_models` folder (pretrained model can also be automatically downloaded during the first inference). The directory structure will be arranged as: ``` pretrained_models |- matanyone2.pth ``` ### Quick Test We provide some examples in the [`inputs`](./inputs) folder. **For each run, we take a video and its first-frame segmenatation mask as input.** The segmenation mask could be obtained from interactive segmentation models such as [SAM2 demo](https://huggingface.co/spaces/fffiloni/SAM2-Image-Predictor). For example, the directory structure can be arranged as: ``` inputs |- video |- test-sample1 # folder containing all frames |- test-sample2.mp4 # .mp4, .mov, .avi |- mask |- test-sample1.png # mask for targer person(s) |- test-sample2.png ``` Run the following command to try it out: ```shell # intput format: video folder matanyone2 -i inputs/video/test-sample1 -m inputs/mask/test-sample1.png # intput format: mp4 matanyone2 -i inputs/video/test-sample2.mp4 -m inputs/mask/test-sample2.png # or via python python inference_matanyone2.py -i inputs/video/test-sample1 -m inputs/mask/test-sample1.png ``` The results will be saved in the `results` folder, including the foreground output video and the alpha output video. ### Python API (recommended 🔥) You can load the model directly from Hugging Face using `from_pretrained` and run inference programmatically: ```python from matanyone2 import MatAnyone2, InferenceCore model = MatAnyone2.from_pretrained("not-lain/matanyone2") processor = InferenceCore(model, device="cuda:0") processor.process_video( input_path="inputs/video/test-sample2.mp4", mask_path="inputs/mask/test-sample2.png", output_path="results", ) ``` - If you want to save the results as per-frame images, you can set `--save-image`. - If you want to set a limit for the maximum input resolution, you can set `--max-size`, and the video will be downsampled if min(w, h) exceeds. By default, we don't set the limit. - Run `matanyone2 --help` for a full list of options. ## 🎪 Interactive Demo To get rid of the preparation for first-frame segmentation mask, we prepare a gradio demo on [hugging face](https://huggingface.co/spaces/PeiqingYang/MatAnyone2) and could also **launch locally**. Just drop your video/image, assign the target masks with a few clicks, and get the the matting results! *We integrate MatAnyone Series in the demo. [MatAnyone 2](https://github.com/pq-yang/MatAnyone2) is the default model. You can also choose [MatAnyone](https://github.com/pq-yang/MatAnyone) as your processing model in "Model Selection".* ```shell cd hugging_face # install GUI dependencies (if not already installed via pip install -e ".[gui]") pip3 install -r requirements.txt # FFmpeg required # launch the demo python app.py ``` By launching, an interactive interface will appear as follow. ![overall_teaser](assets/teaser_demo.gif) ## 📊 Evaluation Please refer to the [evaluation documentation](docs/EVAL.md) for details. ## 🛠️ Data Pipeline ![data_pipeline](assets/data_pipeline.jpg) ## 📑 Citation If you find our repo useful for your research, please consider citing our paper: ```bibtex @InProceedings{yang2026matanyone2, title = {{MatAnyone 2}: Scaling Video Matting via a Learned Quality Evaluator}, author = {Yang, Peiqing and Zhou, Shangchen and Hao, Kai and Tao, Qingyi}, booktitle = {CVPR}, year = {2026} } @inProceedings{yang2025matanyone, title = {{MatAnyone}: Stable Video Matting with Consistent Memory Propagation}, author = {Yang, Peiqing and Zhou, Shangchen and Zhao, Jixin and Tao, Qingyi and Loy, Chen Change}, booktitle = {CVPR}, year = {2025} } ``` ## 📝 License This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license. ## 👏 Acknowledgement This project is built upon [MatAnyone](https://github.com/pq-yang/MatAnyone) and [Cutie](https://github.com/hkchengrex/Cutie), with matting dataset files adapted from [RVM](https://github.com/PeterL1n/RobustVideoMatting). The interactive demo is adapted from [ProPainter](https://github.com/sczhou/ProPainter), leveraging segmentation capabilities from [Segment Anything Model](https://github.com/facebookresearch/segment-anything) and [Segment Anything Model 2](https://github.com/facebookresearch/sam2). Thanks for their awesome works! ## 📧 Contact If you have any questions, please feel free to reach us at `peiqingyang99@outlook.com`.