On GitHub, PeterL1n/RobustVideoMatting has picked up 9.6k stars, AI Tools, Python. Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Official repository for the paper Robust High-Resolution Video Matting with Temporal Guidance. RVM is specifically designed for robust human video matting. Unlike existing neural models that process frames as independent images, RVM uses a recurrent neural network to process videos with temporal memory. RVM can perform matting in real-time on any videos without additional inputs. It achieves 4K 76FPS and HD 104FPS on an Nvidia GTX 1080 Ti GPU. The project was developed at ByteDance Inc.
News
[Nov 03 2021] Fixed a bug in train.py.
[Sep 16 2021] Code is re-released under GPL-3.0 license.
[Aug 25 2021] Source code and pretrained models are published.
[Jul 27 2021] Paper is accepted by WACV 2022.
Showreel
Watch the showreel video (YouTube, Bilibili) to see the model's performance.
All footage in the video are available in Google Drive.
Demo
Webcam Demo: Run the model live in your browser. Visualize recurrent states.
Colab Demo: Test our model on your own videos with free GPU.
Download
We recommend MobileNetv3 models for most use cases. ResNet50 models are the larger variant with small performance improvements. Our model is available on various inference frameworks. See inference documentation for more instructions.
Framework
Download
Notes
PyTorch
rvm_mobilenetv3.pth
rvm_resnet50.pth
Official weights for PyTorch. Doc
TorchHub
Nothing to Download.
Easiest way to use our model in your PyTorch project. Doc
CoreML does not support dynamic resolution. Other resolutions can be exported yourself. Models require iOS 13+. s denotes downsample_ratio. Doc, Exporter
All models are available in Google Drive and Baidu Pan (code: gym7).
PyTorch Example
Install dependencies:
pip install -r requirements_inference.txt
Load the model:
import torch
from model import MattingNetwork
model = MattingNetwork('mobilenetv3').eval().cuda() # or "resnet50"
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))
To convert videos, we provide a simple conversion API:
from inference import convert_video
convert_video(
model, # The model, can be on any device (cpu or cuda).
input_source='input.mp4', # A video file or an image sequence directory.
output_type='video', # Choose "video" or "png_sequence"
output_composition='com.mp4', # File path if video; directory path if png sequence.
output_alpha="pha.mp4", # [Optional] Output the raw alpha prediction.
output_foreground="fgr.mp4", # [Optional] Output the raw foreground prediction.
output_video_mbps=4, # Output video mbps. Not needed for png sequence.
downsample_ratio=None, # A hyperparameter to adjust or use None for auto.
seq_chunk=12, # Process n frames at once for better parallelism.
)
Or write your own inference code:
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
from inference_utils import VideoReader, VideoWriter
reader = VideoReader('input.mp4', transform=ToTensor())
writer = VideoWriter('output.mp4', frame_rate=30)
bgr = torch.tensor([.47, 1, .6]).view(3, 1, 1).cuda() # Green background.
rec = [None] * 4 # Initial recurrent states.
downsample_ratio = 0.25 # Adjust based on your video.
with torch.no_grad():
for src in DataLoader(reader): # RGB tensor normalized to 0 ~ 1.
fgr, pha, *rec = model(src.cuda(), *rec, downsample_ratio) # Cycle the recurrent states.
com = fgr * pha + bgr * (1 - pha) # Composite to green background.
writer.write(com) # Write frame.
The models and converter API are also available through TorchHub.
# Load the model.
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3") # or "resnet50"
# Converter API.
convert_video = torch.hub.load("PeterL1n/RobustVideoMatting", "converter")
Please see inference documentation for details on downsample_ratio hyperparameter, more converter arguments, and more advanced usage.
Training and Evaluation
Please refer to the training documentation to train and evaluate your own model.
Speed
Speed is measured with inference_speed_test.py for reference.
GPU
dType
HD (1920x1080)
4K (3840x2160)
RTX 3090
FP16
172 FPS
154 FPS
RTX 2060 Super
FP16
134 FPS
108 FPS
GTX 1080 Ti
FP32
104 FPS
74 FPS
Note 1: HD uses downsample_ratio=0.25, 4K uses downsample_ratio=0.125. All tests use batch size 1 and frame chunk 1.
Note 2: GPUs before Turing architecture does not support FP16 inference, so GTX 1080 Ti uses FP32.
Note 3: We only measure tensor throughput. The provided video conversion script in this repo is expected to be much slower, because it does not utilize hardware video encoding/decoding and does not have the tensor transfer done on parallel threads. If you are interested in implementing hardware video encoding/decoding in Python, please refer to PyNvCodec.
How active is development on PeterL1n/RobustVideoMatting?
The most recent commit recorded on PeterL1n/RobustVideoMatting was 2.6 years ago, based on the GitHub push timestamp. The repository has 1.2k forks — one of the better signals of community interest.
How does PeterL1n/RobustVideoMatting compare to other AI Tools projects?
PeterL1n/RobustVideoMatting is tracked by TopGit in the AI Tools category, with 9.6k GitHub stars and written in Python. Browse the AI Tools topic page on TopGit to compare it against similar projects by stars and activity.
How many stars does PeterL1n/RobustVideoMatting have?
PeterL1n/RobustVideoMatting has 9.6k GitHub stars — refresh the page for the live number, or check github.com/PeterL1n/RobustVideoMatting. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is PeterL1n/RobustVideoMatting open source?
Yes — PeterL1n/RobustVideoMatting ships under the GPL-3.0 license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/PeterL1n/RobustVideoMatting.
What is PeterL1n/RobustVideoMatting?
PeterL1n/RobustVideoMatting (PeterL1n/RobustVideoMatting) is a Python project on GitHub. From the project's own README: Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!
Where do I read more about PeterL1n/RobustVideoMatting?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/PeterL1n/RobustVideoMatting is the definitive source.
Read full README in the tab above.
Want a second opinion on RobustVideoMatting?
Ask an AI that can read this page — one click and you get its take on RobustVideoMatting.