Robust Omnimatte with 3D Background Modeling
OmnimatteRF
Robust Omnimatte with 3D Background Modeling
ICCV 2023We present OmnimatteRF, which creates mattes with associated effects like shadows from in-the-wild videos with coarse masks.
Abstract
Video matting has broad applications, from adding interesting effects to casually captured movies to assisting video production professionals. Matting with associated effects like shadows and reflections has also attracted increasing research activity, and methods like Omnimatte have been proposed to separate foreground objects of interest into their own layers. However, prior works represent video backgrounds as 2D image layers, limiting their capacity to express more complicated scenes, thus hindering application to real-world videos. In this paper, we propose a novel video matting method, F2B3, that combines 2D foreground layers and a 3D background model. The 2D layers preserve the details of the subjects, while the 3D background robustly reconstructs scenes in real-world videos. Extensive experiments demonstrate that our method reconstructs with better quality on various videos.
Method
OmnimatteRF extends Omnimatting to a larger variety of real-world videos with a combination of 2D foreground layers and a background radiance field.
Data & Results
Download our Movies and Wild datasets here: Google Drive
The Movies dataset contains 5 sequences from 3 Blender movies. They come with ground truth camera poses, object masks, and clean background videos.
Vidoes in Wild are captured by us and come with reconstructed camera poses and coarse masks.
Other videos used in the paper are obtained from their authors: DAVIS, Kubric, dogwalk
Presentation Video
Coming soon!
BibTeX
@InProceedings{Lin_2023_ICCV,
author = {Geng Lin and Chen Gao and Jia-Bin Huang and Changil Kim and Yipeng Wang and Matthias Zwicker and Ayush Saraf},
title = {OmnimatteRF: Robust Omnimatte with 3D Background Modeling},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023}
}