VCLab

 

RESEARCH AREAS   PEOPLE   PUBLICATIONS   COURSES   ABOUT US
Home / Publications

indicator

Computer Vision and Pattern Recognition (CVPR 2024)

 
OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos
 
  Dongyoung Choi Hyeonjoong Jang Min H. Kim  
 
  KAIST  
 
 
  We introduce omnidirectional local radiance fields for photorealistic view synthesis of static scenery from 360-degree videos. Our method effectively removes dynamic objects (including the photographer) without manual interaction. Also, it achieves high-resolution details in the inpainted regions by means of bidirectional observations of omnidirectional local radiance fields.  
     
   
  Supplemental video
     
   
  CVPR 2024 presentation
   
  Abstract
   
 

Omnidirectional cameras are extensively used in various applications to provide a wide field of vision. However, they face a challenge in synthesizing novel views due to the inevitable presence of dynamic objects, including the photographer, in their wide field of view. In this paper, we introduce a new approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can render static-only scene views, removing and inpainting dynamic objects simultaneously. Our approach combines the principles of local radiance fields with the bidirectional optimization of omnidirectional rays. Our input is an omnidirectional video, and we evaluate the mutual observations of the entire angle between the previous and current frames. To reduce ghosting artifacts of dynamic objects and inpaint occlusions, we devise a multi-resolution motion mask prediction module. Unlike existing methods that primarily separate dynamic components through the temporal domain, our method uses multi-resolution neural feature planes for precise segmentation, which is more suitable for long 360-degree videos. Our experiments validate that OmniLocalRF outperforms existing methods in both qualitative and quantitative metrics, especially in scenarios with complex real-world scenes. In particular, our approach eliminates the need for manual interaction, such as drawing motion masks by hand and additional pose estimation, making it a highly effective and efficient solution.

     
  BibTeX
 
@InProceedings{Choi_2024_CVPR,
   author = {Dongyoung Choi and Hyeonjoong Jang and Min H. Kim},
   title = {OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos},
   booktitle = {IEEE Conference on Computer Vision and 
      Pattern Recognition (CVPR)},
   month = {June},
   year = {2024}
} 
   
   
icon
Preprint paper:
PDF (9.6MB)
icon
Supplemental
material #1:
PDF (9.6MB)
www GitHub
code
 

Hosted by Visual Computing Laboratory, School of Computing, KAIST.

KAIST