VCLab

 

RESEARCH AREAS   PEOPLE   PUBLICATIONS   COURSES   ABOUT US
Home / Publications

indicator

European Conference on Computer Vision (ECCV) 2022

 
FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras
 
  Andreas Meuleman   Hakyeong Kim  
  James Tompkin   Min H. Kim  
 
  KAIST   Brown University  
 
   
  (a) Multi-modal smartphone imaging. (b) Reference RGB image. (c) ToF depth reprojected to the reference. (d) Our depth from floating fusion.  
     
   
  ECCV 2022 presentation
   
  Abstract
   
 

High-accuracyper-pixeldepthisvitalforcomputationalpho- tography, so smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. However, producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. Fusing RGB stereo and ToF information is a promising direction to overcome these issues, but a key problem remains: to provide high-quality 2D RGB images, the main color sensor's lens is optically stabilized, resulting in an unknown pose for the floating lens that breaks the geometric relationships between the multimodal image sensors. Leveraging ToF depth estimates and a wide-angle RGB camera, we design an automatic calibration technique based on dense 2D/3D matching that can estimate camera extrinsic, intrinsic, and distortion parameters of a stabilized main RGB sensor from a single snapshot. This lets us fuse stereo and ToF cues via a correlation volume. For fusion, we apply deep learning via a real-world training dataset with depth supervision estimated by a neural reconstruction method. For evaluation, we acquire a test dataset using a commercial high-power depth camera and show that our approach achieves higher accuracy than existing baselines.

   
  BibTeX
 
@InProceedings{Meuleman_2022_ECCV,
author = {Andreas Meuleman and Hakyeong Kim
and James Tompkin and Min H. Kim},
title = {FloatingFusion: Depth from ToF and
Image-stabilized Stereo Cameras},
booktitle = {European Conference on
Computer Vision (ECCV) 2022},
month = {August},
year = {2022}
}
   
   
icon
Preprint paper:
PDF (2.8MB)
icon
Supplemental
document:
PDF (15MB)
www GitHub code
(TBA)
 
 

Hosted by Visual Computing Laboratory, School of Computing, KAIST.

KAIST