Computer Vision and Pattern Recognition (CVPR 2020, Oral)
Single-shot Monocular RGB-D Imaging using Uneven Double Refraction
Andreas Meuleman*
Seung-Hwan Baek*
Felix Heide
Min H. Kim
KAIST
Princeton University
* Equal contribution
(a) Our prototype, consisting of a linear polarizer and a calcite crystal in front of a conventional camera. (b) and (c) estimated depth map and restored color image from the input. Our algorithm provides both sparse depth and clear RGB image within 34 ms.
CVPR 2020 oral presentation
Supplemental video for a real-time demo
Abstract
Cameras that capture color and depth information have become an essential imaging modality for applications in robotics, autonomous driving, virtual, and augmented reality. Existing RGB-D cameras rely on multiple sensors or active illumination with specialized sensors. In this work, we propose a method for monocular single-shot RGB-D imaging. Instead of learning depth from single-image depth cues, we revisit double-refraction imaging using a birefractive medium, measuring depth as the displacement of differently refracted images superimposed in a single capture. However, existing double-refraction methods are orders of magnitudes too slow to be used in real-time applications, e.g., in robotics, and provide only inaccurate depth due to correspondence ambiguity in double reflection.
We resolve this ambiguity optically by leveraging the orthogonality of the two linearly polarized rays in double refraction -- introducing uneven double refraction by adding a linear polarizer to the birefractive medium.
Doing so makes it possible to develop a real-time method for reconstructing sparse depth and color simultaneously in real-time.
We validate the proposed method, both synthetically and experimentally, and demonstrate 3D object detection and photographic applications.
BibTeX
@InProceedings{Meuleman_2020_CVPR,
author = {Andreas Meuleman and Seung-Hwan Baek and Felix Heide and Min H. Kim},
title = {Single-shot Monocular RGB-D Imaging using Uneven Double Refraction},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
Errata
The original supplemental document, published in CVPR 2020, includes an erratum, having n instead of (n+1) interation in Line 6 in Algorithm 1. We fix the erratum to provide the authors' copy (version 2) of the manuscript from this project website.
