VCLab

 

RESEARCH AREAS   PEOPLE   PUBLICATIONS   COURSES   ABOUT US
Home / Publications

indicator

ACM SIGGRAPH 2021 (Transactions on Graphics)

 
DeepFormableTag: End-to-end Generation and Recognition of Deformable Fiducial Markers
 
                     
  Mustafa B. Yaldiz   Andreas Meuleman   Hyeonjoong Jang   Hyunho Ha   Min H. Kim  
 
KAIST
 
 
  Fig. 1. (a) shows our learned deformable markers at the leftmost column and presents detection of multiple deformable markers on curved surfaces. All the deformed markers are detected successfully. (b) presents the captured results of 3D scanning (top) using a 2D array of multiple fiducial markers as structured light patterns (bottom). (c) shows motion tracking of human body motion using our deformable markers. (d) demonstrates rendering of a virtual object on the detected markers with strong motion blur. Refer to the supplemental demo video for more results.  
   
   
  Oral presentation at ACM SIGGRAPH 2021
   
   
  Supplemental material
   
   
  Abstract
   
 

Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. The size of a message or identification code is limited by the spatial resolution of binary patterns in a marker. Markers often fail to be recognized due to various imaging artifacts of optical/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.

   
  BibTeX
 
  @Article{CNNMarker:SIG:2021,
  author  = {Mustafa B. Yaldiz and Andreas Meuleman and
             Hyeonjoong Jang and Hyunho Ha and Min H. Kim},
  title   = {DeepFormableTag: End-to-end Generation and
             Recognition of Deformable Fiducial Markers},
  journal = {ACM Transactions on Graphics (Proc. SIGGRAPH 2021)},
  year    = {2021},
  volume  = {40},
  number  = {4},
  }    
   
   
icon
Preprint paper:
PDF (35.3MB)
icon
Supplemental
document:
PDF (288KB)
SIGGRAPH 2021 slides
PDF (10.3 MB)
www GitHub (TBA)
website
www KAIST marker dataset (TBA)
www ACM
Digital Library
(TBA)
 

Hosted by Visual Computing Laboratory, School of Computing, KAIST.

KAIST