Home / Publications


ACM SIGGRAPH 2021 (Transactions on Graphics)

DeepFormableTag: End-to-end Generation and Recognition of Deformable Fiducial Markers
  Mustafa B. Yaldiz   Andreas Meuleman   Hyeonjoong Jang   Hyunho Ha   Min H. Kim  
  Fig. 1. (a) shows our learned deformable markers at the leftmost column and presents detection of multiple deformable markers on curved surfaces. All the deformed markers are detected successfully. (b) presents the captured results of 3D scanning (top) using a 2D array of multiple fiducial markers as structured light patterns (bottom). (c) shows motion tracking of human body motion using our deformable markers. (d) demonstrates rendering of a virtual object on the detected markers with strong motion blur. Refer to the supplemental demo video for more results.  
  Oral presentation at ACM SIGGRAPH 2021
  Supplemental material

Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. The size of a message or identification code is limited by the spatial resolution of binary patterns in a marker. Markers often fail to be recognized due to various imaging artifacts of optical/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.

  author  = {Mustafa B. Yaldiz and Andreas Meuleman and
             Hyeonjoong Jang and Hyunho Ha and Min H. Kim},
  title   = {DeepFormableTag: End-to-end Generation and
             Recognition of Deformable Fiducial Markers},
  journal = {ACM Transactions on Graphics (Proc. SIGGRAPH 2021)},
  year    = {2021},
  volume  = {40},
  number  = {4},
Preprint paper:
PDF (35.3MB)
PDF (288KB)
SIGGRAPH 2021 slides
PDF (10.3 MB)
www GitHub code &
www ACM
Digital Library

Hosted by Visual Computing Laboratory, School of Computing, KAIST.