Home / Publications


ACM SIGGRAPH Asia 2021 (Transactions on Graphics)

Differentiable Transient Rendering
  Shinyoung Yi   Donggun Kim   Kiseok Choi  
  Adrian Jarabo §   Diego Gutierrez §   Min H. Kim  
  KAIST   § Universidad de Zaragoza, I3A    
  Our general-purpose differentiable transient rendering framework allows to compute derivates of complex, multi-bounce transient sequences with respect to scene parameters, even in the presence of discontinuous light and sensor functions. The figure shows steady-state and transient renders of a table-top scene with light coming from the left, then being bounced back by two off-camera diffusers. The bottom row shows the transient light transport scene derivatives with respect to the index of refraction of the red tea. Please refer to Figure 10 for additional results optimizing the index of refraction of the tea in the teapot, which in turn changes the speed of light.  
  Supplemental video  
  Summary presentation video (5 minutes)  
  Full presentation video (17 minutes)  

Recent differentiable rendering techniques have become key tools to tackle many inverse problems in graphics and vision. Existing models, however, assume steady-state light transport, i.e., infinite speed of light. While this is a safe assumption for many applications, recent advances in ultrafast imaging leverage the wealth of information that can be extracted from the exact time of flight of light. In this context, physically-based transient rendering allows to efficiently simulate and analyze light transport considering that the speed of light is indeed finite. In this paper, we introduce a novel differentiable transient rendering framework, to help bring the potential of differentiable approaches into the transient regime. To differentiate the transient path integral we need to take into account that scattering events at path vertices are no longer independent; instead, tracking the time of flight of light requires treating such scattering events at path vertices jointly as a multidimensional, evolving manifold. We thus turn to the generalized transport theorem, and introduce a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions. Last, we present results in several challenging scenarios where the time of flight of light plays an important role such as optimizing indices of refraction, non-line-of-sight tracking with nonplanar relay walls, and non-line-of-sight tracking around two corners.

  author  = {Yi, Shinyoung and Kim, Donggun and Choi, Kiseok and 
             Jarabo, Adrian and Gutierrez, Diego and Kim, Min H.},
  title   = {Differentiable Transient Rendering},
  journal = {ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2021)},
  year    = {2021},
  volume  = {40},
  number  = {6},
Authors' preprint paper:
PDF (3.1MB)
PDF (951KB)
www SIGGRAPH Asia 2021 slides
www GitHub
www ACM
Digital Library

Hosted by Visual Computing Laboratory, School of Computing, KAIST.