DE-NeRF: Deformable Neural Radiance Fields using RGB and Event Cameras

Qi Ma1, Danda Pani Paudel1, Ajad Chhatkuli 1, Luc Van Gool 1,
1ETH Zurich
Paper arXiv Video Code

Abstract

We present the first method combining RGB and events capable of photorealistically reconstructing a non-rigidly deforming scene using monocular camera.

Modeling Neural Radiance Fields for fast-moving deformable objects from visual data alone is a challenging problem. A major issue arises due to the high deformation and low acquisition rates. To address this problem, we propose to use event cameras that offer very fast acquisition of visual change in an asynchronous manner. In this work, we develop a novel method to model the deformable neural radiance fields using RGB and Event cameras. The proposed method uses the asynchronous stream of events and calibrated sparse RGB frames. In this setup, the pose of the individual events-required to integrate them into the radiance fields-- remains to be unknown. Our method jointly optimizes the pose and the radiance field, in an efficient manner by leveraging the collection of events at once and actively sampling the events during learning.

We show that DE-NeRF can turn RGB-Events pair into deformable NeRF models that allow for highly-deformable scenes, which we dub "de-nerf". We evaluate our method on both realistically rendered and real-world datasets demonstrate a significant benefit of the proposed method over the state-of-the-art and the compared baseline.

Video

Fixed Viewpoint

Using DE-NeRF you can fixed the viewpoint and see the dynamics scene.

Fixed Time

Using DE-NeRF you can fixed the time and change viewpoint to other novel views.

Related Links

We build our work upon excellent deformable neural radiance field work Nerfies.

There's also NeRF using colorful events on static scenes: EventNeRF: Neural Radiance Fields from a Single Colour Event Camera introduces similar Event rendering loss as ours. However the use fixed windows representation and assume poses of events is known.

E-NeRF and EV-NeRF focus also on static scene and using asynchronous events sampling.

BibTeX

@misc{ma2023deformable,
      title={Deformable Neural Radiance Fields using RGB and Event Cameras}, 
      author={Qi Ma and Danda Pani Paudel and Ajad Chhatkuli and Luc Van Gool},
      year={2023},
      eprint={2309.08416},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}