ToFSim: Real-time ToF Camera Simulation...

Commercial 3D imaging systems such as time-of-

flight (ToF) cameras provide depth information in real-time

with decent resolution. In robotics, they have been widely

adopted to provide sensory input to various applications, such

as mapping, segmentation, and object manipulation. As simu-

lation environments become indispensable in design processes,

it is also important to have accurate sensor simulations. Several

effects in the real sensor lead to substantial differences between

standard synthetic depth images, which are easily obtainable in

most simulation frameworks based on methods from computer

graphics, and real ToF camera depth images. In this paper, we

focus on the need for realistic modeling of the characteristics

of ToF cameras and introduce an accurate, user-friendly, real-

time simulation model. The simulation framework is based

on Unity 3D, where scenarios including realistic items and

material distributions can be built. A standard camera object is

used to generate synthetic ToF data, according to the material

properties of the simulated environment. The model considers

the decrease of the signal strength for larger distances, the so-

called flying pixel effect that can occur at object boundaries,

sensor pixel cross-talk, range ambiguity, and saturation effects.

The approach’s validity is demonstrated by comparing real and

simulated data. With update rates in the tens of Hz, the system

enables real-time operation and generation of massive data

with known ground truth, e.g., for sim2real machine learning

approaches and realistic digital twins.