Our reconstruction of a large crane structure, based on images collected by a Skydio drone. Thermal imaging with 3D reconstruction can be used for building and infrastructure inspection, among many other applications ranging from agriculture to search and rescue.

Abstract

Thermal imaging has a variety of applications, from agricultural monitoring to building inspection to imaging under poor visibility, such as in low light, fog, and rain. However, reconstructing thermal scenes in 3D presents several challenges due to the comparatively lower resolution and limited features present in long-wave infrared (LWIR) images. To overcome these challenges, we propose a unified framework for scene reconstruction from a set of LWIR and RGB images, using a multispectral radiance field to represent a scene viewed by both visible and infrared cameras, thus leveraging information across both spectra. We calibrate the RGB and infrared cameras with respect to each other, as a preprocessing step using a simple calibration target. We demonstrate our method on real-world sets of RGB and LWIR photographs captured from a handheld thermal camera, showing the effectiveness of our method at scene representation across the visible and infrared spectra. We show that our method is capable of thermal super-resolution, as well as visually removing obstacles to reveal objects that are occluded in either the RGB or thermal channels.

Broad-Spectrum Radiance Fields

Existing radiance field models, including NeRF and its many variations, typically focus on modeling radiance in the visible spectrum as three color channels (red, green, and blue). These models implicitly assume that each point in space is equally absorptive of all three of these colors of light. While this is a good approximation for RGB visible light, to which most materials are either opaque or transparent, there are certain materials, such as stained glass, for which the approximation is no longer valid.
When we begin to consider radiance fields across a wider spectrum, including our setting of RGB and LWIR thermal radiance field modeling, we find that more materials exhibit differing absorption behavior. We model this behavior by explicitly endowing each spatial location with separate densities (absorption coefficients) for each wavelength, while introducing regularization to encourage these wavelength-specific densities to remain similar for most materials.

Optimization and Regularization

We represent the RGBT scene as a radiance field


To optimize this, we minimize the following objective:


where the 1st and 2nd terms are the standard pixel-wise photometric L2 losses against calibrated ground-truth images, the 3rd term is an L1 regularizer encouraging the RGB and thermal densities to deviate from each other only at sparse 3D positions, the 4th term is is a variation on the cross-channel prior adapted to our radiance field reconstruction task, and the 5th term is a pixelwise total variation regularizer on thermally-unsupervised rendered thermal views.

Additional Results

Revealing Hidden Objects

We can remove occluding objects from RGB or thermal views, thus revealing objects hidden behind other objects by rendering only the parts of the scene with RGB and thermal densities sufficiently similar to each other. Precisely, we render RGB and thermal scenes respectively with densities


using a minimum allowed magnitude of difference between the RGB and thermal densities in the rendered image.

Ground truth RGB Hidden RGB object revealed
Ground truth thermal Hidden thermal object revealed

Citation

Acknowledgements

This material is based upon work supported by the National Science Foundation under award number 2303178 to SFK. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
The website template was borrowed from Michaël Gharbi.