Researchers at Purdue University and Michigan State University, USA, have combined thermal physics, infrared imaging and machine learning in their “heat-assisted detection and ranging” system to improve the nighttime vision of devices such as self-driving cars. [Image: X. Zhou, F. Bao and Z. Jacob / Purdue University]
Scientists in the United States have developed a new kind of thermal imaging that could allow robots and self-driving cars to see their environment at nighttime as well as they can during the day (Nature, doi: 10.1038/s41586-023-06174-6). The researchers say the system uses artificial intelligence (AI) to discern surrounding objects’ temperature and texture within a jumbled stream of infrared photons, allowing it to generate images with accurate depths and otherwise hidden details.
Pros and cons of thermal imaging
Thermal imaging captures the infrared radiation emitted by all objects simply as a result of their finite temperature. This allows those objects to be seen even without a source of visible light, such as the sun. What’s more, its exploitation of naturally occurring electromagnetic waves means that this type of imaging is passive—in contrast to radar, sonar and lidar. That passive nature could prove handy if lots of automated devices are to operate without producing excessive interference or harming nearby humans.
But thermal imaging also has a major limitation. The fine details and texture within a scene come not from the infrared radiation emitted directly from a given object, but instead from the radiation generated elsewhere that is then scattered by the object. The latter waves are usually far less intense than the former, so details tend to get washed out and people appear like faceless ghosts.
HADAR system
Zubin Jacob [Image: Z. Jacob]
In the latest work, Zubin Jacob and Fanglin Bao at Purdue University, with colleagues at Purdue and Michigan State University, have devised a way to disentangle the direct and scattered emissions. Their scheme involves splitting heat signals into three components—temperature (T), emissivity (e) and texture (X). This “TeX decomposition” represents temperature through variations in an image’s color intensity, while texture—the surface geometry of an object—is seen as differences in brightness. Emissivity, a measure of how closely a surface resembles a perfect blackbody emitter, varies from one material to another and is represented by color type.
Jacob and colleagues set up their “heat-assisted detection and ranging” system (HADAR) by first building up a library containing the emissivity of all materials that could conceivably be seen by a wandering device. They then used this information to train a neural network such that, when fed with fresh, nondecomposed data from an infrared camera at a number of different frequencies, the network could reconstruct the scene and give each pixel within it the correct color, color intensity and brightness.
“The training uses synthetic data using a home-built thermal simulator, as well as real-world data,” says Jacob. “However, real-world data is scarce, so we have to embed thermal physics into the network.”
Improved accuracy
The researchers built two prototype HADAR devices. One used a relatively cheap thermal camera and 10 infrared filters, while the other, operated with the US Defense Advanced Research Projects Agency (DARPA) and the US Army, involved a much more expensive hyperspectral imager with 256 frequency bands.
Using their simpler prototype, the researchers showed that, when imaging a person standing in front of a car, HADAR could reduce ranging errors by a factor of about 100 compared with straightforward thermal imaging.
Jacob and colleagues subjected their devices to a number of different tests, including those designed to measure distance. Using their simpler prototype, they showed that, when imaging a person standing in front of a car, HADAR could reduce ranging errors by a factor of about 100 compared with straightforward thermal imaging. They also found that their more sophisticated system could outperform normal thermal imaging enhanced by AI—showing in fact that the absence of “ghosting” allowed HADAR to gauge distances in real-world scenes at night almost as accurately as RGB stereovision can do during the day.
What’s more, the researchers showed that even their simpler device could pick out distinct types of objects at night better than rival systems. In this case, they focused on a black car with two figures standing next to it—one a human being and the other a life-size cardboard cutout of Albert Einstein. They found that only HADAR could image the car and distinguish between the real and pretend human figures—thanks to their different surfaces—while neither optical imaging nor sparse lidar were able to make the distinction. Sparse lidar also struggled to detect the car.
Hurdles to overcome
Manish Bhattarai and Sophia Thompson argue that HADAR “will find immediate applications” when it comes to autonomous vehicles and robotics.
In a commentary in Nature to accompany the paper, Manish Bhattarai at Los Alamos National Laboratory and Sophia Thompson at the University of New Mexico, both in the United States, argue that HADAR “will find immediate applications” when it comes to autonomous vehicles and robotics. They also reckon that its ability to separate temperature from emissivity and texture data could prove a boon in health care; for example, allowing people’s body temperature to be measured efficiently at airports and other venues.
However, Bhattarai and Thompson caution that much remains to be done before HADAR can operate in the real world. One hurdle, they say, is cost—the hyperspectral imager in the second prototype costing over US$1 million. They also point out that the system must be calibrated on the fly to ensure that temperature variations don’t impede reconstruction. Indeed, they say that environmentally induced variations in temperature, emissivity and texture can hamper identification of objects within a scene—although they add that the system’s library could be modified to account for these changes.
Jacob says that he and his colleagues plan to overcome these problems “within the next 3 to 5 years” by improving both hardware and software. “We are working on a new generation of thermal imagers, which can be low cost and compact without the need for cooling,” he says.