Enhancing Thermal Images
We humans cannot see the infrared spectrum, so we measure it and convert it to a colour pallet we can see. This can be very handy to check the temperature of things.
These infrared images can be very difficult to interpret. Three objects that are blue, green and red could all be transmitting the same amount of infrared radiation and therefore appear as one single color on an image, so we cannot distinguish them anymore.
What if we could enhance a thermal image by using information from the RGB? Luckily we can! FLIR has already developed a nice way to do this that they call MSX. This basically takes two input images, an infrared image and a RGB image. It detects the edges from the RGB image and uses these edges to enhance the infrared image (Check the links if you want to learn more about FLIR/MSX). Let's see if we can mimic the results of MSX.
In this article I used a FLIR ONE Pro to capture the images. All information (both IR and RGB) is saved in the metadata of an image. With some Python code you can extract these data. You can even process all your images on the same temperature scale, custom bin the temperatures, 3D thermal models and a whole lot more. But that is a Medium story for an other day (I am not sure people are interested in this). Also we can remove the FLIR logo as we process the raw image values.
The FLIR ONE pro (and all their other sensors making use of MSX) comes with a pre aligned camera. What does this mean? The RGB camera and the IR camera capture information of the same area. Well, at least the RGB camera captures the same of the IR camera. If we extract the raw meta data and convert is back to an image we see that the RGB captures actually much more than the IR camera. The image below shows all the data captured by the IR sensor and the RGB sensor. The IR sensor has a resolution of 160x120 (but it is internally converted to 640x480) and the RGB sensor has a resolution of 1440x1080.
With the FLIR software the images are already aligned on the x axis, and from within the app you can change the y axis alignment (this can also be automated with some Python code). The y axis depends on your distance to the captured object. The color palet might be a bit of from the one directly used by the FLIR ONE Pro. I used the Matplotlib Inferno colormap for the image. Both the original and the processed thermal image range from 6 degrees to 12 degrees.
When looking at the FLIR MSX image, I think that they probably used some high pass filtering technique like a combined Sobel X/Y filter or a Laplacian filter to detect the edges of the RGB image. A high pass filter tends to retain high frequency information, in this case the edges of the image. I can write an entire article about it, but I think there are already plenty.
For now let's assume that our IR and RGB image are perfectly alligned. We start with the IR and RGB image and try to enhance the IR image with the RGB image.
First we load both images into Python:
# Import statements
import cv2
import numpy as np# Open the images
rgb = cv2.imread(‘RGB.png’)
ir = cv2.imread(‘IR.png’)
Now we have the two images loaded into Python we can try to detect the edges of the RGB image and overlay them on the IR image. We can also directly create an additional inverted mask, some people prefer the inverted result.
# Create a mask based on the Laplace operator
mask = cv2.Laplacian(rgb, cv2.CV_8U, ksize=3)
# Invert the mask (black to white, white to black)
mask_inv = cv2.bitwise_not(mask)
Now we have our edges we can actually use some simple Python code thanks to OpenCV and directly add the edges on the thermal image, at least for the normal mask. We can simple add the value array of the images together. If we look at the values of the image a black colour has a value of [0, 0, 0]. Any RGB color + [0, 0, 0] is still the same RGB color. Because we are working with Numpy arrays we could simple do image1 + image2, but this can yield values above 255 and we cause some problems. Instead we will use the cv2.add() operation. It will add the images together, but a value of e.g. 290 will be set to 255.
# Add the edges on top of the IR image
ir_enhanced = cv2.add(ir, mask)For the inverted mask we will use the bitwise_and() operation. This operator will keep the white pixels from the edge that it finds in the thermal image and ignores the black values of the inverted mask.
# Add the edges on top of the IR image
ir_enhanced_inv = cv2.bitwise_and(ir, mask_inv)
Compared to the MSX filter from FLIR is does not look that nice, but is still does the job quite well. I personally use a slightly different mask that is a bit more robust when varying the object distance.
But it is as simple as that, pre aligned images are key here!
