Study of adapted instrumentation architectures and measurements post-processing approaches for infrared thermography applied to thermal monitoring in outdoor conditions

Technological progress in uncooled infrared focal plane array sensors has contributed significantly to enlarge the scope of applications of such sensing technique in many domains: Leisure, Manufacturing, Process Survey, Building insulation diagnostic, Civil Engineering, Road works, etc. Different outdoor situations and objects of interest monitored by an in-house designed measurement architecture are presented. Designed instrumentation architectures and measurements correction from varying environmental conditions and geometrical considerations are discussed. A first step toward joint estimation of emissivity and temperature is introduced for outdoor applications. Then moving object detection by an AI approach applied on thermal image sequences is also presented and discussed. Finally conclusion and perspectives are proposed.


INTRODUCTION
Technological progresses in uncooled infrared focal plane array sensors have bring up new opportunities towards remote and low-cost radiatives fluxes m easurements a nd n ew a pplications i n c ivil e ngineering i nfrastructures or critical systems monitoring. However, one concern about most infrastructures (or other real targets) is their subject to space and time environmental variable parameters in outdoor conditions. The underlying temperature estimation will therefore depend on their intrinsic thermo-optical properties, the varying environmental conditions, geometrical considerations and the chosen instrumentation' performances. As a consequence, the purpose is to identify a number of key parameters thanks to a multi-sensors instrumentation and correct the temperature estimation by merging data from those different s ources. T he p roposed m ethodology presented in this paper relies on three main steps: the calibration of the used instruments, the choice of a multi-sensor instrumentation design and the post-processing of the measurements using the gathered data. Those steps can be fulfilled t hanks t o a i n-house d eveloped D IARITsup f ramework t hat s upervises l ive m easurements, performs Digital Twins models computations and predictions for structures (e.g. other targets) monitoring.
In this paper, a multi-sensor instrumentation design to correct measurements from infrared cameras is presented. Such correction is introduced on the conversion from radiative fluxes t o t emperature e quations a s well as the use of online data to estimate missing environmental parameters. Then, geometrical considerations are discussed and a step toward joint estimation of emissivity and temperature is introduced. Furthermore, moving objects detection by an AI based approach is also introduced . Finally, conclusion and perspectives are proposed.

Radiometric equation model for temperature conversion
An infrared camera measures the radiative flux received on its detectors. However, this radiative flux depends on the geometry of the scene, the observed objects own properties and the meteorological conditions. To overcome the lack of knowledge about environmental and meteorological parameters, an instrumentation made of various sensors is proposed. In fact, equation (1) conjointly with Figure 1 shows the involved parameters in the simplified radiometric equation to solve to retrieve the radiance of the observed object. Simplified radiometric equation at pixel (i, j) for a given spectral band ∆λ and direction ⃗ d: Therefore, environmental quantities can have an important impact on the estimation of temperature. Figure  2 shows the sensitivity of the temperature conversion towards the environmental reflections and the emissivity. In this figure, the error on the estimation of the temperature is shown for different errors on the environment temperature for two different ground truths (up / down lines) and two different emissivities (left / right columns). One can see that both the quantities play an important role on the estimation of the temperature and therefore needs particular care when getting used in the previous equation.
For determining some of the environmental quantities, a weather station can be used. The reflected, optics and atmospherics luminance values can be computed by injecting some knowledge about the monitored objects, components of the infrared system and amospheric conditions. Such step can be embedded at the infrared system level knowing that in such case it depends on the hardware and software of the manufacturer of the infrared camera used. For instance, code samples for FLIR camera conversion parameters can be found. 1 Prior to the actual instrumentation, sensors are calibrated and more particularly the camera to be able to convert the temperature from digital levels, as presented in Section 3. For handling the emissivity, a simultaneous estimation of temperature and emissivity algorithm is used and introduced in section 4.
Such complex instrumentation with a large variety of sensors needs adapted softwares to record, sometimes perform live processing and visualize acquired data. This is achieved with the designed and developed DiaritSUP framework. Figure 2. Difference between the true object temperature and estimated one depending on the error made on the environment temperature and emissivity. Here, the true object is at 293.15K, emissivity at 0.93 (left) and 0.6 (right). How to read: on the left the true object's temperature and emissivity. A point on the image represents the difference between the true object temperature and the estimated one withTenv (x-axis) andε obj (y-axis) values. The images represent different environment temperatures values.

DiaritSUP framework
To handle the amount of data gathered in such instrumentation, a framework called DiaritSUP has been designed and developed. It follows the concept of "system of systems". It aims at assimilating, processing and recording data, managing computation models, returning real-time information and visualizing data. Thanks to its modelview-controller (MVC) architecture, the framework is easily extensible, can be customized and connected to existing applications. It integrates three types of modules, represented in Figure 4. "Instrumentation" with the forecast and device managers that gather data from local sensors and external data providers such as Météo-France Geoservices, Copernicus, METAR, etc. "Computation" with the model manager that launches Structural Health Monitoring (SHM) Digital Twins models at the right time. Finally, a HDF5 Handler records data and metadata in Hierarchical Data Format (HDF5) in Single-Writer-Multiple-Readers feature to save all data (computed and gathered) and enable the live visualization through the User Interface (UI) made on a webserver (see Figure 3). A typical workflow is presented in Figure 5 where the components of the framework are more detailed and the data flow is illustrated. The main communications and access to the data are done with a REST API (HTTPS). The communication between the machines can be done through different protocols such as SSH, SCP, Modbus TCP/IP, etc. The framework deployment is made on Ansible, an information technology automation tool that makes the system configuration and deployment easy thanks to simple tasks defined in a configuration file. Code is hosted on our own Gitlab instance at Inria which enables the use of Continuous Integration and Deployment tools (CI/CD) and the setup of an automatic update mechanism. Finally, for robustness purposes, examples of Vagrant configuration are provided to create virtual machines (Digital Twins) to test deployment before pushing to production.

CALIBRATION
All sensors used in the instrumentation needs to be and are generally calibrated by the vendor with an up-to-date calibration certificate.
For the camera though, we perform the calibration in our laboratory, by using a black-body calibration source and the standard procedure usually found in the literature. This source is put in a climatic chamber which enables the control of the ambient temperature. Measurements are then performed for different blackbody source temperatures that evolves in our range of interest. Then, the Planck version of the Sakuma-Hattori equation 2 is used for which uncertainties have been well established. 3 The real field of view of the camera may also alter the estimation of the temperature. When dealing with particular region of interests, a spatial calibration has to be done. Many different methods can be found in the literature for the infrared thermography field that are mainly inspired from computer vision field.

SIMULTANEOUS ESTIMATION OF EMISSIVITY AND TEMPERATURE: NUMERICAL STUDY
Once the received flux can be estimated thanks to the gathered data and equation 1, the emissivity parameter remains unkown. The problem of estimating at the same time temperature and emissivity is a challenging task often addressed in the litterature. [4][5][6][7] An algorithm that is particularly suited for the live and in-situ estimation of temperature and emissivity is described in. 8 The equations are derived to a linear time variant (LTV) system. Such derivation enables the use of Bayesian methods from the automation field such as Marginalized Particle Filter which aims at estimating the state vector (temperature here) and the parameters (emissivity and dynamic evolution of the temperature here) at the same time.
To simplify the writting and the problem, let assume that only the temperature and the emissivity remain to be estimated. Therefore, we can write the previous equation 1 to: Therefore, the simultaneous estimation of emissivity and temperature algorithm can be applied with the estimated environmental parameters, where C opt,atm is the optical and atmospherical contribution sum.
An example of the estimation of temperature time evolution with such workflow using numerical simulations carried out with a modified implementation of the progressive radiosity 9, 10 algorithm is proposed in Figure 6.

MOVING OBJECT DETECTION: EXAMPLE ON TRAIN AXLES
As an example of AI based image processing, hot boxes detection results obtained by an improved YOLO-v4 model 11 are presented and discussed in this paragraph. In this example, 6365 Infrared thermal images (extracted from on site thermal images sequence) were the data used to test the Y-UC (Yolo -Uncooled Camera). The same data were used to test the baseline models. In this experiments, the target confidence threshold (conf-thresh) was taken equal to 0.45 and IoU threshold to 0.36. The results are shown in Tab. 1. Visual representations of the Y-UC results are presented in Fig. 7. Modification studied and implemented in YOLO-v4 contributed to improve the performance of the global detetection. However, both integrating CBAM and optimizing PANet made more substantial improvements to the network, with precisely better recall and F1score. On the other hand, the precision of the baseline network was found, in most cases, slightly superior than the modified networks. That means low confusing of the hot boxes with the rest of the train body. Nevertheless, with weak recall, the baseline network missed to detect some ground truth hot boxes. In contrast, the improved YOLO-v4 showed a good balance between precision and recall which justify the high F1-score. Based on the detected hot boxes, an axle counter was also studied. The proposed approach rely simply on the following steps: 1. Define a detection zone. 2. Run the improved YOLO-V4 detection algorithm. 3. Crop the image with respect to the bounding box. 4. Update with a new frame and repeat steps 2 and 3. Then, calculate the structural similarity index (SSIM) 12 between the cropped images. In the present study, an index superior to 0.8 was estimated sufficient to validate the count.

CONCLUSION & PERSPECTIVES
A new surveillance instrumentation architecture has been designed for the thermal monitoring of transport infrastructures or any other object of interest, in outdoor conditions. It combines infrared camera sensors with in-situ or cloud-based environmental measurement data to perform measurement corrections on infrared thermal images. It also allows the integration of prediction models or its coupling with other structural health monitoring models. Its deployment includes different stages at the laboratory or in situ level. Moreover, a numerical study of a promising method for joint estimation of emissivity and temperature, in outdoor conditions, was briefly presented. Finally, for the detection of moving objects, an improved YOLO-V4 was presented. The obtained results show that the upgraded YOLO-V4 outperforms the original one. Future research work will focus on studying the robustness and performance of the new architecture currently deployed on a real transport infrastructure. In addition, the new method for joint estimation of emissivity and temperature will be evaluated in the laboratory and then in an outdoor configuration under controlled conditions. Finally, for the detection of moving objects, we plan to improve the information on the blurring characteristics in the infrared images and to compare the temperatures of the detected axles with a reference system on a new dedicated real railway test site.