Main
Main
Main
A R T I C L E I N F O A B S T R A C T
Keywords: Infrared images are often degraded by the issue of random and stripe noises in the business infrared focal plane
Infrared imaging system imaging process. In this study, we proposed a novel infrared image destriping model with unidirectional and
Destriping bidirectional total variation regularization. From the destriped infrared images, the stripe noise only exists in
Image restoration
unidirection. The nature of unidirectional noise is leveraged to converted the total variation regularization and
Regularization
Total variation
constructed the proposed model. For the random noise, the bidirectional total variation is introduced, which can
preserve the image edge and texture details. Furthermore, the alternating direction method of multiplier algo
rithm is introduced to solve the infrared image destriping model. The proposed model is executed on several
public infrared image datasets, thus outperforming the traditional destriping models in the qualitative evaluation
index.
1. Introduction edge can be seen in Fig. 1(c). The strip noise in the infrared focal plane
imaging system can be classified into three types, such as detector-to-
Infrared focal plane imaging systems [1,2] can capture the important detector stripes, periodic stripes and mirror-side stripes. In this work,
cue and image detail in the night vision environment. The infrared im we plan to find an effective stripe noise suppression method to remove
ages are widely used in the fields of science and engineering, such as those three types of stripe noise. The unidirectional nature is revealed in
wildlife protection [3], security monitoring [4], contact-point detection this work to formulate the stripe noise.
[5], head pose estimation [6,7], facial express recognition [8,9] and Benefiting from the hardware performance improvement, the image
small target recognition [10]. Due to the aging of imaging device, the resolution can be significantly improved, and obtained an appealing
acquired images often suffer from the stripe and random noises result [12,15,16]. However, the hardware often costs more and is hard
[2,11,12]. The image resolution is degraded by the objective aperture to use widely. The infrared destriping task is often modeled as an ill-
and imaging component [13]. The infrared image degradation process posed issue [14]. The stripe lines and clean image needs to estimate
can be mathematically formulated as, from the degraded infrared images simultaneously. The noise filter-
based paradigms are proposed to remove the stripes, but the image
u(x, y) = f (x, y) + s(x, y) + ε(x, y) (1)
detail is also smoothed in the iteration process.
where the symbols f and u denotes the clean infrared image and Generally, all infrared image destriping models [2,17,18] can be
degraded stripe image [2,14]. The random noise and stripe noise are grouped into three types, such as filter-based destriping (FBD), model-
represented by the last two symbols s and ε, respectively. In Fig. 1, the based destriping (MBD), and deep learning-based destriping (DLBD).
stripe noise in the infrared image along with the x-axis, while the stripe- For the FBD methods, numerous filters are proposed to remove the stripe
crossed direction is denoted as y-axis. The soil in Fig. 1(a) is degraded by line with the random noise [19–21]. In [22], Munch et al. proposed a
the stripe noise. We calculated the image gradient of the infrared image powerful wavelet-Fourier filter to suppress the ring stripe and line stripe
along the x-axis and y-axis. The unidirectional gradients are shown in simultaneously. The stripes in x-axis and y-axis directions can be
Fig. 1(b) and (c). The stripe lines only exist in Fig. 1(b), and the clean removed with phase projections technology. In [1], Shao et al. proposed
* Corresponding author at: Wuhan Donghu University, Wuhan, Hubei 430212 China.
E-mail address: [email protected] (H. Fang).
https://doi.org/10.1016/j.infrared.2021.104005
Received 12 October 2021; Received in revised form 16 December 2021; Accepted 17 December 2021
Available online 23 December 2021
1350-4495/© 2021 Elsevier B.V. All rights reserved.
G. Liu et al. Infrared Physics and Technology 121 (2022) 104005
2
G. Liu et al. Infrared Physics and Technology 121 (2022) 104005
The total variation regularizations are utilized in Eq. (5), which are
only one form of the sparse representation learning. The deep con
volutional neural network can also be introduced to extract the image
feature. Because the proposed TVDM method comprises four items, the
functions of each item are different. The major function of the proposed
model is to recover the high-quality infrared image and remove the
stripe noise and random noise simultaneously.
3
G. Liu et al. Infrared Physics and Technology 121 (2022) 104005
Fig. 4. Comparisons of mean cross-track profiles of destriped images. (a) Stripe image in 2(b). (b) Result in 2(c) by DLRD method [35]. (c) Result in 2(d) by LSFD
method [1]. (d) Result in 2(e) by TVDM method.
Fig. 5. Comparisons of striped noises by mean cross-track profiles. (a) Stripe noise in 3(a). (b) Result in 3(b) by DLRD method [35]. (c) Result in 3(c) by LSFD method
[1]. (d) Result in 3(d) by TVDM method.
according to their papers. For the TVDM, the regularization parameters intensity is normalized between 0 and 1. In Fig. 2(c) to 2(e), the authors
are indicated as α1 ∈ [0.010, 0.035], α2 ∈ [50, 150], and β ∈ [350, 450]. show the destriped results by DLRD, LSFD, and our TVDM methods,
respectively. The wild cat in the infrared image is restored after the
destriping processing. The wild cat is marked by the red ellipse. Fig. 2(e)
3.1. Simulated experiments and analysis
produces more image structures compared with the destripe results in
Fig. 2(c) and (d). The wild cat in Fig. 2(e) looks visually more natural.
To conduct the simulated experiments, the stripe noise (Fig. 3(a)) is
The stripe lines have been attenuated significantly. In Fig. 2(c), the
produced and added to the clean infrared image (Fig. 2(a)). Then, the
image details and structure are also smoothed with stripe noise removal.
stripe infrared image couples with the random Gaussian noise (20 dB),
Furthermore, the result destriped by the DLRD seems rather blurry. The
which is shown in Fig. 2(b). In the simulated experiment, the image
4
G. Liu et al. Infrared Physics and Technology 121 (2022) 104005
Table 1
Comparisons of stripe noise suppression experiments by DLRD, LSFD, TVDM
methods.
Index Stripe image DLRD [35] LSFD [1] TVDM
Fig. 7. Extend experiment on the remote sensing image. (a) Observed image.
(b) Mean cross-track profiles of (a). (c), (f), (i) are estimated by DLRD method
[35], LSFD method and our TVDM method, respectively. (d), (g), (j) denote the
Fig. 6. Infrared tank image experiments. (a) Tank image suffers from the close-up of red box of (c), (f), (i), respectively. (e), (h), (k) show the stripe noise,
random noise and stripe noise. (b) Destriped result by DLRD method [35]. (c) which are computed by DLRD method [35], LSFD method and our TVDM
Destriped result by LSFD model [1]. (d) Destriped result by our TVDM method. method, respectively. (For interpretation of the references to colour in this
(e)-(f) Stripe noise estimated by the comparing methods. (h)-(i) Close-up from figure legend, the reader is referred to the web version of this article.)
(b)-(d), respectively.
Table 2
large-scale image edges are removed by the DLRD model. The unidi Destriping results of the infrared image on the metric of PSNR and Q-index. The
rectional total variation regularization plays an important role in sup compared methods include the DLRD, LSFD, and proposed TVDM method.
pressing the line shape noise.
IR Images Degraded DLRD LSFD TVDM
In Figs. 3–5, the stripe noise and destriped images are evaluated in
the visualization form. In Fig. 3(a), the stripe noise is simulated by the IR 1 19.35 27.38 31.25 34.76
(18.41) (21.15) (22.22) (24.13)
nonperiodic program and is added to the clean image (Fig. 2(a)). The
IR 2 17.67 27.86 29.72 33.28
estimated stripe noise images by three comparing methods are shown in (17.57) (20.97) (23.52) (24.76)
Fig. 3(b)–(d). The estimated stripe noise images are different from the IR 3 18.24 25.17 33.13 35.57
original simulated stripe noise image. Because the LSFD method (18.34) (21.35) (23.52) (25.93)
IR 4 18.64 24.93 28.94 32.65
removed some large-scale image edges, the line shape of the stripe noise
(19.62) (20.31) (22.78) (24.82)
is not straight. The mean cross-track profiles of the destriped image and IR 5 21.68 29.16 34.23 36.67
striped noises are shown in Figs. 4 and 5. They demonstrated a similar (19.21) (22.37) (22.51) (25.72)
phenomenon. The result in Fig. 4(d) is similar to the original clean
image.
In Table 1, the destriped image quality is evaluated by three indices. noise in Fig. 6(b), it also removes some image structure. The image
The TVDM model achieved the best values, which outperformed the details are marked by the blue arrow in Fig. 6(e). Furthermore, close-ups
DLRD and LSFD methods. The SSIM index can reflect the approach de of the destriped image are provided and shown in Fig. 6(h)–(j). The tank
gree between the recovered infrared image and original infrared image. edges are indicated by the red ellipses. The image detail in Fig. 6(j) is
Without the reference image, the Q-index can obtain the same law with clearer than that in Fig. 6(h) and (i). By comparing the results, inte
the PSNR and SSIM. It shows that the Q-index can be leveraged to grating the unidirectional total variation and bidirectional total varia
evaluate the destriped results on the real image. tion into the destriping process, the proposed TVDM can significantly
preserve the image structure and remove the stripe noise.
The extended experiment was conducted on other type image. The
3.2. Real infrared image destriping experiments
destriping performance was compared among three methods. In Fig. 7
(a), the remote sensing image was degraded by the observable stripe
In the real image scenarios, we conducted the proposed TVDM
noise. Similar experiments can also be found in Fig. 7(c), (f), and (i). The
method on the infrared noisy image and remote sensing image. The
main reason lies in that the unidirectional total variation regularization
infrared image is downloaded from the Internet, while the remote
can distinguish the stripe noise from the original image, namely, the
sensing image is captured by [39]. In the observed images, they are
gradient of stripe noise is significantly penalized by the item ||∇y f ||TV ,
suffering from different levels of strip noise and random noise. As shown
while the gradient of random noise can be smoothed by the item ||∇f ||TV .
in Fig. 6(a), we have shown the original infrared tank image. Fig. 6(b)–
Moreover, the proposed TVDM method was executed on five other
(d) show the destriped image by the DLRD, LSFD, and our TVDM
infrared images. The results are listed in Table 2. Comparing the PSNR
methods, respectively. Although the DLRD can remove most of the stripe
5
G. Liu et al. Infrared Physics and Technology 121 (2022) 104005
and Q-index values, the proposed method has recovered the high image [13] C. Lu, Stripe non-uniformity correction of infrared images using parameter
estimation, Infrared Phys. Technol. 107 (2020), 103313.
quality among the tested approaches.
[14] H. Liu, Z. Zhang, et al., Destriping algorithm with L0 sparsity prior for remote
sensing images, in: 2015 IEEE International Conference on Image Processing
4. Conclusion (ICIP), Quebec City, QC, Canada, 2015, pp. 2295–2299.
[15] T. Liu, et al., Efficient Blind Signal Reconstruction with Wavelet Transforms
Regularization for Educational Robot Infrared Vision Sensing, IEEE/ASME Trans.
In this paper, a novel infrared image destriping model was proposed Mechatron. 24 (2019) 384–394.
to raise the image resolution, which integrated the uni-total variation [16] Z. Huang, L. Chen, et al., Robust contact-point detection from pantograph-catenary
and bi-total variation regularization. From the destriped infrared im infrared images by employing horizontal-vertical enhancement operator, Infrared
Phys. Technol. 101 (2019) 146–155.
ages, the stripe noise only exists in unidirection. The nature of unidi [17] W. Hua, J. Zhao, G. Cui, X. Gong, P. Ge, J. Zhang, Z. Xu, Stripe nonuniformity
rectional noise was leveraged to convert the total variation correction for infrared imaging system based on single image optimization,
regularization and construct the proposed model. For the random noise, Infrared Phys. Technol. 91 (2018) 250–262.
[18] M. Wang, X. Zheng, J. Pan, B. Wang, Unidirectional total variation destriping using
the bidirectional total variation was introduced, which can preserve the difference curvature in MODIS emissive bands, Infrared Phys. Technol. 75 (2016)
image edge and texture details. The proposed TVDM model was opti 1–11.
mized by the AM algorithm, and each sub-problem can be solved easily. [19] C. Jinsong, S. Yun, G. Huadong, W. Weiming, Z. Boqin, Destriping CMODIS data by
power filtering, IEEE Trans. Geosci. Remote Sens. 41 (2003) 2119–2124.
The major novelty of this study is that the unidirectional total variation [20] Z. Huang, Y. Zhang, Joint Analysis and Weighted Synthesis Sparsity Priors for
can suppress the stripe noise and random Gaussian noise simulta Simultaneous Denoising and Destriping Optical Remote Sensing Images, IEEE
neously. The simulated and actual experimental results on the infrared Transactions on Geoscience and Remote Sensing 58 (10) (2020) 6958–6982.
[21] B. Münch, P. Trtik, F. Marone, M. Stampanoni, Stripe and ring artifact removal
images demonstrated that TVDM is overly effective on removing des
with combined wavelet — Fourier filtering, Opt. Express 17 (2009) 8567–8591.
tripe noise and producing image structures. In the future, we will [22] B. Münch, P. Trtik, F. Marone, M. Stampanoni, Stripe and ring artifact removal
introduce the graph neural networks to suppress the stripe noise in the with combined wavelet-Fourier filtering, Opt. Express 17 (2009) 8567–8591.
infrared images. [23] T. Liu, Y. Li, et al., RISIR: rapid infrared spectral imaging restoration model for
industrial material detection in intelligent video systems, IEEE Trans. Ind. Inf.
(2022), https://doi.org/10.1109/TII.2019.2930463.
Declaration of Competing Interest [24] H. Liu, et al., Blind Poissonian reconstruction algorithm via curvelet regularization
for an FTIR spectrometer, Opt. Express 26 (2018) 22837–22856.
[25] T. Liu, H. Liu, Z. Chen, A.M. Lesgold, Fast Blind Instrument Function Estimation
The authors declare that they have no known competing financial Method for Industrial Infrared Spectrometers, IEEE Trans. Ind. Inf. 14 (2018)
interests or personal relationships that could have appeared to influence 5268–5277.
the work reported in this paper. [26] H. Liu, Z. Zhang, et al., Blind spectral deconvolution algorithm for Raman spectrum
with Poisson noise, Photon. Res. 2 (2014) 168–171.
[27] H. Liu, L. Yan, Y. Chang, H. Fang, T. Zhang, Spectral deconvolution and feature
References extraction with robust adaptive Tikhonov regularization, IEEE Trans. Instrum.
Meas. 62 (2013) 315–327.
[1] Y. Shao, Y. Sun, et al., Infrared image stripe noise removing using least squares and [28] T. Liu, H. Liu, Z. Chen, et al., FBRDLR: Fast blind reconstruction approach with
gradient domain guided filtering, Infrared Phys. Technol. 119 (2021), 103968. dictionary learning regularization for infrared microscopy spectra, Infrared Phys.
[2] Q. Zeng, H. Qin, X. Yan, H. Zhou, Fourier Spectrum Guidance for Stripe Noise Technol. 90 (2018) 101–109.
Removal in Thermal Infrared Imagery, IEEE Geosci. Remote Sens. Lett. 17 (2020) [29] H. Liu, T. Zhang, L. Yan, et al., A MAP-based algorithm for spectroscopic semi-blind
1072–1076. deconvolution, Analyst 137 (2012) 3862–3873.
[3] Z. Huang, et al., Optical remote sensing image enhancement with weak structure [30] Y. Huang, C. He, H. Fang, X. Wang, Iteratively reweighted unidirectional
preservation via spatially adaptive gamma correction, Infrared Phys. Technol. 94 variational model for stripe non-uniformity correction, Infrared Phys. Technol. 75
(2018) 38–47. (2016) 107–116.
[4] T. Liu, et al., Flexible FTIR Spectral Imaging Enhancement for Industrial Robot [31] K. Mikelsons, M. Wang, L. Jiang, M. Bouali, Destriping algorithm for improved
Infrared Vision Sensing, IEEE Trans. Ind. Inf. 16 (2020) 544–554. satellite-derived ocean color product imagery, Opt. Express 22 (2014)
[5] Z. Huang, Y. Zhang, et al., Joint horizontal-vertical enhancement and tracking 28058–28070.
scheme for robust contact-point detection from pantograph-catenary infrared [32] H. Liu, S. Fang, et al., MFDNet: Collaborative Poses Perception and Matrix Fisher
images, Infrared Phys. Technol. 105 (2020), 103156. Distribution for Head Pose Estimation, IEEE Trans. Multimedia (2022) 1–13
[6] T. Liu, J. Wang, et al., NGDNet: Nonuniform Gaussian-label distribution learning https://doi.org/10.1109/TMM.2021.3081873.
for infrared head pose estimation and on-task behavior understanding in the [33] Z. Li, et al., Learning knowledge graph embedding with heterogeneous relation
classroom, Neurocomputing 436 (2021) 210–220. attention networks, IEEE Trans. Neural Networks Learning Syst. (2022), https://
[7] H. Liu, X. Wang, et al., Infrared head pose estimation with multi-scales feature doi.org/10.1109/TNNLS.2021.3055147.
fusion on the IRHP database for human attention recognition, Neurocomputing [34] H. Liu, C. Zheng, D. Li, et al., EDMF: efficient deep matrix factorization with review
411 (2020) 510–520. feature learning for industrial recommender system, IEEE Trans. Ind. Inf. (2022)
[8] T. Liu, J. Wang, et al., Facial expression recognition method with multi-label 1–11, https://doi.org/10.1109/TII.2021.3128240.
distribution learning for non-verbal behavior understanding in the classroom, [35] X. Kuang, X. Sui, Y. Liu, C. Liu, Q. Chen, G. Gu, Robust destriping method based on
Infrared Phys. Technol. 112 (2021), 103594. data-driven learning, Infrared Phys. Technol. 94 (2018) 142–150.
[9] G. Zhao, X. Huang, M. Taini, S.Z. Li, M. Pietikäinen, Facial expression recognition [36] J. Lee, Y.M. Ro, Dual-branch structured de-striping convolution network using
from near-infrared videos, Image Vision Computing 29 (2011) 607–619. parametric noise model, IEEE Access 8 (2020) 155519–155528.
[10] T. Liu, et al., Nonlocal low-rank-based blind deconvolution of Raman spectroscopy [37] T. Goldstein, S. Osher, The split Bregman method for L1-regularized problems,
for automatic target recognition, Appl. Opt. 57 (2018) 6461–6469. SIAM J. Imag. Sci. 2 (2009) 323–343.
[11] S.-P. Wang, Stripe noise removal for infrared image by minimizing difference [38] H. Liu, S. Liu, et al., Adaptive total variation-based spectral deconvolution with the
between columns, Infrared Phys. Technol. 77 (2016) 58–64. split Bregman method, Appl. Opt. 53 (2014) 8240–8248.
[12] L. Liu, L. Xu, H. Fang, Simultaneous intensity bias estimation and stripe noise [39] http://ladsweb.nascom.nasa.gov/, in, 2021-12.
removal in infrared images using the global and local sparsity constraints, IEEE
Trans. Geosci. Remote Sens. 58 (2020) 1777–1789.