IEIE Transactions on Smart Processing and Computing, vol. 10, no.

4, August 2021
https://doi.org/10.5573/IEIESPC.2021.10.4.291 291

IEIE Transactions on Smart Processing and Computing

Image Denoising Method based on Deep Learning using

Improved U-net
Jaewook Han, Jinwon Choi, and Changwoo Lee*

School of Information, Communications and Electronics Engineering, The Catholic University of Korea / Pucheon-City, Korea

* Corresponding Author: Changwoo Lee, [email protected]

Received January 31, 2021; Revised April 5, 2021; Accepted April 26, 2021; Published August 30, 2021

* Short Paper

Abstract: Various methods, including block-matching and 3D filtering (BM3D), have been
proposed for image denoising. Recently, studies on deep learning methods for image denoising
have been on the rise. In this paper, we propose a new structure for a deep neural network that
improves image denoising performance. Among the existing deep neural networks, we improve U-
net, which is widely used for image restoration, through the inclusion of pre-processing and post-
processing and by modifying each of its stages. Extensive simulations show that the proposed
structure performs very well for a wide range of noise levels with a single trained parameter, and it
exhibits superior image denoising performance compared to conventional deep neural networks.

Keywords: Image denoising, Deep learning, U-net, New structure, Improved U-net

1. Introduction Extensive computer simulations show that the proposed

method yields superior denoising performance compared
Various image denoising methods have been studied, to BM3D and other deep learning methods.
for example, when images are damaged by Gaussian noise,
impulse noise, and speckle noise [1-9]. The nonlocal mean
(NLM) technique and the block-matching and 3D filtering 2. Image Denoising Method
(BM3D) technique, which eliminate noise by calculating
the weighted sum using similarity for each image patch in To remove noise such as Gaussian noise, impulse noise
an entire image, show very good denoising performance [3, and speckle noise [1, 2], starting with a median filter,
4]. In recent years, deep learning methods, which have various denoising methods using the characteristics of the
excellent performance in various image processing fields, frequency bands of the image and noise have been studied.
have been studied for application to image denoising, A denoising technique that utilizes high-frequency
showing performance superior to conventional image characteristics of noise has a problem in that high-
denoising techniques [5-9]. frequency components of the original image are also lost.
In this paper, we propose an efficient deep neural The NLM method shows very good performance in image
network structure to improve image denoising denoising by calculating the weighted sum from the entire
performance by improving the structure of U-net, which is image using the local similarity of each patch [3]. In
widely used for image restoration. The proposed structure particular, the BM3D technique, which groups image
adds pre-processing and post-processing to the patches into a 3D structure and precisely calculates
conventional U-net structure while also adding a weights, was identified as the technique that showed state-
convolution layer in addition to a shortcut for each stage of of-the-art denoising performance prior to using a deep
U-net. Since the proposed structure improves the learning technique [4].
convergence performance of the deep neural network when Since deep learning methods have shown excellent
generating the target image, it can be used not only for performance in various image processing fields, a lot of
denoising but also for various image restoration research has been conducted into applying deep learning to
applications. By training the proposed structure using image denoising [5-9]. Zhang et al. proved that the deep
images with various noise intensities, noise at various convolutional neural network (CNN) structure can be
intensities can be removed with a single trained parameter. applied to image denoising to achieve excellent denoising
292 Han et al.: Image Denoising Method based on Deep Learning using Improved U-net

performance [5]. A CNN using a variable split technique

was proposed to reduce the number of computations for
image denoising without degrading performance [6], and
FFDNet was proposed, which can handle a wide range of
noise levels and can improve convergence speed using a
noise level map as input [7]. Tian et al. proposed ADNet
using an attention-guided CNN [8]. These CNN methods
have been proven to show superior denoising performance,
compared to the BM3D technique [5-9].

3. Improving U-net for Image Denoising

Because deep learning has shown excellent Fig. 1. U-net.
performance in various fields of image processing, many
studies on deep learning are being conducted. Research on
deep learning is being conducted from various aspects,
such as structures and training methods for deep neural
networks. Among the various structures of deep neural
networks, U-net, shown in Fig. 1, was proposed for
medical image processing, but it has been used in various
image processing fields, including image restoration [10,
11]. U-net improves convergence performance by adding
skip connection to the autoencoder structure. The U-net
encoder consists of a contractive path that extracts feature
vectors from the input image, and the decoder consists of
an expansive path that restores the image from the
extracted feature vectors. In the deep learning process, the
feature vectors extracted from the contractive path are
trained so they are as close as possible to the feature
vectors of the target image. For the expansive path, U-net
is trained to restore the image as closely as possible to the
target image using the extracted feature vectors. Image
characteristics that may be lost in the process of reducing
the size of the feature vectors in the contractive path are
transferred to the expansive path through the skip
connection, and are used in the image restoration process,
thus improving convergence performance compared to the
autoencoder.
Fig. 2. Improved U-net (ImpUnet1 & ImpUnet2).
In this paper, we propose improved structures for U-net,
and we prove that it shows superior denoising performance,
compared to the conventional deep neural networks. The
improved U-net proposed in this paper can be used in
various image restoration fields as well as for denoising.
First, we propose the deep neural networks shown in Figs.
2 and 3. Convergence performance is enhanced by further
processing the U-net input and output through pre-
processing and post-processing, respectively. The input
data from the pre-processing unit are transferred to the
post-processing unit through an additional skip connection.
After concatenation with the data processed in the
expansive path, the image is restored through the final
post-processing step. As shown in Fig. 2, pre-processing,
the additional skip connection, and post-processing all
compose a single module, and convergence performance
can be further improved through cascaded connections of
the modules. Also, as shown in Fig. 3, each stage of U-net
can be modified by applying the so-called ResBlock
structure that adds a convolution layer with a shortcut to
each U-net stage. This structure can be used together with Fig. 3. Improved U-net (ImpUnet3).
the pre-processing and post-processing structures
IEIE Transactions on Smart Processing and Computing, vol. 10, no. 4, August 2021 293

described above in order to maximize the overall Table 1. Average PSNR and SSIM Results (Kodak
performance. As is shown in Section 4, the convergence images).
and denoising performance of the proposed structure are
PSNR (in dB) SSIM
improved compared to the conventional U-net. Since the Method
proposed structure can improve the overall convergence σ = 10 σ = 30 σ = 50 σ = 10 σ = 30 σ = 50
performance of a deep neural network that minimizes the Noisy 28.21 18.85 14.78 0.6595 0.2744 0.1551
difference between the target image and the degraded input BM3D [4] 36.57 30.88 28.62 0.9435 0.8472 0.7788
image, it can be used in various image restoration fields as DnCNN [5] 36.58 31.28 28.95 0.9447 0.8580 0.7917
well as for image denoising.
IRCNN [6] 36.70 31.25 28.94 0.9448 0.8584 0.7943
FFDNet [7] 36.81 31.40 29.11 0.9462 0.8597 0.7952
ADNet [8] 36.73 31.28 28.93 0.9452 0.8576 0.7887
4. Performance Evaluation Unet [10] 36.19 31.29 28.98 0.9430 0.8622 0.7957
ImpUnet1 36.61 31.46 29.16 0.9461 0.8647 0.8025
In order to evaluate the performance of the proposed
method, extensive simulations were performed using a ImpUnet2 36.72 31.56 29.27 0.9466 0.8677 0.8056
program based on TensorLayer [12]. Training images were ImpUnet3 36.52 31.45 29.18 0.9452 0.8640 0.8027
generated using the DIV2K image database [13]. BSD68 ImpUnet4 36.88 31.63 29.30 0.9478 0.8688 0.8079
images and Kodak images, which are the most widely used
standard test images [14, 15], were used to measure Table 2. Average PSNR and SSIM Results (BSD68
performance. Image patches at 64 × 64 were extracted from images).
the training images, and training was performed to
minimize the mean square error (MSE) loss over a total of PSNR SSIM
Method
20,000 epochs using the Adam optimizer [16]. The step σ = 10 σ = 30 σ = 50 σ = 10 σ = 30 σ = 50
size started at 10 −4 and was decreased by 1/2 for every Noisy 28.30 19.03 14.99 0.7069 0.3299 0.1944
4,000 epochs. Additive white Gaussian noise with standard BM3D [4] 36.18 30.25 27.80 0.9541 0.8541 0.7776
deviation that varied between 5 and 50 was added to the DnCNN [5] 36.44 30.67 28.25 0.9562 0.8687 0.7987
input training images for the deep neural network, hence, IRCNN [6] 36.37 30.57 28.19 0.9557 0.8675 0.7985
training the deep neural network to operate regardless of
FFDNet [7] 36.50 30.70 28.31 0.9567 0.8682 0.7984
the noise level.
Performance comparisons of the deep neural networks ADNet [8] 36.38 30.56 28.13 0.9555 0.8660 0.7931
are presented in Figs. 4-9 and Tables 1 and 2, where Unet [10] 35.84 30.56 28.22 0.9527 0.8690 0.8001
ImpUnet1 to ImpUnet4 represent stages of the improved ImpUnet1 36.20 30.71 28.33 0.9557 0.8721 0.8050
U-net as proposed in this paper. ImpUnet1 improves U-net ImpUnet2 36.30 30.75 28.39 0.9560 0.8741 0.8064
by using only one pre-processing and post-processing unit,
ImpUnet3 36.15 30.70 28.32 0.9549 0.8721 0.8043
while ImpUnet2 improves U-net by using three pre-
processing and post-processing units. ImpUnet3 improves ImpUnet4 36.39 30.79 28.38 0.9570 0.8749 0.8078
U-net by using ResBlock, and ImpUnet4 improves U-net
by using three pre-processing and post-processing units
and ResBlock. First, to analyze the convergence noise reduction performance of the proposed deep neural
performance of the deep neural network, the MSE network is superior to that of the BM3D technique and the
convergence curves are presented in Fig. 4. We can see existing deep neural networks, and detailed characteristics
that the convergence performance of the proposed of the image are restored well.
structure improves, compared to the conventional U-net.
When pre-processing, post-processing, and ResBlock are
used together, the convergence performance is at its best. 5. Conclusion
Tables 1 and 2 show the average peak signal-to-noise ratio
(PSNR) and the average structural similarity index In this paper, a deep learning–based image denoising
measure (SSIM) [17] for 68 BSD68 test images and 24 method using an improved U-net was proposed. The
Kodak images. For comparison with the proposed method, convergence and denoising performance of the proposed
the denoising performance of the BM3D technique and deep neural network is improved by adding pre-processing
deep neural networks that provide excellent performance and post-processing to the conventional U-net. The
from among the existing deep neural networks used for performance is further enhanced by adding a convolution
image denoising, was compared for various noise standard layer together with a shortcut in each stage of U-net. In
deviations, σ. We can see that the proposed deep neural particular, pre-processing and post-processing have a
network shows significant PSNR and SSIM gain, modular structure, and performance can be further
compared to BM3D and the existing deep neural networks, improved through adopting a cascaded connection between
respectively. The proposed method outperforms the modules. Extensive simulations confirmed that the
conventional U-net by up to 0.7 dB for PSNR, and shows proposed method has superior denoising performance
better performance than BM3D and existing neural compared to BM3D and existing deep learning methods.
networks for all noise levels. As shown in Figs. 5-9, the Since the proposed structure improves the overall
294 Han et al.: Image Denoising Method based on Deep Learning using Improved U-net

Fig. 6. Comparison of denoising results (Kodak image

(a) 7, σ=50).

(b) Fig. 7. Comparison of denoising results (BSD68 image

47, σ=50).
Fig. 4. MSE convergence (a) MSE for all 2,000 epochs,
(b) MSE for the last 500 epochs.

(a) (b)

Fig. 8. Comparison of denoising results (Kodak image

1, σ=30).

(c) (d)

Fig. 5. Test images for subjective comparison of

denoising results (a) Kodak image 7, (b) BSD68 image
47, (c) Kodak image 1, (d) BSD68 image 18.

convergence performance of U-net, it can be used not only

for image denoising but also for various image restoration
applications.
Fig. 9. Comparison of denoising results (BSD68 image
18, σ=30).
IEIE Transactions on Smart Processing and Computing, vol. 10, no. 4, August 2021 295

Acknowledgement [12] H. Dong, A. Supratak, L. Mai, F. Liu, A. Oehmichen,

S. Yu and Y. Guo, “TensorLayer: A versatile library
This study was supported by Research Fund 2020 of for efficient deep learning development,” in Proc.
The Catholic University of Korea and by the Basic Science ACM-MM 2017, pp. 1201–1204, 2017. Article
Research Program through the National Research (CrossRef Link)
Foundation of Korea (NRF) funded by the Ministry of [13] E. Agustsson and R. Timofte, “NTIRE 2017
Education (No. 2017R 1D 1A 1B03030585). challenge on single image super-resolution: Dataset
and study,” in CVPRW 2017. Article (CrossRef Link)
References [14] R. Franzen, “Kodak lossless true color image suite,”
source: http://r0k.us/graphics/kodak, vol. 4, 1999.
Article (CrossRef Link)
[1] M. Mafi, S. Tabarestani, M. Cabrerizo, A. Barreto, [15] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A
and M. Adjouadi, “Denoising of ultrasound images database of human segmented natural images and its
affected by combined speckle and Gaussian noise,” application to evaluating segmentation algorithms
IET Image Processing, vol. 12, np. 12, pp.2346–2351, and measuring ecological statistics,” in ICCV 2001.
2018. Article (CrossRef Link) Article (CrossRef Link)
[2] Y. Dong and S. Xu, “A new directional weighted [16] D. Kingma and J. B. Adam, “Adam: A method for
median filter for removal of random-valued impulse stochastic optimization,” International Conference on
noise,” IEEE Signal Processing Letters, vol. 14, no. 3, Learning Representations, 2015. Article (CrossRef
pp. 193–196, 2007. Article (CrossRef Link) Link)
[3] A. Buades, B. Coll and J.-M. Morel, "A non-local [17] A. Horé and D. Ziou, “Image quality metrics: PSNR vs.
algorithm for image denoising," in Proc. of Computer SSIM,” 20th International Conference on Pattern
Vision and Pattern Recognition 2005 (CVPR 2005), Recognition, 2010. Article (CrossRef Link)
pp. 60-65, June 2005. Article (CrossRef Link)
[4] K. Dabov, A. Foi and V. Katkovnik and K.
Egiazarian, "Image denoising by sparse 3-D
transform domain collaborative filtering," IEEE Jaewook Han is a student at the
Trans. on Image Processing, vol. 16, no. 8, pp. 2080- School of Information, Communi-
2095, Aug. 2007. Article (CrossRef Link) cations and Electronics Engineering,
[5] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, the Catholic University of Korea. His
“Beyond a gaussian denoiser: Residual learning of current interests lie in the area of
deep cnn for image denoising,” IEEE Transactions image processing and deep learning.
on Image Processing, 26(7): 3142–3155, 2017.
Article (CrossRef Link)
[6] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning
deep CNN denoiser prior for image restoration,” in
CVPR 2017. Article (CrossRef Link)
[7] K. Zhang, W. Zuo, and L. Zhang, “FFDNet: Toward Jinwon Choi is a student at the School
a fast and flexible solution for CNN-based image of Information, Communications and
denoising,” IEEE Transactions on Image Processing, Electronics Engineering, the Catholic
vol. 27, no. 9, pp. 4608–4622, 2018. Article University of Korea. His current
(CrossRef Link) interests lie in the area of image
[8] C. Tian, Y. Xu, Z. Li, W. Zuo, L. Fei and H. Liu, processing and deep learning.
“Attention-guided CNN for image denoising,” Neural
networks, vol. 124, pp. 117-129, Aprial 2020. Article
(CrossRef Link)
[9] C. Tian, L. Fei, W. Zheng, Y. W. Zuo, C-W. Lin,
“Deep learning on image denoising: An overview,”
Neural networks, vol. 131, pp. 251-275, Nov. 2020. Changwoo Lee received a BSc and an
Article (CrossRef Link) MSc in control and instrumentation
[10] O. Ronneberger, P. Fischer and T. Brox, “U-Net: engineering from Seoul National
Convolutional networks for biomedical image University. After receiving a PhD in
segmentation,” MICCAI 2015: Medical Image image processing area from Seoul
Computing and Computer-Assisted Intervention 2015, National University in 1996, he
pp. 234-241, 2015. Article (CrossRef Link) worked as a Senior Researcher with
[11] Y. J. Kim and C. W. Lee, “Deep Learning Method Samsung Electronics. He is currently a
for Extending Image Intensity Using Hybrid Log- Professor at the School of Information, Communications
Gamma,” IEIE Transactions on Smart Processing and Electronics Engineering, the Catholic University of
and Computing, vol. 9, no. 4, pp. 312-316, August Korea. His current interests lie in the area of image
2020. Article (CrossRef Link) processing and deep learning.
Copyrights © 2021 The Institute of Electronics and Information Engineers