Multi-focus Image Fusion for Extended Depth of Field

Wisarut Chantara and Yo-Sung Ho

School of Electrical Engineering and Computer Science
Gwangju Institute of Science and Technology (GIST)
{wisarut,hoyo}@gist.ac.kr

ABSTRACT is the process which combines two or more different focal

In this paper, we address the extended depth of field method, image with the same scene into a single image, maintaining
which appropriately applicable to a set of stack images. The the significant information from each of the stack images.
proposed fusion algorithm improved a conventional discrete Therefore, all important objects in the image are in-focused.
wavelet transform using spatial frequency and sum-modified- Generally, the digital camera can set only one distance focus
Laplacian. Both methods are applied to the approximation at a time. Then, to acquire a set of stack images with the
coefficient and the detail coefficients (horizontal, vertical and same scene by captured in a single shot, a light field camera
diagonal) for calculating in-focused regions. After that, the such as Lytro camera [8] and Raytrix camera [10] provide
sharp areas of each stack images are combined with the fu- a set of the stack images with different focal display in the
sion image process. Finally, the inverse wavelet transform same view.
is utilized to obtain a final result image. The performance In recent year, many depth of field extension techniques
of the proposed method is conducted and compared with have been developed, e.g., Li et al. [6], introduced a method
conventional fusion methods. Experiment results can demon- based on the selection of image blocks from source images
strate that the proposed method outperforms other reference to construct the fused image using spatial frequency. While
methods. Desale et al. [2], proposed a study and analysis of image fusion
technique by way of PCA, DCT, and DWT. Moreover, Wang
CCS CONCEPTS et al. [11], presented multisource image fusion using spatial
frequency and simplified pulse coupled neural network. In this
• Computing methodologies → Image processing;
paper, we proposed an improved discrete wavelet transform
using spatial frequency and sum-modified-Laplacian to detect
KEYWORDS
focal regions and reconstruct in-focused regions to an all-in-
Depth of field, discrete wavelet transform, spatial frequency, focused image.
sum-modified-Laplacian
ACM Reference Format: 2 PROPOSED METHOD
Wisarut Chantara and Yo-Sung Ho. 2018. Multi-focus Image Fusion
for Extended Depth of Field. In The 10th International Conference The following techniques are involved in the proposed meth-
on Internet Multimedia Computing and Service(ICIMCS’18), Au- ods implementation. Light field camera [8] is utilized as a
gust 17-19,2018, Nanjing, China. ACM, Nanjing, China, 4 pages. capture device. Light field picture splitter [9] is applied to
https://doi.org/10.1145/3240876.3240894 split the raw file into different focus level images. This process
provided a set of different focal images which display the
1 INTRODUCTION same position as shown in Fig. 1.
The depth of field (DOF), also called focus range or effective
focus range, is the distance between the nearest and farthest
objects in a scene that appears acceptably sharp in an image.
Extended DOF creates an image with an extended focal
range from a series of images focused at a different depth,
particularly useful for computer vision, digital photography,
and macro photography. So that multi-focus image fusion

Permission to make digital or hard copies of all or part of this work

for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage
and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than the au-
thor(s) must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions
from [email protected].
ICIMCS’18, August 17-19,2018, Nanjing, China
© 2018 Copyright held by the owner/author(s). Publication rights
licensed to ACM.
ACM ISBN 978-1-4503-6520-8/18/08. . . $15.00 Figure 1: A set of different focal images.
https://doi.org/10.1145/3240876.3240894
ICIMCS’18, August 17-19,2018, Nanjing, China W. Chantara et al.

𝑓 𝑜𝑟 ∇2𝑀 𝐿 𝐼(𝑖, 𝑗) ≥ 𝑇𝑆𝑀 𝐿

⃒ 2 ⃒ ⃒ 2 ⃒
⃒𝜕 𝐼 ⃒ ⃒𝜕 𝐼 ⃒
𝑤ℎ𝑒𝑟𝑒 ∇2𝑀 𝐿 𝐼(𝑖, 𝑗) = ⃒⃒ 2 ⃒⃒ + ⃒⃒ 2 ⃒⃒
𝜕𝑥 𝜕𝑦

2.4 Enhancement SML (eSML)

In the homogeneous region, the focus measure can be affected
by the pixel noise [1]. In order to decrease this effect, the
SML values are computed in a small window to determine
the eSML focus measure of the center pixel (x, y) as:
Figure 2: DWT decomposition.
∑︁ 𝑗=𝑦+𝑁
𝑖=𝑥+𝑁 ∑︁
𝑒𝑆𝑀 𝐿 = 𝑆𝑀 𝐿(𝑖, 𝑗) (5)
2.1 Discrete wavelet transform (DWT) 𝑖=𝑥−𝑁 𝑗=𝑦−𝑁
Discrete wavelet transform uses a cascade of special low-
pass and high-pass filter and a sub-sampling operation. The 2.5 Image fusion
output from first-order of DWT contains four decomposition The wavelet decomposition coefficient structures (approxima-
parts as shown in Fig. 2. Those are CA, CH, CV and CD, tion and detail coefficients) are combined as:
where CA is the approximation coefficient, which is sensitive
to human eyes [5]. While CH, CV, and CD are the detail 𝑚
𝐹𝑖,𝑗 = 𝐷𝑖,𝑗 (6)
coefficient (horizontal, vertical and diagonal) which have 𝑡
more detail information more than CA. Since DWT of image 𝑤ℎ𝑒𝑟𝑒 𝑚= arg max𝑡 𝐶𝑖,𝑗 ,𝑡 = 1, 2, ..., 𝑁
signals produces a nonredundant image representation, it
can provide better spatial and spectral localization of image where 𝐹𝑖,𝑗 is a final coefficient in an all-focused image at (i,
𝑚
information. j). While 𝐷𝑖,𝑗 is a maximum coefficient information at (i, j).
𝑡
𝐶𝑖,𝑗 is a coefficient information at t stack image. N is a set
2.2 Spatial frequency (SF) of input stack images.
Spatial frequency, which originated from the human visual
system, indicates the overall active level in an image. It is 2.6 The proposed method procedure
difficult to completely comprehend the human visual system A summary of the proposed method is provided as follow:
with current physiologic means. While the use of SF has led (1) Apply the DWT on each image of the input stack
to an effective contrast criterion for image fusion [7]. images,
SF is defined as: (2) Apply the SF by using Eq. (1) to (3) for the approxi-
√︀ mation coefficients (CA),
𝑆𝐹 = (𝐶𝐹 )2 + (𝑅𝐹 )2 (1) (3) Apply the SML by using Eq. (4) for the detail coeffi-
cients (CH, CV, and CD),
whereRF and CF are the row frequency and column frequen- (4) Apply the eSML by using Eq. (5) for the noise reduction
cy respectively: improvement in the detail coefficients,
⎯ (5) Combine each coefficient of the approximation and
⎸ 𝑀 ∑︁𝑁
⎸ 1 ∑︁ detail coefficients by using Eq. (6), and
𝑅𝐹 =⎷ [𝐼(𝑥, 𝑦) − 𝐼(𝑥, 𝑦 − 1)]2 (2)
𝑀 × 𝑁 𝑖=1 𝑗=2 (6) Apply an inverse discrete wavelet transform to obtain
a multi-focus image with extended depth of field as a
⎯ final image.
⎸ 𝑁 ∑︁𝑀
⎸ 1 ∑︁
𝐶𝐹 =⎷ [𝐼(𝑥, 𝑦) − 𝐼(𝑥 − 1, 𝑦)]2 (3) 3 EXPERIMENT RESULTS
𝑀 × 𝑁 𝑗=1 𝑖=2
The experiments are conducted to compare the performance
for M x N image with a grayscale I(x, y) at (x, y) coordinate. of the proposed method with those of popular widely con-
ventional methods (such as pixel averaging method, DWT-
2.3 Sum-modified-Laplacian averaging method [2], DWT-maximum method [2], and the
spatial frequency method). We conducted experiments on
Sum-modified-Laplacian is developed to compute a local
512 × 512 pixels sample images. The test images are a set
measure of the quality of image focus. SML can provide
of cup and stuff images, which contain multiple objects at
better performance in focus measurement criterion [4]. SML
different focus level images capturing by light filed camer-
is defined as:
a [8]. In all our experiments, we set 𝑇𝑆𝑀 𝐿 to be 10. The
experiment results are presented in Fig. 3 and 4. From the
∑︁ 𝑗=𝑦+𝑁
𝑖=𝑥+𝑁 ∑︁
𝑆𝑀 𝐿(𝑥, 𝑦) = ∇2𝑀 𝐿 𝐼(𝑖, 𝑗) (4) fusion results, we can easily observe that the results of the
𝑖=𝑥−𝑁 𝑗=𝑦−𝑁 pixel averaging and DWT methods have a lower contrast
Multi-focus Image Fusion for Extended Depth of Field ICIMCS’18, August 17-19,2018, Nanjing, China

Table 1: Objective evaluation of the extended depth of field image (non-reference fusion metrics)

Image Criteria Pixel DWT[2] DWT[2] SF Proposed

averaging Averaging Maximum method

Cup FMI 0.9227 0.9186 0.9122 0.9262 0.9264

𝑄𝐴𝐵/𝐹 0.6292 0.6159 0.5207 0.6533 0.6552

Stuff FMI 0.9221 0.9178 0.9059 0.9228 0.9230

𝑄𝐴𝐵/𝐹 0.6129 0.5837 0.5055 0.6498 0.6500

than those of the spatial frequency method and the proposed REFERENCES
method. However, it is hard to tell the difference between [1] Xiangzhi Bai, Yu Zhang, Fugen Zhou, and Bindang Xue. 2015.
the results of the spatial frequency method and the proposed Quadtree-based multi-focus image fusion using a weighted focus-
measure. Inf. Fusion 22 (March 2015), 105–118.
method by subjective evaluation. Hence, the paper applies [2] Rajenda Pandit Desale and Sarita V. Verma. 2013. Study and
some non-reference fusion metrics such as Feature Mutual analysis of PCA, DCT & DWT based Image Fusion Techniques. In
Int. Conf. on Signal Processing, Image Processing and Pattern
Information (FMI) [3] and Petrovics metric (𝑄𝐴𝐵/𝐹 ) [12] Recognition. IEEE, Coimbatore, India, 66–69.
are then introduced and employed. These evaluation criteria [3] Mohammad Haghighat and Masoud Amirkabiri Razian. 2014. Fast-
metrics are calculated without respect to the reference im- FMI: non-reference image fusion metric. In Proceedings of 8th
Int. Conf. on Application of Information and Communication
ages. FMI measures the amount of information that the fused Technologies. IEEE, Astana, Kazakhstan, 1–3.
image contains the source images, while 𝑄𝐴𝐵/𝐹 measures [4] Wei Huang and Zhongliang Jing. 2007. Evaluation of focus mea-
the relative amount of edge information that is transferred sures in multi-focus image fusion. Pattern Recognition Letters
28, 4 (March 2007), 493–500.
from the source into the fused image. The higher the FMI or [5] Chu-Hui Lee and Zheng-Wei Zhou. 2012. Comparison of image
𝑄𝐴𝐵/𝐹 value, the better the fused image performance. The fusion based on DCT-STD and DWT-STD. In Proceedings of
the International MultiConference of Engineers and Computer
comparison results are summarized in Table 1. Scientists (IMECS 2012). Hong Kong, 720–725.
The above two evaluation criteria are then applied to [6] Shutao Li, James T. Kwok, and Yaonan Wang. 2001. Combination
evaluate the four fusion methods in Fig. 3 and 4, the de- of images with diverse focuses using the spatial frequency. Inf.
Fusion 2, 3 (Sept. 2001), 169–176.
tailed quantitative results are given in Table 1. From Table [7] Shutao Li and Bin Yang. 2008. Multifocus image fusion using
1, we can observe that the values of all quality indices of the region segmentation and spatial frequency. Image and Vision
proposed method are larger than those of pixel averaging, Computing 26, 7 (July 2008), 971–979.
[8] Lytro. [n. d.]. The Lytro camera. http://www.lytro.com
DWT-averaging, DWT-Maximum, and the conventional spa- [9] Nirav Patel. 2013. lfptools. http://github.com/nrpatel/lfptools/
tial frequency methods, which means the proposed algorithm [10] Raytrix. [n. d.]. The Raytrix camera. http://www.raytrix.de
[11] Nianyi Wang, Yide Ma, and Weilan Wang. 2014. DWT-based
can effectively combine sharp parts of the original image to multisource image fusion using spatial frequency and simplified
the fused image, and yield superior quality than conventional pulse coupled neural network. Journal of Multimedia 9, 1 (Jan.
methods. 2014), 159–165.
[12] C.S. Xydeas and Vladimir Petrovic. 2000. Objective image fusion
performance measure. Electronics Letters 36, 4 (Feb. 2000),
4 CONCLUSIONS 308–309.

In this paper, we proposed a multi-focus image fusion for

extended depth of field imaging. The reconstructed image
provides all-focused scene. We apply the enhanced discrete
wavelet transform algorithm to measure focus regions and
fuse the final image. The main contribution of this work is
that we reform the conventional DWT algorithm with the
spatial frequency and the sum-modified-Laplacian algorithms.
As a result of that, the proposed method has more efficiently
than other conventional methods.

ACKNOWLEDGMENTS
This work was supported by the National Research Founda-
tion of Korea (NRF) Grant funded by the Korean Government
(MSIP) (No. 2011-0030079).
ICIMCS’18, August 17-19,2018, Nanjing, China W. Chantara et al.

(a) (b) (c)

(d) (e)

Figure 3: Comparison of ”Cup”: (a) Pixel Avg., (b) DWT-Avg., (c) DWT-Max, (d) SF, (e) Proposed method.

(a) (b) (c)

(d) (e)

Figure 4: Comparison of ”Stuff”: (a) Pixel Avg., (b) DWT-Avg., (c) DWT-Max, (d) SF, (e) Proposed method.