Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding
Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding
Road Crack Detection Using Deep Convolutional Neural Network and Adaptive Thresholding
Abstract— Crack is one of the most common road distresses of gradient features using gradient filters, such as Sobel
which may pose road safety hazards. Generally, crack detection [12], [13], for each image pixel, and then using a binary
is performed by either certified inspectors or structural engi- classifier to determine whether an image pixel is part of a
neers. This task is, however, time-consuming, subjective and
arXiv:1904.08582v1 [cs.CV] 18 Apr 2019
labor-intensive. In this paper, we propose a novel road crack crack region or not [8]. In early methods, such as [14] and
detection algorithm based on deep learning and adaptive image [15], the authors used threshold-based approaches to find
segmentation. Firstly, a deep convolutional neural network is crack regions based on the assumption that a pixel lying
trained to determine whether an image contains cracks or in a crack area is consistently darker than others [7], [16].
not. The images containing cracks are then smoothed using Furthermore, many researchers [10], [11], [17], [18] tried
bilateral filtering, which greatly minimizes the number of noisy
pixels. Finally, we utilize an adaptive thresholding method to to suppress the inference of noise by considering additional
extract the cracks from road surface. The experimental results local features, such as the mean and the standard deviation
illustrate that our network can classify images with an accuracy of an image region. However, these methods are still very
of 99.92%, and the cracks can be successfully extracted from sensitive to noise because only the brightness features are
the images using our proposed thresholding algorithm. taken into consideration.
I. INTRODUCTION In recent years, some novel algorithms, such as minimal
path selection (MPS) [19]–[21], minimum spanning tree
Road damages in the form of cracks may reduce the road (MST) [22], [23] and crack fundamental element (CFE) [24],
performance and pose potential road safety hazards [1]. Ev- [25], have been proposed to improve the existing crack de-
ery year, government bodies across the globe allocate funds tection approaches. In addition, Hu and Zhao [26] proposed
to enhance the quality of their road networks [2]. Road safety a crack detection algorithm based on local binary patterns
should be taken very seriously and authorities are fully aware (LBP), whereas the authors of [27] utilized Gabor filter for
of the need for suitable road inspection and maintenance the same purpose. In [22], an automated crack detection
techniques [3]. Crack detection is an essential part of road algorithm based on a tree structure, referred to as CrackTree,
maintenance systems, and it has attracted growing interest was introduced. Moreover, Oliveira et al. [7], [28] utilized
from researchers in this field over the past few years [4]. Tra- a comprehensive set of image analysis algorithms to detect
ditional manual road crack detection approaches are known and characterize cracks from road pavements. Although the
to be very time-consuming, dangerous, labor-intensive and above-mentioned algorithms have been widely used in crack
subjective [5]. Therefore, the slow and subjective traditional detection and they perform well on high-quality datasets [22],
methods have been gradually replaced by automated crack [28], [29], it is important to note that these algorithms are
detection systems which provide fast and reliable analysis in not accurate enough to distinguish cracks from the complex
intelligent transportation systems (ITS) [6]. Automated crack background in low-quality images.
detection systems can effectively assess the quality of the Furthermore, some machine learning-based crack detec-
road surfaces and help governments plan and prioritize the tion approaches [30]–[35] have been proposed in recent
maintenance of the road network, thereby keeping the roads years, and the features produced by neural network are very
in good condition and extending their service life [7]. likely to replace the local features utilized in traditional
With the development of image analysis techniques, road methods [36]. For example, restricted Boltzmann machine
crack detection and recognition have been widely investi- (RBM) anto encoder and their variants are capable of de-
gated over the past few decades [8]–[11]. The traditional tecting cracks, when the training samples are limited [37].
framework for crack detection consists of defining a variety In addition, deep convolutional neural networks (DCNNs)
1 R. Fan, J. Jiao and M. Liu are with the Robotics and Multi-Perception are popular for feature-learning and supervised classification
Laboratory in Robotics Institute at the Hong Kong University of Science [38]. Zhang et al. [36] trained a neural network to determine
and Technology, Hong Kong. whether the patches in an images contain cracks or not.
2 M. J. Bocus is with the Visual Information Institute at the University
Hence in this paper, we build on the recent successful
of Bristol, Bristol, United Kingdom.
3 Y. Zhu and F. Ma are with Unity-Drive Technology Inc, Shenzhen, application of deep neural network to image classification
China. and train a convolutional neural network (CNN) to find
4 L. Wang and S. Cheng are with the National Engineering Research
the images that contain cracks. Then, we present a novel
Center of Road Maintenance Technologies, Beijing, China.
∗ These two authors contributed equally to this work and therefore are thresholding method to extract cracks from classified color
joint first authors. Corresponding author: Rui Fan. Email: [email protected]. images.
227×227×3 227×227×2
56×56×8
28×28×8
7×7×32 7×7×64 3×3×64 3×3×128 1×1×2
14×14×32
14×14×16
28×28×16
56×56×4 convolution + batch normalization + ReLU
113×113×4 max pooling
113×113×2 fully connected + softmax
Image Input
Fig. 1. The structure of the proposed deep neural network for image classification.
input convolution
batch
normalization
output ReLU
Fig. 2. The structure of the green block in Fig. 1. (a) (b) (c)
Region 4 Region 2
within-cluster
Fig. 4. Image downsampling. sum squares
(b)
Fig. 6. Experimental results of image classification. (a) True positive images. (b) True negative images.
TABLE I
C OMPARISON BETWEEN OUR PROPOSED METHOD AND OTSU ’ S
(a) THRESHOLDING METHOD .
both four). The true positive and true negative examples are
Fig. 7. The failed classification results. (a) False positive images. (b) False
negative images. shown in Fig. 6, while the false positive and false negative
results are shown in Fig. 7. The image classification takes
about 4.8 ms on an Intel Core i7-8700K CPU processed with
a single core (3.7 GHz).
In our practical experiments, we randomly select 15000 Furthermore, we evaluate the performance of the image
positive images and 15000 negative images from the dataset, segmentation at pixel level. Some experimental results of
to train the neural network. The rest of the images are utilized image filtering and segmentation are shown in Fig. 8. Since
to evaluate the performance of the proposed approach. The the dataset we use does not contain pixel-level ground truth,
initial learning rate, the maximum number of epochs and the we manually label the crack areas in a set of images and use
validation frequency are set to 0.01, 16 and 60, respectively. these ground truth images to quantify the performance of our
The stochastic gradient descent with momentum (SGDM) is proposed segmentation method. Moreover, we compare our
utilized as the optimizer, and the value of momentum is set method with Otsu’s thresholding method [44] with respect to
to 0.9. pixel-level precision, recall, accuracy and F1 -measure. The
To quantify the accuracy of our proposed image classier, notations ntp , nfp , nfn and ntn in (6), (7) and (8) represent the
we compute ntp , nfp , nfn and ntn , which represent the number of pixels that are true positive, false positive, false
number of testing images that are true positive, false positive, negative and true negative, respectively. The comparison
false negative and true negative, respectively. The precision, between these two methods is shown in Table I. It is to be
recall, accuracy and F1 -measure can be computed using the noted here that the crack areas with less than 100 pixels
following equation: are ignored in our experiments. From Table I, it can be
ntp observed that our method outperforms Otsu’s thresholding
precision = , (6)
ntp + nfp method in terms of precision, accuracy and F1 -measure,
ntp and the proposed segmentation method achieves the best
recall = , (7) performance when τ is set to 1.
ntp + nfn
ntp + ntn
accuracy = , (8) IV. C ONCLUSION AND F UTURE WORK
ntp + ntn + nfp + nfn
A novel crack detection approach was proposed in this
precision · recall
F1 = 2 . (9) paper. The main novelties include a fully connected neural
precision + recall network for image classification and a k-mean clustering
The values of precision, recall, accuracy and F1 -measure based image segmentation algorithm. Firstly, our neural
achieved using the proposed network are 99.92% (the num- network classified the input images as either positive (crack
ber of false positive images and false negative images are present) or negative (crack absent). The positive images
Fig. 8. Experimental results of image segmentation; (a) input images; (b) filtered images; (c) results obtained using Otsu’s thresholding method; (d)-(g)
results obtained using the proposed method when τ is set to 1, 2, 3 and 4, respectively; (h) ground truth.
were then processed using a bilateral filter, which not only (road surface), some color images with a large number of
minimized the number of noisy pixels but also preserved noisy pixels cannot be properly segmented. Therefore, as a
the edges between the cracks and road surface. Finally, the future work, a deep neural network can be trained to segment
filtered images were downsampled, and an adaptive threshold the positive images into a set of semantically meaningful
was computed by minimizing the within-cluster sum squares. regions, i.e., cracks and road surface.
The cracks can therefore be detected by segmenting the
filtered images using the adaptively determined threshold. ACKNOWLEDGMENT
The experimental results showed that the precision of the
image classification is 99.92% and the pixel-level segmenta- This work is supported by grants from the Research
tion accuracy is around 98.70%. Grants Council of the Hong Kong SAR Government, China
(No. 11210017 and No. 21202816) awarded to Prof. Ming
Although the proposed image segmentation algorithm per- Liu. This work is also supported by grants from the
forms better than Otsu’s thresholding method in terms of Shenzhen Science, Technology and Innovation Commission,
distinguishing between foreground (cracks) and background JCYJ20170818153518789, and National Natural Science
Foundation of China (No. 61603376) awarded to Dr. Lujia [22] Q. Zou, Y. Cao, Q. Li, Q. Mao, and S. Wang, “Cracktree: Automatic
Wang. crack detection from pavement images,” Pattern Recognition Letters,
vol. 33, no. 3, pp. 227–238, 2012.
R EFERENCES [23] K. Fernandes and L. Ciobanu, “Pavement pathologies classification
using graph-based features,” in Image Processing (ICIP), 2014 IEEE
[1] R. Fan, Y. Liu, X. Yang, M. J. Bocus, N. Dahnoun, and S. Tancock, International Conference on. IEEE, 2014, pp. 793–797.
“Real-time stereo vision for road surface 3-d reconstruction,” in 2018 [24] Y.-C. Tsai, C. Jiang, and Y. Huang, “Multiscale crack fundamental
IEEE International Conference on Imaging Systems and Techniques element model for real-world pavement crack classification,” Journal
(IST). IEEE, 2018, pp. 1–6. of Computing in Civil Engineering, vol. 28, no. 4, p. 04014012, 2012.
[2] H. Oliveira and P. L. Correia, “Automatic road crack segmentation [25] Y. J. Tsai, C. Jiang, and Z. Wang, “Implementation of automatic crack
using entropy and image dynamic thresholding,” in Signal Processing evaluation using crack fundamental element,” in Image Processing
Conference, 2009 17th European. IEEE, 2009, pp. 622–626. (ICIP), 2014 IEEE International Conference on. IEEE, 2014, pp.
[3] R. Fan, X. Ai, and N. Dahnoun, “Road surface 3D reconstruction 773–777.
based on dense subpixel disparity map estimation,” vol. 27, no. 6, pp. [26] Y. Hu and C.-x. Zhao, “A novel lbp based methods for pavement crack
3025–3035, Jun. 2018. detection,” Journal of pattern Recognition research, vol. 5, no. 1, pp.
[4] T. S. Nguyen, M. Avila, and S. Begot, “Automatic detection and 140–147, 2010.
classification of defect on road pavement using anisotropy measure,” [27] M. Salman, S. Mathavan, K. Kamal, and M. Rahman, “Pavement crack
in Signal Processing Conference, 2009 17th European. IEEE, 2009, detection using the gabor filter,” in 16th international IEEE conference
pp. 617–621. on intelligent transportation systems (ITSC 2013). IEEE, 2013, pp.
[5] R. Fan, “Real-time computer stereo vision for automotive applica- 2039–2044.
tions,” Ph.D. dissertation, University of Bristol, 2018. [28] H. Oliveira and P. L. Correia, “Crackit-an image processing toolbox
[6] R. Fan, J. Jiao, J. Pan, H. Huang, S. Shen, and M. Liu, “Real-time for crack detection and characterization,” in Image Processing (ICIP),
dense stereo embedded in a uav for road inspection,” arXiv preprint 2014 IEEE International Conference on. IEEE, 2014, pp. 798–802.
arXiv:1904.06017. [29] S. Varadharajan, S. Jose, K. Sharma, L. Wander, and C. Mertz, “Vision
[7] H. Oliveira and P. L. Correia, “Automatic road crack detection and for road inspection,” in Applications of Computer Vision (WACV), 2014
characterization,” IEEE Transactions on Intelligent Transportation IEEE Winter Conference on. IEEE, 2014, pp. 115–122.
Systems, vol. 14, no. 1, pp. 155–168, 2013. [30] H. R. Roth, L. Lu, J. Liu, J. Yao, A. Seff, K. Cherry, L. Kim, and R. M.
[8] H. Oh, N. W. Garrick, and L. E. Achenie, “Segmentation algorithm Summers, “Improving computer-aided detection using convolutional
using iterative clipping for processing noisy pavement images,” in neural networks and random view aggregation,” IEEE transactions on
Imaging Technologies: Techniques and Applications in Civil En- medical imaging, vol. 35, no. 5, pp. 1170–1181, 2016.
gineering. Second International ConferenceEngineering Foundation; [31] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Deep
and Imaging Technologies Committee of the Technical Council on neural networks segment neuronal membranes in electron microscopy
Computer Practices, American Society of Civil Engineers., 1998. images,” in Advances in neural information processing systems, 2012,
[9] M. Petrou, J. Kittler, and K. Song, “Automatic surface crack detection pp. 2843–2851.
on textured materials,” Journal of materials processing technology, [32] D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber,
vol. 56, no. 1-4, pp. 158–167, 1996. “Mitosis detection in breast cancer histology images with deep neural
[10] Y. Huang and B. Xu, “Automatic inspection of pavement cracking networks,” in International Conference on Medical Image Computing
distress,” Journal of Electronic Imaging, vol. 15, no. 1, p. 013017, and Computer-assisted Intervention. Springer, 2013, pp. 411–418.
2006. [33] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
[11] M. Gavilán, D. Balcones, O. Marcos, D. F. Llorca, M. A. Sotelo, with deep convolutional neural networks,” in Advances in neural
I. Parra, M. Ocaña, P. Aliseda, P. Yarza, and A. Amı́rola, “Adaptive information processing systems, 2012, pp. 1097–1105.
road crack detection system by pavement classification,” Sensors, [34] Y. Zhang, K. Sohn, R. Villegas, G. Pan, and H. Lee, “Improving object
vol. 11, no. 10, pp. 9628–9657, 2011. detection with deep convolutional networks via bayesian optimization
[12] U. Ozgunalp, R. Fan, X. Ai, and N. Dahnoun, “Multiple lane detection and structured prediction,” in Proceedings of the IEEE Conference on
algorithm based on novel dense vanishing point estimation,” IEEE Computer Vision and Pattern Recognition, 2015, pp. 249–258.
Transactions on Intelligent Transportation Systems, vol. 18, no. 3, pp. [35] J. Kivinen, C. Williams, and N. Heess, “Visual boundary prediction:
621–632, 2017. A deep neural prediction network and quality dissection,” in Artificial
[13] R. Fan, V. Prokhorov, and N. Dahnoun, “Faster-than-real-time linear Intelligence and Statistics, 2014, pp. 512–521.
lane detection implementation using soc dsp tms320c6678,” in 2016 [36] L. Zhang, F. Yang, Y. D. Zhang, and Y. J. Zhu, “Road crack detection
IEEE International Conference on Imaging Systems and Techniques using deep convolutional neural network,” in Image Processing (ICIP),
(IST). IEEE, 2016, pp. 306–311. 2016 IEEE International Conference on. IEEE, 2016, pp. 3708–3712.
[14] M. S. Kaseko and S. G. Ritchie, “A neural network-based methodol- [37] Y. Xu, S. Li, D. Zhang, Y. Jin, F. Zhang, N. Li, and H. Li,
ogy for pavement crack detection and classification,” Transportation “Identification framework for cracks on a steel structure surface by
Research Part C: Emerging Technologies, vol. 1, no. 4, pp. 275–291, a restricted boltzmann machines algorithm based on consumer-grade
1993. camera images,” Structural Control and Health Monitoring, vol. 25,
[15] Q. Li and X. Liu, “Novel approach to pavement image segmentation no. 2, p. e2075, 2018.
based on neighboring difference histogram method,” in Image and [38] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol.
Signal Processing, 2008. CISP’08. Congress on, vol. 2. IEEE, 2008, 521, no. 7553, p. 436, 2015.
pp. 792–796. [39] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
[16] R. Fan, M. J. Bocus, and N. Dahnoun, “A novel disparity transforma- network training by reducing internal covariate shift,” arXiv preprint
tion algorithm for road segmentation,” Information Processing Letters, arXiv:1502.03167, 2015.
vol. 140, pp. 18–24, 2018. [40] R. Fan and N. Dahnoun, “Real-time stereo vision-based lane detection
[17] N. Tanaka and K. Uematsu, “A crack detection method in road surface system,” Measurement Science and Technology, vol. 29, no. 7, p.
images using morphology.” MVA, vol. 98, pp. 17–19, 1998. 074005, 2018.
[18] H. Oliveira and P. L. Correia, “Supervised strategies for cracks [41] R. Fan, Y. Liu, M. J. Bocus, L. Wang, and M. Liu, “Real-time subpixel
detection in images of road pavement flexible surfaces,” in Signal fast bilateral stereo,” arXiv preprint arXiv:1807.02044.
Processing Conference, 2008 16th European. IEEE, 2008, pp. 1–5. [42] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy
[19] J. Jiao, R. Fan, H. Ma, and M. Liu, “Using dp towards a shortest path minimization via graph cuts,” vol. 23, no. 11, pp. 1222–1239, Nov.
problem-related application,” arXiv preprint arXiv:1903.02765, 2019. 2001.
[20] M. Avila, S. Begot, F. Duculty, and T. S. Nguyen, “2d image based [43] A. Ahmad and L. Dey, “A k-mean clustering algorithm for mixed nu-
road pavement crack detection by calculating minimal paths and meric and categorical data,” Data & Knowledge Engineering, vol. 63,
dynamic programming,” in Image Processing (ICIP), 2014 IEEE no. 2, pp. 503–527, 2007.
International Conference on. IEEE, 2014, pp. 783–787. [44] N. Otsu, “A threshold selection method from gray-level histograms,”
[21] R. Amhaz, S. Chambon, J. Idier, and V. Baltazart, “Automatic crack and Cybernetics IEEE Transactions on Systems, Man, vol. 9, no. 1,
detection on 2d pavement images: An algorithm based on minimal pp. 62–66, Jan. 1979.
path selection, accepted to ieee trans,” Intell. Transp. Syst, 2015.