Real-Time Lane

and Road Surface Marking

Detection
 Abstract
 We propose a novel, real-time, integrated, unsupervised learning framework for lane detection, tracking and road surface marking
detection and recognition using live feed from a camera mounted on the dashboard of a moving vehicle. Accurate and automated
lane and road surface marking detection and recognition from cameras play an important role in intelligent vehicles.

 It helps increase driving safety in Advanced Driver Assistant Systems and enhance global localisation and path planning in
autonomous driving vehicles. Map building and updation can also benefit from automated detection of lanes and road surface
markings.

 In our framework, we propose a spatio-temporal incremental clustering algorithm coupled with curve fitting for simultaneously
detecting lanes and road surface markings. The clusters that do not represent the lanes are further processed using Grassmann
manifold learning to accurately recognise the road surface markings.

 Our framework is independent of the number and type of lanes, be it continuous, non-continuous, straight or curved. Moreover,
the framework works for all types, sizes and orientations of road surface markings. It is only limited by the type of road surface
markings present in the training data. We have measured the performance of our framework on state of the art datasets.
 Introduction
 Lane detection and tracking as well as road surface markings detection and recognition are an integral part of an
Advanced Driver Assistance System (ADAS). They provide helpful and correct information to the driver and help to
reduce driving errors such as unintended lane departure, thereby decreasing the number of accidents and increasing
safety in driving. This project describe the importance of lane and road surface markings detection in driver assistance
systems by explaining various scenarios of driver behaviour while driving.

 Lane detection, tracking and road surface marking are also an integral part of an autonomous driving vehicle.
Autonomous driving vehicles need to estimate the pose (position and orientation) of the vehicle in real time. Another
important component of autonomous vehicle navigation is to have highly accurate maps with important road features.
These maps, along with the pose information estimated by the vehicle, provide information that is necessary for both
vehicle localisation as well as path planning. In general, in ADAS systems Global Navigation Satellite System (GNSS)
is used to get the position data. Path planning is to derive an optimal trajectory or path of the vehicle as a function of time
from its current position, orientation and map information.
Domain - Machine Learning
 Machine learning is a branch of artificial intelligence (AI) and computer science which
focuses on the use of data and algorithms to imitate the way that humans learn, gradually
improving its accuracy.

 Machine learning is an important component of the growing field of data science. Through
the use of statistical methods, algorithms are trained to make classifications or predictions,
and to uncover key insights in data mining projects. These insights subsequently drive
decision making within applications and businesses, ideally impacting key growth metrics.
As big data continues to expand and grow, the market demand for data scientists will
increase. They will be required to help identify the most relevant business questions and the
data to answer them.
 Existing System
 Robust detection of lanes and road surface markings is very challenging because of the quality of appearance of
lanes and road surface markings.

 The appearance of these markings can be affected by dirt, illumination, environmental conditions, possible
occlusion by other vehicles and high volume of traffic.

 The illumination changes which could be caused by sunlight falling on the road, sudden illumination change
when the vehicle comes out of a tunnel, etc., also affects the appearance of these markings.

 Road surface markings may vary in color, size and orientation, depending on the position of view of the camera
on the vehicle. Therefore, automatic detection and recognition of lanes and road surface markings is challenging
specially in cases where visibility of markings is affected because of shadows, illumination variation, color
variation and various environmental conditions such as rain, fog, snow, etc.
 Proposed System
 We discuss our novel integrated, unsupervised framework for lane detection, tracking and road surface marking detection and
recognition.

 We assume that a camera is mounted behind the wind-shield of the vehicle and this camera sends the captured view to the on-
board computer.

 We apply our integrated spatio-temporal incremental clustering algorithm and Grassmann manifold learning framework for
lanes and road surface markings detection and recognition.

 Our spatio-temporal incremental clustering algorithm coupled with curve fitting is used for clustering the foreground regions.

 These clusters are of two types, namely, those that represent the lanes across space and time and those that represent either road
surface markings or noise.

 The clusters are separated into these two categories based on the cluster size and orientation. The clusters identified as lanes are
then displayed by curves fitted on the cluster elements. The clusters identified as nonlanes are then further processed in the
Grassmann learning framework to recognise the road surface markings as its actual information or to be classified as noise as
Advantages
 As a frame is captured by the camera and sent to our spatio-temporal clustering algorithm, the frame is
processed and the lane markings and road surface markings are simultaneously detected on-the-fly.

 Our spatio-temporal incremental clustering algorithm does not require any a priori information and works
well for various kind of lanes whether it is straight or curved, continuous or discontinuous.

 For road surface marking recognition, during the offline training, we create subspaces of each unique
road surface marking and apply Graph-Embedded Grassmann Discriminant Analysis (GGDA) for
recognition on-the fly.

 Our integrated framework for road surface markings detection and recognition works well for various
types of road surface marking, such as, directional arrows, speed limit signs and text painted on road
surface.
Modules
1. Region of interest (ROI) selection
2. Image gray-scaling and smoothing
3. Foreground detection and segmentation
4. Connected component detection
5. Lane Detection
6. Road surface markings
 System Requirements
 HARDWARE REQUIREMENTS
Processor - Intel Pentium Dual Core
Speed - 3.0 Ghz
RAM - 4 GB
Hard Disk - 500 GB
Key Board - Standard Windows Keyboard
Mouse - Optical Mouse
Monitor - LED Monitor
 SOFTWARE REQUIREMENTS
Operating system : Windows 8.1
Coding Language : Python 3.2
Tool : Anaconda 3
IDE : Notebook
Region of interest (ROI) selection
 We select a region of interest to reduce the number of pixels needed to be
processed, thereby reducing the computational complexity and time of our
integrated system for lane detection, tracking and road surface markings
detection. This makes our system work fast and efficiently in real time. We
choose the lower part of the image containing the road since the upper part
containing sky, trees, etc., does not provide information that is meaningful for
our system. We may use the terms ROI and image inter-changeably to imply
ROI unless otherwise mentioned.
 Image gray-scaling and smoothing
We then convert the ROI form RGB to gray-scale to improve the segmentation of the lanes and road
surface markings.

Since these are painted on the road surface using colors such as white or yellow, converting the ROI to
gray-scale converts these markings to a uniform gray-scale. This makes our system independent to the
requirements of having any prior information of the road. Each pixel of gray scaled image denotes the
intensity of pixel in the original image. The low intensity value will be given to dark pixels and high
intensity value will be given to bright pixels in RGB form. To reduce the noise present in the captured
image, we apply a 3 x 3 Gaussian smoothing kernel on the gray-scaled ROI. We choose a 3 x 3 smoothing
kernel to ensure that while the noise is reduced, important features are not removed. Smoothing the ROI
effectively removes unnecessary sharp intensity changes that hamper foreground detection.
Foreground detection and segmentation
In the next step, we apply adaptive thresholding to segment the ROI into foreground objects of interest and
the background. It converts the grayscaled image into binary image.

The white portion of the image considered as foreground contains the important markings which are
relevant for this work and the black portion is considered as background.

Figure shows the results of thresholding on the grayscaled images of dataset. The foreground region can
also contain noise because of high intensity regions of nearby vehicles, rails, fences, reflection of sunlight or
street light, etc., which get filtered out at a later step in our system.
Foreground detection and segmentation

 Fig. : (a) the ROI from original images, (b) corresponding grayscaled images and (c) the corresponding
thresholded images after applying adaptive thresholding on dataset [25] that has been taken from internet.
These are the steps we use for detecting the foreground regions in our proposed framework.
 Connected Component Detection
A connected component is a consecutive collection of boundary pixels of extracted foreground regions.
These connected components may vary in size due to variation in the type of lanes such as continuous or non-
continuous, different shapes of road surface markings and perspective distortion due to the view of the camera.
Some connected components may be very big in size due to a large area reflected by sunlight or street light on
the road having high intensity value and some may be very small because of the noise.

We eliminate these noisy connected components based on their area as they represent neither lane
markings nor road surface markings and are not relevant for our work. We find orientation of these connected
components by applying Principal Component Analysis (PCA). We then apply our spatio temporal incremental
clustering algorithm coupled with curve fitting on the relevant connected components for lane detection,
tracking and road surface markings detection as discussed in Section III-E.
 Spatio-temporal Incremental Clustering (STIC)

 In this stage, we cluster the foreground connected components to discover

the lanes and detect the road surface markings. The clustering is done across
space and time, that is, within and across frames such that the clusters
formed till frame fi are used for clustering the connected components of
frame fi+1. Then, these clusters are further processed to differentiate between
lanes, road surface markings and noise. To cluster the connected foreground
regions, the first connected component of the first frame forms the first
cluster and its orientation is set as the cluster orientation. For every other
foreground region, the following methodology is followed.
Spatio-temporal Incremental Clustering (STIC)
Processing of non-lane clusters

 The non-lane clusters could represent a road surface mark­ing or noise. We apply
Graph embedding Grassmann discrim­inant analysis (GGDA) for recognising the
road surface mark­ings and eliminating noise. A Grassmann manifold G(k; n) is a
set of linear subspaces in Rn such that all the subspaces have the same dimension,
k. A point Q(k; n) is a unique subspace Sk Rn, that is represented by a nk
orthonormal matrix HRn such that Sk = span(H). Q1 and Q2 are two distinct
points on G(k; n) if and only if span(Q1), span (Q2). The projection kernel in
Equation 3 gives the similarity between two points Q1 and Q2G(k; n).
Processing of non-lane clusters

Fig. 5: Bounded rectangle on road surface markings of dataset

Fig. 6: Road surface markings detected as non-lane clusters with normalised orientation of Dataset
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Output Screen
Conclusion
 We have proposed a novel, vision based, integrated framework based on spatio-temporal incremental
clustering coupled with curve fitting and Grassmann manifold learning for lane detection, tracking and road
surface marking detection and recognition from images by a camera mounted on the front of a moving
vehicle. The lanes are detected and tracked as clusters across space and time while the road surface markings
are recognised by creating a subspace of each unique road surface marking during the training phase and
applying Graph Embedded Grassmann Discriminant Analysis to recognize them on-the-fly. These subspaces
encompass the variability and uncertainty associated with the detected road surface markings when seen from
the camera mounted on the moving vehicle. Automated lane detection, tracking and road surface marking
detection and recognition are challenging problems because of the variations in illumination, appearance and
environmental conditions. In the future, deep learning-based feature extraction methods can be explored for
making the system more robust. Autonomous driving and ADAS in intelligent vehicles require accurate lane
detection and tracking and road surface marking for safe driving, global localization and path planning. We
have evaluated our system on standard datasets and shown that its performance is comparable with the state-
of-the-art methods.