1

Local Binary Patterns and Its Application to

Facial Image Analysis: A Survey
Di Huang, Caifeng Shan, Mohsen Ardebilian, Yunhong Wang, and Liming Chen

lance and security, and computer animation etc. LBP has been
Abstract—Local Binary Patterns (LBP) is a non-parametric exploited for facial representation in different tasks containing
descriptor whose aim is to efficiently summarize the local struc- face detection [4], [17]-[19], face recognition [20]-[26], facial
tures of images. In recent years, it has aroused increasing interest expression analysis [27]-[31], demographic (gender, race, age,
in many areas of image processing and computer vision, and has
shown its effectiveness in a number of applications, in particular
etc.) classification [32]-[33], and other related applications
for facial image analysis, including tasks as diverse as face detec- [34]-[35]. The development of LBP methodology can be well
tion, face recognition, facial expression analysis, demographic illustrated in facial image analysis, and most of its recent vari-
classification, etc. This paper presents a comprehensive survey of ations are proposed in this area.
LBP methodology including several more recent variations. As a Some brief surveys on image analysis [36] or face analysis
typical application of the LBP approach, LBP-based facial image [37]-[39] using LBP were given, but all these works discussed
analysis is extensively reviewed, while its successful extensions in
dealing with various tasks of facial image analysis are also hig-
limited papers of the literature, and many new related methods
hlighted. have appeared in more recent years. In this paper, we present a
comprehensive survey of the LBP methodology, including its
Index Terms—Local Binary Patterns (LBP), local features, face recent variations and LBP-based feature selection, as well as
detection, face recognition, facial expression analysis. the application to facial image analysis. To the best of our
knowledge, this paper is the first survey that extensively re-
I. INTRODUCTION views LBP methodology and its application to facial image
analysis, with more than 100 related literatures reviewed.
D uring the last few years, Local Binary Patterns (LBP) [1]
has aroused increasing interest in image processing and
computer vision. As a non-parametric method, LBP summa-
The remainder of this paper is organized as follows. The
LBP methodology is introduced in Section II. Section III
rizes local structures of images efficiently by comparing each presents the recent variations of LBP. LBP-based feature se-
pixel with its neighboring pixels. The most important proper- lection methods are discussed in section IV. Section V de-
ties of LBP are its tolerance regarding monotonic illumination scribes different facets of its applications on facial image
changes and its computational simplicity. LBP was originally analysis. Finally, Section VI concludes the paper.
proposed for texture analysis [2], and has proved a simple yet
powerful approach to describe local structures. It has been II. LOCAL BINARY PATTERNS
extensively exploited in many applications, for instance, face The original LBP operator labels the pixels of an image with
image analysis [3]-[4], image and video retrieval [5]-[6], en- decimal numbers, called Local Binary Patterns or LBP codes,
vironment modeling [7]-[8], visual inspection [9]-[10], motion which encode the local structure around each pixel. It proceeds
analysis [11]-[12], biomedical and aerial image analysis thus, as illustrated in Fig.1: Each pixel is compared with its
[13]-[14], remote sensing [15], so forth (see a comprehensive eight neighbors in a 3x3 neighborhood by subtracting the
bibliography of LBP methodology online [16]). center pixel value; The resulting strictly negative values are
LBP-based facial image analysis has been one of the most encoded with 0 and the others with 1; A binary number is ob-
popular and successful applications in recent years. Facial tained by concatenating all these binary codes in a clockwise
image analysis is an active research topic in computer vision, direction starting from the top-left one and its corresponding
with a wide range of important applications, e.g., hu- decimal value is used for labeling. The derived binary numbers
man-computer interaction, biometric identification, surveil- are referred to as Local Binary Patterns or LBP codes.

Di Huang is with MI Department, LIRIS Laboratory, CNRS 5205, Ecole

Centrale de Lyon, 69134, Lyon, France (e-mail: [email protected]).
Caifeng Shan is with Philips Research, HTC 36, 5656 AE Eindhoven,
Netherlands (e-mail: [email protected])
Mohsen Ardebilian is with MI Department, LIRIS Laboratory, CNRS 5205,
Ecole Centrale de Lyon, 69134, Lyon, France.
Yunhong Wang is with School of Computer Science and Engineering, IRIP
Lab, Beihang University, 100091, Beijing, China.
Liming Chen is with MI Department, LIRIS Laboratory, CNRS 5205, Ecole Fig. 1. An example of the basic LBP operator.
Centrale de Lyon, 69134, Lyon, France.
 2

One limitation of the basic LBP operator is that its small 3x3 binary patterns to describe the texture of images. Ojala et al.
neighborhood cannot capture dominant features with large named these patterns uniform patterns, denoted LBP U2 (P, R) . A
scale structures. To deal with the texture at different scales, the local binary pattern is called uniform if it contains at most two
operator was later generalized to use neighborhoods of dif- bitwise transitions from 0 to 1 or vice versa when the corres-
ferent sizes [1]. A local neighborhood is defined as a set of ponding bit string is considered circular. For instance,
sampling points evenly spaced on a circle which is centered at 00000000 (0 transitions) and 01110000 (2 transitions) are both
the pixel to be labeled, and the sampling points that do not fall uniform whereas 11001001 (4 transitions) and 01010011 (6
within the pixels are interpolated using bilinear interpolation, transitions) are not. It is observed that the uniform patterns
thus allowing for any radius and any number of sampling points account for around 90% of all the patterns in a (8, 1) neigh-
in the neighborhood. Fig. 2 shows some examples of the ex-
borhood and around 70% in a (16, 2) neighborhood in texture
tended LBP operator, where the notation (P, R) denotes a
images [1]. A similar experiment was conducted on the FERET
neighborhood of P sampling points on a circle of radius of R.
database, and it was found that 90.6% of the patterns in a (8, 1)
neighborhood and 85.2% in a (8, 2) neighborhood are uniform
[20]. More recently, Shan and Gritti [41] verified validity of
uniform patterns for representing faces from the viewpoint of
machine learning. Specifically, they applied AdaBoost to select
the discriminative patterns for facial expression recognition,
and their experiments demonstrated that, using LBP (8, 2) oper-
ator, 91.1% of these selected patterns are uniform. Accumu-
Fig. 2. Examples of the extended LBP operator [20]: the circular (8, 1), (16, 2), lating the non-uniform patterns into a single bin yields an LBP
and (24, 3) neighborhoods. operator with less than 2p labels. For example, the number of
labels with the neighborhood of 8 pixels is 256 for the standard
Formally, given a pixel at (xc, yc), the resulting LBP can be
LBP but only 59 for LBPU2.
expressed in decimal form as:
P −1 It should be noted that, at about the same time that the
LBPP , R ( xc , yc ) = ∑ s(iP − ic )2 P (1)
original LBP operator was proposed, Zabih and Woodfill in-
P =0
troduced a Census Transform (CT) method [42] which is very
where ic and iP are respectively gray-level values of the central similar to LBP. CT also maps the local neighborhood sur-
pixel and P surrounding pixels in the circle neighborhood with rounding a pixel onto a binary string, and the only difference
a radius R, and function s(x) is defined as: between LBP and CT is the opposite order of bit string. Later,
⎧1 if x ≥ 0 (2)
s( x) = ⎨ CT and its variations were exploited for facial image analysis
⎩0 if x < 0 [43]-[45].
By the definition above, the basic LBP operator is invariant The C/C++ and Matlab implementations of the LBP operator
to monotonic gray-scale transformations preserving pixel in- can be found online [46].
tensity order in the local neighborhoods. The histogram of LBP
labels calculated over a region can be exploited as a texture III. RECENT VARIATIONS OF LBP
descriptor.
LBP methodology has been developed recently with plenty
The operator LBP (P, R) produces 2p different output values,
corresponding to 2p different binary patterns formed by P pix- of variations for improved performance in different applica-
els in the neighborhood. If the image is rotated, these sur- tions. These variations focus on different aspects of the original
rounding pixels in each neighborhood will move correspon- LBP operator: (1) improvement of its discriminative capability;
dingly along the perimeter of the circle, resulting in a different (2) enhancement of its robustness; (3) selection of its neigh-
LBP value, except patterns with only 1s and 0s. In order to borhood; (4) extension to 3D data; (5) combination with other
remove rotation effect, a rotation-invariant LBP is proposed in approaches. In this section, we review recent variations of LBP
[1]: (See Table. 1 for the overview).
LBPPri, R = min{ROR( LBPP , R , i) | i = 0, 1, ..., P − 1} (3)
A. Enhancing the discriminative capability
where ROR(x, i) performs a circular bit-wise right shift on the The LBP operator defines a certain number of patterns for
P-bit number x i times. The LBPri(P, R) operator quantifies occur-
describing the local structures. To enhance their discriminative
rence statistics of individual rotation invariant patterns cor-
capability, more patterns or information could be encoded. In
responding to certain micro-features in the image; hence, the
[17], Jin et al. modified the LBP operator to describe more local
patterns can be considered as a feature detector [1]. However,
in [40], it was shown that such a rotation-invariant LBP oper- structure information under certain circumstances. Specifically,
ator does not necessarily provide discriminative information, they proposed an Improved LBP (ILBP) which compares all
since the occurrence frequencies of the individual patterns the pixels (including the central pixel) with the mean intensity
incorporated in LBPri(P, R) vary greatly and the crude quantization of all the pixels in the patch (as shown in Fig. 3). For instance,
of the angular spaces at 45° intervals. the LBP (8, 1) operator produces only 256 (28) in a 3x3 neigh-
It has been shown that certain patterns contain more infor- borhood, while ILBP has 511 patterns (29-1, as all zeros and all
mation than others [1]. It is possible to use only a subset of 2p ones are the same). Later, ILBP was extended to use the
 3

neighborhoods of any size instead of the original 3x3 patch [48]. A Mean LBP is presented in [49], similar to ILBP, but
[47]. Almost at the same time, a similar strategy was introduced without considering the
to extend CT to Modified CT [43], named Modified LBP in
Subsection Variations Properties Year & Reference
2004 [17], 2005 [48]
Improved LBP (Mean LBP) Consider the effects of central pixels; present complete structure patterns.
2008 [49]

Hamming LBP Incorporate non-uniform patterns into uniform patterns 2007 [50]
A: Enhancing the discrimina-
tive ability
Extended LBP Discriminate the same local binary patterns; cause high dimensionality. 2007 [51, 52]

Completed LBP Include both the sign and the magnitude information of the given local region 2010 [53]

Local Ternary Patterns Bring in new threshold; no longer strictly invariant to gray-level transformation. 2007 [22]
B: Improving the robustness
Soft LBP Not invariant to monotonic grayscale changes; cause high computational complexity. 2007 [54]

Elongated LBP Extract the anisotropic information and lose anisotropic information; not invariant to rotation. 2007 [23]

C: Choosing the neighbor-

Multi-Block LBP Capture micro- and macro- structure information 2007 [18, 55]
hood

Three/Four Patch LBP Encode patch type of texture information 2008 [57]

3D LBP Extend LBP to 3D volume data 2007 [58], 2008 [59]

D: Extending to 3D
Volume LBP (LBP-TOP) Describe dynamic texture; cause high dimensionality. 2007 [30, 60]

2005 [67, 70], 2006 [68]

LBP and Gabor Wavelet Combine advantages of Gabor and LBP; increase time cost and cause high dimensionality.
2007 [64, 65], 2008 [66, 71]

E: Combining with other 2006 [72], 2009 [73]

LBP and SIFT Bring in the advantages of SIFT; reduce feature vector length
features 2010 [76]

LBP Histogram Fourier Obtain rotation invariance globally for the whole region 2009 [77]

Table.1 List of recent LBP variations.

central pixels. improve the discriminative capability of LBP. The ELBP op-
erator not only performs binary comparison between the central
pixel and its neighbors, but also encodes their exact gray-value
differences using some additional binary units. Specifically, the
ELBP feature consists of several LBP codes at multiple layers
which encode the gray-value difference (GD) between the
central pixel and its neighboring pixels. As shown in Fig. 4, the
first layer of ELBP is actually the original LBP code encoding
the sign of GD. The following layers of ELBP then encode the
absolute value of GD. Basically, each absolute GD value is first
Fig. 3. An example of ILBP operator [17]. encoded in its binary representation, and then all the binary
values at a given layer result in an additional local binary pat-
In [50], Yang and Wang proposed Hamming LBP to im- tern. For example, in Fig. 4, the first layer is the original LBP
prove the discriminative ability of the original LBP. They rec- code that encodes the sign of GD, thus yielding a decimal
lassified non-uniform patterns based on Hamming distance, number of 211 from its binary form (11010011)2. The absolute
instead of collecting them into a single bin as LBPu2 does. In the values of GD, i.e. 1, 5, 3, 2, 1, 2, 3, 0, are first encoded in their
Hamming LBP, these non-uniform patterns are incorporated binary numbers: (001)2, (101)2, (011) 2, (010) 2 , …, etc. Using a
into existing uniform patterns by minimizing the Hamming same weight scheme of LBP on all the binary bits, its ELBP
distance between them. E.g., the non-uniform pattern code of the corresponding layer can be generated, e.g., L2 is
(10001110)2 is converted into the uniform one (10001111)2 composed of (01000000)2 and its decimal value is 64; L3 is
since their Hamming distance is one. When several uniform composed of (00110110)2 and its decimal value is 54; finally L4
patterns have the same Hamming distance with a non-uniform is composed of (11101010)2 and its decimal value is 234. As a
pattern, the one with the minimum Euclidian distance will be result, when describing similar local textures, although the first
selected. layer LBP is not discriminative enough, the information en-
The Extended LBP (ELBP) [51]-[52] is another approach to coded in the other additional layers can be utilized to distin-
 4

guish them. Its downside is that ELBP greatly increases feature ploys two fuzzy membership functions instead of (2) for thre-
dimensionality. sholding:
More recently, Guo et al. proposed a Complete LBP (CLBP) ⎧0 x < −d
⎪
[53], which, in our opinion, is quite similar with ELBP. CLBP ⎪ x (5)
s1,d ( x) = ⎨0.5 + 0.5 −d ≤ x ≤ d
also includes both the sign and the gray-value differences be- ⎪ d
tween a given central pixel and its neighbors in order to im- ⎪⎩1 x>d
prove the discriminative power of the original LBP operator. s0, d ( x ) = 1 − s1, d ( x ) (6)
Unlike the binary bit coding strategy used by ELBP, CLBP
Parameter d controls the amount of fuzzification that the fuzzy
compares the absolute value of GD with the given central pixel
function performs. When the local neighborhood consists of P
again to generate an LBP liked code.
sampling points, the histogram with a uniform pattern operator
has bins numbered 0, 1, …, 2P-1. The contribution of a single
pixel (xc, yc) to bin h of the histogram is:
P −1
SLBP( xc , yc , h) = ∏ [bp (h)s1,d (i p − ic ) + (1 − bp (h))s0,d (i p − ic )] (7)
p =0

where bp(h)∈{0,1} denotes the numerical value of the pth bit of

binary representation of h.
With SLBP, one pixel contributes to more than one bin, but
the sum of the contributions of the pixel to all bins is always 1.
SLBP enhances the robustness in the sense that a small change
in the input image causes only a small change in output. But, it
loses the invariance to monotonic variations and also increases
the computation complexity. As with LTP, a proper value of d
should be set.
Fig. 4. An example of the ELBP operator. C. Choosing the neighborhood
The choice of an appropriate neighborhood for LBP-based
B. Improving the robustness techniques has a significant impact on the final performance. It
LBP is sensitive to noise since the operator thresholds ex- involves the number of sampling points, the distribution of the
actly at the value of central pixel. To address this problem, Tan sampling points, the shape of the neighborhood, and the size of
and Triggs [22] extended the original LBP to a version with the neighborhood as well.
3-value codes, called Local Ternary Patterns (LTP). In LTP,
indicator s(x) in (1) is replaced by:
⎧ 1 in ≥ ic + t
⎪ (4)
s ( in , ic , t ) = ⎨ 0 in − ic < t
⎪−1 in ≤ ic − t
⎩
where t is a user-specified threshold. The LTP codes are more
resistant to noise, but no longer strictly invariant to gray-level
transformations. A coding scheme is used to split each ternary Fig. 6. Two examples of Elongated LBP operator [23].
pattern into two parts: the positive one and the negative one, as
illustrated in Fig. 5. One problem of LTP is that threshold t Neighboring pixels in the original LBP are defined on a
needs to be set, which is not simple. circle. Liao and Chung [23] argued that the main reason to
define the neighborhood in such an isotropic manner is to ob-
tain rotation invariance for texture description. But this is not
suitable for all applications; on the contrary, the anisotropic
information could also be an important feature. As a result, they
proposed Elongated LBP with neighboring pixels lying on an
ellipse. Fig. 6 gives two examples of the Elongated LBP, where
A, B denote the long axis and short axis respectively, and m is
the number of neighboring pixels. Following original LBP,
bilinear interpolation technique is adopted for neighboring
pixels that do not fall exactly at the pixels. The Elongated LBP
operator could be rotated around the central pixel, with a spe-
Fig. 5. An example of the LTP operator [22].
cific angle to characterize elongated local structures in different
orientations, to achieve multi-orientation analysis.
The Soft LBP (SLBP) was introduced in [54], which em-
In order to capture not only the microstructures but also the
 5

macrostructures, Li et al. [18] [55] proposed Multi-Block LBP used for analyzing image sequences or videos. It should be
(MB-LBP) which, instead of comparing pixels, compares av- noted that this approach makes use of dynamic texture analysis
erage intensities of neighboring sub-regions. The original LBP of 2D time series instead of full 3D volumetric data. The VLBP
can be regarded as a special case of the MB-LBP. Fig. 7 shows features are not only insensitive to translation and rotation
an example of MB-LBP, where each sub-region consists of six (toward rotations around the z axis), but robust to monotonic
pixels. The sub-region can either be a rectangle or a square. The grayscale changes as well. Compared with LBP (P, R), VLBP (L, P,
average intensities over the blocks can be computed efficiently R) takes time domain into account, and the parameter L denotes
by using summed-area table [56] or integral image. A similar the length of the time interval. From a small local neighborhood
scheme is introduced in [57]: Three-Patch LBP (TP-LBP) and in volume, comparing neighboring pixels with the central pixel,
Four-Patch LBP (FP-LBP) are proposed to compare distances a number of binary units are obtained, and the weights for these
between the whole blocks (patches) concerned, instead of sin- units are given as a spiral line (see Fig. 8). In order to make
gle pixel [1] or average intensity in [55], and any distance VLBP computationally simple and easy to extend, only
function can be used (e.g., L2 norm of their gray level differ- co-occurrences on three separate planes are considered. The
ences). textures are modeled with the concatenated LBP histograms
extracted from Three Orthogonal Planes XY, XT, and YT, and
thus this simpler version of VLBP is named LBP-TOP. The
traditional circular sampling is replaced by an ellipse so that
different radius parameters can be set in the space and time
domain.
In [58], Fehr exploited the Spherical Harmonic transform to
produce an orthogonal basis on the 2-Sphere, and then compute
the LBP features in the frequency domain. This method over-
Fig. 7. An example of the MB-LBP operator [18].
comes both problems mentioned above. Paulhac et al. proposed
another solution to apply LBP to 3D [59]. They used a number
D. Extending to 3D LBP of circles to represent the sphere, adding the parameter S, thus
Several researchers have been trying to extend the LBP from the operator denotes LBP(S, P, R) (see Fig. 8) and they also de-
2D plane to 3D volume [30] [58]-[59] [60]; however, it is not fined the uniform rule as in 2D. This method causes the prob-
so straightforward as it appears at first glance. There are two lem that different textures could have the same LBP descrip-
difficulties: first, equidistant sampling on a sphere is a difficult tion.
job, and second, it is also difficult to set an order to those
sampling points, which is important to achieve rotation inva-
riance.

Fig. 9. Representation of a 3D local binary pattern (S=9, P=16, R=2) [59].

E. Combining with other features

As a method for describing local features, LBP can be
combined with other approaches. For example, a set of ap-
proaches were proposed to combine Gabor wavelets [61]-[63]
and LBP features using different methods. It was concluded in
[64]-[66] that Gabor wavelet and LBP based features are mu-
tually complementary, because LBP captures the local ap-
pearance detail; while Gabor-wavelets extract shape informa-
Fig. 8. Procedure of VLBP1 ,4, 1 [60].
tion over a broader range of scales. A simple fusion strategy is
To endow the LBP with the ability to capture dynamic tex- to first extract Gabor and LBP features in the parallel way, and
ture information, in [30] and [60], Zhao and Pietikäinen ex- then fuse two kinds of features on feature level, matching score
tended the LBP neighborhood from 2D plane to 3D space. The level, or decision level [65]-[66].
operator is named as Volume LBP (VLBP or 3D-LBP). VLBP Another combination way is the serial strategy which con-
combines motion and appearance information, and can thus be sists in first applying Gabor filters and then LBP to the raw
 6

image [24] [67]. The Multiple Gabor feature maps (GFM) are histogram with DFT. That means, rotation invariance is ob-
computed by convolving input images with multi-scale and tained globally, and features are thus invariant to rotations of
multi-orientation Gabor filters. Each GFM is divided into small the whole input signal, but they still retain information about
non-overlapped regions from which LBP histograms are ex- the relative distribution of different orientations of uniform
tracted and finally concatenated into a single feature histogram. LBP.
Multi-resolution Histograms of Local Variation Patterns
(MHLVP) [24] as well as Local Gabor Binary Pattern Histo- IV. LBP FEATURE SELECTION
gram (LGBPH) [67]-[69], have been proposed based on such a In most existing work, the input image is divided into small
procedure. Recently, He et al. [70] proposed a similar serial regions, from which LBP histograms are extracted, and the
method using both wavelets and LBP, which firstly uses local histograms are further concatenated into a spatially en-
wavelets to decompose raw images into four frequency images, hanced feature vector of the dimensionality of O (103). More-
i.e., low frequency, horizontal high frequency, vertical high over, some recent variations even increase the feature vector
frequency and diagonal high frequency, as the inputs of the length dramatically, such as Extended LBP, VLBP and Gabor
original LBP. Wavelets based LBP. It is believed that the derived LBP-based
Motivated by LBP-TOP and LGBPH, Lei et al. proposed to feature vector provides an over-complete representation with
construct a 3rd-order Gabor image based volume and then redundant information [78], which could be reduced to be more
apply the original LBP to three orthogonal planes, to extract the compact and discriminative. Furthermore, when building
discriminative information not only in the spatial domain, but real-time systems, it is also desired to have LBP-based repre-
also in the Gabor frequency and orientation domains [71]. To sentation with reduced feature length. For all the reasons, the
reduce the computation complexity, a GV-LBP operator is problem of LBP feature selection has recently been addressed
introduced to describe the variations in spatial, frequency and in many literatures. We classify these techniques into two
orientation domains simultaneously by defining the orientation categories: the first one is to reduce the feature length based on
and scale neighboring points in different Gabor images. some rules (like uniform patterns), whilst the other one exploits
feature selection techniques to choose the discriminative pat-
terns. Both streams have their own merits and drawbacks: the
first one is simple, but has limited feature selection ability; on
the contrary, the second has a better feature selection capacity,
but usually requires off-line training that could be computa-
tionally expensive.
A. Rule-based Strategy
Uniform pattern is an effective rule to select LBP features,
and it has been widely adopted in existing work. There are also
Fig. 10. LBP and CS-LBP features for a neighborhood of 8 pixels [73]. other rules which could be used. For instance, Lahdenoja et al.
[79] proposed a symmetry level scheme for uniform patterns to
A Center-Symmetric LBP (CS-LBP) [72]-[73], was pro- further reduce the length of LBP feature vectors. The symmetry
posed by only comparing pairs of neighboring pixels which are level Lsym of each pattern is defined as the minimum of the total
in the same diameter of the circle. This variation combines the number of ones and zeros in that pattern. For example, Lsym of
LBP operator with the SIFT [74] definition and thus produces both patterns (00111111)2 and (00011000)2 are 2. The sym-
fewer binary units than the original LBP does. The difference metry level is rotation invariant according to the definition. The
between CS-LBP and LBP with 8 neighboring pixels is given most symmetric pattern contains the same number of ones and
in Fig. 10. Later, in [75], Fu and Wei introduced Centralized zeros, indicating a symmetric edge, while the patterns with the
Binary Patterns (CBP), making a small modification to this lowest symmetry level are the ones consisting of only ones or
scheme; by comparing the central pixel with the mean of all the zeros. It is claimed that the patterns with high symmetry level
pixels in the neighborhood to produce an additional binary unit occur more frequently in the images with more discriminative
assigned the largest weight to strengthen its effect. More re- power [79]. This conclusion is supported by experiments: the
cently, Huang et al. [76] proposed to combine the LBP and comparative performance was obtained using only the patterns
SIFT approach in a serial way: first adopts the original LBP of high symmetry level, but the length of feature vectors was
operator with different parameter settings to extract MS-LBP reduced by a quarter.
images based on which SIFT is exploited to do matching.
Ahonen et al. proposed an approach, named LBP Histogram B. Boosting LBP Features
Fourier features (LBP-HF) [77], to combine LBP and the Dis- Boosting learning [80] provides an effective way for feature
crete Fourier Transform (DFT). Unlike the existing local rota- selection. In [78], by shifting and scaling a sub-window over
tion-invariant LBP features, the LBP-HF descriptor is pro- face image, many more sub-regions are obtained to extract
duced by computing an LBP histogram over the whole region local LBP histograms; the distance between the corresponding
and then constructing rotationally invariant features from the histograms of two images is utilized as the discriminative fea-
 7

ture, and AdaBoost is used to learn a few of the most efficient Local Binary Feature predicates as an improvement to tradi-
features. Compared with [3], the approach achieves slightly tional LBP. A feature selection method, named Binary Histo-
better performance but with fewer histograms computed from gram Intersection Minimization (BHIM), is introduced to
the local regions. A similar approach was also adopted in [27]. generate the predicates, which comprise individual point fea-
In these works, the n-th bin of a local histogram is utilized as a tures from multiple scales. The experiments illustrate that
whole for region description, and feature selection is performed BHIM establishes less redundant LBP feature sets than CMI
at region level. AdaBoost can also be exploited to learn the and AdaBoost do, and it produced promising performance [97].
discriminative bins of an LBP histogram [41], since not all the In [98], Nanni and Lumini adopted Sequential Forward Float-
bins are necessary to supply useful information. Their expe- ing Selection (SFFS) to select the LBP feature extracted from
riments illustrate that the selected LBP bins provide a much both 2D and 3D images.
more compact representation with a greatly reduced length of
feature vector, while producing better performance. AdaBoost V. LBP BASED FACIAL IMAGE ANALYSIS
has been widely adopted for LBP feature selection in various Machine-based face recognition involves two crucial aspects,
tasks [18] [25] [27] [32]-[34] [47] [51] [55] [81]-[88]. In [69], i.e. facial representation [3] [63] [99]-[103] and classifier de-
Yao et al. exploited RankBoost with domain-partitioning weak sign [104]-[106]. Facial representation consists in deriving a set
hypotheses to select the most discriminative LGBPH features. of relevant features from original images for describing faces,
C. LBP Subspace Learning in order to facilitate effective machine-based recognition.
“Good” facial features are desired to have the following prop-
Subspace learning (or Dimensionality Reduction) [89] maps
erties [4]: first, they can tolerate within-class variations while
data set from a high dimensional space to a lower dimensional
discriminate different classes well; second, they can be easily
space, and thus can be applied to LBP-based features to derive
extracted from the raw images to allow fast processing; and
a low-dimensional compact representation. For example, Chan
finally, they lie in a space with low dimensionality to avoid
et al. introduced Linear Discriminant Analysis (LDA) to
computationally expensive classifiers. Since it was introduced
project high-dimensional Multi-Scale LBP features into a dis-
for face representation [3], LBP has proved to be an efficient
criminant space [21], and the same scheme was later exploited
descriptor for facial image analysis as it fulfills the above cri-
with the Multi-Spectral LBP features calculated from color
teria quite well, and recent years have witnessed increasing
images [90]. To deal with the small sample size problem of
interest in LBP features for facial representation.
LDA, Shan et al. [68] proposed an Ensemble of Piecewise
In this section, we first present LBP-based facial description,
LDA, which partitions the entire LGBP feature vector into
and then review existing works on different tasks including
segments; then applies LDA to each segment separately. Their
face detection, face recognition, facial expression analysis,
approach was verified to be more effective than applying LDA
demographic classification, and other applications.
to high-dimensional holistic feature vector. Combining Gabor
wavelets and LBP features for face recognition, Tan and Triggs
[65] firstly projected original feature vectors into the Principal
Component Analysis (PCA) space, and then utilized Kernel
Discriminative Common Vectors (KDCV) to extract the dis-
criminative features.
Dual-Space LDA was also adopted to select discriminative
LBP features, and proved to be effective [91]. Zhao et al. [92]
employed Laplacian PCA (LPCA) for LBP feature selection,
and their experiments showed that LPCA outperforms PCA and
KPCA on selecting LBP-based feature. Wolf and Guttmann
Fig. 11. LBP based face description [27].
[93] adopted max-plus PCA to select LBP feature, and
achieved a better performance than traditional PCA. In [94], A. LBP-based Face Description
Shan et al. applied Locality Preserving Projections for mani- A face image can be considered as a composition of the mi-
fold learning. Gao and Wang [95] proposed to select LBP cro-patterns described by LBP. One can build an LBP histo-
feature by applying boosting learning in random subspaces. gram computed over the whole face image. However, such a
Specifically, multiple low-dimensional subspaces are randomly representation only encodes the occurrences of micro-patterns
generated from original high-dimension feature space as the without any indication about their locations. To also consider
input to boosting. the shape information of faces, Ahonen et al. [3] proposed to
D. Other Methods divide face images into m local regions from which local LBP
histograms can be extracted, and then to concatenate them into
Shan et al. [96] adopted the Conditional Mutual Information
a single, spatially enhanced feature histogram (as shown in Fig.
(CMI) maximization criterion for LBP feature selection. Their
11). The resulting histogram encodes both the local texture and
experiments show that selected LBP features perform very
global shape of face images.
well. Raja and Gong [97] proposed the Multi-Scale Selected
 8

proposed method detected 221 faces without any false posi-

tives. Later, they [109] proposed a hybrid method to address
face detection under unconstrained environments. Their me-
thod first searched for the potential skin regions in an input
image to avoid scanning the entire image as was done in [110].
Then a coarse-to-fine strategy is employed to determine
Fig. 12. The top four selected sub-regions [78]. whether the scanned regions are faces or not: in the coarse
stage, LBP feature vector extracted from the whole region is
Most of the existing works adopt the above scheme to extract utilized as the input to a polynomial SVM; patterns which are
LBP features for facial representation. However, dividing face not rejected by the first SVM classifier are further analyzed by
images into a grid of sub-regions is somewhat arbitrary, and the the second finer one whose inputs are extracted from over-
sub-regions are not necessary well aligned with facial features. lapped blocks inside the region. The detection rate reported is
Moreover, the resulting facial description depends on the 93.4% with 13 false positives.
chosen size and the positions of these sub-regions. To address Motivated by the fact that LBP is invariant to monotonic
this issue, in [78] and [96], many more sub-regions are obtained transformations, Zhang and Zhao [19] proposed to compute the
by shifting and scaling a sub-window over the face images, and spatial histograms on color measurements for face detection in
boost learning [80] is adopted to select the most discriminative color images. After extracting five measurements, Y, R, G, B
sub-regions in term of LBP histograms (as shown in Fig. 12). In and θ, in the RGB and YUV color space from the original
their experiments, sub-regions of various positions and dif- images, LBP was utilized to transform the obtained measures to
ferent sizes were selected. More recently, facial representation histograms, which are computed as facial description with 23
based on LBP histograms extracted from overlapped different spatial templates to preserve the shape information of
sub-regions was evaluated in [31]. Furthermore, the faces. A hierarchical classifier combining histogram matching
sub-regions do not need to be rectangular. For example, they and SVM was used to discriminate between faces and
can also be circular [20] or triangular [107] regions. non-faces. The experiments were conducted on 251 color im-
To achieve a more comprehensive description of local facial ages including 356 frontal faces with variations in color, posi-
patterns, the LBP operators with different numbers of sampling tion, size and also expression, and precision of 91.7% was
points and various neighborhood radii can be combined. E.g., reported. Jin et al. [17] exploited Improved LBP features for
in [21], [64] and [108], the Multi-Scale LBP or Multi-Radius facial representation, and modeled faces and non-faces using
LBP were introduced for facial description, to reduce sensitiv-
the multivariable Gaussian distribution. Given the ILBP based
ity of LBP-based face representations to the scale of face im-
features of an input image, the likelihood of the face class and
ages (Fig. 13). More recently, for facial expression recognition,
non-face class are calculated separately; the Bayesian decision
Shan and Gritti [41] first extracted LBP features of different
scales, and then ran AdaBoost to learn the most discriminative rule is then applied to decide whether the image is a face or not.
features. It proved that a boosted classifier of Multi-Scale LBP The Yale B and MIT-CMU databases were used for evaluation,
consistently outperforms that of single scale LBP, and the and the detection rate was more than 90% with a false positive
selected LBP bins are distributed at all scales on the rate of 2.99x10-7. They later extracted ILBP based features
Cohn-Kanade database. from larger neighborhoods [47], and trained a cascade Ada-
Boost detector, which achieved detection result of 93.0% on the
MIT-CMU database and 94.6% on the Yale B database. In [18],
Zhang et al. exploited MB-LBP for face detection. Experi-
mented on a dataset containing 13,000 faces and 50,000
non-faces with a false alarm rate set to 0.001, the MB-LBP
based approach achieved superior accuracy which was 15%
higher than Haar-like features and 8% higher than original LBP
features. The experiments on the MIT-CMU database also
showed that the approach displayed a comparable performance
Fig. 13. The evolvement from the LBP to Multi-Radius LBP operator [64]. with that in [80], but with fewer features.
B. Face Detection C. Face Recognition
The purpose of face detection is to determine the locations Face recognition aims to identify or verify a person from a
and sizes of human faces in digital images. In [4], Hadid et al. digital image or a video sequence. Ahonen et al [3] introduced
first used LBP for face detection. To describe low-resolution LBP in face recognition with Nearest Neighbor (NN) classifier
faces, a 4-neighborhood LBP operator, LBP(4, 1), was applied to and Chi square distance as the dissimilarity measure. The ex-
overlapping small regions. The Support Vector Machine perimental results showed that their approach outperforms
(SVM) classifier was adopted to discriminate faces from PCA, EBGM and the Bayesian Intra/Extra-personal Classifier
non-faces. To compare with the state-of-the-art methods, they on all four probe sets of the FERET database. They later in-
performed their experiments on the MIT-CMU dataset, and the vestigated whether these good results are due to the use of local
 9

regions or the discriminative capacity of LBP methodology Nanni and Lumini [98] also utilized LBP to extract both 2D and
[111]. Based on comparisons with three other texture descrip- 3D facial features; their experiments were conducted on a
tors extracting features from the same local patches, the subset of 198 persons from the Notre-Dame database collection
strength of LBP for representing faces was clearly confirmed. D; the reported EER was 3.5%.
In [4], face recognition experiments were also carried out on Li et al. [114]-[115] later applied LBP with NIR images to
the MoBo database, which is challenging because of the images obtain robust facial descriptions under illumination variations.
with low resolution. As mentioned above, AdaBoost later was The method achieved a verification rate of 90% at FAR=0.001
applied to select a few of the most effective LBP based features and 95% at FAR=0.01 on a database with 870 subjects. The
for face recognition [78]. Compared to the approach in [3], the same method was utilized with Enhanced NIR images for face
boosting-LBP based method achieves a slightly better recog- verification outdoor, especially in sunlight [83]. Pan et al. [86]
nition rate while using fewer LBP features. proposed to improve the robustness of this work to variations of
Zhang et al. [24] introduced MHLVP for face recognition pose. NIR face images were decomposed into several parts in
based on histogram intersection. Their experiments on FERET accordance with key facial components, and LBP features
database showed that their algorithm provides better accuracy extracted from these parts were selected by AdaBoost; the
than some milestone approaches, containing the best ones in outputs of part classifiers were then fused to give the final
FERET’97. In particular, they achieved 95.9% accuracy on the score. The verification rate of their approach is 96.03% with
fc set with illumination changes. They employed LGBPHS FAR=0.001. Huang et al. [51] adopted AdaBoost to learn
[67], similar to MHLVP but with weighted rules, for the same ELBP features for NIR face recognition, and obtained a rec-
task. In addition to the FERET dataset, they also ran experi- ognition rate of 95.74% on a database with 60 individuals.
ments on the AR database. The results on both databases were Yan et al. examined Multi-Radius LBP for face recognition
very promising. Furthermore, they [68] exploited LGBP with [64]. Their experiments on Purdue (90 subjects) and CMU-PIE
an ensemble of piecewise LDA, which not only reduces the (68 subjects) datasets showed that LBP and Gabor features are
feature dimension, but also improves the performance on the mutually complementary and a combination of similarity
FERET database. Yao et al. [69] adopted the Do- scores can bring performance improvement. In [21], Chan et al.
main-Partitioning RankBoost to select LGBPH features for employed the Multi-Scale LBP with LDA for face recognition.
face recognition. The subsets fb and Dup I from the FERET The reported performance on the FERET and XM2VTS data-
database were used for evaluating the approach, and compara- bases was better than the state-of-the-art approaches. They also
ble results were achieved with only 50 selected features. Zhang projected multi-spectral LBP features extracted from local
et al. [112] argued that Gabor phases are also useful for face regions into an LDA subspace as the discriminative regional
recognition. By encoding Gabor phases through LBP and description. They proved the effectiveness of their method on
forming local feature histograms, impressive recognition rates the FRGC and XM2VTS databases. Hadid et al. [82] [85]
were obtained on FERET database (99% for fb, 96% for fc, introduced VLBP to extract local facial dynamics for spa-
78% for Dup I, 77% for Dup II). tio-temporal face recognition from video sequences. AdaBoost
Zhao et al. [26] applied Kernel LDA with the LBP features was applied to learn the specific facial dynamics of each subject
for face recognition where their kernel function was designed from LBP-based features while ignoring intra-personal tem-
using the Chi square distance and Radial Basis Function. Their poral information such as facial expressions. Their approach
method has been proved effective on FRGC Exp2.0.1, achieved superior performances on various databases: MoBo
achieving a verification rate of 97.4% with FAR at 0.001 and (97.9%), Honda/UCSD (96.0%) and CRIM (98.5%). Lei et al.
99.2% with FAR at 0.01. Rodriguez and Marcel [113] proposed [71] used GV-LBP-TOP and E-GV-LBP based features for
an LBP-based generative method for face authentication. Spe- face recognition, and both methods achieved encouraging
cifically, LBP histogram is interpreted as a probability distri- results on FERET and FRGC2.0 databases. Yang and Wang
bution, and a generic face model is considered as a collection of [50] introduced Hamming LBP for face recognition on the
LBP histograms. A client-specific model is then obtained by FRGC dataset. The experimental results reveal that the Ham-
Maximum a Posteriori (MAP) from a generic face model. The ming LBP outperforms the original LBP, especially when
outcomes on XM2VTS and BANCA, reveals that their ap- variations of illumination and facial expression exist. Liao and
proach outperforms the approaches in [3] and [78]. Chung [23] exploited Elongated LBP to capture the anisotropic
Li et al. [25] designed a framework for fusing 2D and 3D structures of faces. Average Maximum Distance Gradient
face recognition based on LBP features at both feature and Magnitude was proposed to embed the information on
decision levels. AdaBoost was used for LBP feature selection. gray-level difference between the reference and the neighbor-
The experiments on a database containing 252 subjects illu- ing pixel in each Elongated LBP pattern. With a subset of 70
strate the advantages of two-level fusion over decision-level persons randomly selected from the FERET database, their
fusion. To our knowledge, this is the first work to apply LBP to method obtained 93.16% accuracy, and 98.50% on the ORL
3D domain. Later Huang et al. [52] extended LBP to 3DLBP, database.
actually similar to Extended LBP as in [65], for 3D face rec- Tan and Triggs [22] proposed a method for face recognition
ognition based on range images; their approach achieved the under illumination variations, which includes preprocessing for
promising result of 9.4% EER on FRGC v2.0 Exp3. ROC I. reducing sensitivity to illumination changes and LTP for
 10

solving the problem caused by LBP’s sensitivity to random and illumination discontinuity at each pixel. In particular, the kernel
quantization noise. A distance transform-based similarity me- weights are encoded into a local binary pattern to achieve fast
tric was used for decision. The method showed promising and memory-efficient processing. 633 frontal face images were
performance on three datasets with illumination variations: selected from the Yale B database, and average recognition
FRGC Exp 1.0.4 (86.3%); Yale-B (100%); as well as CMU PIE accuracy was 99.74% with 0.038s time consumed for each
(100%). They also fused Gabor and LBP features to construct image.
heterogeneous features for face recognition in [65]. With the Table 2 summarizes the performance of different approaches
features extracted by KDCV, they achieved satisfying results on the FERET database.
on the FRGC 1.0.4, FRGC 2.0.4 and FERET databases. Park D. Facial Expression Analysis
and Kim [116] presented an adaptive smoothing approach for
Machine-based facial expression recognition aims to recog-
face image normalization under changing lighting. The illu-
nize facial affect states automatically and may rely on both of
mination is estimated by iteratively convolving the input image
with a 3x3 averaging kernel weighted by a simple measure of
Author, Year, Reference  Facial Feature  Core Matching Algorithm  Reported Accuracy 

fb 0.970, Dup I 0.660, 
Ahonen 2004 [3, 23]  LBPU2 
(8, 2)  Weighted χ2 
fc 0.790, Dup II 0.640. 

Zhang 2004 [39]  Boosting LBP  χ2   fb 0.979 

fb 0.942, Dup I 0.676, 
Zhang 2005 [27]  MHLVP  Histogram Intersection 
fc 0.959, Dup II 0.594. 

Weighted  fb 0.980, Dup I 0.740, 
Zhang 2005 [74]  LGBPHS 
Histogram Intersection  fc 0.970, Dup II 0.710. 

fb 0.996, Dup I 0.920, 
Shan 2006 [40]  LGBP  EPFDA 
fc 0.990, Dup II 0.889. 

Yao 2007 [75]  LGBPH  CDP‐RankBoost  fb 0.970, Dup I 0.550. 

fb 0.986, Dup I 0.722, 
Chan 2007 [24]  Multi‐Scale LBP  LDA 
fc 0.711, Dup II 0.474. 

fb 0.980, Dup I 0.900, 
Tan 2007 [72]  Gabor+LBP  KDCV 
fc 0.980, Dup II 0.850. 

0.9316  
Liao 2007 [26]  Elongated LBP  AMDGM 
(only 70 samples for test) 

Weighted  fb 0.990, Dup I 0.780, 
Zhang 2008 [111]  ELGBP (Mag + Pha) 
Histogram Intersection  fc 0.960, Dup II 0.770. 

Weighted  0.8873 
Lei 2008 [77]  E‐GV‐LBP 
Histogram Intersection  (all four probe together) 

Table.2 Some performances of LBP-based face recognition on the FERET database.

audio and visual clues [117]. In this paper, we focus our atten- statistics and SVM were adopted to classify the basic proto-
tion on works purely based on visual information, using facial typical facial expressions; and the best performance obtained
motion or facial features [118]-[119]. Most of these works only on the Cohn-Kanade Database reached 88.4% using SVM. In
consider the prototypical emotional states which include seven many applications involving facial expression recognition, the
basic universal categories, namely neutral, anger, disgust, fear, input face images are with low-resolution. In [124]-[125], they
happiness, sadness, and surprise further studied this topic. They not only performed evaluation
Feng et al. [28] [120] exploited a coarse-to-fine classifica- on face images with different resolutions, but also ran experi-
tion scheme with LBP for facial expression recognition making ments on real-world low-resolution video sequences. It was
use of images. More precisely, at the coarse stage, a 7-class observed that LBP features perform stably and robustly over a
problem was first reduced to a 2-class one, while at fine stage; a useful range of face images with low resolutions. In [96], Shan
k-nearest neighbor classifier performed the final decision. Their et al. introduced CMI maximization criterion for LBP feature
approach produced 77% average recognition accuracy on selection, and the selected features improved recognition ac-
JAFFE dataset. In [121]-[123], with the same facial descrip- curacy compared with that using AdaBoost. Later, Shan et al.
tion, a Linear Programming technique was applied for expres- [94] also studied facial expression manifold learning by em-
sion classification. A 7-class problem was decomposed into 21 bedding image sequences in a high-dimensional LBP space to a
binary classifications using the one-against-one scheme. With low-dimensional manifold. Their experiments on the
this method, they obtained over 90% accuracy on both the Cohn-Kanade database illustrated that meaningful projections
JAFFE database and some real videos. could be obtained. Shan and Gritti [41] used AdaBoost to learn
Shan et al. [124] also investigated LBP for facial expression a set of discriminative bins of an LBP histogram for facial
recognition. The template matching with weighted Chi-square expression recognition. Their experiments indicated that the
 11

selected bins provide a much more compact facial description. scription with SVM for multi-view gender classification, and
It was also evidenced that it is necessary to consider the mul- reported an average accuracy of 94.08% on the CAS-PEAL
ti-scale LBP for facial description. By applying SVM to the face dataset. Yang and Ai [33] exploited the LBP based fea-
selected multi-scale LBP bins, they obtained recognition rate of tures for a face-based demographic classification which in-
93.1% on the Cohn-Kanade database, comparable to the best volved gender, ethnicity as well as age classification. Given a
results so far reported on this database. local patch, Chi square distance between achieved LBP histo-
In [70], He et al. used LBP on four kinds of frequency im- grams was utilized as a confidence measure for classification.
ages decomposed by Gabor wavelets for facial expression The positive mean histogram was utilized for initialization, and
recognition. Their approach provided better performance than the steep descent method was applied to find an optimal ref-
LBP did on the JAFFE dataset. To consider multiple cues, Liao erence template. They adopted the Real AdaBoost for training a
et al. [29] extracted LBPri(P, R) features in both intensity and gra- strong classifier. The achieved error rates for gender classifi-
dient maps, and then computed the Tsallis entropy of the Gabor cation on the FERET, PIE and a snapshot database were 6.7%,
filter responses as the first feature set and performed null-space 8.9% and 3.68% respectively. Their method also produced
LDA for the second feature set. With the SVM classifier, they promising performance for ethnicity and age classification.
achieved 94.59% accuracy for images of 64x64 pixels and Huang et al. [108] proposed an improved ASM framework,
84.62% for 16x16 pixels on the JAFFE database. With an namely ELBP-ASM, in which local appearance patterns of key
Active Appearance Model (AAM), Feng et al. [126] extracted points are modeled by the Extended LBP. The experiments on a
the local texture feature by applying LBP to facial feature dataset with 250 samples show that ELBP-ASM achieves more
points; the direction between each pair of feature points was accurate results than the original ASM. In order to extend ASM
also considered as shape information. In addition, they used to improve robustness against illumination variations, Marcel
LBP with the entire image to get global texture information. et al. [131] later presented a divided-square based LBP-ASM to
Combining these three types of feature, a NN-based classifier extract histograms from a square region divided into four
with weighted Chi square statistic was introduced for classifi- blocks around each landmark instead of the normal profile.
cation. Subject-independent recognition rate of 72% was re- Histograms were then concatenated into a single feature vector
ported on the JAFFE dataset. Cao et al. [127] combined LBP representing local appearance. The comparative experiments
with embedded Hidden Markov Model to recognize facial on XM2VTS dataset showed that this method outperforms
expressions by using an Active Shape Model (ASM), and ELBP-ASM [108], and requires only raw images for facial
achieved 65% accuracy on the JAFFE dataset. keypoint localization.
In [30] and [60], Zhao and Pietikäinen employed VLBP and In [35], Ma et al. introduced the Local Gabor Binary Pattern
LBP-TOP for facial expression recognition in video sequences. (LGBP) to encode the local facial characteristics for head pose
A recognition rate of 96.26% was achieved on the estimation. With an SVM classifier, estimation rate of 97.14%
Cohn-Kanade database; the evaluation over a range of image for 7 poses was gained on a subset of the CAS-PEAL dataset
resolutions and frame rates demonstrated that both approaches containing 200 subjects. Cao et al. [34] used a facial symme-
outperform the state-of-the-art methods. They also compute try-based approach to standardize face image quality. With this
LBP-TOP at multiple resolutions for describing dynamic method, facial asymmetries caused by non-frontal illumination
events [87]. AdaBoost technique was used to learn the principal and improper facial pose can be measured. The effectiveness
appearance and motion from the spatiotemporal descriptors. Fu was evaluated on images of 10 persons of the Yale B dataset.
and Wei [75] utilized the CBP instead of LBP for facial ex- LBP can also be used as a preprocessing technique on face
pression recognition, and recognition rates of 94.76% and images. For instance, Heusch et al. [132] considered LBP as a
94.86% were achieved on the JAFFE and Cohn-Kanade data- preprocessing step to remove lighting effects (Fig. 14). Com-
bases respectively. The capacity of LBP to describe faces was pared with other preprocessing methods, including Histogram
further demonstrated in [31], where Gritti et al. compared Equalization and the technique proposed by Gross & Brajovic
different local features: LBP, LTP, Histogram of Oriented [133], LBP provided better results on the XM2VTS database.
Gradients (HOG) [128], Gabor wavelets, with various para- Cardinaux et al. showed that LBP is better combined with
meter settings for facial expression recognition. Based on their feature-based HMM than with appearance-based LDA for face
experiments, LBP with an overlapping strategy achieved the recognition on the BANCA dataset [134]. Poh et al. presented a
best result, 92.9%, on the Cohn-Kanade database. Furthermore, similar comparative study in [135], and their experiments fur-
it was indicated that the overlapping LBP is the most robust to ther supported that LBP is effective for face preprocessing, but
deal with registration errors. combining LBP with feature-based Gaussian Mixtures Models
did not perform as well as combined with LDA. LBP applica-
E. Demographic Classification and Other Applications
tion for preprocessing was also addressed in [48]; a compara-
Demographic classification is used to classify by age, gender tive study on 5 preprocessing in 16 different Eigenspace-based
and ethnicity, based on face images. Sun et al. [32] adopted the face recognition systems evidenced that the Modified LBP
boosting-LBP based approach [78] for gender recognition, and (MLBP) achieves promising results for illumination compen-
obtained the performance of 95.75% on the FERET dataset. In sation and normalization. More recently, in order to highlight
[129]-[130], Lian and Lu combined the LBP-based facial de- the details of facial images, Huang et al. [136]-[137] proposed
 12

to use LBP to extract the range and texture LBP faces, and cessor. Cascade AdaBoost with Haar features was applied for
Canonical Correlation Analysis (CCA) was then applied to face and eye detection, while LBP was exploited for face au-
learn the relationship of two types of LBP faces for asymmetric thentication. Though the CPU and memory capabilities of
face recognition. The reported result was 95.61% on the FRGC mobile phones are limited, the experiments showed encourag-
v2.0 dataset [137]. ing performance on face detection and displayed recognition
rates of 82% for faces of 40x40 pixels and 96% for faces of
80x80 pixels. The system ran at 2 fps on video sequences with a
resolution of 320x240 pixels. Abbo et al. [146] recently
mapped an LBP-based facial expression recognition algorithm
proposed in [124] on a low-power smart camera, which was
assembled with a massively-parallel processor for low level
and intermediate vision processing, and an 8051 microcon-
Fig. 14. The original image (left) processed by the LBP operator (right) [132]. troller for high level decision making and the camera control
tasks.
F. Face Analysis Systems
Hannuksela et al. [88] proposed a head tracking system for
Advantages of LBP make it very attractive to build real-time controlling the user interface on hand-held mobile devices.
face analysis systems. Furthermore, related hardware designed Face and eye detection were realized using boosting-LBP ap-
for high-speed LBP computation [138]-[141] also boosts the proach. It worked in real-time on a resource-limited mobile
development of LBP-based real-world applications. device. In an interactive photo annotation system [147], LBP
Hadid et al. built an access control system using LBP-based was also used to extract facial features for face clustering and
face recognition [142]. In their system, a camera was set on a re-ranking.
door for capturing video frames; LBP features were extracted
for both background subtraction and face recognition. The face G. Discussion
detection approach in [110] was adopted for face detection in The techniques developed so far for facial representation can
color images, and the face recognition method in [3] was ap- be roughly classified into two main categories: holistic-based
plied for person identification. The face recognition accuracy ones and local-based ones [148]-[149]. The holistic approaches
of 71.6% was obtained on 20 video sequences of 10 subjects. use the whole facial region to construct a subspace using, e.g.
Trinh et al. [143] presented a system for detecting multiple Principal Component Analysis (PCA) [99], Linear Discrimi-
faces in video sequences where faces are not limited to frontal nant Analysis (LDA) [100], Independent Component Analysis
views. An adaptive selection approach from two skin models in (ICA) [150], or Locally Linear Embedding (LLE) [151]. On the
RGB and ratio RGB spaces is used to overcome the illumina- other hand, the local-based ones, e.g. [63] [101] [152], proceed
tion problem by automatic focus of the camera. The experi- firstly to locate a number of features or components from a face;
mental result of 93% accuracy was reported on the NRC-IIT then classify them by combining and comparing corresponding
database, which consists of 23 single-face video sequences of local statistics. The local approaches have shown promising
11 persons with different poses. The system runs at 2.57 fps for performances in recent years. It has been proved by Heisele et
image sequences of 320x240 pixels on a standard PC (Pentium al. [153] the component-based face recognition techniques
4 2.6GHz, 512MB RAM) in the Visual C++ environment. (local-based) outperform the global ones (holistic-based), as
Based on LBP features extracted from Near Infrared (NIR) the global approaches require face images to be accurately
faces, Li et al. [114]-[115] designed an illumination-invariant normalized according to pose, lighting and scale. In addition,
face recognition system for cooperative users in an indoor global features are also more sensitive to facial expression
situation. AdaBoost was used to learn Haar features for face variations and occlusions. Since local-based methods extract
detection and eye detection, and to select LBP features for face features from local points or patches, there always remain some
recognition. All three parts achieve outstanding results with invariant features even in the presence of facial expression or
low cost on a large dataset. The system can operate in real time occlusion variations, and recognition can still be achieved by
with an EER below 0.3%. matching remaining invariant features. Therefore, local-based
Ekenel et al. [144] introduced a portable face recognition methods are potentially more robust than holistic ones to ex-
system deployed on a laptop using a standard webcam for pression variations and occlusions. Moreover, unlike holistic
image acquisition. Based on the relevant regions determined by approaches, local-based ones require few samples for enroll-
skin color, the two eyes were first detected with a cascade ment, and can even achieve analysis with a single face image in
AdaBoost classifier of Haar features. These were then used to the gallery set [154]. From this viewpoint, if LBP can be used
register face images. LBP was used to preprocess facial regions in a straightforward manner as a global representation, it is its
for reducing illumination influences. Their system was eva- local- or component-oriented variants that prove to be efficient
luated on a small dataset consisting of 42 sequences from 14 descriptors for facial image analysis as the previous overview
subjects, and produced 79% accuracy. highlights. This motivates the increasing interests in
In [145], Hadid et al. implemented face detection and au- LBP-based features for facial representation since this method
thentication on mobile phones equipped with an ARM9 pro- was applied for face representation [3].
 13

Compared to other popular local-based features as discussed techniques of feature selection schemes for reducing the feature
in [148] [155]-[156], namely Gabor wavelets [63] [157]-[159] length LBP codes, including rule-based strategy, boosting and
and SIFT [101] [155], Luo et al. [155] showed that SIFT is not subspace learning, etc.
as robust as LBP to illumination effects for face recognition on As the most typical and important application of LBP, facial
the FERET dataset. Zou et al. [148] compared Gabor wavelets image analysis provides a very good demonstration of the use,
and LBP using the same database for the same task, and con- development and performance of LBP. From this comprehen-
cluded that Gabor wavelets are more insensitive to illumination sive overview, following conclusions can be drawn : (1) local-
changes since they detect amplitude-invariant spatial frequen- or component-oriented LBP representations are effective re-
cies of gray values of pixels, whilst LBP is greatly affected by presentations for facial image analysis as they encode the in-
non-monotonic gray value transformations. Ruiz-del-Solar et formation of facial configuration while providing local struc-
al. [156] evaluated these three methods extensively for face ture patterns; (2) using the local- or component-oriented LBP
recognition not only on controlled datasets, such as the FERET facial representations, feature selection is particularly impor-
and FRGC, but also on the UCH FaceHRI database, designed tant for various tasks in facial image analysis, since this facial
for Human-Robot Interaction, and the LFW database, captured description scheme greatly increases feature length.
in unconstrained environments. Concerning robustness to il- As the conclusions based on their experiments drawn in [39]
lumination changes, the study showed that Gabor wavelets revealed, like all texture based methods, LBP seems to be sen-
achieved the best performance on the FERET database, LBP sitive to severe illumination changes, and to blurred and noisy
was not far behind; while SIFT was the last, thus further sup- images. The former one can be seen as non-monotonic lighting
porting the previous conclusions. On the UCH FaceHRI data- variations, as is mostly the case in facial image, since compared
base, the LBP approach gained the best results in all the spe- with flat texture images, human faces depict 3D volume
cially-designed experiments for indoor and outdoor lighting, structures; therefore non-monotonic transformations, e.g.,
expression, scaling, and rotation, followed by Gabor wavelets shadows and bright spots can typically occur and change their
and SIFT. On LFW database, LBP and Gabor wavelets ob- positions depending on illumination directions. While the latter
tained a slightly better result than each other with aligned face one is often caused by the bad quality of camera sensors and
and non-aligned face respectively, both of which surpassed that poor user cooperation of capture condition etc. As a result, in
of SIFT. On the other hand, in their investigation on computa- such environments, it is necessary and useful to preprocess the
tion cost, LBP ran much faster than Gabor wavelet and SIFT. images before applying LBP.
In addition, some open questions for sub-region based LBP
VI. CONCLUDING REMARKS description, e.g. facial description, concern the relevant number
LBP is one of the most powerful descriptors for representing of components and the corresponding neighborhood of a cer-
local structures. Thanks to its advantages, i.e., its tolerance of tain LBP operator for the best analysis result. Although these
monotonic illumination changes and its computational sim- questions have been discussed in several works and even with
plicity, LBP has been successfully used for many different machine learning techniques, these conclusions drawn so far
image analysis tasks, such as facial image analysis, biomedical have always been dependent on the datasets used and on some
image analysis, aerial image analysis, motion analysis, and given parameters.
image and video retrieval.
During the development of LBP methodology, a large REFERENCES
number of variations are designed to expand the scope of ap- [1] T. Ojala, M. Pietikäinen, and T. Maenpaa, “Multiresolution gray-scale
and rotation invariant texture classification with local binary patterns,”
plication, offering better performance as well as improving IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 7,
robustness in one or more aspects of the original LBP. ILBP, pp. 971–987, 2002.
Hamming LBP and ELBP enhance the discriminative ability of [2] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of
texture measures with classification based on featured distribution,”
LBP; LTP and SLBP focus on improving the robustness of
Pattern Recognition, vol. 29, no.1, pp. 51–59, 1996.
LBP on noisy images; MB-LBP, Elongated LBP, TPLBP, and [3] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition with local
FPLBP, change the scale of LBP to provide other categories of binary patterns,” in Proc. Euro. Conf. Computer Vision (ECCV), 2004,
local information; Gabor wavelet-based LBP, CS-LBP as well pp. 469–481.
[4] A. Hadid, M. Pietikäinen, and T. Ahonen, “A discriminative feature space
as LBP-HF combine other methods with LBP to bring in new for detecting and recognizing faces,” in Proc. Int. Conf. Computer Vision
merits. However, the previous extensions only operate on tra- and Pattern Recognition (CVPR), 2004, pp. 797–804.
ditional 2D data, the variant 3DLBP and VLBP should be [5] D. P. Huijsmans and N. Sebe, “Content-based indexing performance: a
class size normalized precision, recall, generality evaluation,” in Proc.
highlighted, since both of them expand the scope of LBP ap- International Conference on Image Processing (ICIP), 2003, pp.
plications: 3DLBP extends the LBP operator to describe 3D 733-736.
volume data, whilst VLBP endows LBP with the ability to [6] D. Grangier and S. Bengio, “A discriminative kernel-based approach to
rank images from text queries,” IEEE Trans. Pattern Analysis and Ma-
capture dynamic information. chine Intelligence, vol. 30, no. 8, pp. 1371-1384, 2008.
To obtain a small set of the most discriminative LBP-based [7] W. Ali, F. Georgsson, and T. Hellström, “Visual tree detection for auto-
features for better performance and dimensionality reduction, nomous navigation in forest environment,” in Proc. IEEE Intelligent Ve-
LBP-based representations are associated with some popular hicles Symposium, 2008, pp. 560-565.
 14

[8] L. Nanni and A. Lumini, “Ensemble of multiple pedestrian representa- [31] T. Gritti, C. Shan, V. Jeanne, and R. Braspenning, “Local features based
tions,” IEEE Trans. on Intelligent Transportation Systems, vol. 9, no. 2, facial expression recognition with face registration errors,” in Proc. IEEE
pp. 365-369, 2008. Int. Conf. Automatic Face and Gesture Recognition (FG), 2008.
[9] T. Mäenpää, J. Viertola, and M. Pietikäinen, “Optimising colour and [32] N. Sun, W. Zheng, C. Sun, C. Zou, and L, Zhao, “Gender classification
texture features for real-time visual inspection,” Pattern Analysis and based on boosting local binary pattern,” in Proc. Int. Symposium on
Applications, vol. 6, no. 3, pp.169-175, 2003. Neural Networks (ISNN), 2006, pp. II: 194-201.
[10] M. Turtinen, M. Pietikäinen, and O. Silven, “Visual characterization of [33] Z. Yang and H. Ai, “Demographic classification with local binary pat-
paper using Isomap and local binary patterns,” IEICE Transactions on terns,” in Proc. Int. Conf. Biometrics (ICB), 2007, pp. 464-473.
Information and System, vol. E89D, no. 7, pp. 2076-2083, 2006. [34] X. Gao, S. Z. Li, R. Liu, and P. Zhang, “Standardization of face image
[11] M. Heikkilä and M. Pietikäinen, “A texture-based method for modeling sample quality,” in Proc. Int. Conf. Biometrics (ICB), 2007, pp. 242-251.
the background and detecting moving objects,” IEEE Trans. Pattern [35] B. Ma, W. Zhang, S. Shan, X. Chen, and W. Gao, “Robust head pose
Analysis and Machine Intelligence, vol. 28, no. 4, pp. 657-662, 2006. estimation using LGBP,” in Proc. Int. Conf. Pattern Recognition (ICPR),
[12] V. Kellokumpu, G. Zhao, and M. Pietikäinen, “Human activity recogni- 2006.
tion using a dynamic texture based method,” in Proc. The British Ma- [36] M. Pietikäinen, “Image analysis with local binary patterns,” in Proc.
chine Vision Conference (BMVC), 2008. Scandinavian Conference on Image Analysis (SCIA), 2005.
[13] A. Oliver, X. Lladó, J. Freixenet, and J. Martí, “False positive reduction in [37] S. Marcel, Y. Rodriguez, G. Heusch, “On the recent use of local binary
mammographic mass detection using local binary patterns,” in Proc. patterns for face authentication,” Research Report IDIAP-RR-34-2006,
Medical Image Computing and Computer-Assisted Intervention 2006.
(MICCAI), 2007. [38] A. Hadid, T. Ahonen, and M. Pietikäinen, “Face analysis using local
[14] S. Kluckner, G. Pacher, H. Grabner, H. Bischof, and J. Bauer, “A 3D binary patterns,” in M. Mirmehdi, X. Xie, and J. Suri (eds) Handbook of
teacher for car detection in aerial images,” in Proc. IEEE International Texture Analysis,.Imperial College Press, pp 347-373, 2008.
Conference on Computer Vision (ICCV), 2007. [39] A. Hadid, “The local binary pattern and its applications to face analysis,”
[15] A. Lucieer, A. Stein, and P. Fisher, “Multivariate texture-based segmen- in Proc.International Workshops on Image Processing Theory, Tools and
tation of remotely sensed imagery for extraction of objects and their un- Applications, pp. 28-36, 2008.
certainty,” International Journal of Remote Sensing, vol. 26, no. 14, [40] M. Pietikäinen, T, Ojala, and Z. Xu, “Rotation-Invariant Texture Classi-
pp. 2917-2936, 2005. fication using feature distributions,” Pattern Recognition (PR), vol.33,
[16] http://www.ee.oulu.fi/mvg/page/lbp_bibliography. The availability of the pp. 43-52, 2000.
link was last checked on 15 Nov., 2010. [41] C. Shan and T. Gritti, “Learning discriminative LBP-histogram bins for
[17] H. Jin, Q. Liu, H. Lu, and X. Tong, “Face detection using improved LBP facial expression recognition,” in Proc. British Machine Vision Confe-
under Bayesian framework,” in Proc Int. Conf. Image and Graphics rence (BMVC), 2008.
(ICIG), 2004, pp. 306–309. [42] R. Zabih and J. Woodfill, “Non-parametric local transforms for compu-
[18] L. Zhang, R. Chu, S. Xiang, and S. Z. Li, “Face detection based on Mul- ting visual correspondence,” in Proc. Euro. Conf. Computer Vision
ti-Block LBP representation,” in Proc. Int. Conf. Biometrics (ICB), 2007, (ECCV), 1994, pp. 151-158.
pp. 11-18. [43] B. Froba and A. Ernst, “Face detection with the modified census trans-
[19] H. Zhang and D. Zhao, “Spatial histogram features for face detection in form,” in Proc. IEEE Int. Conf. Automatic Face and Gesture Recognition
color images,” in Proc. Advances in Multimedia Information Processing: (FG), 2004, pp. 91-96.
Pacific Rim Conference on Multimedia, 2004, pp. I: 377-384. [44] J. H. Kim, J. G. Park, and C. Lee, “Illumination normalization for face
[20] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face description with local recognition using the census transform,” in Proc. SPIE, vol. 6814, 2008,
binary patterns: application to face recognition”, IEEE Transactions on pp. 161-169
Pattern Analysis and Machine Intelligence (PAMI), vol. 28, no. 12, pp. [45] X. Wang, H. Xu, H. Wang, and H. Li, “Robust real-time face detection
2037–2041, 2006. with skin color detection and the modified census transform,” in Proc.
[21] C. Chan, J. Kittler, and K. Messer, “Multi-scale local binary pattern IEEE Int. Conf. Information and Automation, 2008, pp. 590-595.
histograms for face recognition,” in Proc. Int. Conf. Biometrics (ICB), [46] http://www.ee.oulu.fi/mvg/page/downloads. The availability of this link
2007, pp. 809-818. was last checked on 15 Nov., 2010.
[22] X. Tan and B. Triggs, “Enhanced local texture feature sets for face rec- [47] H. Jin, Q. Liu, X. Tang, and H. Lu, “Learning local descriptors for face
ognition under difficult lighting conditions,” in Proc. Analysis and detection,” in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), 2005,
Modeling of Faces and Gestures (AMFG), 2007, pp. 168-182. pp. 928-931.
[23] S. Liao and A. C. S. Chung, “Face recognition by using elongated local [48] J. Ruiz-del-Solar, J. Quinteros, “Illumination compensation and norma-
binary patterns with average maximum distance gradient magnitude,” in lization in eigenspace-based face recognition: a comparative study of
Proc. Asian Conf. Computer Vision (ACCV), 2007, pp. 672-679. different pre-processing approaches,” Pattern Recognition Letters (PRL),
[24] W. Zhang, S. Shan, H. Zhang, W. Gao, and X. Chen, “Multi-resolution vol. 29, no. 14, pp. 1966–1979, 2008.
Histograms of Local Variation Patterns (MHLVP) for robust face recog- [49] G. Bai, Y. Zhu, and Z. Ding, “A hierarchical face recognition method
nition,” in Proc. Audio- and Video-based Biometric Person Authentica- based on local binary pattern,” in Proc. Congress on Image and Signal
tion (AVBPA), 2005, pp. 937-944. Processing, 2008, pp. II: 610 – 614.
[25] S. Z. Li, C. Zhao, M. Ao, and Z. Lei, “Learning to fuse 3D+2D based face [50] H. Yang and Y. Wang, “A LBP-based face recognition method with
recognition at both feature and decision levels,” in Proc. Int. Workshop on Hamming distance constraint,” in Proc. Int. Conf. Image and Graphics
Analysis and Modeling of Faces and Gestures (AMFG), 2005, pp. 44-54. (ICIG), 2007, pp. 645-649.
[26] J. Zhao, H. Wang, H. Ren, and S.-C. Kee, “LBP discriminant analysis for [51] D. Huang, Y. Wang, and Y. Wang, “A robust method for near infrared
face verification,” in Proc. IEEE Workshop on Face Recognition Grand face recognition based on extended local binary pattern,” in Proc. Int.
Challenge Experiments, 2005. Symposium on Visual Computing (ISVC), 2007, pp. 437-446.
[27] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition [52] Y. Huang, Y. Wang, and T. Tan, “Combining statistics of geometrical and
based on local binary patterns: a comprehensive study,” Image and Vision correlative features for 3D face recognition,” in Proc. British Machine
Computing, 2009. Vision Conference (BMVC), 2006, pp. III: 879-888.
[28] X. Feng, A. Hadid, and M. Pietikäinen, “A coarse-to-fine classification [53] Z. Guo, L. Zhang, and D. Zhang, “A completed modeling of local binary
scheme for facial expression recognition,” in Proc. Int. Conf. Image pattern operator for texture classification,” IEEE Transactions on Image
Analysis and Recognition (ICIAR), 2004, pp. 668–675. Processing (TIP), vol. 19, no. 6, pp. 1657-1663, 2010.
[29] S. Liao, W. Fan, A. C. S. Chung, and D. Y. Yeung, “Facial expression [54] T. Ahonen and M. Pietikäinen, “Soft histograms for local binary pat-
recognition using advanced local binary patterns, tsallis entropies and terns,” in Proc. Finnish Signal Processing Symposium (FINSIG), 2007.
global appearance features,” in Proc. IEEE Int. Conf. Image Processing [55] S. Liao and S. Z. Li, “Learning multi-scale block local binary patterns for
(ICIP), 2006, pp. 665 – 668. face recognition,” in Proc. Int. Conf. Biometrics, 2007, pp. 828-837.
[30] G. Zhao and M. Pietikäinen, “Experiments with facial expression recog- [56] P. Y. Simard, L. Bottou, P. Haffner, and Y. L. Cun, “Boxlets: a fast
nition using spatiotemporal local binary patterns,” in Proc. Int. Conf. convolution algorithm for signal processing and neural networks,” Ad-
Multimedia and Expo (ICME), 2007, pp. 1091-1094.
 15

vances in Neural Information Processing Systems, MIT Press, vol. 11, pp. [78] G. Zhang, X. Huang, S. Z. Li, Y. Wang, and X. Wu, “Boosting Local
571-577, 1998. Binary Pattern-based face recognition,” in Proc. Advances in Biometric
[57] L. Wolf, T. Hassner, Y. Taigman, “Descriptor Based Methods in the Person Authentication, 2004, pp. 179-186.
Wild,” in Proc. ECCV Workshop on Faces in 'Real-Life' Images: Detec- [79] O. Lahdenoja, M. Laiho, and A. Paasio, “Reducing the feature vector
tion, Alignment, and Recognition, 2008. length in local binary pattern based face recognition,” in Proc. IEEE Int.
[58] J. Fehr, “Rotational invariant uniform local binary patterns for full 3D Conf. Image Processing (ICIP), 2005, pp. II: 914-917.
volume texture analysis,” in Proc. Finnish Signal Processing Symposium [80] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of
(FINSIG), 2007. simple features,” in Proc. IEEE Int. Conf. Computer Vision and Pattern
[59] L. Paulhac, P. Makris, and J.-Y. Ramel, “Comparison between 2D and 3D Recognition (CVPR), 2001, pp. I: 511-518.
local binary pattern methods for characterization of three-dimensional [81] S. Z. Li, L. Zhang, S. Liao, X. Zhu, R. Chu, M. Ao, and R. He, “A
textures,” in Proc. Int. Conf. Image Analysis and Recognition, 2008. near-infrared image based face recognition system,” in Proc. Int. Conf.
[60] G. Zhao and M. Pietikäinen, “Dynamic texture recognition using local Automatic Face and Gesture Recognition (FG), 2006, pp. 455-460.
binary patterns with an application to facial expressions,” IEEE Trans- [82] A. Hadid, M. Pietikäinen, and S. Z. Li, “Boosting spatio-temporal LBP
actions on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. patterns for face recognition from video,” in Proc. Asia-Pacific Workshop
915-928, 2007. on Visual Information Processing (VIP), 2006, pp. 75-80.
[61] S. Marcelja, “Mathematical description of the responses of simple cortical [83] D. Yi, R. Liu, R. Chu, R. Wang, D. Liu and S. Z. Li, “Outdoor face
cells,” Journal of the Optical Society of America, vol. 70, no. 11, pp. recognition using enhanced near infrared imaging,” in Proc. Int. Conf.
1297–1300, 1980. Biometrics (ICB), 2007, pp. 415-423.
[62] J. G. Daugman, “Complete discrete 2-D Gabor transforms by neural [84] R. Liu, X. Gao, R. Chu, X. Zhu, and S. Z. Li, “Tracking and recognition of
networks for image analysis and compression,” IEEE Transactions on multiple faces at distances,” in Proc. Int. Conf. Biometrics (ICB), 2007,
Acoustics, Speech and Signal Processing (ASSP), vol.36, No.7, pp. pp. 513-522.
1169-1179, 1988. [85] A. Hadid, M. Pietikäinen, and S. Z. Li, “Learning personal specific facial
[63] L. Wiskott, J. M. Fellous, N. Kruger, and C. v. d. Malsburg, “Face rec- dynamics for face recognition from videos,” in Proc. Analysis and Mod-
ognition by elastic bunch graph matching,” IEEE Transactions on Pattern eling of Faces and Gestures (AMFG), 2007, pp. 1-15.
Analysis and Machine Intelligence, vol. 19, no. 7, pp. 775–779, 1997. [86] K. Pan, S. Liao, Z. Zhang, S. Z. Li, and P. Zhang, “Part-based face rec-
[64] S. Yan, H. Wang, X. Tang, and T. S. Huang, “Exploring feature de- ognition using near infrared images,” in Proc. IEEE Int. Conf. Computer
scriptors for face recognition,” in Proc. Int. Conf. Acoustics, Speech, and Vision and Pattern Recognition (CVPR), 2007, pp. 1-6.
Signal Processing (ICASSP), 2007, pp. I: 629-632. [87] G. Zhao and M. Pietikäinen, “Principal appearance and motion from
[65] X. Tan and B. Triggs, “Fusing Gabor and LBP feature sets for ker- boosted spatiotemporal descriptors,” in Proc. IEEE Workshop on CVPR
nel-based face recognition,” in Proc. Analysis and Modeling of Faces and for Human Communicative Behavior Analysis (CVPR4HB), 2008, pp.
Gestures (AMFG), 2007, pp. 235-249. 1-8.
[66] R. Singh, M. Vatsa, and A. Noore, “Integrated multilevel image fusion [88] J. Hannuksela, P. Sangi, M. Turtinen, and J. Heikkilä, “Face tracking for
and match score fusion of visible and infrared face images for robust face spatially aware mobile user interfaces,” in Proc. Int. Conf. Image and
recognition,” Pattern Recognition, vol. 41, no. 3, pp. 880-893, 2008. Signal Processing (ICISP), 2008, pp. 405-412.
[67] W. Zhang, S. Shan, W. Gao, X. Chen, and H. Zhang, “Local Gabor Binary [89] G. Shakhnarovich and B. Moghaddam, “Face recognition in subspaces,”
Pattern Histogram Sequence (LGBPHS): A novel non-statistical model in Handbook of Face Recognition, Eds. Stan Z. Li & Anil K. Jain,
for face representation and recognition,” in Proc. IEEE Int. Conf. Com- Springer-Verlag, 2004.
puter Vision (ICCV), 2005, pp. I: 786-791. [90] C. Chan, J. Kittler, and K. Messer, “Multispectral local binary pattern
[68] S. Shan, W. Zhang, Y. Su, X. Chen, and W. Gao, “Ensemble of piecewise histogram for component-based color face verification,” in Proc. IEEE
FDA based on spatial histograms of local (Gabor) binary patterns for face Int. Conf. Biometrics: Theory, Applications, and Systems (BTAS), 2007,
recognition,” in Proc. Int. Conf. Pattern Recognition (ICPR), 2006, pp. pp. 1-7.
IV: 606-609. [91] D. Zhao, Z. Lin, and X. Tang, “Contextual distance for data perception,”
[69] B. Yao, H. Al, Y. Ijiri, and S. Lao, “Domain-partitioning rankboost for in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2007.
face recognition,” in Proc. IEEE Int. Conf. Image Processing (ICIP), [92] D. Zhao, Z. Lin, and X. Tang, “Laplacian PCA and its applications,” in
2007, pp. I: 129-132. Proc. IEEE Int. Conf. Computer Vision (ICCV), 2007.
[70] L. He, C. Zou, L. Zhao, and D. Hu, “An enhanced LBP feature based on [93] L. Wolf and M. Guttmann, “Artificial complex cells via the tropical
facial expression recognition,” in Proc. Annual Int. Conf. Engineering in semiring,” in Proc. IEEE Int. Conf. Computer Vision and Pattern Rec-
Medicine and Biology Society (EMBS), 2005, pp. 3300-3303. ognition (CVPR), 2007.
[71] Z. Lei, S. Liao, R. He, M. Pietikäinen, and S. Z. Li, “Gabor volume based [94] C. Shan, S. Gong, and P. McOwan, “Appearance manifold of facial
local binary pattern for face representation and recognition,” in Proc. expression,” in Proc. Computer Vision in Human Computer Interaction:
IEEE Int. Conf. Automatic Face and Gesture Recognition (FG), 2008. ICCV Workshop on HCI, 2005, pp. 221-230.
[72] M. Heikkilä, M. Pietikäinen and C. Schmid, “Description of interest [95] Y. Gao and Y. Wang, “Boosting in random subspaces for face recogni-
regions with center-symmetric local binary patterns,” in Proc. Indian tion,” in Proc. Int. Conf. Pattern Recognition (ICPR), 2006, pp. I:
Conference on Computer Vision, Graphics and Image Processing 519-522.
(ICVGIP), 2006. [96] C. Shan, S. Gong, and P. McOwan, “Conditional mutual information
[73] M. Heikkilä, M. Pietikäinen and C. Schmid, “Description of interest based boosting for facial expression recognition,” in Proc. British Ma-
regions with local binary patterns,” Pattern Recognition, vol. 42, no. 3, pp. chine Vision Conference (BMVC), 2005.
425-436, 2009. [97] Y. Raja and S. Gong, “Sparse multi-scale local binary patterns,” in Proc.
[74] D. Lowe, “Distinctive image features from scale-invariant keypoints,” British Machine Vision Conference (BMVC), 2006.
International Journal on Computer Vision, vol. 2, no. 60, pp. 91-110, [98] L. Nanni and A. Lumini, “RegionBoost learning for 2D+3D based face
2004. recognition,” Pattern Recognition Letters, vol. 28, no. 15, pp. 2063-2070,
[75] X. Fu and W. Wei, “Centralized binary patterns embedded with image 2007.
Euclidean distance for facial expression recognition,” in Proc. Int. Conf. [99] M. Turk, and A. Pentland, “Eigenfaces for recognition,” J. Cognitive
Neural Computation (ICNC), 2008, pp. IV: 115-119. Neuroscience, vol. 13, no. 1, pp. 71–86, 1991.
[76] D. Huang, G. Zhang, M. Ardabilian, Y. Wang, and L. Chen, “3D face [100] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigenfaces vs.
recognition using distinctiveness enhanced facial representations and fisherfaces: recognition using class specific linear projection,” IEEE
local feature hybrid matching,” in Proc. IEEE International Conference Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp.
on Biometrics: Theory, Applications and Systems, 2010. 711–720, 1997.
[77] T. Ahonen, J. Matas, C. He, and M. Pietikäinen, “Rotation invariant [101] M. Bicego, A. Lagorio, E. Grosso, and M. Tistarelli, “On the use of SIFT
image description with local binary pattern histogram fourier features,” in features for face authentication,” in Proc. Conference on Computer Vi-
Proc. Scandinavian Conference on Image Analysis (SCIA), 2009. sion and Pattern Recognition Workshop (CVPRW), 2006.
[102] Y. Pang, X. Li, Y. Yuan, D. Tao, and J. Pan, “Fast Haar transform based
feature extraction for face representation and recognition,” IEEE Trans-
 16

actions on Information Forensics and Security, vol. 4, no. 3, pp. 441-450, [127] J. Cao and C. Tong, “Facial expression recognition based on
2009. LBP-EHMM,” in Proc. Congress on Image and Signal Processing, 2008
[103] Y. Pang, Y. Yuan, and X. Li, “Iterative subspace analysis based on feature [128] N. Dalal and B. Triggs, “Histogram of oriented gradients for human
line distance,” IEEE Transactions on Image Processing, vol. 18, no. 4, pp. detection,” in Proc. IEEE Int. Conf. Computer Vision and Pattern Rec-
903-907, 2009. ognition (CVPR), 2005, pp. 886-893.
[104] T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE [129] H. Lian and B. Lu, “Multi-view gender classification using local binary
Transactions on Information Theory, vol. 13 no. 1, pp. 21–27, 1967. patterns and support vector machines,” in Proc. Int. Symposium on Neural
[105] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face Networks (ISNN), 2006, pp. II: 202-209.
recognition via sparse representation,” IEEE Transactions on Pattern [130] H. Lian and B. Lu, “Multi-view gender classification using Mul-
Analysis and Machine Intelligence (PAMI), vol. 31, no. 2, 2009. ti-resolution local binary patterns and support vector machines,” Int. J. of
[106] C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, Neural Systems, vol. 17, no. 6, pp. 479-487, 2007.
vol. 20, no. 3, pp. 273-297, 1995. [131] S. Marcel, J. Keomany, and Y. Rodriguez, “Robust-to-illumination face
[107] H. M. Vazquez, E. G. Reyes, and Y. C. Molleda, “A new image division localization using active shape models and local binary patterns,” IDIAP
for LBP method to improve face recognition under varying lighting con- Technical Report IDIAP-RR 06-47, 2006.
ditions,” in Proc. International Conference on Pattern Recognition [132] G. Heusch, Y. Rodriguez, and S. Marcel, “Local binary patterns as an
(ICPR), 2008. image preprocessing for face authentication,” in Proc. Int. Conf. Auto-
[108] X. Huang, S. Z. Li, and Y. Wang, “Shape localization based on statistical matic Face and Gesture Recognition (FG), 2006, pp. 9-14.
method using extended local binary pattern,” in Proc. Int. Conf. Image [133] R. Gross and V. Brajovic, “An image preprocessing algorithm for illu-
and Graphics (ICIG), 2004, pp. 184–187. mination invariant face recognition,” in Proc. Audio- and Video-based
[109] A. Hadid and M. Pietikäinen, “A hybrid approach to face detection under Biometric Person Authentication (AVBPA), 2003.
unconstrained environments,” in Proc. Int. Conf. Pattern Recognition [134] F. Cardinaux, C. Sanderson, and S. Bengio, “Face verification using
(ICPR), 2006, pp. I: 227-230. adapted generative models,” in Proc. IEEE Int. Conf. Automatic Face and
[110] A. Hadid, M. Pietikäinen, and B. Martinkauppi, “Color-based face de- Gesture Recognition (FG), 2004.
tection using skin locus model and hierarchical filtering,” in Proc. Int. [135] N. Poh, G. Heusch, and J. Kittler, “On combination of face authentication
Conf. Pattern Recognition (ICPR), 2002, pp. 196-200. experts by a mixture of quality dependent fusion classifiers,” in Proc.
[111] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition based on the Multiple Classifier Systems (MCS), 2007, pp. 344-356.
appearance of local regions,” in Proc. Int. Conf. Pattern Recognition [136] D. Huang, M. Ardabilian, Y. Wang, and L. Chen, “Asymmetric 3D-2D
(ICPR), 2004, pp. III: 153-156. face recognition based on LBP facial representation and canonical cor-
[112] W. Zhang, S. Shan, L. Qing, X. Chen, and W. Gao, “Are Gabor phases relation analysis,” in Proc. Int. Conf. on Image Processing (ICIP), 2009.
really useless for face recognition,” Pattern Analysis and Applications, [137] D. Huang, M. Ardabilian, Y. Wang, and L. Chen, “Automatic asymmetric
2008. 3D-2D face recognition,” in Proc. Int. Conf. Pattern Recognition (ICPR),
[113] Y. Rodriguez and S. Marcel, “Face authentication using adapted local 2010.
binary pattern histograms,” in Proc. Euro. Conf. Computer Vision [138] O. Lahdenoja, J. Maunu, M. Laiho, and A. Paasio, “A massively parallel
(ECCV), 2006, pp. IV: 321-332. algorithm for local binary pattern based face recognition,” in Proc. IEEE
[114] S. Z. Li, R. Chu, M. Ao, L. Zhang, and R. He, “Highly accurate and fast Int. Symposium on Circuits and Systems (ISCAS), 2006.
face recognition using near infrared images,” in Proc. Advances in Bio- [139] O. Lahdenoja, J. Maunu, M. Laiho, and A. Paasio, “A massively parallel
metrics, International Conference (ICB), 2006, pp. 151-158. face recognition system,” EURASIP J. Embedded Systems - Special Issue
[115] S. Z. Li, R. Chu, S. Liao, and L. Zhang, “Illumination invariant face on Embedded Vision System, 2007.
recognition using near-infrared images,” IEEE Trans. Pattern Analysis [140] M. Laiho, O. Lahdenoja, and A. Paasio, “Dedicated hardware for parallel
and Machine Intelligence, vol. 29, no. 4, pp. 627-639, 2007. extraction of local binary pattern feature vectors,” in Proc. Int. Workshop
[116] Y. K. Park and J. K. Kim, “Fast adaptive smoothing based on LBP for on Cellular Neural Networks and Their Applications, 2005, pp. 27-30.
robust face recognition,” IEE Electronics Letters, vol. 43, no. 24, 2007. [141] O. Lahdenoja, M. Laiho, and A. Paasio, “Local binary pattern feature
[117] Z. Zeng, M. Pantic, G.I. Roisman and T. S. Huang, “A survey of affect vector extraction with CNN,” in Proc. Int. Workshop on Cellular Neural
recognition methods: audio, visual, and spontaneous expressions,” IEEE Networks and Their Applications, 2005, pp. 202-205.
Transaction on Pattern Analysis and Machine Intelligence (PAMI), [142] A. Hadid, M. Heikkilä, T. Ahonen, and M. Pietikäinen, “A novel ap-
vol.31, No.1, pp.39- 58, 2009. proach to access control based on face recognition,” in Proc. Workshop
[118] B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,” on Processing Sensory Information for Proactive Systems (PSIPS), 2004,
Pattern Recognition, vol. 36, no. 2003, pp. 259–275, 2002. pp. 68-74.
[119] N. Sebe, M.S. Lew, Y. Sun, I. Cohen, T. Gevers and T.S. Huang, “Au- [143] P. Trinh, P. Ngoc, and K.-H. Jo, “Multi-face detection system in video
thentic facial expression analysis,” Image and Vision Computing, vol. 25, sequence,” in Proc. International Forum on Strategic Technology, 2006,
pp. 1856–1863, 2007. pp. 146-150.
[120] X. Feng, “Facial expression recognition based on local binary patterns [144] H. K. Ekenel, J. Stallkamp, H. Gao, M. Fischer, and R. Stiefelhagen,
and coarse-to-fine classification,” in Proc. Int. Conf. Computer and In- “Face recognition for smart interactions,” in Proc. IEEE Int. Conf. Mul-
formation Technology (CIT), 2004, pp. 178-183. timedia and Expo, 2007, pp. 1007-1010.
[121] X. Feng, A. Hadid, and M. Pietikäinen, “Facial expression recognition [145] A. Hadid, J. Heikkilä, O. Silven, and M. Pietikäinen, “Face and eye
with local binary patterns and linear programming,” in Proc. Int. Conf. detection for person authentication in mobile phones,” in Proc.
Pattern Recognition and Image Analysis, 2004, pp. 666-669. ACM/IEEE Int. Conf. Distributed Smart Cameras, 2007.
[122] X. Feng, J. Cui, A. Hadid, and M. Pietikäinen, “Real time facial expres- [146] A. Abbo, V. Jeanne, M. Ouwerkerk, C. Shan, R. Braspenning, A. Ganesh,
sion recognition using local binary patterns and linear programming,” in and H. Corporaal, “Mapping facial expression recognition algorithms on a
Proc. Mexican Int. Conf. Artificial Intelligence, 2005. Low-power smart camera,” in Proc. ACM/IEEE Int. Conf. Distributed
[123] X. Feng, A. Hadid, and M. Pietikäinen, “Facial expression recognition Smart Cameras, 2008.
with local binary patterns and linear programming,” Pattern Recognition [147] J. Cui, F. Wen, R. Xiao, Y. Tian, and X. Tang, “EasyAlbum: an interactive
and Image Analysis, vol. 15, no. 2, pp. 550-552, 2005. photo annotation system based on face clustering and re-ranking,” in
[124] C. Shan, S. Gong, and P. W. McOwan, “Robust facial expression recog- Proc. SIGCHI conf. Human Factors in Computing Systems (CHI), 2007,
nition using local binary patterns,” in Proc. IEEE Int. Conf. Image pp. 367-376.
Processing (ICIP), 2005, pp. II: 370-373. [148] J. Zou, Q. Ji, and G. Nagy, “A comparative study of local matching
[125] C. Shan, S. Gong, and P. W. McOwan, “Recognizing facial expressions at approach for face recognition”, IEEE Trans. on Image Processing, vol.
low resolution,” in Proc. IEEE Conf. Advanced Video and Signal Based 16, no. 10, pp. 2617 – 2628, 2007.
Surveillance (AVSS), 2005, pp. 330-335. [149] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, “Face recognition:
[126] X. Feng, B. Lv, Z. Li, and J. Zhang, “A novel feature extraction method a literature survey.” in ACM Computing Survey, pp. 399–458, 2003.
for facial expression recognition,” in Proc. Joint Conf. Information [150] M. S. Bartlett, J. R. Movellan, and T. J. Sejnowski, “Face recognition by
Sciences (JCIS) Issue on Advances in Intelligent Systems Research, 2006. independent component analysis,” IEEE Trans. Neural Network, vol. 13,
no. 6, pp. 1450–1464, 2002.
 17

[151] X. Li, S. Lin, S. Yan, and D. Xu, “Discriminant locally linear embedding
with high-order tensor data,” IEEE Transactions on Systems, Man, and
Cybernetics, Part B: Cybernetics, vol. 38, no. 2, pp. 342-352, 2008.
[152] M. Song, Z. Liu, D. Tao, X. Li, and M. Zhou, “Image Ratio Features for
Facial Expression Recognition,” IEEE Transactions on Systems, Man,
and Cybernetics, Part B: Cybernetics, vol. 40, no. 3, pp. 779-788, 2010.
[153] B. Heisele, P. Ho, J. Wu, and T. Poggio, “Face recognition: compo-
nent-based versus global approaches,” Computer Vision and Image Un-
derstanding, vol. 91, no. 1, pp. 6–12, 2003.
[154] X. Tan, S. Chen, Z. Zhou, and F. Zhang, “Face recognition from a single
image per person: a survey”, Pattern Recognition, vol. 39, no. 9, pp.
1725-1745, 2006.
[155] J. Luo, Y. Ma, E. Takikawa, S.Lao, M. Kawade, and B.-L. Lu, “Per-
son-specific SIFT features for face recognition,” in Proc. IEEE Interna-
tional Conference on Acoustics, Speech and Signal Processing (ICASSP),
2007.
[156] J. Ruiz-del-Solar, R. Verschae, and M. Correa, “Recognition of faces in
unconstrained environments: a comparative study,” in EURASIP Journal
on Advances in Signal Processing, 2009.
[157] M. Lyons, S. Akamastu, M. Kamachi, and J. Gyoba, “Coding facial
expressions with Gabor wavelets,” in Proc. IEEE Int. Conf. Face and
Gesture Recognition (FG), 1998, pp. 200-205.
[158] Y. Pang, Y. Yuan, and X. Li, “Gabor-based region covariance matrices
for face recognition,” IEEE Transactions on Circuits and Systems for
Video Technology, vol. 18, no. 7, pp. 989-993, 2008.
[159] Y. Pang, Y. Yuan, and X. Li, “Effective feature extraction in high
dimensional space,” IEEE Transactions on Systems, Man, and Cyber-
netics, Part B: Cybernetics, vol. 38, no. 6, pp. 1652-1656, 2008.