Computers and Electrical Engineering 38 (2012) 11861195

Contents lists available at SciVerse ScienceDirect

Computers and Electrical Engineering

journal homepage: www.elsevier.com/locate/compeleceng

Automatic recognition vision system guided for apple harvesting robot q

Wei Ji a,, Dean Zhao a, Fengyi Cheng a, Bo Xu a, Ying Zhang a, Jinjing Wang b
a
b

The School of Electrical and Information Engineering, Jiangsu University, Zhenjiang 212013, China
Chinese Academy of Agricultural Mechanization Sciences, Beijing 100083, China

a r t i c l e

i n f o

Article history:
Available online 30 November 2011

a b s t r a c t
In apple harvesting robot, the rst key part is the machine vision system, which is used to
recognize and locate the apples. In this paper, the procedure on how to develop an automatic recognition vision system guided for apple harvesting robot, is proposed. We rst
use a color charge coupled device camera to capture apple images, and then utilize an
industrial computer to process images for recognising fruit. Meanwhile, the vector median
lter is applied to remove the color images noise of apple, and images segmentation
method based on region growing and color feature is investigated. After that the color feature and shape feature of image are extract, a new classication algorithm based on support vector machine for apple recognition is introduced to improve recognition accuracy
and efciency. Finally, these procedures proposed have been tested on apple harvesting
robot under natural conditions in September 2009, and showed a recognition success rate
of approximately 89% and average recognition time of 352 ms.
Crown Copyright 2011 Published by Elsevier Ltd. All rights reserved.

1. Introduction
The challenge of developing a cost-effective robotic system for fruit picking has been taken up by researchers at several
places in the world [13]. The principal problem required to be solved for a robotic harvesting system is the machine vision
system, which affects the robots dependability and also determines its ability to directly, quickly and accurately recognize
the fruit in a real-time [4,5].
In the literature, there are already some results on the research of the harvesting robot vision system. In [6], a review of
different vision systems to recognize fruits for automated harvesting was presented, where a model of the attenuation process was presented and used to restore images and derive additional information, which was used to recognize the fruit by
color and shape analysis algorithms. Later, the RedBlue (RB) chromatic aberration information of the images has been used
in [7] to recognize oranges on the tree, and the fruit in the conditions of front lighting and back lighting have also been
considered, respectively. In [8], the technology of threshold segmentation and recognition based on hue statistic in HueIntensity-Saturation (HIS) color space was researched. Based on the obvious differences of color between the apples and
background, Bulanon et al. [9] made use of light and color model to check the Fuji apples, and threshold to segment images.
In Ref. [10], a new modied Hue-Saturation-Value (HSV) color space was used to segment images and extract the relevant
information of the fruit in the segmentation phase before the localization process. Meanwhile, for images segmentation procedure, a well-known Bayesian classier for the classication of each pixel as fruit or background was used to classify defects
on apples [11]. However, the Bayesian classier is dependent on a training image to provide information for class mean and
the covariance matrix. As can be seen from the literatures above, these methods exist some unsolved disadvantages, such as
low accuracy rate, too much time consuming, etc. These disadvantages, to some extent, restrict the real-time and multitask

Reviews processed and approved for publication by Editor-in-Chief Dr. Manu Malek.

Corresponding author.

E-mail address: [email protected] (W. Ji).

0045-7906/$ - see front matter Crown Copyright 2011 Published by Elsevier Ltd. All rights reserved.
doi:10.1016/j.compeleceng.2011.11.005

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

1187

ability of the apple harvesting robot operations in a natural environment. To this end, it is necessary to introduce a better
fruit recognition method for the apple harvesting robot.
Support vector machine (SVM) is a new pattern recognition method which has been deeply developed in recent years
[12,13]. Bases on the principles of minimum frame hazard and kernel functions, this method resolves the inconsistency
between complexity and popularization of the model classication efciently. It also transfers the model recognition problem to a seeking excellent value problem, and assures the value best and overall theoretically. And thus the local constringency phenomenon can be avoided. Especially, the SVM reects unique advantages and good application foreground in
resolving problems such as small samples recognition, nonlinear and high-dimension problems. Consequently, this method
has been attracting much attention of the researchers in pattern recognition eld [14,15].
Based on the method mentioned above, this paper will try to describe the development of a real-time machine vision recognition system to guide a harvesting robotic for picking Fuji apples in different conditions. By applying SVM, a new recognizing method is developed for improving apple recognition accuracy and efciency. The outline of the paper is organized as
follows. In Section 2, the materials and method of real-time machine vision recognition system is presented. Section 3 reports on a eld test result with the apple harvesting robot vision system. And nally, conclusions and suggestions for future
research are drawn in Section 4.
2. Materials and methods
2.1. Vision system setup and image acquisition
This robot vision system consists of a VGA (640  480) color charge coupled devices (CCD) video camera with 30 images
per second, and an industrial computer with Intel Pentium4 1.7 GHz processor and 512 M memory. CCD video camera is
used to acquire original apple images, and the video for windows (VFW) capture technology is adopted to capture the video
image. The industrial computer is for dealing with original images and detecting objective, whose software platform is Visual
C++ 6.0.
Since the Fuji apples are the most popular in China, our research will focus on this variety. Color images of Fuji apple,
which will be examined in the following, are acquired under natural daylight condition on the apple demonstration orchard
of Feng Country, Xuzhou City, Jiangsu Province. The color signals from the camera are transferred as a 24-bit red, green, blue
(RGB) color image data (640 pixels by 480 pixels in each color band) and processed by an industrial computer.
2.2. Image processing
2.2.1. Image pre-processing
Due to the natural environment and the image acquisition device used, the original unprocessed color apple image inevitably includes noise that inuences its quality. In this study, a vector median lter is applied to image enhancement preprocessing [16]. It can not only removes or weakens noise information effectively and highlights the apple fruit in foreground, but also maintains good image edges. The basic process of vector median lter includes three steps. First of all,
achieve the average vector Xi; j of all known color image pixel vector Xi; j. Secondly, calculate the distance Sij between
Xi; j and Xi; j. Finally, make the Smin , which is minimum of the Sij , be the output value of the window central pixel.
In this study, by assuming that the size of original two-dimensional color image setting is M  N pixel (M is the rows
number, N is the columns number), we adopt a vector median lter with a n  n window to deal with the noise. The arithmetic is as follows.
(a) Take the pixel as a vector Xi; j. By letting ri; j; gi; j; bi; j be three parameters standing for pixels in RGB color
images, then we have Xi; j ri; j; gi; j; bi; jT i 1; . . . M; j 1; . . . N.
(b) Calculate the averages of r, g and b, then we get the average vector of the window as

 jT
 j r i; j; gi; j; bi;
Xi;

where

8
M P
N
P
>
>
r
>
ri; j=n  n
>
>
>
i1 j1
>
>
>
<
M P
N
P
gi; j=n  n
g
>
i1 j1
>
>
>
>
M N
>
>
 P P bi; j=n  n
>
>
:b
i1 j1

(c) Calculate the distance Sij between each vector and the average vector, and obtain the minimum distance Smin of them.

 jk
Sij kXi; j  Xi;

1188

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

(d) Make the pixel barX min corresponding to Smin as the vector median of the window, then let it replace the central pixel
vector of the window.
In our study, the size of original color image setting is 640  480 pixel. Vector median lter of color images with the n  n
window being 3  3, is adopted to carry out enhancement for the apple color image. The image lter result is shown in Fig. 1.
It can be shown that vector median lter of color images can wipe off noise efciently, and stand out the apple fruit in foreground. Furthermore, the solution can keep the edge and detail. In additional, the window central pixel is replaced by the
pixel corresponding to minimum distance, while not replaced by the synthetical vector. And thus the color feature will remain unchanged.

2.2.2. Image segmentation

Most apple images acquired in the natural conditions usually include branches and leaves which complicate matters. By
only using the conventional image segmentation algorithm, it is difcult to achieve the desirable result. The seeded region
growing method and color feature are employed here to develop an image segmentation algorithm for identifying apple fruit
from complex background.
The segmentation effect of seeded region growing is determined by the selection of initial seed point and the growing
rules. In this study, the images are colorized in RGB model. First, we map the image pixels into column diagram of RGB model. Then, select some pixel as the preparative growing seed in these colors whose probability is the biggest. After that, according to [17,18], we tentatively select the 5  5 pixel region of the preparative seed, and set a threshold value, then calculate
the number of pixels which are less than the threshold value. If the number of pixels is more than 20, we regard the region
around these pixels as the goal object, namely the apple fruit. If the number is less than 20, we consider that the preparative
seed picks some isolated points whose color is close to that of the object. Then, we abnegate this preparative seed and choose
a new one. Finally, we select a growing seed to grow it under the growing rules.
The growing threshold value obeys the following rules: the color of growing seed color is just the main color of the apple
fruit. Thus, we can choose the absolute value as the threshold, which is the difference between the color of the growing seed
and the average value of the whole image. In this study, the difference between the color of apple fruit and background is
distinct, so we can select a bigger value as the threshold one.
The growing rules, which are the key ingredient, determine the dependability and running time of the region growing
algorithm. In this study, we investigate 8 neighbor pixels starting from the growing point. If the color difference between
growing point and that of the central point is less than the threshold value, then we add this pixel to the growing region.
After that, the central point is labeled. Whether label or unlabel its neighbored pixels is according to their color homogeneity.
Do the same procedure to the neighbor pixels neighbor pixels iteratively until no homogeneity pixels can be found.
For a color image, to judge whether the colors are homogeneous or not can be measured by the Euclid distance. It means
that there are two pixels P1 (r1, g1, b1) and P2 (r2, g2, b2) in RGB model, and their distance is

dP 1 ; P 2

q
r 1  r 2 2 g 1  g 2 2 b1  b2 2

Based on the method proposed above, we can divide the apple fruit image into two parts including fruit and background.
The segment image is used to extract features. However, during the process of segmenting, the isolated dots, burrs and holes
are usually exist in the image. To decrease the inuence of these noises for subsequent image recognizing steps, we adopt
opening and closing operations in mathematical morphology to remove the noises. The concrete process is to adopt the

Fig. 1. Results of vector median lter applied apple color images.

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

1189

open-shut lter sequence for executing the task iteratively. At the beginning, the opening operation is utilized to clear the
isolated dots and burrs. Nextly, the closing operation is applied to ll up the small holes. And then, we can obtain the ideal
image. The segmentation and mathematical morphology operation results of apple image are shown in Fig. 2.
2.2.3. Image feature extraction
2.2.3.1. Apple image color feature extraction. The images are taken from the apple tree under natural daylight conditions. Four
lighting conditions are investigated: (a) front lighting, (b) back lighting, (c) fruit in the shade, and (d) cloudy. Here, conditions
(a), (b) and (c) are taken under sunny weather. Changes in lighting condition result in different quality images, and affecting
subsequent image processing steps [9]. Therefore, it is necessary to choose a color model which can t most lighting conditions. The HIS color model is a common color perceptive model, which describes color with its three components of H, I and
S. Note that the chromaticity is hardly inuenced by the lighting conditions. Here, we use the H and S components of HIS
model to deal with this problem.
Compared with the RGB model, the HIS model is easier for one perceiving the colors, and also accords with peoples habits
to describe colors. This model is based on two important facts: (a) the component of I does nothing with the color information of image; (b) the chroma components of H and S have close relation with people manner to perceive colors. The wellknown non-linear transformation from RGB components to the HIS color model is employed as follows [19]

GPB

h;

2p  h; G < B
2

6 0:5R  G R  B 7
h cos1 4q5
R  G2 R  BG  B

3
minR; G; B
R G B

S1

1
R G B
3

In view of the varieties of the light intensity, the inuence of the light and shade in the image should be avoided. The
components of H and S which do nothing with lightness are chosen to extract image features. Each point in the HIS colors
space of apple fruit image can be regarded as a point in three-dimensional space of HIS model. Therefore, the difference of
two colors can be measured by the Euclid distance of two color points. The chromatic aberration formula is as follows

DE

q
DH2 DS2

By this way, the colors can be compared directly in HIS color space, and the color and intensity information can be controlled well. Moreover, it is efcient in distinguishing small difference of the colors. And thus, H and S components in HIS
color space are employed to recognize the color feature.
2.2.3.2. Apple image shape feature extraction. The shape feature is important in object recognition. Different kinds of objects
have great differences in shape [5]. In this study, apple fruit, branches and leaves have themselves specic shapes, and their

Fig. 2. Segmentation and mathematical morphology operation results of apple fruit images.

1190

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

differences in shape are signicant. So, according to the edge of apple fruit, the shape feature used in following classied
process, can be extracted.
The biggest character of objects geometry shape is that they are not changed with the change of the objects position, size
and angle in image. Consequently, we should extract the feature vectors that can satisfy RST (round, scale, transfer) invariability. In all the features of apple, feature parameters of round variance, ellipse variance, tightness, and ratio of perimeter
and square area can describe furthest the outline feature of apple efciently, so the above four feature parameter vectors are
extracted for further research. The extraction process detailed can be described as follow.
2.2.3.2.1. Reckon image outline position. Outline position, namely, position of boundary pixels in image, can be obtained by
edge detection method such as Soble or Robert arithmetic operators [19]. Assuming the outline edge coordinate is
pi xi ; yi T , the outline is a setting P of N stochastic vectors, P fpi g; i 1; 2; . . . ; N. The average vector is l, namely, the object centroid coordinate is

N
1X
p
N i1 i

The average radius lr is

lr

N
1X
jp  lj
N i1 i

10

2.2.3.2.2. Calculate parameters of outline region area and perimeter. Assuming A is the outline region area of image, which is
the summation of the numbers of region pixels. So is the region perimeter. The way to calculate So is as follows. In the boundary pixels of the region, we suppose the distance between some pixel and
p another one that lies up, down, left or right, is 1,
and the distance between it and another one lying slanting position is 2. Then So is the summation of these distances.
2.2.3.2.3. Extract feature parameter vectors of the round variance, ellipse variance, tightness, ratio of perimeter and square. Round
variance rc reects the resemblance extent between the outline of object and the round. The smaller rc indicates the more
chance of the object to be the round. It is dened as

rc

N


1 X
kpi  lk  lr
2
Nl i;r1

11

Ellipse variance re reects the resemblance extent between the outline of object and the ellipse whose principal axis is
axis, the smaller re indicates the object is more like an ellipse. This value is applied when the apple fruit image is inuenced
by noise, and the fruit image cannot take on an absolute round. It is dened as

re
where

1
Nlrc

lrc N1

2
N q
X
pi  lT C 1 pi  l  lrc

12

i1

q
pi  lT C 1 pi  l .
i1

PN

Assuming Co is the tightness of the object, the smaller value reects that the composition of object is more incompact. It is
dened as

Co

p
2 Ap
So

13

The ratio of perimeter and square area D is dened as

D S2o =A

14

These features are regarded as the eigenvectors of each sample, which will be employed to the following training and
classication.
2.3. Apple automatic recognition based on SVM
2.3.1. Support vector machine
SVM is a learning system that uses a hypothesis space of linear functions in a high dimensional feature space, trained with
a learning algorithm from optimization theory that implements a learning bias derived from statistical learning theory [13].
It includes linear separable question and non-linear separable question. The reader is referred to [12] for a more comprehensive introduction.
2.3.1.1. Linear separable question. For binary linear separable classication, the basic thought of SVM is to nd an optimal
hyperplane between the two kinds of examples setting to classify them and make their distance furthest. For example, in
Fig. 3, solid dots and hollow loop stand for the rst and the second training examples, respectively. H is the optimal hyperplane, H1 and H2 are parallel with H. The points on H1 are the rst samples whose distance to H are the shortest, and the

1191

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

Fig. 3. The sketch of two-dimensional linear separable optimal hyperplane.

points on H2 are the second samples whose distance to H are the shortest too. The points on H1 and H2 are on the edge of
isolation belt, and these examples are called support vectors. They determine the isolation belt. In this study, the classied
judged function is selected as follows.


f x sgnx  x b sgn

N
X

!
ai yi xi

 x b

15

i1


where the coefcient a is the Lagrange multiplier and b is the classication threshold.
2.3.1.2. Non-linear separable question. An important advantage of SVM is to deal with non-linear separable question. When
the question is non-linear separable, the feature mapping method can be adopted. It can map the non-linear separable feature vector space to a high dimensional feature space which is linear separable. Then, the examples are classied based on
linear separable SVM.
The principle map of feature mapping method is shown as in Fig. 4. Fig. 4(a) shows two kinds of non-linear separable
examples in the original feature space, and Fig. 4(b) shows the result of non-linear separable question mapping into linear
separable question. The feature mapping method is fullled by using kernel functions, which can complete the transform
from the non-linear separable question to a linear separable one. Certainly, the results of classication are different when
different kernel functions are adopted. The three kernel functions that are usually used as follows [12]
(1) Poly kernel function

Kx; xi x  xi 1q

16

(2) Radial Basis Function (RBF) kernel function

Kx; xi exp 

jx  xi j2

!
17

r2

(3) Sigmoid kernel function

Kx; xi tan hax  xi b

18

(
(

)
(

(a) Non-linear separable question

(b) linear separable question after mapping

Fig. 4. Non-linear separable question mapping to linear separable question.

1192

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

2.3.2. Recognition of apple fruit based on SVM with color feature and shape feature
Select 150 Fuji apple images as a training set, and establish a recognition model. Then, select another 50 apple images as a
testing set, which is used to validate the dependence of the model. In the experiment, each apple color image is pre-processed, segmented and extracted for features, and then recognized based on SVM. In allusion to feature data of the apple fruit
image, SVM is classied and tested by different kernel functions to distinguish whether SVM has different classication
capacity or not and to conrm which kind of SVM is much tter for the apple fruit recognition.
When recognizing an apple manually, the red pericarp (viz. color information) and round fruit (viz. shape information) are
shown. So rstly, SVM based on only color feature or shape feature is used in apple recognition, respectively. And then we
use SVM based on both color feature and shape feature to recognize apple. The three recognition simulated experiments are
contrasted and discussed in following subsections.
2.3.2.1. Recognition of apple fruit based on color feature. In the recognition experiment based on color feature, the color features of the apple fruit image are extracted after pre-processing. The sample set and testing set are recognized by three different kinds of SVM kernel functions, and then their identication capabilities are compared. In SVM algorithm, we choose
penalized coefcient C 1, slack variable n 0:001, and q 3 in Poly kernel function, r2 3=2 in RBF kernel function, a 3,
b 10 in Sigmoid kernel function. The results are showed in Table 1.
2.3.2.2. Recognition of apple fruit based on shape feature. In the recognition experiment based on shape feature, every image
pre-processed and segmented, is classied and experimented after shape feature are extracted integrated with SVM. Here,
we choose C 1, n 0:001, and q 2 in Poly kernel function, r2 1 in RBF kernel function, a 1=3, b 1 in Sigmoid kernel function. The results are showed in Table 2.
2.3.2.3. Recognition of apple fruit based on color feature and shape feature. In the recognition experiment based on both color
feature and shape feature, by using SVM for apple fruit image, we take c 1, n 0:001; q 3 in Poly kernel function; r2 6
in RBF kernel function and a 1=6, b 1 in Sigmoid kernel function. The results are showed in Table 3.
As shown in the three tables above, we can nd that apple fruit recognition based on only color feature or shape feature is
inferior to that integrated both of them, either in the aspect of recognition accurate rate or running time. The RBF function is
the one with the highest recognition accurate rate in the three SVM functions. The Sigmoid function has the shortest running
time, but its recognition rate is the lowest. The other functions are almost the same in running time. By taking into account
all the factors mentioned above, we can conclude that the SVM method with RBF kernel function based on both color feature
and shape feature is the best for apple recognition.

Table 1
Recognition results of apple fruit based on color feature.
Result

The average of recognition rate (%)

Running time (ms)

Method
SVM
Poly

RBF

Sigmoid

62.3
283

89.1
247

58.5
212

Poly

RBF

Sigmoid

82.9
262

90.1
253

69.2
171

Table 2
Recognition results of apple fruit based on shape feature.
Result

The average of recognition rate (%)

Running time (ms)

Method
SVM

Table 3
Recognition results of apple fruit based on color feature and shape feature.
Result

The average of recognition rate (%)

Running time (ms)

Method
SVM
Poly

RBF

Sigmoid

85.7
268

93.3
256

57.2
175

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

1193

3. Field test results and discussion of apple harvesting robot vision system
The apple harvesting robot developed independently by ourselves is shown in Fig. 5. The automatic recognition vision
system for apple proposed above is applied to guide robot for picking. In the sequel, the results of the two experiments
for apple recognition are presented in terms of the success rate and execution real-time. The experiment was done in September 2009 at Demonstration Orchard of Changping, Beijing.

Fig. 5. The harvest robot in apple demonstration orchard during the eld test.

Fig. 6. Apple automatic recognition result based on SVM under different conditions.

1194

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

Table 4
Images recognition time.
Image frame
Recognition time
(ms)
Image frame
Recognition time
(ms)
Image frame
Recognition time
(ms)
Image frame
Recognition time
(ms)

1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
235 235 315 390 315 310 390 390 310 390 390 310 390 310 235 315 390 390 310 390 390 390 315 390 395
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
310 390 390 310 315 390 390 315 310 390 390 310 390 390 310 390 390 310 315 390 315 310 390 390 390
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70
315 390 395 310 390 390 310 315 390 390 315 310 390 390 310 390 390 310 390 390

71 72 73 74 75
31 315 390 395 310

76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
315 390 310 315 390 395 310 390 390 310 390 390 310 315 390 390 310 315 390 390 315 390 390 315 390

3.1. Recognition success rate

In different orchard, the apple variety is usually different, and so does the apple color. Furthermore, with the difference of
planting areas, the protected measures for apple are also different. To validate the availability for the method mentioned
above, these factors have been considered in the recognition algorithm experiments for the machine vision system. Also,
in order to meet with visibility, the recognized apple is labeled by symbol +. Meanwhile, the obscured apple by leaves,
branches, or overlapped with other apples are not labeled with any symbol.
Fig. 6 shows the apple automatic recognition results under different conditions. The sample image Fig. 6(a) shows apple
orchard scene taken under natural outdoors. The lowest height of the apples in tree is 1 m above the ground. The interval
distance of two rows tree is about 5 m. Thus the apple harvesting robot can perform apple searching, recognizing and picking
smoothly according to designated path. Fig. 6(b) shows red apple recognition result without protected bag under front lighting. Here, 13 apples have been recognized. The error rate is almost zero. Fig. 6(c) shows red apple recognition result under
back lighting. Approximately 89% of the fruit is successfully detected. Fig. 6(d) shows red apple recognition result with protected bag. There, 7 apples with protected bag have been recognized accurately, and only 2 apples cannot be detected. Foliage occlusion caused nondetection cases. The experiment results show that the developed method is effective, and thus
could be applied to exactly recognize apple under different conditions.
3.2. Recognition execution real-time
In recognition execution real-time experiment, 100 frame dynamic video images (320 pixels by 240 pixels) are acquired
continuously by the CCD video camera. The apple recognition algorithm is implemented using Visual C++ 6.0. Table 4 gives
the recognition time of each frame image. The average recognition time of 100 frame images is 352 ms. From these results, it
can be concluded that the developed algorithm can be used to guide a robot manipulator as it approaches the apple in realtime.
4. Conclusions
A real-time vision recognition system to guide a harvesting robotic for apple picking in different conditions is developed.
Firstly, the apple fruit images acquired by CCD camera are pre-processed by vector median lter. Then, segmentation method
based on seeded region growing method and color feature is applied, and color feature and shape feature of color image are
extracted. After that, a new classication algorithm based on SVM for recognition of apple fruit is developed to improve recognition accuracy and efciency. Simulation experiment indicates that the recognition rate of the apple based on SVM of
color and shape feature is higher than that of only using the color or shape feature, and the running time is also shorter. This
machine vision system described has been tested on apple harvesting robot under natural conditions in September 2009, and
it is successful in recognizing the apples under different conditions. Approximately 89% of the fruit are successfully detected.
The average recognition time is 352 ms. In general, the machine vision system meets the requirement of apple harvesting
robot for the recognition accuracy rate and time efciency. However, nondetection case caused by the foliage occlusion
has an average error rate of 11%. Future work will be performed on this area to minimize unrecognition of apples. In addition,
reducing the recognition execution time is still a challenge, since it is too high to allow this algorithm to be implemented in a
real-time system.
Acknowledgments
The authors would like to thank the Handling Editor and the anonymous reviewers for their constructive comments,
which led to signicant improvement of the paper. This work was supported in part by Research Fund for the Doctoral Program of Higher Education of China under Grant 20093227120013, in part by National High Technology Research and Devel-

W. Ji et al. / Computers and Electrical Engineering 38 (2012) 11861195

1195

opment Program of China under Grant 2006AA10Z254, in part by a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions, and in part by Innovate Foundation for Graduate Student of Jiangsu Province under Grant CXZZ11_0573.
References
[1] Sarig Y. Robotics of fruit harvesting: a state-of-the-art review. J Agric Eng Res 1993;54(4):26580.
[2] Special Issue. Agricultural robotics. Autonomous Robots 2002;13:1.
[3] Muscato G, Prestilippo M, Abbate Nunzio, Rizzuto Ivan. A prototype of an orange picking robot: past history, the new robot and experimental results.
Industrial Robot 2005;32(2):12838.
[4] Bulanon DM, Kataoka T, Ukamoto H. Development of a real-time machine vision system for the apple harvesting robot. In: SICE annual conference in
Sapporo; 2004. p. 5958.
[5] Pla F, Juste F, Ferri F. Feature extraction of spherical objects in image analysis: an application to robotic citrus harvesting. Comput Electro Agric
1993;8(1):5772.
[6] Jimenez AR, Jain AK, Ceres R, Pons JL. Automatic fruit recognition: a survey and new results using range/attenuation images. Pattern Recogn
1999;32(10):171936.
[7] Xu HR, Ye ZZ, Ying YB. Identication of citrus fruit in a tree canopy using color information. Trans CSAE 2005;21(5):98101.
[8] Zhao JW, Liu MH, Yang GB. Discrimination of mature tomato based on HIS color space in natural outdoor scenes. Trans CSAM 2004;35(5):1224.
[9] Bulanon DM, Kataoka T, Ota Y. A segmentation algorithm for the automatic recognition of Fuji Apples at harvest. Biosyst Eng 2002;83(4):40512.
[10] Plebe A, Grasso G. Localization of spherical fruits for robotic harvesting. Machine Vision Appl 2001;13(2):709.
[11] Leemans V, Magein H, Destain M. Defect segmentation of Jonagold apples using color vision and Bayesian classication method. Comput Electro Agric
1999;23(1):4353.
[12] Burges C. A tutorial on support vector machines for pattern recognition. Data Mining Knowledge Discovery 1998;2(2):12167.
[13] Cristianini N, John ST. An introduction to support vector machines: and other kernel-based learning methods. UK: Cambridge University Press; 2000.
[14] Liu XD, Zhu ML, Chen ZQ. Support vector machine and its applications in pattern recognition. Comput Sci 2003;30(6):1137.
[15] Jonsson K, Kittler J, Li YP. Support vector machine and its applications in pattern recognition. Image Vision Comput 2002;20(56):36975.
[16] Jia YD. Machine Vision. Beijing: Science Press; 2000.
[17] Jianping Fan, Guihua Zeng, Mathurin Body. Seeded region growing: an extensive and comparative study. Pattern Recogn Lett 2005;26:113956.
[18] Wang JJ, Zhao DA, Ji W, Tu JJ, Zhang Y. Application of support vector machine to apple recognition using in apple harvesting robot. In: International
conference on information and automation in Zhuhai; 2009. p. 225.
[19] Zhang YJ. Image Segmentation. Beijing: Science Press; 2001.
Wei Ji received the B.Sc. and M.Sc. degrees in Electrical Engineering from China University of Mining and Technology, Xuzhou, China, in 1999 and 2002,
respectively, and the Ph.D. degree in Electrical Engineering from Southeast University, Nanjing, China, in 2007. Since 2007, he has been with the School of
Electrical and Information Engineering, Jiangsu University, Zhenjiang, China, where he is currently an associate professor. His current research interests
include robot for fruit harvesting, machine vision, and articial intelligence.
Dean Zhao received the B.Sc. and M.Sc. degrees in Electrical Engineering from Jiangsu University, Zhenjiang, China, in 1978 and 1982, respectively, and the
Ph.D. degree in Power Electronics from Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2006. Currently he is a professor in the School
of Electrical and Information Engineering, Jiangsu University, Zhenjiang, China. His research interests are in introducing computers and electronics to
agriculture, and in developing harvesting machinery for fruits and vegetables.
Fengyi Cheng received the B.Sc. degrees in Automatic Control from Changzhou University, Changzhou, China, in 2010. Currently he is a M.Sc. candidate at
the School of Electrical and Information Engineering in Jiangsu University, Zhenjiang, China. His current research interests include robotics, and articial
intelligence.
Bo Xu received the B.Sc. degrees in Automatic Control from China University of Mining and Technology, Xuzhou, China, in 1999, and the M.Sc. degree in
Computer Science from Jiangsu University, Zhenjiang, China, in 2005. She is currently pursuing the Ph.D. degree in the School of Electrical and Information
Engineering in Jiangsu University, Zhenjiang, China. Since 1999, she has been with the School of Electrical and Information Engineering, Jiangsu University,
Zhenjiang, China, where she is currently an associate professor. Her current research interests include motion control, and machine vision.
Ying Zhang received the B.Sc. degrees in Automatic Control from Jiangsu University, Zhenjiang, China, in 2009. Currently she is a M.Sc. candidate at the
School of Electrical and Information Engineering in Jiangsu University, Zhenjiang, China. Her current research interests include machine vision, and image
processing.
Jinjing Wang received the B.Sc. and M.Sc. degrees in Electrical Engineering from Jiangsu University, Zhenjiang, China, in 2006 and 2009, respectively.
Currently he is an engineer in the Chinese Academy of Agricultural Mechanization Sciences, Beijing, China. His research interests are in developing
harvesting machinery for fruits and vegetables.