Computer vision algorithm development for

classification of palm fruit ripeness

Cite as: AIP Conference Proceedings 2233, 030012 (2020); https://doi.org/10.1063/5.0002188
Published Online: 05 May 2020

Zhin Yue Wong, Wei Jen Chew, and Swee King Phang

ARTICLES YOU MAY BE INTERESTED IN

Hybrid approach for fruit recognition in high data variance

AIP Conference Proceedings 2233, 030003 (2020); https://doi.org/10.1063/5.0001362

Ripeness detection simulation of oil palm fruit bunches using laser-based imaging system
AIP Conference Proceedings 1801, 050003 (2017); https://doi.org/10.1063/1.4973101

Sign language gesture recognition with bispectrum features using SVM

AIP Conference Proceedings 2233, 030001 (2020); https://doi.org/10.1063/5.0002344

AIP Conference Proceedings 2233, 030012 (2020); https://doi.org/10.1063/5.0002188 2233, 030012

© 2020 Author(s).
 Computer Vision Algorithm Development for Classification
of Palm Fruit Ripeness
Zhin Yue Wong1, Wei Jen Chew1, a), Swee King Phang1
1
School of Engineering, Taylor’s University, Taylor’s Lakeside Campus,
No.1, Jalan Taylor’s, 47500 Subang Jaya, Selangor Darul Ehsan, Malaysia
a)
Corresponding author: [email protected]

Abstract. This paper presents a novel computer vision algorithm that focus on detecting the ripeness of palm fruit and the
developed computer vision algorithm is implemented into a low cost processor that can be integrated into a portable
standalone device. The computer vision algorithm mainly consists of two major functions, which are segmenting out the
section that consists of tree from the image and classifying the ripeness of palm fruit after locating the fresh fruit bunch
(FFB) on the palm tree. A sliding window method is used to separate the image of a palm oil plantation into various
sections. By retraining the Convolutional Neural Network (CNN), which is AlexNet, using fully labelled dataset, it is
capable to identify the existence of palm tree in each segmented section. For the ripeness detection of palm fruit, it is
achieved by analyzing dataset that consists of 100 palm fruit images from different category of ripeness. Those images are
analyzed in Hue, Saturation and Value (HSV) color spaces. Palm fruit that belongs to the ripe category has a unique range
of value that is not found in other categories. Therefore, the algorithm to classify the ripeness of palm fruit is developed
accordingly. The novel computer vision algorithm is then converted to Python programming language which is compatible
to run in Tinker Board. Tinker Board is one of the Single Board Computer (SBC) that consists of Graphical Processing
Unit (GPU) that is vital in digital image processing field. A high definition camera is equipped with the Tinker Board to
capture the image of palm oil plantation and palm fruit. The integrated device that consists of Tinker Board and camera
provides mobility to end-user to classify the ripeness of palm fruit in the palm oil plantation. The proposed algorithm
successfully yielded an accuracy of 85% as there were a total of 85 images which were correctly classified out of 100
images of palm fruit.

INTRODUCTION
The palm oil industry is growing rapidly in recent years especially in Southeast Asia as the benefit and usage of
palm oil is being introduced to different fields and promoted to the world. Even the palm oil mill effluent (POME)
generated from this industry can be used to produce biomass and biogases which can be used as renewable energy
sources [1-2]. As the usage and demand of palm oil is getting wider around the world, the establishment of palm oil
plantations are flourishing too. Due to its rapid expansion, the demand for land to establish palm oil plantation has
risen gradually and drastically [3]. This leads to conversion of forest and other agricultural land to palm oil plantation
to fulfill the demand. Therefore, the efficiency and utilization of extracting palm oil from fresh fruit bunch (FFB) from
current palm oil plantation has to be increased to avoid further land exploitation and expansion.
Malaysia, as one of the largest palm oil producers in the world [4], has also faced with the problem stated earlier
and is constantly developing and looking for solutions. In general, the ripeness of palm oil FFB does heavily affect
the quantity and quality of palm oil that can be extracted [5]. Before the usage of computer vision, the grading of palm
fruit is usually conducted manually through workers themselves. The shape, colour [6] and the number of loose fruits
on the FFB [7] are the three most significant factors to determine the ripeness of the palm fruit which is heavily
correlated to the quantity and quality of palm oil extracted. By observing the FFB, workers have to grade them into
two categories which are unripe and ripe. However, the present judgement of palm fruit ripeness, which is manually
grading them using workers, is unreliable. Workers may have different opinion and perception on the ripeness of palm
fruit and misjudgement could happen due to lack of experience. These human errors could lead to significant loss to

13th International Engineering Research Conference (13th EURECA 2019)

AIP Conf. Proc. 2233, 030012-1–030012-8; https://doi.org/10.1063/5.0002188
Published by AIP Publishing. 978-0-7354-1992-6/$30.00

030012-1
the profitability of the palm oil industry. To achieve better efficiency in harvesting FFB in current palm oil plantation,
computer vision which developed gradually in recent years is introduced to help classify the ripeness of palm fruit.
In this project, a novel computer vision algorithm will be developed and run on a Single Board Computer (SBC)
to classify the ripeness of palm fruit without detaching fresh fruit bunches from those palm trees. By applying digital
image processing techniques in appropriate color space, the proposed algorithm will be able to classify the ripeness
of FFB to two different classes which are unripe and ripe. This project is expected to eliminate the manual grading
system conducted by workers to help yield better quality and quantity of oil extracted in a palm oil plantation.

METHODOLOGY

Training of Convolutional Neural Network (CNN), AlexNet

AlexNet, which is one of the best convolutional neural networks nowadays, is selected as the classifier to determine
the location of palm tree in a palm oil plantation. The image processing toolbox will first be installed in MATLAB
software. AlexNet is formed by 5 convolution layers, 3 pooling layers and 2 fully connected layers and it is a pre-
trained CNN that is capable of classifying up to 1000 different objects [8-9]. Figure 1 shows the architecture of
AlexNet.

FIGURE 1. Architecture of pre-trained CNN, AlexNet [9]

Images that is used to train AlexNet are obtained by manually cropping the image of palm oil plantation into
sections that contain palm tree and sections that does not contain palm tree. By manually cropping the image of palm
oil plantation, 50 images of each section are created and used as a dataset to train AlexNet.

Identification of Palm Tree using AlexNet and Sliding Window

After successfully training AlexNet to identify the existence of palm tree in each section, a MATLAB code which
utilize a sliding window method and AlexNet is fully developed to detect the location of palm tree in the image of
palm oil plantation by showing section that consists of palm tree after analyzing the image. In the field of computer
vision, a sliding window is a rectangular region with predetermined height and width that is used to slide across an
image. Sliding window technique is often used with an image classifier in an object detection field to determine the
content that is located in the window. The content of the window will pass through an image classifier before it
proceeds to the next window. Sliding window method plays a critical role to determine the location of palm tree in a
palm oil plantation and the image classifier used is the retrained AlexNet. The image of palm oil plantation will first
be standardized to an image with 863 x 1300 height and width. The window size is determined by finding out the size
of each palm tree in the image. The size of each palm tree in the image has a width of 230. Therefore, window size of
863 x 230 has been chosen. The sliding window will move by width of 10 if no palm tree has been identified by
AlexNet in the section while it will move by width of 230 if AlexNet detects that there is a palm tree in the section.
This is to avoid repetitive results of showing the same palm tree.

030012-2
 Preparation of Palm Fruit Dataset
In Malaysia, the general palm tree species in the palm oil plantation is “Elaeis Guineensis” and they will grow up
to 20 meters height at maturity [7]. After detecting the palm tree using CNN and the sliding window technique, the
FFB on the tree can be segmented out by zooming in to the top part of the tree trunk. The location of the FFB on each
tree will be at a similar height during harvesting period since the palm fruit will only be harvested when it has reached
maturity to extract the optimum quantity of palm oil. The ripeness of the FFB can then be determined using the
proposed classification method. To test the algorithm, a dataset of palm fruit was used. The dataset of palm fruit used
in this article is provided by Mr. Sabri and his team [10-11] from Universiti Teknologi Mara due to the lack of publicly
available palm fruit datasets online. There are a total of 200 palm fruit images of different categories in the dataset.
Those images originated from a palm oil plantation that is located at Batu Pahat, Johor, Malaysia [10]. To develop the
computer vision algorithm that is able to classify the ripeness of palm fruit accordingly, 50 palm fruit images from the
ripe category and 50 images from the unripe category are randomly selected to analyze in hue, saturation and value
colour spaces.

Analysis of Palm Fruit Dataset

The basic knowledge on digital image processing is image is formed by three layers. The commonly used three
layers are red, green and blue layers (RGB). In the field of ripeness classification, image in Hue, Saturation, Value
(HSV) color space is less affected by illumination variation compared to RGB color space [10] therefore it has a higher
ripeness classification rate than RGB [12]. In this article, palm fruit images are analyzed in HSV color space. First,
those FFB images are converted from RGB color space to HSV color space using MATLAB built in function,
‘‫’ݒݏ݄ʹܾ݃ݎ‬. The image is converted into matrix form that shows every value of pixels in HSV color space. The mean
value of each plane in HSV color space is computed by summing up all the non-zero pixels value in each plane and
divide by the total amount of non-zero pixels using MATLAB command ‘݉݁ܽ݊ሺ݊‫’))(ݏ݋ݎ݁ݖ݊݋‬. The mean value of
pixels of 6 images of palm fruit from each ripeness category are tabulated and shown in Table 1. Although Table 1
only shows the results of 6 palm fruit images from each category, an observation has been made after concluding the
result of analyzation of all palm fruit images, which is the mean value of pixels in hue layer of a ripe palm fruit is
distributed between the range of 0.7 – 0.8. This range possessed by a ripe palm fruit is unique and the computer vision
algorithm to classify the ripeness of palm fruit is developed according to this observation.

Conversion of MATLAB code to Python programming language

After the novel computer vision algorithm is tested and successfully executed using MATLAB software, it is
converted to Python programming language which can be implemented in most Single Board Computer (SBC). The
Single Board Computer used to perform the ripeness classification in this paper is Tinker Board as it consists of
Graphical Processing Unit (GPU) which is a crucial component in digital image processing.
To convert the MATLAB code to Python programming language, the logic of the MATLAB code would need to
be deciphered. The function for each line of the MATLAB code is realized and the corresponding function is then
developed into Python programming language. Various important packages must be imported before writing the
Python code and they are scikit-image which is a collection of algorithms for image processing and OpenCV which
consists of various computer vision algorithm. The Python code will first be tested on a personal computer before
running it on the Tinker Board.

030012-3
 TABLE 1. Mean value of pixels in hue, saturation, value layer.
Unripe Palm Fruit Hue Saturation Value
UNR1 0.56 0.2843 0.453
UNR2 0.3682 0.2492 0.4272
UNR3 0.4199 0.2781 0.383
UNR4 0.6144 0.247 0.4585
UNR5 0.4064 0.2643 0.4869
UNR6 0.3873 0.2588 0.3919
Underripe Palm Fruit
UR1 0.6234 0.2995 0.6704
UR2 0.6408 0.2731 0.6974
UR3 0.6429 0.3318 0.5661
UR4 0.5869 0.3227 0.555
UR5 0.6309 0.3226 0.513
UR6 0.4806 0.3734 0.4819
Ripe Palm Fruit
R1 0.7607 0.3527 0.529
R2 0.7633 0.3475 0.5629
R3 0.7634 0.3897 0.5304
R4 0.7353 0.4365 0.4366
R5 0.7378 0.351 0.5381
R6 0.7809 0.5044 0.5883

Implementation on Low Cost Processor

After successfully implementing the Python code which yield the same output as the original MATLAB code, it
will be transferred and executed on the Tinker Board to test its functionality. Tinker Board is boot with ‘TinkerOS’
which is an operating system that is similar to Linux Operating System. After setting up the environment for Tinker
Board, a C615 Logitech HD Webcam is connected to the Tinker Board via USB port in order to provide input to the
Tinker Board to perform the ripeness classification and tree segmentation. For ripeness classification of FFB, the
Python code is designed to generate a rectangle box in the middle of the live video streamed by the camera. Due to
the portability of SBC, the SBC can be integrated into a drone system in the future and by controlling the drone, the
FFB of palm tree can be placed in the rectangle box and analyzation of palm fruit ripeness will only be executed within
the rectangle box. By doing so, the workload of Tinker Board can be reduced and it can further prevent the Tinker
Board from overheating. The analyzation of palm fruit ripeness will be executed every 5 seconds and it can be changed
depending on the moving speed of the drone. The image within the rectangle box will be captured every 5 seconds
and saved in Tinker Board with the result of ripeness classification and time captured. The Python code is set to be
run once Tinker Board is boot up so that it can perform the function without receiving any input command from the
end-user. After powering up the Tinker Board, it will perform the function as explained above automatically.

RESULTS AND DISCUSSION

Identification of Location of Tree in an Image.

After training AlexNet with self-crafted dataset, various images of plantation have also been collected to verify
the functionality of AlexNet. After importing various images that consists of multiple tree to the MATLAB software,
the section that consists of tree is segmented from the original image using the developed MATLAB code. The results
in Figure 2 and Figure 3 show the original image and the segmented section that consists of tree. By using the
algorithm, the tree that is located at the front most of the image will be detected.

030012-4
 (a) (b)
FIGURE 2. Original Image (a) and the results generated (b) after executing MATLAB code that perform identification of
existence of tree.

(a) (b)
FIGURE 3. Original Image (a) and the results generated (b) after executing MATLAB code that perform identification of
existence of tree.

030012-5
 In Figure 2, the original image consists of two trees. After executing the MATLAB code that utilizes the sliding
window method and AlexNet, the section that consists of tree are segmented out into 2 separate images.
In Figure 3, the original image consists of 1 tree only that is located at the most front of the image. Therefore, after
executing the MATLAB code, it shows that there is only 1 section that consists of tree and it is displayed in Figure 3.

Ripeness Classification of Palm Fruit

By analyzing the results obtained from the palm fruit dataset, the mean value of pixels in hue layer of ripe palm
fruit are distributed between the range of 0.7 – 0.8. According to this finding, palm fruit that do not possess mean
value of pixels in the hue layer between the range of 0.7-0.8 is classified as unripe category and vice versa. There are
a total of 200 images of palm fruit that is available in the dataset. 100 of them were used to help develop the algorithm
to classify the ripeness of palm fruit while the remaining images were used to verify the accuracy of the algorithm. A
sample of the results obtained from 6 images are shown in Figure 4 where the first 3 images are ripe palm fruit and
next 3 images are unripe palm fruit. The results show that ripe palm fruit will have a hue mean value between 0.7-0.8
while unripe palm fruit will have a hue mean value of lower than 0.7. From the 100 images tested, the algorithm
successfully yielded an accuracy of 85% as there were a total of 85 images which were correctly classified out of 100
images of palm fruit.

0.7021(ripe) 0.7257(ripe) 0.7197(ripe)

0.6263(unripe) 0.3685(unripe) 0.4443(unripe)

FIGURE 4. Images of Palm Fruit and the Mean value of Pixels in Hue Layer.

030012-6
 Implementation on Low Cost Processor
After converting the relevant MATLAB code to Python programming language and successfully implementing in
a personal computer, the code is imported to Tinker Board and being set to run once Tinker Board is boot up. Figure
5 shows the component needed in order to perform the ripeness classification of palm fruit and identification of
existence of tree which are a camera, Tinker Board and power supply. They can be integrated into a standalone device
to carry around to deliver the function mentioned.

FIGURE 5. Components needed to perform ripeness classification of palm fruit and identification of existence of tree.

FIGURE 6. Rectangle box indicated in the live streaming video and the results of ripeness detection within the rectangle box.

Figure 6 shows the rectangle box in the live streaming video and the results of ripeness detection within the
rectangle box. The size and location of the rectangle box can be adjusted to capture the FFB precisely. The results of
ripeness detection using algorithm developed earlier is shown in Figure 6. It will first display the mean value of pixels
of hue layer for the image within the rectangle box and determine the ripeness of palm fruit according to the computed
value. The results are generated every 5 seconds and can be adjusted according to the usage of end-user. The live
streaming video and the results of ripeness classification of palm fruit can be shown in a personal computer with the
help of a router by enabling remote access function [13].
After obtaining the ripeness classification result, the image within the rectangle box will be cropped and saved
automatically to the folder of ‘Palm Pictures’. Each image will be named automatically according to the results of

030012-7
ripeness detection with the time when the results are generated. Because of the results are generated every 5 seconds,
therefore the time difference of every pictures are approximately 5 seconds too.

CONCLUSION
This research project mainly focus on developing a novel computer vision algorithm to assist the palm oil industry
to classify the ripeness of palm fruit efficiently. The conventional grading method which is manually grading by
workers is not standardized and is prone to human errors. The section that consists of palm tree in an image of a palm
oil plantation will first be identified, and the FFB located on the palm tree will be analyzed afterwards by computing
the mean value of pixels in the hue layer of the image. The novel computer vision algorithm is first developed in
MATLAB software and converted into Python programming language that is compatible to run in a SBC with GPU
which is the Tinker Board. After transferring the Python code to Tinker Board, the code is set to be run once the Tinker
Board is boot up. It will analyze the FFB located in the rectangle box indicated in the live streaming video and generate
the result every 5 seconds and save the results in the Tinker Board with correctly labelled category of ripeness and
time of capture. Results obtained shows that the proposed algorithm was able to determine the ripeness category of
palm fruit up to an accuracy of 85%.

REFERENCES

1. S. Hassan, L. S. Kee, Hussain H. Al-Kayiem. “Experimental study of palm oil mill effluent and palm oil frond
waste mixture as an alternative biomass fuel,” Journal of Engineering Science and Technology, vol. 8, issue 6,
pp 703-712, December 2013.
2. I. N. Mohamad, R. Rohani, M. T. M. Nor, P. Claassen, M. S. A. Rahaman, M. S. Mastar@Masdar, M. I. Rosli,
“An overview of gas-upgrading technologies for biohydrogen produced from treatment of palm oil mill effluent,”
Journal of Engineering Science and Technology, vol. 12, issue 3, pp 725-755, March 2017.
3. S. L. Pimm, C. N. Jenkins, and S. J. Smith, “The Impacts of Oil Palm on Recent Deforestation and Biodiversity
Loss,” PLOS ONE, vol. 11, no. 7, p. e0159668, Jul. 2016.
4. M. F. Awalludin, O. Sulaiman, R. Hashim, and W. N. A. W. Nadhari, “An overview of the oil palm industry in
Malaysia and its waste utilization through thermochemical conversion, specifically via liquefaction,” Renewable
and Sustainable Energy Reviews, vol. 50, pp. 1469–1484, Oct. 2015.
5. Fadilah N, Saleh J M, Ibrahim H, Halim Z A. “Oil palm fresh fruit bunch ripeness classification using artificial
neural network,” In 4th IEEE International Conference on Intelligent and Advanced Systems, pp. 18-21, 2012.
6. Roseleena J, Nursuriati J, Ahmed J, Low CY. “Assessment of palm oil fresh fruit bunches using photogrammetric
grading system,” International Food Research Journal, pp 999-1005, 2011.
7. MPOB, Manual Penggredan Buah Kelapa Sawit “Oil Palm Fresh Fruit Bunches Grading Manual,” Third ed.
Malaysian Palm Oil Board (MPOB) Selangor, pp 955-957, 2006.
8. A. Krizhevsky, I. Sutskever, and H. Geoffrey E., “ImageNet Classification with Deep Convolutional Neural
Networks,” Adv. Neural Inf. Process., pp. 1–9, 2012.
9. H. C. Shin, “Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset
Characteristics and Transfer Learning,” IEEE Trans. Med. Imaging, vol. 35, no. 5, pp. 1285–1298, 2016.
10. Z. Ibrahim, N. Sabri, and D. Isa, “Palm Oil Fresh Fruit Bunch Ripeness Grading Recognition Using
Convolutional Neural Network,” Journal of Telecommunication, Electronic and Computer Engineering (JTEC),
vol. 10, no. 3–2, pp. 109-113, September 2018.
11. N. Sabri, Z. Ibrahim, and D. Isa, “Evaluation of Color Models for Palm Oil Fresh Fruit Bunch Ripeness
Classification,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 11, no. 2, pp. 549–
557, August 2018.
12. Fadilah, N., Mohamad-Saleh, J., Abdul Halim, Z., Ibrahim, H., and Syed Ali, S. , “Intelligent Color Vision
System for Ripeness Classification of Oil Palm Fresh Fruit Bunch,” Sensors, pp 7-9, 2012.
13. S. K. Phang, M. A.-A. Hassan, Z. Y. Wong, Z. Y. Ng, and Y. L. Lai, “Development of Autonomous UAV
Systems for Low Light Surveillance Applications Using Night Vision Camera,” Journal of Advanced Research
in Dynamical and Control Systems, vol. 10, no. 13-Special Issue, pp. 1379-1391, 2018.

030012-8