Food Recognition and Health Monitoring System

for Recommending Daily Calorie Intake

Rutuja Rewane P. M. Chouragade
Department of Computer Science and Engineering Department of Computer Science and Engineering
Government College of Engineering, Amravati Government College of Engineering, Amravati
Amravati, India Amravati, India
[email protected] [email protected]

Abstract—Growing concern of human life is about their games, accessing cell phones for longer hours, and watching
health and fitness. People are generally inattentive about the television for whole day, thus giving no time for outdoor
contents present in the food item they are consuming. Also games and sports. The efforts taken to reduce weight and
today’s sedentary lifestyle is giving rise to increased obesity in excessive body fats like gymming, exercising prove great
some people and also weakness and malnutrition in some and provide fabulous results. But maintaining that proper
others. This paper presents a study of various food recognition
weight is often difficult. Once all these exercises are
and health monitoring systems. The structures based on
wearable gadgets and android implementation are considered stopped, they produce a reversible effect and weight is
along with the structures based on augmented reality. The increased. For this, taking proper diet with appropriate and
structures are arranged on the basis of algorithmic details used well maintained amount of calories and nutrients is very
for various steps in the activity of food identification and also important.
for estimating the calorie count. Performance and the
execution rate of these systems are compared along with their The use of android devices has increased rapidly
dominance factors and pitfalls and depending on which, a worldwide in last few years. Therefore, many health
system is proposed for food recognition and monitoring health monitoring systems and also applications for food
by recommending the daily nutrition and calorie intake. recognition and nutrition and calorie estimation have been
developed for these android devices. Along with this, many
wearable gadgets for health monitoring are also developed
Keywords— Food nutrition, Recommendation, Visualization, which help the users to count heart beat rate, walking steps
Segmentation, Feature extraction, Classification
and also measuring the calorie count. Gadgets are also
developed for detecting the masticating moments and
I. INTRODUCTION differentiating between solid and liquid foods and then
Collection of massive amount of fats in the body and detecting the nutrition and calories present in the food. Also
stacking up of it, is termed in simple terms as obesity. Large some devices take input using user’s speech. The concept of
percentage of the population is affected by this medical augmented reality is also used in many food recognition
condition. Upon grasping a certain stage, it can have applications.
unpromising effects on health. Generally, person with BMI The study of all such devices for food recognition and
over 30 kg/m2 is considered as obesed and with 25-30 kg/m2 nutritional detection health monitoring systems have been
as overweight. Increased obesity give rise to number of done here and a brief review of all such systems is presented
disorders and sometimes depression also. Factors which are along with their efficiency, accuracy and various
responsible for causing obesity are mainly reduction in the technologies used in them.
amount of physical movements of body, some genetic
inherited factors, and the most important one is improper A. Applications
diet, which includes excessive food intake as well as The population of people having such excessive body fats
improper calorie consumption and nutrient intake. is increasing rapidly. So, these food recognition and
To lessen the extra body fats, the factors to be taken care nutritional detection systems can be used by such people to
of are proper and well maintained daily food intake, more monitor their daily nutrition and calorie intake. It is useful
amount of physical activity, yoga, jogging and daily for people affected with diabetes, cardiovascular diseases,
exercise, among which selection of food contents to be types of cancer and many diet related diseases. People living
consumed should be considered with high priority. Food fast life in urban areas and who are not having time for
items having high amount of fibers such as leafy vegetables gymming and exercising can use these systems and
should be consumed in large quantity and consumption of applications to keep their calorie count in control and also to
food items with large amounts of fats should be avoided. If get recommendations on daily intake of food items. Also,
not reduced in appropriate time, these excessive body fats can be used by doctors for recommending it to their patients.
can be more destructive and dangerous. The working culture Such applications can be proved of greater importance for
consisting of reduced amount of physical work and working college going students and young generation in their
continuously for large hours sitting at one place is also one extremely busy life.
of the important cause. Also, equipments have been
developed for performing many tasks in daily life as well as
at industrial level, which leads to zero physical efforts.
Nowadays, children are mostly attracted towards video

978-1-5386-8158-9/19/$31.00©2019IEEE
 II. LITERATURE SURVEY reduction technique is employed. Accuracy level of 99% is
attained for single food portions.
A. Augmented Reality based food recognition systems
D. Mobile cloud based system
The paper [1] proposed a wearable gadget for food
monitoring which is based on the concept of Augmented In the paper [8], authors proposed a structure for the
Reality. The system allows to make the nutritional measurement of calories. The cloud computing environment
information of the food available and visible in various real makes use of the Naive Bayes algorithm with machine
–life scenarios, such as supermarket. Here, the use of learning for the purpose of food recognition. Images of food
reverse image search (RIS) and text mining has been done to are clicked before and after eating. Food is also check for
recognize the food. The main components are food its freshness by using the textural extraction method. Three
recognition, retrieval of nutrition information, image parameters are used for observing outcomes time, accuracy
tracking and visualization. The overall accuracy of the RIS and freshness. Proposed system which is using Naive Bayes
on images from real scene is 75.9% and on photos from the is more efficient as compared to systems which used SVM.
image database is 87.9%. The Google glass which is used E. Crowdsourcing
for visualization run much hotter, also more power
The work in the paper [9] is the study to address the
consumption and currently does not allow graphic
problem of obesity by using personalized messaging. The
processing unit acceleration on a regular Android survey was conducted by taking the help from both
smartphone and tablet, therefore, the system cannot expertise and non-expertise, whom the training was
optimize the runtime much as things stand.
provided for certain period. Here, the main attraction is
A system based on android implementation [7], which crowdsourcing which allows to engage with lot of people to
shows the information of contents present in the food by provide either data or input in any other form. Opinions
taking the images of food. To get the nutrition information from a crowd of experts or non-experts can be aggregated to
and display it FatSecret server is used along with FatSecret provide collective wisdom. The crowd of experts was more
API. Here, the accuracy obtained for food identification accurate as compared to crowd of non-experts.
from the image on an average is 92% and the whole process
F. Nutrition monitoring systems based on wearable
of identifying the food item, detecting its name and nutrient
devices
contents present in it take the total time of 9.295 seconds.
In the paper [13], authors introduced a wearable gadget
In Paper [22] a mobile application to help users to see the based monitoring system for keeping track of nutrition
nutrition contents present in the food based on augmented intake. The data is provided to the wearable gadget through
reality is presented. Images taken with the help of the use of handheld mobile device. Main task is to identify
smartphone camera were used in their 3D form. Volume the gulping movements.
estimation is done to accurately calculate the nutrition
information. Authors [21] proposed a structure for assisting people
with diabetes. It allows them to keep the track of their
B. Machine learning based food detection system calorie count. Also the goal is towards achieving daily food
The researchers [2] attempts to develop an android based identification in semi-automatic manner. 75% of the classes
implementation for food recognition that could be used as a showed improvement and the remaining 25% were mostly
health awareness tool. The system allows user to take the maintained.
photo of the food and show its nutritional contents. One goal
G. System for multiple food recognition
of this study is also to determine the level of awareness of
users in terms of their eating habits and food nutrition. The paper [16], gave a solution for the problem of
Machine learning is used for detecting food from images. multiple food identification. To detect the multiple items
The study shows that, the respondent’s level of food present on the plate, candidate regions are drawn
nutritional awareness is remarkably low as to the status of surrounding each food item to differentiate them from one
cognizance about food intake. another. For performance evaluation a classification rate
(CR) is used. 55.8% classification rate was achieved for
C. Use of SVM for classification multiple food items and 68.9% for single food item.
In the dietary management system [4], the calorie values H. CNN based food identification system
are calculated by using segmentation, feature extraction and
classification. After that, the calorie value is also calculated In paper [19], authors present a system to assist doctors as
with food area volume and nutrition measure based on the well as patients regarding diet related health issues. Taking
mass value. Classification rate of 95% is achieved. the images of food before consuming it helps in determining
the calories present in it. Applying neural networks along
Researchers [15] proposed a system for identifying food with the segmentation methods helps to attain a good
item along with its nutrition elements. Bounding box are accuracy level.
drawn around the food item in plate to identify it.
Evaluation count of 81.55% is achieved. I. Food recognition system based on speech input
The applications capturing the pictures of food and then
In the paper [18], researchers proposed a measurement
calculating the volume of food portions present on the plate
system for estimating calories. Measurement of calories is
to detect its nutrient contents face some challenges. To
done by tracking the intake of food. Mainly the map overcome those challenges, a framework [20] is proposed
by authors. It makes use of the voice input, increasing the
performance of the system. The structure taking the voice detail of foods and activities. This is to prevent redundancies
input provides the ease to add new classes of food items. of data.
J. Android based food recognition and nutrition In work [17], authors presented an approach for calorie
estimation applications content estimation, which is based on analysis of images.
Authors [3] present a suggestion providing system for Multiple image features used for similarity matching are
estimation of contents present in the food. Health profile of color histograms, color correlograms and SURF features.
user is build and accordingly nutritional recommendation is Results show that 79% of the estimates are correct within
provided. The system can provide the advice for using ±40% error and 35% correct within ±20% error.
typical calabrian foods, according to particular’s health
III. PROPOSED METHODOLOGY
conditions and related pathologies. In its current version, the
system can provide recommendations for users affected by The proposed system will allow the user to register
various diseases and also for healthy users. themself on the given mobile application. This will include
collecting the information from the user which includes the
The researchers [5] proposed an application for self-care. basic knowledge like age, height, lifestyle, disease (if any)
The application help users to manage their health and and food taken. This information is given as input to calorie
monitor daily intake of food. The analysis of daily calories and BMI calculator for calculating the BMI and calorie
and nutrition is made available to the user. FoodForCare is consumed by that person. Total fats on the body are
specially designed for patients with NCDs diseases. The calculated on the basis of weight and height. After
evaluation of FoodForCare is done by measuring the calculating BMI and calories the result is given to the
satisfaction of the users. recommender engine, where rules are applied and it will
The paper [6] recommends a model for systematic diet give the options for healthy food and also the requirements
and healthy food. Two dimensions for food of nutrition and calories for that person.
recommendations are considered i.e. calories intake by the The system mainly consist of three main components for
person and the calories which are left unused for that nutrition visualization:
person. Nutrition limit for particular age group is decided on 1) Food Recognition
the basis of Recommended Dietary Intake (RDI) table, 2) Retrieval of nutrition information
called as Recommended Daily Intake. Food options are 3) Image tracking and visualization
provided to the user according to BMI, healthy immune
system and calorie intake.
Authors [10] have proposed a device for identifying
artificially ripened fruits by using image processing. Use of
histogram is done for comparing the features of given fruit.
Histogram calculated using MATLAB provides the
efficiency of 91%. Also the android application was
developed for calculating the histogram using the threshold
values. Fruits ripened by using chemical compounds are
identified.
The work in [11] is an implementation for health
suggestions. Various algorithms were used for retrieving
information from database. In the proposed system, the
user’s profile is used to categorize the associated
knowledge, so that the users can make the food as well as
exercise enquiries. The application and techniques are
efficient, feasible and user friendly that would be less time
consuming. Fig. 1. Proposed system for food identification.

The study [12] proposed a methodology for automized When a person wishes to consume some food item, the
food identification. Performance evaluation is carried out by image of that food item is clicked before consuming it. On
considering the accuracy of how correctly the food item can the server, the following steps are performed on the image:
be identified from the provided image. The results obtained
proved to be good but the given method proved to be a) Segmentation: The segmentation process holds the
somewhat expensive. The best result achieved was 77.8%. food image as input image and segment it significantly. The
main task here is to categorize and labeling of points
Daily calorie manager [14] is an android application to present on the image into various classes. It belongs to the
manage the daily calorie intake of the users. The application class of data clustering methods. Each point on the image
was build keeping in mind the health issues of Malaysian represent some data. Points representing same data or
people especially diabetes. Rational Unified Process model features are grouped together in one group called as cluster.
(RUP) was used to build the application. As compared to Here, the mean field is incorporated in the objective
other applications, it proved to be much more easier to use. function to improve its performance and evaluation rate.
One limitation of application is in adding another calorie
 b) Feature Extraction: The features of every food f) Crowdsourcing: Inputs are taken from a crowd of
hold the significant demonstration of the image. Attributes people belonging to different fields. People providing input
such as texture, color, shape are obtained from the can be expertize in that particular field or non-expertize
segmented image of the food. F- Shift Invariant Feature also. Collection of data from large category of users
Transform algorithm is used for feature extraction. improves the accuracy level of data and also its evaluation.
c) Classification: After the segmentation and feature When designing personalized application for obesity
extraction the food items are classified using the CNN prevention and management, a crowd of experts and non-
algorithm, which is one of the popular technique used for experts should be included to estimate calories in foods
classification. After the food items are classified by the from photographs taken on a smartphone, which may have
CNN, the classified results from the image are used to potential in contributing to obesity prevention and
estimate the nutrition value of the food. management.

IV. CONCLUSIONS
Survey of various food recognition, nutrition and calorie
detection systems have been done. Here, many android
based food recognition implementations along with the
wearable gadgets used as monitoring devices are studied.
Augmented reality based systems with the use of Google
glass have been observed with great interest along with their
efficiency and accuracy. All these systems are mainly
categorized according to the working of their modules and
theories used in them. Lastly, various systems providing
suggestions regarding health issues are also studied.
Efficiency and accuracy of the systems and applications
have been observed and compared. Proposed system is a
promising application for food nutrition information
retrieval and visualization. The proposed food monitoring
system recognizes food, retrieves the nutrition information
from a nutrition database and overlays the dietary facts on
the food surface. The system also recommend the daily
nutrition requirements for the user. Also, the freshness of
food is determined along with indication provided for
chemically ripened fruits.
REFERENCES
[1] Haotian Jiang, Jammes Starkman, Menghan Liu, and Ming-Chun
Huang “Food Nutrition Visualization on Google Glass”, IEEE
Consumer Electronics Magazine , May 2018, pp. 21-31.
[2] Arnel B. Ocay, Jane M. Fernandez, Thelma D. Palaoag, “NutriTrack:
Fig. 2. Food identification and nutrition estimation module. Android-based Food Recognition App for Nutrition Awareness”, 3rd
IEEE International Conference on Computer and Communications,
2017, pp. 2099-2104.
d) Freshness of the food: Freshness of the food will [3] Agapito G., Calabrese B., Guzzi P. H., Cannataro M., Simeoni M.,
be found before eating it. If the food is not fresh then, the Car´e I., Lamprinoudi T., Fuiano G., Pujia A., “DIETOS: a
message will be displayed to the user that the food is not recommender system for adaptive diet monitoring and personalized
food Suggestion”, Fourth International IEEE Workshop on e-Health
fresh don’t eat it. Mainly by using the color features, Pervasive Wireless Applications and Services, 2016.
freshness of the food will be determined. Here, in the testing [4] S. Jasmine Minija, W.R. Sam Emmanuel, “Food image Classification
dataset for every food type average, higher and lower values using Sphere Shaped-Support Vector Machine”, Proceedings of the
for color and texture features will be calculated. Depending International Conference on Inventive Computing and Informatics
(ICICI 2017) IEEE Xplore Compliant - Part Number: CFP17L34-
upon these values the food will be put in either fresh or stale ART, ISBN: 978-1-5386-4031-9, pp. 109-113.
category. [5] Natnicha Suthumchai, Sirin Thongsukh, Pacharamai Yusuksataporn,
e) Detecting artificially ripened fruits: Human eye is Songsri Tangsripairoj, “FoodForCare: An Android Application for
Self-Care with Healthy Food”, Fifth ICT International Student Project
not so powerful to detect the use of chemicals being made to Conference (ICT-ISPC), 2016.
ripe the fruits. The proposed system gets an image of the [6] Neha Gaur, Archana Singh, “Recommender System – Making
fruit and its histogram value is calculated. The highest value LifestyleHealthy Using Information Retrieval”, 2nd International
obtained upon calculations from histogram is taken as a Conference on Next Generation Computing Technologies (NGCT-
reference value. The threshold of histogram value will be set 2016) Dehradun, India 14-16 October 2016, pp. 479-484.
after taking up several trials using different input images. [7] Ridho Rahman Hariadi, Wijayanti Nurul Khotimah, Eko Adhi
Wiyono, “Design and Implementation of Food Nutrition Information
The histogram value calculated greater than this value will System using SURF and FatSecret API”, pp. 181-183.
be considered to be a fruit ripened artificially and if it is [8] Diptee Kumbhar, Prof. Sarita Patil, “Mobile Cloud based System
lesser it will be considered to be naturally ripened fruit. Recognizing Nutrition and Freshness of Food Image”, International
Conference on Energy, Communication, Data Analytics and Soft
Computing (ICECDS-2017), pp. 709-714.
[9] Dr. Anne Moorhead, Dr Raymond Bond, Dr Huri Zheng, “Smart [17] Tatsuya Miyazaki , Gamhewage C. de Silva, Kiyoharu Aizawa,
Food: Crowdsourcing of experts in nutrition and non-experts in “Image-based Calorie Content Estimation for Dietary Assessment”,
identifying calories of meals using smartphone as a potential tool IEEE International Symposium on Multimedia, 2011, pp.363-368.
contributing to obesity prevention and management”, IEEE [18] Parisa Pouladzadeh, Pallavi Kuhad, Sri Vijay Bharat Peddi,
International Conference on Bioinformatics and Biomedicine Abdulsalam Yassine, Shervin Shirmohammadi,“ MOBILE CLOUD
(BlBM), 2015, pp. 1777-1779. BASED FOOD CALORIE MEASUREMENT”.
[10] Dr.S.Maheswaran, S.Sathesh, P.Priyadharshini, Vivek. B, [19] Parisa Pouladzadeh, Pallavi Kuhad, Sri Vijay Bharat Peddi,
“Identification of Artificially Ripened Fruits Using Smart Phones”, Abdulsalam Yassine, Shervin Shirmohammadi, “Food Calorie
International Conference on Intelligent Computing and Control Measurement Using Deep Learning Neural Network”.
(I2C2), 2017.
[20] Manika Puri, Zhiwei Zhu ,Qian Yu, Ajay Divakaran, Harpreet
[11] Miss Shreya B.Ahire, Ms. Harmeet Kaur Khanuja, “A Personalized Sawhney, “Recognition and Volume Estimation of Food Intake using
Framework for HealthCare Recommendation”, International a Mobile Device”, 978-1-4244-5498-3/09, 2009 IEEE.
Conference on Computing Communication Control and Automation,
2015, pp. 442-445. [21] Geeta Shroff, Asim Smailagic, Daniel P. Siewiorek, “Wearable
Context-Aware Food Recognition for Calorie Monitoring”, 978-1-
[12] Marios M. Anthimopoulos, Lauro Gianola, Luca Scarnato, Peter 4244-2638-6/08, 2008 IEEE.
Diem, Stavroula G. Mougiakakou, “A Food Recognition System for
[22] Thomas Stütz, Radomir Dinic, Michael Domhardt, Simon Ginzinger,
Diabetic Patients Based on an Optimized Bag-of-Features Model”,
“Can Mobile Augmented Reality Systems Assist in Portion
IEEE JOURNAL OF BIOMEDICAL AND HEALTH
INFORMATICS, VOL. 18, NO. 4, JULY 2014, pp. 1261-1271. Estimation?A User Study”, IEEE International Symposium on Mixed
and Augmented Reality 2014 Media, Art, Social Science, Humanities
[13] Haik Kalantarian, Nabil Alshurafa, Majid Sarrafzadeh, “A Wearable and Design Proceedings 10 - 12 September 2014, Munich, Germany,
Nutrition Monitoring System”, 11th International Conference on pp. 51-57.
Wearable and Implantable Body Sensor Networks, 2014, pp.75-80.
[14] Shahreen Kasim, Faten Azmira Zakaria, “Daily calorie manager for
basic daily use”, 978-1-4799-0048-0/13, pp. 437-442.
[15] Yoshiyuki kawano, Keiji yanai, “Real-time Mobile Food Recognition
System”, IEEE Conference on Computer Vision and Pattern
Recognition Workshops, 2013.
[16] Yuji Matsuda, Hajime Hoashi, Keiji Yanai, “RECOGNITION OF
MULTIPLE-FOOD IMAGES BY DETECTING CANDIDATE
REGIONS”, IEEE International Conference on Multimedia and Expo,
2012, pp. 25-30.