5-Pesticide00027_3-PRIMARY.

qxd 8/29/17 11:25 AM Page 228

ORIGINAL ARTICLE

Pesticide applicators questionnaire content validation:

A fuzzy delphi method

Sujith Kumar Manakandan, MPH1, Rosnah Ismail, DrPH1, Mohd Ridhuan Mohd Jamil, PhD2, Priya Ragunath,
MPH3

1
Occupational Health Unit, Department of Community Health, UKM Medical Centre, The National University of Malaysia,
Kuala Lumpur, Malaysia, 2Department of Mechanical Engineering, Politeknik Nilai, Negeri Sembilan, 3Occupational Health
Unit, Disease Control Division, Ministry of Health, Putrajaya, Malaysia

SUMMARY set were obtained, hence it is now ready for further

Background: The most crucial step in forming a set of construct validation process.
survey questionnaire is deciding the appropriate items in a
construct. Retaining irrelevant items and removing KEY WORDS:
important items will certainly mislead the direction of a
particular study. This article demonstrates Fuzzy Delphi
Fuzzy Delphi, survey questionnaire, validation, noise exposure,

method as one of the scientific analysis technique to

chemical exposure

consolidate consensus agreement within a panel of experts

pertaining to each item's appropriateness. This method INTRODUCTION
reduces the ambiguity, diversity, and discrepancy of the
opinions among the experts hence enhances the quality of
The questionnaire is commonly used as a measurement tool

the selected items. The main purpose of this study was to

in Public Health research. Today, varieties of validated

obtain experts' consensus on the suitability of the pre-

questionnaires are easily accessible and retrievable from

selected items on the questionnaire.

various databases. However, the main challenge as Public
Health researcher is determining the items’ suitability of the

Methods: The panel consists of sixteen experts from the

questionnaire to be used for the intended research scope.

Occupational and Environmental Health Unit of Ministry of

Consulting the experts of the research scope is one of the

Health, Vector-borne Disease Control Unit of Ministry of

ways to solve the challenge.

Health and Occupational and Safety Health Unit of both

public and private universities. A set of questionnaires
Fuzzy Delphi method is the current trend in consulting those

related to noise and chemical exposure were compiled

experts. It is the modification method of former classic Delphi

based on the literature search. There was a total of six

method developed by two scientists, Olaf Holmer and

constructs with 60 items in which three constructs for

Norman Dalkey, which has been used widely to get the

knowledge, attitude, and practice of noise exposure and

expert opinions via surveys.1 It has few disadvantages, such

three constructs for knowledge, attitude, and practice of

as misinterpretation of experts’ opinions due to neglecting

chemical exposure. The validation process replicated

the fuzziness, no dedicated rules to yield the desired outcome,

recent Fuzzy Delphi method that using a concept of

loss of experts' interest and data due to its time-consuming

Triangular Fuzzy Numbers and Defuzzification process.

process which will lead to repeated surveys and ultimately
make the study more expensive.2,3,4 In view of the importance

Results: A 100% response rate was obtained from all the

to solve the ambiguity of the experts, whom might have a

sixteen experts with an average Likert scoring of four to five.

common understanding,3 Fuzzy Delphi Method (FDM) was

Post FDM analysis, the first prerequisite was fulfilled with a

introduced over three decades ago5 which was again revised

threshold value (d) ≤ 0.2, hence all the six constructs were
by previous scholars.6, 7 It uses fuzzy set numbers or fuzzy set

accepted. For the second prerequisite, three items (21%)

theory whereby each set will have a value from 0 to 1. This

from noise-attitude construct and four items (40%) from

method reduces cost and time during evaluating each item in

chemical-practice construct had expert consensus lesser

a questionnaire. It reduces the survey rounds and increases

than 75%, which giving rise to about 12% from the total
items recovery rate, allows the experts to express their

items in the questionnaire. The third prerequisite was used

opinions without any ambiguity biases, which enhances the

to rank the items within the constructs by calculating the

completeness and consistency of opinion8 and to get the

average fuzzy numbers. The seven items which did not fulfill
consensus from the experts without jeopardising their

the second prerequisite similarly had lower ranks during the

original opinion and by giving their real reaction towards the

analysis, therefore those items were discarded from the final

questions.9

draft. As far as concern, there are no studies available pertaining to

Conclusion: Post FDM analysis, the experts' consensus on

the Pesticide Applicators (Foggers) of the Ministry of Health.

the suitability of the pre-selected items on the questionnaire

Their nature of work, which exposes them to both noise and
chemical hazards warrants a set of questionnaires from the
This article was accepted: 1 February 2017
Corresponding Author: Rosnah Ismail
Email: [email protected]

228 Med J Malaysia Vol 72 No 4 August 2017

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 229

Pesticide applicators questionnaire content validation: A fuzzy delphi method

experts from both occupational health and vector-borne of years of experience. Each chosen experts was at least one
disease control unit of Ministry of Health. Therefore, we feel, of the following; 1 a public health physician that has
FDM is the most suitable method to be used to form a set of published an article related pesticide applicators, 2 an
questionnaire. In this article, Fuzzy Delphi Method was used administrator who manages the pesticide applicators at
prior to constructing validation process of pesticide district/state/national level, 3 had previously worked or
applicators questionnaire pertaining to knowledge, attitude experienced in pesticide application related job,4 minimum
and practice related to noise and chemical exposure in this five years of experience in the related field of noise and
study. The main purpose of this study was to obtain experts’ pesticide exposure,5 an academician or tutor in the
consensus on the suitability of the pre-selected items on the occupational health related field. A total of sixteen experts
questionnaire. were recruited as the panel of experts via non-probable,
purposive sampling method. The number was considered

MATERIALS AND METHODS

optimum and complied with previous suggestions which
required 10 to 50 experts.18 Lesser amount of experts is
Pesticide applicator questionnaire required, i.e. 10 to 15, if they are homogenous experts.2,4
A number of items related to knowledge, attitude and
practice to noise and chemical exposure were compiled based The panel of experts was from various part of Malaysia. The
on the literature search of previous studies instruments10, 12,14 panel consisted of eight Public Health Physicians of
using keywords noise induce hearing loss, noise, pesticide Occupational and Environmental Health Unit of Ministry of
and, knowledge, attitude and practice (KAP).Three main Health, three academicians of Occupational Health from the
databases were explored i.e. PubMed, Ovid, and Google public and private universities, three Health Inspectors and
Scholar. Some items were obtained directly from the authors two Entomologists who are presently working in the Vector-
via email.11, 13 All those studies were done on sawmill workers, borne Disease Control unit of Ministry of Health. They were
vector control workers, industrial workers and the general contacted by the researcher via a phone call to brief the FDM
population. A total of three constructs were finalized for each and get their verbal informed consent. A set of 60 items
noise and chemical exposure. For noise exposure, the selected questionnaire was distributed to each expert via email
items for knowledge, attitude, and practice were 10, 14 and 6 between October and December 2016. They were instructed to
items, respectively. Meanwhile, for chemical exposure, there indicate their agreement level for each item using five-point
were 10 items for each construct of knowledge, attitude, and Likert scale i.e. 1= highly disagree to 5= highly agree. Upon
practice. Only 33% of the selected items were in the English successful completion, each answer sheets were delivered to
language. Due to limited resources, those items were primary researcher through emails.
translated from English to Malay version using simplest,
traditional forward translation15 to best of the first author’s Data Analysis
ability. The items were constructed based on the nature of The analysis of the data was replicated from the latest Malay
work environment faced by the Pesticide Applicators and version published material,8 which discusses two important
considering the purpose and conceptual basis of the concepts of FDM, namely Triangular Fuzzy Numbers and
questionnaire measurement. The items were developed in Defuzzification process (refer Figure 1).
terms of routine, simple terminologies without deviating
from the original theoretical meaning of the questions. For Triangular Fuzzy Numbers
example, "Chemical enters the body through breathing in" Triangular Fuzzy Numbers (TFN) provided an opportunity for
and "I am confident that I can use PPE properly”10 were each recorded response made by an expert in the form of
translated into “Racun memasuki tubuh badan melalui Likert scale scoring to be translated into fuzzy scoring (Refer
pernafasan” and “Saya pasti saya boleh menggunakan alat Table I). Each recorded response had three values to consider,
pelindung diri dengan betul”, respectively. Apart from that, namely the average minimum value (n1), most reasonable
few items were adopted and modified to suit the study value (n2), and the maximum value (n3). The rationale of
population, whereby originally those items were in TFN was to show the fuzziness or inexactness in the opinion
behavioural questions, modified into a practical statement. made by an expert. Every opinion had a certain amount of
Example, “how often do you wash your hands before putting ambiguity which can't be addressed by using a Likert scale
on gloves” into “Saya mencuci tangan sebelum memakai sarung because it is a fixed score. Let us say an item “Racun
tangan semasa mengendalikan racun serangga”. This memasuki tubuh badan melalui pernafasan” was scored 5
compilation of 60 items was later presented to a panel of (highly agree) by an expert. The score is converted into
experts. minimum, most reasonable, and the maximum value of 0.6,
0.8 and 1.0 fuzzy scores, respectively. It indicated the expert
Panel of experts agreeable to the item is 60%, 80%, and 100%, respectively.
A panel of experts is defined as a group of persons who are The fuzzy scores were averaged as indicated by m1, m2 and
skilful in the scope of a study area. They are selected based m3 values for further Defuzzification process.
on leading position in public health care system with a
significant practical knowledge in their field of practice.16 Defuzzification process
They also should represent his/her circle of professional Defuzzification process (Amax) is a ranking process of each
Occupational Health group as suggested by the previous item to identify the importance level of each item. This
scholar.17 In this study, the inclusion criteria for the experts ranking process was very helpful to determine whether to
were occupational health related specialization, familiarity keep or discard certain items based on the following formula:
with the working zone, authority in the field, and the number Amax = 1/3 * (m1 + m2 + m3)

Med J Malaysia Vol 72 No 4 August 2017 229

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 230

Original Article

Table I: The difference between Likert scale scoring and Fuzzy scoring for a five-point scale
VariaLikert Scale Scoring Linguistic variable Fuzzy Scoring
5 Highly Agree 0.6, 0.8, 1.0
4 Agree 0.4, 0.6, 0.8
3 Moderately/Not Sure 0.2, 0.4, 0.6
2 Not Agree 0.0, 0.2, 0.4
1 Highly Not Agree 0.0, 0.0, 0.2

Table II: The summary of All Three Pre-requisites Post Fuzzy Delphi Analysis (Noise)
Construct/Items Average Likert Threshold Percentage of Average of Ranking Verdict
Score Value Experts’ Fuzzy
(d) ≤ 0.2 Consensus (%) Numbers
Noise-Knowledge 0.00 Acceptable
NK-1 5 75 0.738 2 Retained
NK-2 5 75 0.738 2 Retained
NK-3 5 75 0.738 2 Retained
NK-4 5 75 0.738 2 Retained
NK-5 5 81 0.750 1 Retained
NK-6 5 75 0.738 2 Retained
NK-7 5 75 0.738 2 Retained
NK-8 5 81 0.750 1 Retained
NK-9 5 81 0.750 1 Retained
NK-10 5 81 0.750 1 Retained
Noise-Attitude 0.01 Acceptable
NA-1 5 94 0.738 3 Retained
NA-2 5 94 0.700 5 Retained
NA-3 5 94 0.725 4 Retained
NA-4 5 81 0.738 3 Retained
NA-5 4 31* 0.588 6 Discarded
NA-6 5 81 0.763 1 Retained
NA-7 5 75 0.750 2 Retained
NA-8 5 81 0.763 1 Retained
NA-9 5 94 0.725 4 Retained
NA-10 5 94 0.725 4 Retained
NA-11 5 88 0.763 1 Retained
NA-12 4 25* 0.583 7 Discarded
NA-13 5 88 0.725 4 Retained
NA-14 4 38* 0.533 8 Discarded
Noise-Practice 0.01 Acceptable
NP-1 5 75 0.738 3 Retained
NP-2 5 81 0.750 2 Retained
NP-3 5 81 0.738 3 Retained
NP-4 5 81 0.738 3 Retained
NP-5 5 88 0.775 1 Retained
NP-6 5 88 0.725 4 Retained
* Item with Experts’ consensus ≤ 75% and lowest ranking within their construct

Determination of item acceptability Once the value was obtained, a threshold value (d-construct)
There were three prerequisites to be fulfilled to determine the was calculated by using the formula below:
acceptability of the constructs and its respective items. The Threshold Value ∑ Average Threshold Value, (d) for each item
=
prerequisites were (1) threshold value, d-construct ≤ 0.219, (2) (d-Construct) Total Experts x Total Items in Constructs
experts agreement on evaluated items ≥75%20 and (3)
ranking of the item. The threshold value,d-construct indicates Based on the value, the acceptability of the construct was
the selection of certain construct based on the consensus of determined, whereby a construct was accepted if the
the experts for each construct. However, prior to that, a Threshold value (d-construct) ≤ 0.2. Expert agreement on
threshold value (d) for each itemwas found, by calculating each evaluated item was also based on threshold value (d) for
the difference between average fuzzy number and each each item, whereby (d) ≤ 0.2 are accepted. The frequency of
expert fuzzy number (refer Figure 2& 3)using the formula accepted values was presented as percentage as shown in
below: Figure 3. Items with expert agreement of less than 75% were
discarded. The rank of an item within a similar construct was
determined after Defuzzification process as mentioned earlier
(refer Figure 1). All respondents data were entered and
analysed using Microsoft excel version 2013. A complete

230 Med J Malaysia Vol 72 No 4 August 2017

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 231

Pesticide applicators questionnaire content validation: A fuzzy delphi method

Table III: The summary of All Three Pre-requisites Post Fuzzy Delphi Analysis (Chemical)
Construct/Items Average Threshold Value Percentage of Average of Ranking Verdict
Likert Score (d) ≤ 0.2 Experts’ Fuzzy
Consensus (%) Numbers
Chemical-Knowledge 0.00 Acceptable
CK-1 5 94 0.788 1 Retained
CK-2 5 88 0.696 6 Retained
CK-3 5 94 0.788 1 Retained
CK-4 5 88 0.750 3 Retained
CK-5 5 88 0.788 1 Retained
CK-6 5 81 0.733 4 Retained
CK-7 5 88 0.775 2 Retained
CK-8 5 88 0.788 1 Retained
CK-9 5 94 0.713 5 Retained
CK-10 5 94 0.775 2 Retained
Chemical-Attitude 0.01 Acceptable
CA-1 5 94 0.729 4 Retained
CA-2 5 88 0.742 3 Retained
CA-3 5 88 0.704 6 Retained
CA-4 5 88 0.717 5 Retained
CA-5 5 94 0.788 1 Retained
CA-6 5 94 0.717 5 Retained
CA-7 5 88 0.775 2 Retained
CA-8 4 81 0.692 7 Retained
CA-9 5 88 0.717 5 Retained
CA-10 5 94 0.729 4 Retained
Chemical-Practice 0.01 Acceptable
CP-1 4 31* 0.600 6 Discarded
CP-2 4 38* 0.575 8 Discarded
CP-3 4 94 0.725 3 Discarded
CP-4 4 75 0.750 2 Discarded
CP-5 5 94 0.692 4 Retained
CP-6 4 94 0.679 5 Retained
CP-7 4 94 0.725 3 Discarded
CP-8 4 88 0.775 1 Discarded
CP-9 5 13* 0.592 7 Retained
CP-10 5 13* 0.592 7 Retained
* Item with Experts’ consensus ≤ 75% and lowest ranking within their construct

summary of the study flow process has been illustrated in terms of the structure, position and wordings were done based
Figure 4. on the comments by the experts. These were some minor
changes and it didn’t alter the objective and nature of the

RESULTS
items. As a final draft, a total of six constructs with 53 items
were finalised as the result of this Fuzzy Delphi analysis.
A 100% response rate was obtained from all the sixteen

DISCUSSION
experts. All the items within the six constructs had scored
average Likert scoring of four to five, which was in the scale
of agree to highly agree. These scores were converted into This article demonstrated the study objective which was the
fuzzy numbers. Post FDM analysis, the first prerequisite was content validation of pesticide applicators questionnaire by
fulfilled whereby all the six constructs had threshold value (d) obtaining the experts’ consensus on suitability of the pre-
≤ 0.2. For the second prerequisite, three items (21%) from selected items on the questionnaire and using FDM to
noise-attitude construct and four items (40%) from chemical- ultimately remove the unfit items. This study found that the
practice construct had expert consensus lesser than 75%, average Likert scale scoring by the experts for all the items
which giving rise to about 12% from the total items in the are from agreeable to highly agreeable range, which means
questionnaire. The third prerequisite was used to rank the all 60 items can be accepted. However, post FDM analysis,
items within the constructs by calculating the average fuzzy only 53 items were fulfilled all the pre-requisites. About 12%
numbers. The seven items which did not fulfill the second of the items didn’t match the terms, hence those items were
prerequisite similarly had lower ranks during the analysis. regarded as failure to achieve consensus from the expert
The whole findings were summarised in the Table II and panel and removed. This 12% is the fuzziness or uncertainty
Table III. among the expert panel which was not detected by the usual
Likert Scale scoring system. Every expert will have their own
Those seven items were discarded and the remaining which uncertainty towards certain variable, which often regarded as
fulfilled the pre-requisites was retained for the final draft for the “grey area”. The use of FDM is to deal with those “grey
content validation process. Apart from discarding items area”, ensuring a qualified analysis outcome. Furthermore,
based on these prerequisites, little modification of items in this method catered all the experts’ opinion, considering

Med J Malaysia Vol 72 No 4 August 2017 231

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 232

Original Article

Fig. 1: Triangular Fuzzy Number and Defuzzification Process.

Fig. 2: Method to obtain Threshold value (d) for each item.

232 Med J Malaysia Vol 72 No 4 August 2017

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 233

Pesticide applicators questionnaire content validation: A fuzzy delphi method

Fig. 3: Construct and items acceptability based on experts' consensus.

some expert are more experienced, some are more to a construct validation process. Most importantly, this
knowledgeable, some with relevant skills and some has the method gives a proper quantitative approach to usual group
policy making authority in the field. This variety of opinions discussions or meetings which are in a qualitative manner.
is merged together to support each other’s deficiency to derive This questionnaire can be considered as accepted by the
at the desirable outcome. Moreover, the final draft of the experts without any prejudice and it can be used for the
questions was arranged based on priority ranking derived by targeted population after confirmatory validation process.
the analysis. On the positive note, although the items were
picked from variety of literature which was very unusual However, there are some limitations with this method,
compared to the traditional practice of selecting a whereby, the researcher or a person who is conducting this
questionnaire, the difference between the initial selection of FDM should have some pre-existing background knowledge
items and the level of experts’ opinion was very minimal regarding the subject, whereby he/she must be an expert too.
(12%). This could be possibly due to majority of the items are Moreover, FDM requires existing kinds of literature or matter,
originally from the local language and the remaining items to begin with, and this method is not suitable for developing
were hand-picked, translated and modified by the author brand new items. On the other note, this study required
who is equally experienced and knowledgeable in the similar constant reminder to the experts to give their response. This
field. is mainly due to limited time factor and this might lead to the
emotional bias among the experts. In addition to that, the
Generally, an indoor meetings or workshops will be selection of the expert was by purposive sampling method
conducted to gather the experts under a roof in order to get based on their willingness and availability. A probable
their consensus. This involves tedious process, starting from sampling method among the experts and more time frame
the calling letter, arranging the venue, travelling expenses, would have been yielded a different result.
refreshment beverages and obviously plenty of time will be
spent. The main significant advantage of this study was, it As a recommendation, FDM should be widely used in medical
was conducted in a very short span of time, with zero costing related studies, to get expert’s opinion and consensus
involved. It was also a hustle-free job for the experts as well. especially in developing a protocol or guidelines related to
The experts’ responses were gathered via emails and medical practices. Although limited, there are some studies
messages at their convenience. This method will certainly which use this method for medically related researches. It was
reduce the risk of bias by ensuring anonymity and welcoming used in one of the studies to find consensus for Asthma
the opinion of atypical views among the experts and the management guidelines.22 Another study in Mexico which
responses are totally independent without the fear of used this technique to determine the socio-ecological factors
judgemental by others which usually present in any routine that influence adherence to mammography screening.3
group discussions or meetings.21 However, locally in Malaysia, this method is yet to be
introduced in the field of medicine. Furthermore, it is hoped
Pertaining to this study, it introduces that FDM can be used that this study can be beneficial as a guidance for any future
to get expert’s opinion and consensus in order to achieve a medical or health related research which intends to use FDM
decision. This method can be used as a pre-construct for their studies.
validation tool to select the suitable items before subjecting it

Med J Malaysia Vol 72 No 4 August 2017 233

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 234

Original Article

Fig. 4: Summary of content validation using Fuzzy Delphi Method.

CONCLUSION to Associate Professor Dr. Retneswari Masilamani and

Post FDM analysis, the experts’ consensus on suitability of the Associate Professor Dr. Razman Mohd Rus for their inputs in
pre-selected items on the questionnaire set were obtained, forming the items. Last but not least, we express our gratitude
hence it is now ready for further construct validation process. to the Department of Community Health, PPUKM and to
those who had extended their help in contributing to this

ACKNOWLEDGEMENT
manuscript.

REFERENCES
We would like to thank the Director General of Health
Malaysia for his permission to publish this article. This study
is part of doctorate research which is supported by the Dana 1. Hsu Y-L, Lee C-H, Kreng VB. The application of Fuzzy Delphi Method and
Fuzzy AHP in lubricant regenerative technology selection. Expert Systems
Fundamental PPUKM (Project code: FF-2016-291) and ethical
with Applications. 2010;37(1):419-25.
approval from the Medical Research and Ethics Committee 2. Adler M, Ziglio E. Gazing into the oracle: The Delphi method and its
(MREC), Ministry of Health (NMRR-16-660-30666-IIR). The application to social policy and public health: Jessica Kingsley Publishers;
research team would like to thank the sixteen experts for 1996.
their contribution to this study. Our sincere acknowledgment

234 Med J Malaysia Vol 72 No 4 August 2017

5-Pesticide00027_3-PRIMARY.qxd 8/29/17 11:25 AM Page 235

Pesticide applicators questionnaire content validation: A fuzzy delphi method

3. Sanchez-Lezama AP, Cavazos-Arroyo J, Albavera-Hernandez C. Applying 13. Razman M, Naing L, Aziah D, Kamarul I. Validation of Noise Induced
the Fuzzy Delphi Method for determining socio-ecological factors that Hearing Loss Questionnaire Among Malay Sawmill Workers in Kelantan
influence adherence to mammography screening in rural areas of Mexico. Malaysia. The International Medical Journal of Malaysia. 2010;9(2).
Cadernos de saúde pública. 2014;30(2):245-58. 14. Reilly MJ, Rosenman KD, Kalinowski DJ. Occupational noise-induced
4. Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate hearing loss surveillance in Michigan. Journal of occupational and
research. Journal of information technology education. 2007;6:1. environmental medicine. 1998;40(8):667-74.
5. Murray TJ, Pipino LL, van Gigch JP. A pilot study of fuzzy set modification 15. Harkness JA, Schoua-Glusberg A. Questionnaires in translation. DEU;
of Delphi. Human Systems Management. 1985;5(1):76-80. 1998.
6. Alias N, Rosman F, Rahman MNA, Dewitt D. The potential of video game 16. Baker J, Lovell K, Harris N. How expert are the experts? An exploration of
in Malay language learning for foreign students in a public higher the concept of ‘expert’within Delphi panel techniques. Nurse researcher.
education institution. Procedia-Social and Behavioral Sciences. 2006;14(1):59-70.
2015;176:1020-7. 17. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods:
7. Kaufman A, Gupta M. Introduction to Fuzzy Arithmetic: theory and characteristics and guidelines for use. American journal of public health.
Application, van no strand Reinhold. New York. 1988. 1984;74(9):979-83.
8. Mohd Ridhuan Mohd Jamil SS, Zaharah Hussin, Nurulrabihah Mat Noh, 18. Jones H, Twiss B. Forecasting technology for planning decisions: Springer;
Ahmad Arifin Sapar. Pengenalan Asas Kaedah Fuzzy Delphi Dalam 1978.
Penyelidikan Rekabentuk Pembangunan. Bandar Baru Bangi: Minda 19. Cheng C-H, Lin Y. Evaluating the best main battle tank using fuzzy
Intelek Agency; 2014. decision theory with linguistic criteria evaluation. European Journal of
9. Noh NM, Razak SHA, Alias N, Siraj S, Jamil MRM, Hussin Z. Usage of Operational Research. 2002;142(1):174-86.
Facebook: The future impact of curriculum implementation on students in 20. Chu H-C, Hwang G-J. A Delphi-based approach to developing expert
Malaysia. Procedia-Social and Behavioral Sciences. 2013;103:1261-70. systems with the cooperation of multiple experts. Expert systems with
10. Geer LA, Curbow BA, Anna DH, Lees PS, Buckley TJ. Development of a applications. 2008;34(4):2826-40.
questionnaire to assess worker knowledge, attitudes and perceptions 21. Horner K, Islam M, Flygare L, Tsiklakis K, Whaites E. Basic principles for
underlying dermal exposure. Scandinavian journal of work, environment use of dental cone beam computed tomography: consensus guidelines of
& health. 2006:209-18. the European Academy of Dental and Maxillofacial Radiology.
11. Masilamani R, Rasib A, Darus A, Ting AS. Noise-induced hearing loss and Dentomaxillofacial Radiology. 2014.
associated factors among vector control workers in a Malaysian State. 22. Paris MB. The asthma consensus in the light of the Delphi method.
Asia-Pacific Journal of Public Health. 2014;26(6):642-50. Allergologia et immunopathologia. 2010;38(6):293-4.
12. Nor Saleha IT, Noor Hassim I. A study on compliance to hearing
conservation programme among industries in Negeri Sembilan, Malaysia.
Industrial health. 2006;44(4):584-91.

Med J Malaysia Vol 72 No 4 August 2017 235