COMMON DATA COLLECTION STRATEGIES EFFECTIVE IN QUALITATIVE STUDIES USING ACTION RESEARCH IN TECHNICAL/OPERATIONAL TRAINING PROGRAMS

by: Robin D. McClure (September 2002)

Abstract Action research, in simple terms, is an approach that teachers and facilitators use to critically evaluate problems encountered on a regular basis in the classroom. In order to conduct action research, one must determine the best method for data collection. For purposes of this paper, interviewing, observation, and questionnaires have been selected because they provide the most accurate information regarding the effectiveness of technical and operational training in the researchers current environment. The purpose of an interview is to understand the experiences of others better to get their individual story. Observation, an intuitive process, allows individuals to collect information about others by viewing their actions and behaviors in their natural surroundings is also discussed. Questionnaires are used to measure attitudes, facts, or values held by individuals. Sample documents demonstrating the different research techniques discussed are located in the appendices. The paper concludes by evaluating the effectiveness of the different techniques by reviewing how the sample documents were used.

Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

Question Identify common data collection strategies effective in qualitative studies using action research in a technical or operational training program. Evaluate how each strategy could play a key role in a research study designed to evaluate the effectiveness of training. Action research, in simple terms, is an approach that teachers and facilitators employ to critically evaluate problems encountered in the classroom on a regular basis. The goal of action research is to discover techniques to improve current practices through reflection and metaanalysis. This systematic research approach, which is not limited to any specific subject matter, involves collaborative efforts with other teachers, facilitators, students, and administrators. The purpose of an action research study is not to publish reports or validate results through independent research rather to apply and validate the findings through practice and practical application. Grounded in curriculum theory, action research involves three primary steps (1) defining a problem, (2) developing an action plan that includes testing a hypothesis and monitoring applied changes, and (3) reflecting on and sharing any findings or results (Arhar, Holly, & Kasten, 2001; Gay & Airasian, 1996; Leedy & Ormrod, 2001; McKernan, 1996). As observed correlations among human behavior and environmental stimuli increase, so does the value of self-evaluation and self-critical inquiry. For example, students familiar with their environment are more apt to behave and respond normally, whereas students tested in a foreign, sterile environment tend to behave and respond abnormally. Consequently, conducting studies in the students naturalistic environment is more valuable because the focus remains on heuristics, realism, and relevance even though some of the predictability is lost (McKernan, 1996, p. 7). In many respects, action research is equivalent to problem solving. Kurt Lewin, a social psychologist, coined the term action research and was one of the first researchers to develop a comprehensive model. Lewins model explained that the decision-making process involves a series of action steps including analysis, reconnaissance, problem reconceptulization, planning, implementation of social action, and evaluation. Lewins emphasis on planning, fact-finding, execution, and analysis for all action research studies has helped practitioners become more objective in examining their personal biases (Arhar, Holly, & Kasten, 2001; McKernan, 1996). This paper reviews three data collection methods (1) interviewing, (2) observation, and (3) questionnaires. Descriptions of various techniques used for data collection provide readers with practical approaches to consider when developing technical and operational training materials. Interviewing Interviews are basic fact-finding interactions where one individual asks questions while another responds. By conducting interviews, researchers obtain a clearer understanding of an individuals background and experience. Knowledge of this experience helps the researcher better understand the context for an individuals behavior and decision-making rationale (Seidman, 1998). Depending upon the type of information sought, interviews can be informal or formal. Informal interviews elicit purposeful conversation by soliciting responses that may lead the interview down an unplanned or unexpected path. Compassion, empathy, flexibility, and
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

reciprocity from the interviewer to the interviewee are necessary in order for informal interviews to succeed (Arhar, Holly, & Kasten, 2001; Patton, 1987; Seidman, 1998). According to Patton (1987), interviewing becomes an art and science requiring skill, sensitivity, concentration, interpersonal understanding, insight, mental acuity, and discipline (p. 108). Using informal interviews, trainers can gather data to enhance the learning experience of the class. By interviewing the class participants before conducting the class, the facilitator is able to customize the class to better meet the needs of the participants. Formal (or structured) interviews, conversely, involve a great deal of planning, scheduling, and preparing. Strict adherence to general rules such as maintaining objectivity, withholding opinions, and asking the same prepared questions consistently is required in formal interviewing. Because interviewers ask the same questions in the same order, coding of responses often occurs. Unlike informal interviews where interviewers work hard to establish rapport, formal interviewers may not interpret or modify questions, nor can they respond to the interviewee in any way. Formal interviews often incorporate tools such as questionnaires or attitude scales to collect the information needed. As a result, oral or written results from interviews are acceptable (Arhar, Holly, & Kasten, 2001; Patton, 1987). Focused group and critical-incident interviews are two other techniques used to collect data. Focused group interviews, conducted with three to six people, are small group discussions that concentrate on specific topics. Small groups encourage collaboration among individuals, create memorable learning experiences, increase learner participation, and limit anxiety. Criticalincident interviews ask participants to recall a period in time that had particular meaning and resulted in a memorable change or experience. Questions asked during the interview flush out procedural, emotional, and factual details surrounding the event. Unfortunately, individuals may reveal information that they believe is socially acceptable rather than a true representation of their experience (Arhar, Holly, & Kasten, 2001; Patton, 1987). In-depth interviews commonly use a three-interview series approach. During the first interview, interviewees provide a focused life history where they reconstruct how their experiences during their formative years have influenced a specific event. At the second interview, interviewees concentrate on specific details that surround the researched area of study. These details may involve day-to-day operations, specific procedures, or recent related experiences. In the third interview, the interviewees reflect on the intellectual and emotional connections of their experience that have shaped their lives (Seidman, 1998). Two common concerns that revolve around interviewing are (1) how long should an interview last and (2) how much time should there be between the interviews when the threeinterview format is used. Although the length of the interview may vary depending on the interviewee, 90-minute session generally prove to be sufficiently long enough to collect an adequate amount of information while still maintaining the attention and motivation of the interviewee. Although the time between interviews depends upon the schedule of both the interviewee and interviewer, intervals of three days to a week tend to work best because continuity of content and rapport built between the two participating parties can be maintained. Expansive gaps between interviews may cause information to lose validity and reliability because the connection and the context of the responses may be lost (Patton, 1987; Seidman, 1998). The most important aspect of an interviewers job is listening. Zeno of Citium in 300 B.C. gave some excellent advice about this subject: The reason why we have two ears and only one mouth is that we may listen the more and talk the less (Patton, 1987, p. 108). Interviewers
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

should focus on: (1) what the participant is saying, (2) what the participant is not saying, and (3) what status, energy level, and nonverbal cues the participant is communicating. By actively listening, an interviewer can ascertain that the interviewee understands the question and its intent as well as follow up on any outstanding issues not addressed during the initial response. To glean clarification or additional information about a topic, interviewers should ask questions that reflect experience, behavior, opinion, belief, feelings, knowledge, background, demographics, period, or sequence. Interviewers should be careful not to ask inappropriate or leading questions. When relaying a story, ask for concrete details rather than relying on the interviewees memory to reconstruct the process or event (Patton, 1987; Seidman, 1998). Interviewers should address administrative responsibilities before, during, and after the interview process. Prior to beginning an interview, the interviewer should (1) contact those that will be participating in the process, (2) schedule meeting times, (3) make necessary arrangements regarding the location, (4) reserve any equipment that may be needed, and (5) gain informed written consent from each participant. During the interview, it is important that all equipment is working properly, notes and background materials are available, and the participants are comfortable. After completing the interview, the interviewer needs to have a solid plan in place for managing, storing, securing, transcribing, analyzing, interpreting, and disseminating the data. When transcribing interviews, transcribers should always record the data word for word indicating any pauses, coughs, laughs, or interruptions. This consistency provides the reader with a truer picture of the interview, stimulates the interviewers memory to any unusual nuances, and serves as an accurate record if later claims of inaccuracy or inappropriateness occur. The actual process of transcription is very time consuming. Generally, four to six hours are necessary to transcribe a 90-minute tape accurately. Interviewers search for common themes found in the transcribed data by marking interesting passages from the transcripts, labeling them, and putting them into appropriate files (Seidman, 1998, p.107). After coding and classifying the data, the interviewer reviews the new files and continues to identify commonalities and interests. Next, the interviewer interprets the data collected by asking key questions such as (1) Is there a connection among the experiences of the participants? (2) If a connection existed between the experiences of the participants, what was it? (3) Is there an explanation for the connection? (4) Were there any surprises in the information collected? (5) Was the original hypothesis proved or disproved? (6) Was the information consistent with the literature? Using formal interview questions can help trainers compile a user profile for a variety software programs available within the organization. Questions regarding specific training needs can also asked. Interviews that identify roles and responsibilities while better defining the correlation among the percentage of time programs are used, document type, and workers position can also be beneficial. For example, if a large percentage of the interviewees indicate that classroom training is ineffective and unproductive, trainers may develop quick reference cards containing basic procedures and short cuts for each program to replace the classroom training. For students interested in learning advanced computer skills, the training department may purchase computerbased training programs. In-depth interviewing allows the interviewer to better understand the details associated with the lives of the individuals interviewed. This knowledge can lead to increased understanding of how experience relates to individual interactions with others resulting in the discovery of similarities amongst everyone.
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

Observation Observation is, in some part, an intuitive process that allows individuals to collect information about others by viewing their actions and behaviors in their natural surroundings. The role of the observer can vary depending upon the level of involvement desired (e.g., remain as part of the background, no interaction; or, actively participate with the group by asking questions and responding). Tools used to collect information vary depending upon the type of data gathered. Informal observational tools may include writing field notes, making entries into a log, or keeping a journal. When observation is more formal, tools such as audiotapes, videotapes, checklists, and rating scales may be used (Arhar, Holly, & Kasten, 2001). Field notes are direct, written observations (dialogue, impressions, or feelings) about what is occurring that contain rich, detailed information that creates a basis for the study. Field notes can be formatted as (1) a running record to track regularly scheduled occurrences, (2) a time log to record events at designated interval, (3) an event log indicating things such as participation, (4) a critical incident log to identify pivotal events, or (5) an anecdotal record to track growth over time. Establishing predetermined abbreviations and format before beginning the note taking process saves time (Arhar, Holly, & Kasten, 2001). Additional methods used to record observed information include logs, diaries, and journals. Logs generally record entry times, identify individuals involved, and detail events targeting information identified for future study. Diaries are usually unedited accounts of events that also include personal thoughts and feelings. Acceptable diary formats include written and oral (video or audio) accounts of events. Journals contain information that relates to specific events, answers questions, or analyzes information. Generally, journals serve as an event reminder during the analytical and evaluative phases of research. Checklists are structured observation tools used when specific, predictable results are expected. Items on the checklist help the observer remain focused on key points by asking for the frequency, absence, or presence of specific events. Category, numeric, graphic, and pictorial rating scales help the observer record information. The frequent use of Likert-type scales that range from three to seven points and offer a continuum of options help decrease bias as does the use of multiple raters (Aiken, 1996; Arhar, Holly, & Kasten, 2001). Trainers and facilitators use checklists to track development progress, follow critical path objectives, and monitor all aspects of the project. Checklists address key areas and help focus on desired outcomes. By providing a common format for everyone to follow, there are fewer questions and individuals working in remote locations feel more involved and comfortable because they are aware of the process and expectations. Trained and skilled observers are descriptive writers who separate important facts from trivial information by requiring physical, psychological, intellectual, and material preparation followed by descriptive, accurate, factual, and thorough reporting (Patton, 1987). Direct observation provides an inductive advantage to the observer limiting personal perceptions and biases because the event is experienced first-hand. Another advantage to direct observation is that the observer identifies routine, unconscious things, often overlooked by the participants. Disclosure of facts that individuals generally do not share during an interview may also surface. Before beginning research, making a decision regarding the active or passive involvement of the observer is necessary. Participant observation simultaneously combines document analysis, interviewing of respondents and informants, direct participation and
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

observation, and introspection (Denzin, 1978, p. 183; Patton, 1987, p. 75). This method is often chosen when the observer feels it will be most beneficial to experience the event as well as record it. However, depending upon the population observed, full participation may not be feasible. Another, often controversial, decision is whether the observation should be covert or overt. Proponents of covert observations argue that (1) people frequently behave differently when scrutinized, and (2) individuals are generally cooperative in nature, therefore, they would not mind covert observation. Opponents to covert observations argue that it is morally and ethically wrong to not inform individuals that they are part of an active research study. However, when the researcher selects overt observation, the range of information disclosed to the participants regarding the research problem should be determined (Patton, 1987). How long the observation of the individual occurs is another decision the researcher makes. Many sociological and anthropological studies involve years of observation. Program evaluators, on the other hand, generally observe for shorter periods. However, this is often determined only after identifying the scope and desired results of the study. Informal Observation helps trainers and facilitators to realize how they come across to the participants. Awareness of participant perceptions allows trainers and facilitators to develop an action plan that focuses on skills needing improvement. Trainer and facilitator improvement using observation may incorporate several processes discussed below. (1) Observe other trainers. (2) Write down observed rapport and behavior between the students and trainer or facilitator. (3) Identify methods used to establish authority and respect. (4) Note any behaviors believed to enhance or detract from the learning experience. (5) Videotape a training session to observe ones own performance. Questionnaires Action research may employ the use of questionnaires when it is impossible to interview every respondent. Questionnaires generally consist of open- or closed-ended questions or items that measure facts, attitudes, or values. For example, an action research study might try to collect information about a particular teachers presentation style after every class for a month. Because this involves the comments of 150 students regarding 20 lectures, it would be impossible to interview each one every time. Consequently, a standard questionnaire would be developed to gather the desired information. Both qualitative and quantitative research use questionnaires to collect data. Closed-ended questions force a response, score quickly, and are easy to evaluate. To ensure reliability, inventories often restate the question or item several times. Open-ended questions allow the participant to provide a more complete or comprehensive response. Although open-ended responses are difficult to analyze, they often provide specific and meaningful information (Arhar, Holly, & Kasten, 2001; Patten, 1998). Simultaneous administration of questionnaires to multiple people at various locations is one of many positive advantages associated with this methodology. Anonymity is another benefit of questionnaires that makes them an effective tool for collecting data on sensitive or illegal activities. Since duplication and postage are generally the only costs associated with questionnaires, data collection is economical (Patten, 1998; Salkind, 1991). Questionnaires have several disadvantages including low response rates, especially when part of mass mailings. Questionnaires generally contain objective items written as multiple choice, fill in the blank, or short answer items. Although the aforementioned item types are easy
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

to score, the responses only provide a very limited picture of the situation to the researcher. Another problem associated with questionnaires is the frequency that respondents select socially desirable rather than truthful responses. Well-written items should be neutral to eliminate bias. To ensure validity researchers should follow the guidelines listed below. (1) Ask questions based on short-term rather than long-term recollection due to decreased accuracy. (2) Avoid using negatives because they often cause confusion. (3) Use an underlined or boldface font when the use of negatives is unavoidable to draw attention to them. (4) Ask more than one question for one response. (5) Separate items that require multiple answers (Haladyna, 1999; Patten, 1998; Salkind, 1991; Schoer, 1970). Other considerations for writing items include (1) spelling out acronyms and defining difficult terms, (2) using other to exhaust choices when asking a participant to check all answers that apply, (3) avoiding the use of do not know as an option, (4) asking specifically for the information wanted, and (5) limiting the use of open-ended and ranked items (Patten, 1998; Schoer, 1970). Many questionnaires use the Likert scale to measure standard responses. As discussed earlier, Likert scales range from three to seven levels such as 1 (very strongly agree), 2 (strongly agree), 3 (agree), 4 (neutral), 5 (disagree), 6 (strongly disagree), and 7 (very strongly disagree). Once the measurement scale has been determined, the researcher should create evaluative categories as well as the individual items. Using declarative statements and creating hypothetical situations increases the possibility of collecting the desired data (Arhar, Holly, & Kasten, 2001; Patten, 1998). The use of 360-degree feedback surveys is an example of one type of questionnaire used in training. These questionnaires ask direct reports, colleagues, and superiors to rate an individuals performance. Based on the compilation of results, individuals develop an action plan to improve their skills in the areas measured. The anonymity associated with completing this type of survey provides employees the opportunity to express their perspective without fear of reprisal. Consequently, individuals are able to identify needs quicker and find ways to improve issues with communication and teamwork more effectively. Before distributing any questionnaire, researchers should have several other people review the items to identify any confusion associated with verbiage and evaluate the appropriateness of the scales used. Patten (1998), suggests having approximately ten individuals, similar to the target population, answer the items and write down any concerns, frustrations, problems, or errors they have. Next, the researcher should conduct an item analysis with twentyfive or more randomly selected people. Once this second group has completed the questionnaire, number of respondents who selected specific choices should be tallied. Based on the results, it may be prudent to revise or even eliminate certain items. The second phase of the item analysis should compare the responses of high and low groups on individual items. To do this, the researcher assigns each response level a weighted score based on points to identify the emergence of patterns after totaling responses. The final step is to complete an analysis on identified patterns to determine the need for changes (Aiken, 1996; Patten, 1998; Schoer, 1970). Once the questionnaire is ready for distribution, the research should write an accompanying cover letter. The cover letter should introduce the questionnaire by describing the purpose of the questionnaire, requesting a response, and identifying the date that the response is due (if applicable). Assurances of confidentiality, approximate time it will take to complete the questionnaire, where it needs to be returned, and appreciation for the individuals time and consideration should also be expressed in the cover letter (Patten, 1998; Salkind, 1991).
Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

Concerning the questionnaires format, detailed instructions should be listed at the beginning followed by subsequent instructions to communicate any changes. Categorizing items by content and including subheadings where appropriate helps respondent complete the questionnaire quickly and accurately. It is also important to match response types. Using different value scales for items or reversing the value of the scale occasionally (Very Strongly Agree for the first two items and Very Strongly Disagree for the next two) confuses the participants (Patten, 1998; Salkind, 1991). Consideration to the administration and distribution of questionnaires is also necessary. A set of written instructions should be provided to the administrator to read when they are administering the questionnaire to the participants. This helps ensure receipt of a consistent message. When personal administration is not feasible due to dispersed populations located throughout large geographical regions, self-administering questionnaire may be more practical. If mail is the method of distribution, researchers should package the questionnaire in a manner that does not resemble junk mail. A follow-up letter that acknowledges the busy lifestyle of questionnaire recipients yet encourages them to complete the survey is recommended as well (Patten, 1998; Salkind, 1991). Without a cover letter to introduce a questionnaire to the respondents, questions about the purpose and expectations regarding the questionnaire are ubiquitous. Instructions are essential, without them, respondent rarely complete questionnaires correctly. A needs analysis is one example of training questionnaire. This document ensures that the training material developed meets the needs and expectations of the audience. To increase acceptance, researchers should solicit input from individuals from multiple locations. Summary This paper identified interviews, observation, and questionnaires as three effective strategies to use when appraising technical and operational training programs. Proper application and evaluation of information collected using these methods can result in effective evaluation of technical and operation training programs. All of the methods presented in this paper serve as effective training tools and quintessentially represent action research. Analysis, reconnaissance, problem reconceptulization, planning, implementation of social action, and evaluation of training and design issues helps ensure that the efforts afforded by teachers and trainers are as productive as possible. Continuous learning and improvement through meta-analysis based around the concepts of action research ensures that students receive positive learning experiences.

Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

References Aiken, L. R. (1996). Rating scales & checklists: Evaluating behavior, personality, and attitude. New York: John Wiley & Sons. Arhar, J. M., Holly, M. L., & Kasten, W. C. (2001). Action research for teachers: Traveling the yellow brick road. Upper Saddle River, NJ: Merrill/Prentice Hall Booth, W. C., Colomb, G. G., & Williams, J. M. (1995). The craft of research. Chicago, IL: The University of Chicago Press. Borg, W. R. (1987). Applying education research: A practical guide for teachers (2nd ed.). White Plains, NY: Longman. Denzin, N. K. (1978). The research act. New York: McGraw-Hill. Gay, L. R., & Airasian, P. (1996). Educational research: Competencies for analysis and application (6th ed.). Upper Saddle River, NJ: Merrill/Prentice Hall. Hadley, R. G. (1995). Counseling research and program evaluation. Pacific Grove, CA: Brooks/Cole Publishing Company. Hill, B. C., & Ruptic, C. (1994). Practical aspects of authentic assessment: putting the pieces together. Norwood, MA: Christopher-Gordon Publishers. Keeves, J. P., & Lakomski, G. (Eds.). (1999). Issues in educational research. Oxford: Pergamon. Krathwohl, D. R. (1993). Methods of educational and social science research: An integrated approach. New York: Longman. LeCompte, M. D., Millroy, W. L., & Preissle, J. (Eds.). (1992). The handbook of qualitative research in education. San Diego, CA: Academic Press. Leedy, P. D., & Ormrod, J. E. (2001). Practical research: Planning and design (7th ed.). Upper Saddle River, NJ: Merrill/Prentice Hall. Macintyre, C. (1988). The art of action research in the classroom. London: David Fulton Publishers. McKernan, J. (1996). Curriculum action research: A handbook of methods and resources for the reflective practitioner (2nd ed.). New York: NY: St. Martins Press. Oja, S. N., & Smulyan, L. (1989). Collaborative action research: A developmental approach. Philadelphia, PA: The Falmer Press.

Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1

Patten, M. L. (1998). Questionnaire research: A practical guide. Los Angeles, CA: Pyrczak Publishing. Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage Publications. Salkind, N. J. (1991). Exploring research. New York, NY: Macmillan Publishing Company. Schoer, L. A. (1970). Test construction: A programmed guide. Boston, MA: Allyn and Bacon. Schuler, H., Farr, J. L., & Smith, M. (Eds.). (1993). Personnel selection and assessment: Individual and organizational perspectives. Hillsdale, NJ: Lawrence Erlbaum Associates. Seidman, I. (1998). Interviewing as qualitative research: A guide for researchers in education and the social sciences (2nd ed.). New York, NY: Teachers College Press.

Common Data Collection Strategies Effective in Qualitative Studies Using Action Research in Technical/Operational Training Programs - 1