Test Review: The Intercultural Development Inventory manual

Journal of Psychoeducational Assessment http://jpa.sagepub.com/ Test Review: The Intercultural Development Inventory manual Jingzhu Zhang Journal of Psychoeducational Assessment 2014 32: 178 DOI: 10.1177/0734282913505075 The online version of this article can be found at: http://jpa.sagepub.com/content/32/2/178 Published by: http://www.sagepublications.com Additional services and information for Journal of Psychoeducational Assessment can be found at: Email Alerts: http://jpa.sagepub.com/cgi/alerts Subscriptions: http://jpa.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://jpa.sagepub.com/content/32/2/178.refs.html >> Version of Record - Feb 20, 2014 What is This? Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 505075 JPA32210.1177/0734282913505075Journal of Psychoeducational AssessmentTest Review research-article2013 Test Review Test Review Journal of Psychoeducational Assessment 2014, Vol. 32(2) 178–183 © 2013 SAGE Publications Reprints and permissions: sagepub.com/journalsPermissions.nav jpa.sagepub.com M. R. Hammer The Intercultural Development Inventory manual (Vol. 3). Ocean Pines, MD: IDI, LLC, 2007. Reviewed by: Jingzhu Zhang, College of William and Mary, Williamsburg, VA, USA DOI: 10.1177/0734282913505075 Test Description Introduction The Intercultural Development Inventory v.3 (IDI; Hammer, 2011) was constructed to measure individuals’ orientations toward cultural differences. The theoretical framework for this instrument was Bennett’s (1993) developmental model of intercultural sensitivity (DMIS). The DMIS identifies three ethnocentric orientations (i.e., denial, defense, minimization) and three ethnorelative orientations (i.e., acceptance, adaptation, integration; Bennett, 1993). Ethnocentric orientation is defined as an attitude that regards one’s own culture as the center of the world. The first stage of ethnocentric thought is denial, in which people who are isolated from other cultures consciously ignore those other cultures’ values. Defense is the next stage of ethnocentrism, in which one begins to acknowledge the existence of other cultures but believes that one’s own culture is superior to other cultures. The last stage of ethnocentric orientations is minimization. People in the minimization stage focus only on personal similarities (e.g., physical, biological, philosophical), but do not legitimize other societies’ broad cultural frameworks. The second HALF of the DMIS continuum includes three stages of ethnorelative orientations, each of which recognizes to some extent the influence of other cultures on one’s own culture. The first ethnorelative orientation is acceptance in which people accept the existence of various cultures in the world and respect their values. The second stage is adaptation, in which people adjust their attitudes and behaviors to accommodate and exist within a specific cultural context. According to Bennett (1993), to achieve this latter level of ethnorelative orientation, pure exposure to foreign culture is insufficient. More critical to adaptation, people must consciously construct their cultural experiences and views to adapt fully. The last stage of ethnorelative orientation is integration, in which one is comfortable switching identities within one culture versus another. Bennett (1993) emphasized that integration is not necessarily a better orientation than adaptation; it is simply an orientation that some people assume after many years of experience in or interacting with another culture, or when a person may have inherited and been raised within multiple cultures. The IDI was based on the DMIS model and is purported to be appropriate for any cultural group (Hammer, 2011). As such, the IDI includes seven subscales, each measuring one of the six dimensions of the DMIS, plus an additional scale (i.e., Cultural Disengagement [CD]). Hammer (2011) pointed out that CD is not a dimension along the DMIS orientation continuum; rather it is a separate dimension that measures how people relate to their own cultural group and other cultures. The entire IDI is composed of 50 items, with10 additional demographic items. The scale is self-report and can be administered either in paper and pencil format or in an online administration in 15 languages (i.e., Arabic, Bahasa Indonesian, Chinese, Czech, English, Finnish, French, German, Italian, Japanese, Korean, Norwegian, Portuguese, Russian, and Spanish). The IDI is Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 179 Test Review appropriate for individuals who are ages 15 or older and who are reading at least at a 10th-grade level (Hammer, 2011). Also, the instrument can be administered in either individual or group settings. One significant drawback of the instrument is that to use the IDI, examiners are required to attend an IDI Qualifying Seminar and consent to a licensing agreement. The training seminar requires a 2.5-day commitment, and as of 2013, the training costs US$1,800. Moreover, participants must complete the entire seminar to receive their IDI Qualified Administrator Certification. Purpose According to the IDI official website, the instrument can be used for a wide variety of purposes, including individual assessment, group analysis, and organization-wide needs assessment, program evaluation, and research. The IDI has been used for assessing intercultural sensitivity in short-term (Anderson, Lawton, Rexeisen, & Hubbard, 2006) and year-long (Pedersen, 2010) study abroad programs. The IDI was also used in assessing the long-term impact of study abroad programs on intercultural development in longitudinal studies (Hansel, 2009; Meyer-Lee & Evans, 2006; Rexeisen, Anderson, Lawton, & Hubbard, 2008). Scoring Administration of the IDI generates a diagnostic profile, which represents the following six cultural orientations: (a) Perceived Orientation (PO), computed using an unweighted formula, reflects one’s perception of intercultural development; (b) Developmental Orientation (DO), using a weighted formula, that is defined by the IDI assessment; (c) Orientation Gap (OG) is the difference between clients’ perception and the assessment, and is calculated by subtracting PO from DO; (d) Trailing Orientation (TO) presents a score that is intended to reflect unresolved orientations from lower DMIS stages, and is determined by a score less than 4 in each subscale (e.g., a score of 3.2 in the subscale of Minimization indicates a TO); (e) Leading Orientation (LO) is the stage above the examinee’s DO (e.g., Adaptation is the LO to Acceptance); and (f) CD, which addresses how disconnected examinees feel toward their own respective cultural group; a score less than 4.00 indicates an unresolved feeling toward one’s own culture (Hammer, 2011). PO and DO scores are represented as standardized IQ metric scores (i.e., M = 100, SD = 15), ranging from 55 to 145. A score below 85 demonstrates a cultural orientation in Polarization (i.e., defense and reversal), scores between 85 and 114.99 indicate an orientation in Minimization, and scores of 115 and above represent degrees of Acceptance and Adaptation (Hammer, 2009b; Sample, 2009). The gap between PO and DO is the OG score, and “a gap score of 7 points or higher can be considered a meaningful difference between where you perceive you are on the developmental continuum and where the IDI places your level of intercultural competence” (Hammer, 2009b, p. 6). TO scores range from 1 to 5: any scores below 4 suggest an unresolved TO. Scores of 4 and higher suggest that a person has resolved that particular TO (Sample, 2009). Technical Adequacy Development The development of the IDI consisted of three phases. During the beginning phase (226 respondents), the researchers developed an initial (60-item) version of the IDI (Hammer, 1999). The items represented all six DMIS orientations. In Phase 2 (591 respondents), the researchers conducted a confirmatory factor analysis, eliminated 10 items and determined the final 50-item IDI, with its five dimensions (Hammer, Bennett, & Wiseman, 2003). In addition, in 2007, a sample of Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 180 Journal of Psychoeducational Assessment 32(2) 4,763 respondents from a wide range of ages and professions completed the IDI in their native languages (six languages were used). The translation process included a rigorous back-translated methodology. The participants came from the profit sector, international organizations, nonprofit organizations, and high schools and colleges (Hammer, 2011). Phase 3 in 2007 yielded a new IDI version (v.3) with seven subscales (Hammer, 2009a). The authors also created two composite measures: PO (α = .82) and DO (α = .83; Hammer, 2009b). Reliability The IDI researchers provided detailed information on scale reliability estimates during each stage of the instrument’s development, which is described below. Internal consistency. Based on the 2007 Phase 3 test, the IDI v.3 achieved the following Cronbach’s alpha internal consistency coefficients for the seven subscales: Denial (seven items, α = .66), Defense (six items, α = .72), Reversal (nine items, α = .78), Minimization (nine items, α = .74), Acceptance (five items, α = .69), Adaptation (nine items, α = .71), and CD (five items, α = .79). None of these reliability estimates meet the lower bounds considered acceptable for research or low-stakes personal applications; a minimum reliability of .80 is considered appropriate for these purposes (Crocker & Algina, 1986). Each subscale in the second version of the IDI met the .80 minimal reliability criterion (Paige, Jacobs-Cassuto, Yershova, & DeJaeghere, 2003). This outcome suggests that additional study with comparable sample sizes is needed to reexamine the IDI internal consistencies. Further item and scale development appears warranted to render a scale that is sufficiently reliable for the intended purposes and applications of this instrument. Validity There are various forms of validity associated with an instrument such as the IDI. Among them, face validity, content-related validity, and internal validity can be determined through self- and expert review, pilot studies, and data analysis (Cohen & Swerdlik, 2010; Gall, Gall, & Borg, 2007); additional criterion-related validation is needed to demonstrate the predictive value of the construct assessed. IDI content-related validity. Content experts typically assess the content-related validity of an instrument by determining how well the test items represent the intended domain of the content (Gall et al., 2007). The IDI was constructed to measure intercultural competence based on Bennett’s (1993) DMIS. During Phase 1, the researcher calculated interrater reliability to determine whether the categorization of the IDI dimensions was aligned with Bennett’s (1993) DMIS. Interrater reliabilities ranged from .66 to .86 (Cohen’s Kappa); interrater reliabilities in this vicinity are considered good to excellent (DeVellis, 1991). In addition, the content validity of the IDI was addressed through the in-depth interviews and expert review (Hammer et al., 2003). It appears that adequate evidence of content validity exists to support the construction of the instrument. Construct validity. The researchers assessed construct validity by examining the correlations between the IDI subscales and two theoretically related instruments: a six-item Worldmindedness Scale (Sampson & Smith, 1957; Wiseman, Hammer, & Nishida, 1989) and a modified Intercultural Anxiety scale (Stephan & Stephan, 1985). According to Paige et al. (2003), “Higher ethnorelativism scores correlated with higher Worldmindedness and lower Social Anxiety Scale scores as theoretically predicted” (p. 473). This result demonstrated that the IDI was supported by at least minimal evidence of its construct validity as the constructs evidenced appropriate Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 181 Test Review convergent correlations. Many more studies are needed to support the utility of the instrument for its intended purposes (e.g., additional concurrent validation, criterion-referenced validation, contrasted-groups validation) and its predictive value. Internal validity. The IDI v.3 is composed of seven subcategories: Denial, Defense, Reversal, Minimization, Acceptance, Adaptation, and CD. With correlation analyses among all seven dimensions, significant intercorrelations have been identified, which supports the developmental continuum, shared underlying content, and the relationships among core orientations (Hammer, 2010). Predictive validity. Paige et al. (2003) examined the predictive validity of the IDI by using a sample of 353 participants. Using one-way ANOVA, Paige et al. found that five background variables (i.e., age, prior intercultural experience, prior language-culture study, friends from other cultures, socialize with people from other cultures) were associated with the IDI score as predicted by theory and prior research. Although these correlations do not support the future predictive abilities of the IDI, they do support the concurrent relationships between convergent characteristics and orientations. Only one background variable, gender, was not associated with the IDI score in a meaningful way. This finding indicated that the IDI scores can be used to predict respondents’ past intercultural experiences as a developmental phenomenon. Higher IDI scores indicate a higher possibility that the respondent has had intercultural interactions previously; however, according to Bennett (1993), exposure to different cultures itself does not increase one’s intercultural competence. It appears that intercultural competence only progresses with constructive self-reflection. Again, additional research is needed to more adequately demonstrate the usefulness of the instrument for its intended applications. Test Bias The researchers also addressed the possible effects of gender, age, and education on IDI scale scores. No significant differences were found between groups, except the male mean Denial/ Defense score was significantly higher than female mean (t = 4.84, df = 553, p < .01). As there was no significant difference on other scales, the researchers concluded that the IDI scores in general were not significantly influenced by gender differences. To investigate test bias, Hammer et al. (2003) and Paige et al. (2003) tested for social desirability bias for the IDI using the (10- or 20-item) Marlowe–Crowne Social Desirability Scale short form (Strahan & Gerbasi, 1972). Neither group produced significant correlations between the Social Desirability scale and the IDI subscales, suggesting that the IDI scale scores were not influenced by a tendency for participants to provide socially desirable responses. Commentary and Recommendations Overall, the IDI v.3 is a promising and comprehensive instrument that measures individuals’ orientations toward cultural differences. The subscales were based on a solid theoretical foundation; then the instrument was tested with factor analysis in Phase 1 and confirmatory factor analyses in Phases 2 and 3 of development. The findings demonstrated that the IDI has an acceptable fit with Bennett’s (1993) DMIS. As it has been widely used in the fields of intercultural competence assessment, intercultural program evaluation, and research during the past 10 years, the IDI appears to be an acceptable tool to assess examinees for the previously mentioned applications and contexts (Hammer, 2011; Paige et al., 2003; Sample, 2009). The population for whom the instrument was intended includes people from any cultural background and individuals age 15 or older who have at least a 10th-grade reading level (Hammer, 2011). The IDI scores in general Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 182 Journal of Psychoeducational Assessment 32(2) were not significantly influenced by gender differences and were not influenced by a tendency for the participants to provide socially desirable responses. The researchers described the detailed information on the development process of the IDI, which provides other researchers abundant resources for future study and analysis. Because the 2007 test evidenced low internal consistency for each of the seven subscales, another large sample test is needed in the future to retest the internal reliability after the scale is revised. The potential evidence for test–retest reliability would be helpful as well. A revised IDI with better internal and test–retest reliability is extremely valuable and in demand in evaluating study abroad programs and international students’ development of intercultural competence in the United States. Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding The author(s) received no financial support for the research, authorship, and/or publication of this article. References Anderson, P. H., Lawton, L., Rexeisen, R. J., & Hubbard, A. C. (2006). Short-term study abroad and intercultural sensitivity: A pilot study. International Journal of Intercultural Relations, 30, 457-469. Bennett, M. J. (1993). Towards ethnorelativism: A developmental model of intercultural sensitivity. In R. M. Paige (Ed.), Education for the intercultural experience (2nd ed., pp. 21-71). Yarmouth, ME: Intercultural Press. Cohen, R. J., & Swerdlik, M. E. (2010). Psychological testing and assessment: An Introduction to tests & measurement (7th ed.). Boston, MA: McGraw-Hill Higher Education. Crocker, L., & Algina, J. (Eds.). (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt, Brace, & Jovanovich College Publishers. DeVellis, R. F. (1991). Scale development. Newbury Park, CA: Sage. Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston, MA: Pearson. Hammer, M. R. (1999). A measure of intercultural sensitivity: The Intercultural Development Inventory. In S. M. Fowler & M. G. Fowler (Eds.), The intercultural sourcebook (vol. 2, pp. 61-72). Yarmouth, ME: Intercultural Press. Hammer, M. R. (2009a). The Intercultural Development Inventory: An approach for assessing and building intercultural competence. In M. A. Moodian (Ed.), Contemporary leadership and intercultural competence: Exploring the cross-cultural dynamics within organizations (pp. 203-217). Thousand Oaks, CA: Sage. Hammer, M. R. (2009b). Intercultural Development Inventory v.3 (IDI) individual profile report. Retrieved from http://www.idiinventory.com/pdf/idi_sample.pdf Hammer, M. R. (2010). IDI validity. Retrieved from http://www.idiinventory.com/pdf/idi_validity.pdf Hammer, M. R. (2011). Additional cross-cultural validity testing of the Intercultural Development Inventory. International Journal of Intercultural Relations, 35, 474-487. Hammer, M. R., Bennett, M. J., & Wiseman, R. (2003). Measuring intercultural sensitivity: The intercultural development inventory. International Journal of Intercultural Relations, 27, 421-443. Hansel, B. (2009, August). Using the IDI for impact assessment: Findings, strengths, challenges, and practical considerations. Paper presented at IAIR conference, Honolulu, HI. Meyer-Lee, E., & Evans, J. (2006, February). Multi-method longitudinal assessment of intercultural engagement, IC sensitivity, identity, and goals: Preliminary results. Paper presented at AIEA Annual Conference, San Diego, CA. Paige, R. M., Jacobs-Cassuto, M., Yershova, Y. A., & DeJaeghere, J. (2003). Assessing intercultural sensitivity: An empirical analysis of the Hammer and Bennett Intercultural Development Inventory. International Journal of Intercultural Relations, 27, 467-486. Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014 183 Test Review Pedersen, P. J. (2010). Assessing intercultural effectiveness outcomes in a year-long study abroad program. International Journal of Intercultural Relations, 34, 70-80. Rexeisen, R. J., Anderson, P. H., Lawton, L., & Hubbard, A. C. (2008). Study abroad and intercultural development: A longitudinal study. Frontiers: The Interdisciplinary Journal of Study Abroad, 17, 1-20. Sample, S. G. (2009, October 29-30). Intercultural development and the international curriculum (Top Paper Award). Paper presented at the IDI Conference, Minneapolis, MN. Sampson, D. L., & Smith, H. P. (1957). A scale to measure world-minded attitudes. Journal of Social Psychology, 45, 99-106. Stephan, W. G., & Stephan, C. W. (1985). Intergroup anxiety. Journal of Social Issues, 41, 157-176. Strahan, R., & Gerbasi, K. C. (1972). Short homogeneous version of the Marlowe-Crowne Social Desirability Scale. Journal of Clinical Psychology, 28, 191-193. Wiseman, R. L., Hammer, M. R., & Nishida, H. (1989). Predictors of intercultural communication competence. International Journal of Intercultural Relations, 13, 349-370. Downloaded from jpa.sagepub.com at COLLEGE OF WILLIAM AND MARY on March 4, 2014