Intercultural Competence Assessment Formats
Reliability and Validity Formats
Abstract
Globalization has encouraged a demand for increased intercultural competence (ICC) in higher education. Although in the modern world, the assessments and concepts of ICC have been given a wide attention, few assessments have designed to meet the standards in areas of innovative item types and reliability and validity evidence. This study seeks to discuss the possible item types and their strengths and weaknesses within the category of selected response items. In addition, this paper aims to discuss the reliability evidence for the previously designed ICC assessments and then discuss the validity evidence concerning the internal structure, the relationship with conceptually related constructs, and the relationship with criteria. To summarize, the researcher found that the reliability evidence of existing assessments includes no major issue with reliability at the total test level. Regarding validity, it was found that for most assessments, insufficient validity exists, particularly criterion-related validity.
References
American Educational Research Association, American Psychological Association, & National Council On Measurement in Education. (2014). Standards for educational and psychological testing. Washington,DC: Author.
Ang, S., Van Dyne, L., & Koh, C. (2006). Personality correlates of the four-factor model of cultural intelligence. Group & Organization Management, 31(1), 100–123.
Ang, S., Van Dyne, L., Koh, C., Ng, K. Y., Templer, K. J., Tay, C., & Chandrasekar, N. A. (2007). Cultural intelligence: Its measurement and effects on cultural judgment and decision making, cultural adaptation and task performance. Management and Organization Review, 3(3), 335–371.
Arasaratnam, L. A. (2008). Acculturation process of Sri Lankan Tamil immigrants in Sydney: An ethnographic analysis using the Bidirectional model (BDM). Australian Journal of Communication, 35, 57–68.
Arasaratnam, L. A. (2009).The development of a new instrument of intercultural communication competence. Journal of Intercultural Communication, 20, 1–11
Ascalon, M. E., Schleicher, D. J., & Born, M. P. (2008). Cross-cultural social intelligence: An assessment for employees working in cross-national contexts. Cross Cultural Management: An International Journal, 15(2), 109–130.
Bazgan,M., & Norel,M. (2013). Explicit and implicit assessment of intercultural competence. Procedia- Social and Behavioral Sciences, 76(15), 95–99. doi:10.1016/j.sbspro.2013.04.080
Bhawuk, D. P. (2001). Evolution of culture assimilators: Toward theory-based assimilators. International Journal of Intercultural Relations, 25(2), 141–163.
Bhawuk, D. P., & Brislin, R. (1992).The measurement of intercultural sensitivity using the concepts of individualism and collectivism. International Journal of Intercultural Relations, 16, 413–436.
Bhawuk, D. P., & Brislin, R. (2000). Cross-cultural training: A review. Applied Psychology, 49(1), 162–191.
Bing, J. W. (2001, February). Developing a consulting tool to measure process change on global teams: The Global Team ProcessQuestionnaireâ„¢. Paper presented at the national conference of the Academy of Human Resource Development, Tulsa, OK.
Bird, A., Stevens, M. J., Mendenhall, M. E., & Oddou, G. (2002). The global competencies inventory. The Kozai Group, Inc., St. Louis, MO.
Caligiuri, P., & Tarique, I. (2012). Dynamic cross-cultural competencies and global leadership effectiveness. Journal ofWorld Business, 47, 612–622.
Corbitt, J. N. (1998). Global awareness profile (1). Yarmouth, ME: Intercultural Press.
Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82(1), 143–159.
Chen, G. M. (1992). A test of intercultural communication competence. Intercultural Communication Studies, 2, 62–83.
Chen,G.M.,&Starosta,W. J. (1996). Intercultural communication competence:Asynthesis. Communication Yearbook, 19(1), 353–384.
Chen, G.M., & Starosta,W. J. (2000).The development and validation of the Intercultural Sensitivity Scale. Human Communication, 3, 1–15.
Corbitt, J. N. (1998). Global awareness profile (1). Yarmouth, ME: Intercultural Press.
Cushner, K. (1986).The Inventory of Cross-Cultural Sensitivity. Kent,OH: School of Education, Kent StateUniversity.
Davis, S. L., & Finney, S. J. (2006). A factor analytic study of the Cross-Cultural Adaptability Inventory. Educational and Psychological Measurement, 66(2), 318–330.
D’Andrea,M., Daniels, J., & Heck, R. (1991). Evaluating the impact ofmulticultural counseling training. Journal of Counseling&Development, 70(1), 143–150.
Deardorff, D. K. (2006). Identification and assessment of intercultural competence as a student outcome of internalization. Journal of Studies in International Education, 10(3), 241–266. doi:10.1177/1028315306287002
Der-Karabetian, A. (1992).World-mindedness and the nuclear threat: A multinational study. Journal of Social Behavior & Personality, 7(2), 293–298.
Elmer, M. I. (1987). Intercultural effectiveness: Development of an intercultural competency scale (Unpublished doctoral dissertation). Michigan State University, East Lansing, MI.
Engle, R. L., & Crowne, K. A. (2014).The impact of international experience on cultural intelligence: An application of contact theory in a structured short-term programme. Human Resource Development International, 17(1), 30–46.
Engle, R. L., & Crowne, K. A. (2014).The impact of international experience on cultural intelligence: An application of contact theory in a structured short-term programme. Human Resource Development International, 17(1), 30–46.
Erez, M., & Gati, E. (2004). A dynamic, multi-level model of culture: From the micro level of the individual to the macro level of a global culture. Applied Psychology, 53, 583–598.
Erez, M., Lisak, A., Harush, R., Glikson, E., Nouri, R., & Shokef, E. (2013). Going global: Developing management students’ cultural intelligence and global identity in virtual culturally diverse teams. Academy of Management Learning & Education, 12, 330–355
Fantini, A. E., & Tirmizi, A. (2006). Exploring and assessing intercultural competence.World Learning Publications, Paper 1. Brattleboro VT: SIT Digital Collections. Retrieved from
http://digitalcollections.sit.edu/worldlearning_publications/1
Fuertes, J. N. (2000). Factor structure and short formof theMiville-Guzman Universality-Diversity Scale.Measurement and Evaluation in Counseling and Development, 33(3), 157–169.
Global Leadership Excellence. (2010). RetrievedMay 8, 2015, from http:www.globallycompetent.com sectors/GCAAeducation.html. Goldberg, L. R. (1999). A broad-bandwidth, public domain personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7, 7–28.
Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7, 7–28.
Graf,A.,&Mertesacker,M. (2009). Intercultural training: Sixmeasures assessing training needs. Journal of European Industrial Training, 33(6), 539–558.
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the implicit association test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97(1), 17–41. doi:10.1037/a0015575
Hammer, M. R. (2005). Assessment of the impact of the AFS study abroad experience. Executive summary: Overall findings. Retrieved from http://idiinventory.com/wp-content/uploads/2014/02/afs_study.pdf
Hammer, M. R. (2011). Additional cross-cultural validity testing of the Intercultural Development Inventory. International Journal of Intercultural Relations, 35(4), 474–487.
Hammer, M. R. (2012). The Intercultural Development Inventory: A new frontier in assessment and development of intercultural competence. In M. Vande Berg, R. M. Paige, & K. H. Lou (Eds.), Student learning abroad (pp. 115–136). Sterling, VA: Stylus.
Hammer, M. R., Bennett, M. J., &Wiseman, R. (2003). Measuring intercultural sensitivity: The Intercultural Development Inventory. International Journal of Intercultural Relations, 27(4), 421–443.
Hammer, M. R., Wiseman, R. L., Rasmussen, J. L., & Bruschke, J. C. (1998)
Hanvey, R. G. (1982). An attainable global perspective. Theory Into Practice, 21(3), 162–167. doi:10.1080/00405848209543001
Harrison, J. K. (1992). Individual and combined effects of behavior modeling and the culture assimilator in cross-cultural management training. Journal of Applied Psychology, 77, 952–962.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. NewYork, NY: SAGE.
Hunter, W. D., White, G. P., & Godbey, G. C. (2006). What does it mean to be globally competent? Journal of Studies in International Education, 10(3), 267–285.
Hofstede, G. (2010).The GLOBE debate: Back to relevance. Journal of International Business Studies, 41(8), 1339-1346.
Ingulsrud, J. E., Kai, K., Kadowaki, S., Kurobane, S., & Shiobara, M. (2002). The assessment of cross-cultural experience: Measurin awareness through critical text analysis. International Journal of Intercultural Relations, 26(5), 473–491.
Isley, E. B., Shealy, C. N., Crandall,K. S., Sivo, S.A.,&Reifsteck, J. B. (1999,August). Relevance of the “BEVI” for research in developmental psychopathology. Poster session presented at the meeting of the American Psychological Association, Boston, MA.
Jacobson,W., Sleicher,D., &Maureen, B. (1999). Portfolio assessment of intercultural competence. International Journal of Intercultural Relations, 23(3), 467–492.
Kelley, C., & Meyers, J. (1995). Cross-Cultural Adaptability Inventory. Minneapolis, MN: National Computer Systems.
Klein, J. (1995). Intelligence and cross-cultural sensitivity. Psychology: A Journal of Human Behavior, 32(1), 31–32.
Koban, L., & Pourtois, G. (2014). Brain systems underlying the affective and social monitoring of actions: An integrative review. Neuroscience & Biobehavioral Reviews, 46, 71–84.
Koester, J., & Olebe, M. (1989).The behavioral assessment scale for intercultural communication effectiveness. International Journal of Intercultural Relations, 12(3), 233–246. doi:10.1016/0147-1767(88)90017-X
Kupka, B. (2008). Creation of an instrument to assess intercultural communication competence for strategic international human resource management (Unpublished doctoral dissertation). University of Otago, Otago, New Zealand.
Kupka, B., & Everett, A. M. (2008). Moscow on the Hudson: Assessing expatriates’ affective fit in host cultures with the intercultural communication affinity scale. European Journal of International Management, 2(3), 234–249.
Leung, K., Ang, S.,&Tan,M. L. (2014). Intercultural competence. Annual Review of Organizational Psychology and Organization Behavior, 1(1), 489–519.
Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426–441.
Lin, Y. C., Chen, A. S. Y., & Song, Y. C. (2012). Does your intelligence help to survive in a foreign jungle? The effects of cultural intelligence and emotional intelligence on cross-cultural adjustment . International Journal of Intercultural Relations, 36, 541–552.
Lisak, A., & Erez, M. (2015). Leadership emergence in multicultural teams:The power of global characteristics. Journal ofWorld Business, 50(1), 3–14.
Matsumoto, D., LeRoux, J. A., Ratzlaff, C., Tatani, H., Uchida, H., Kim, C., & Araki, S. (2001). Development and validation of a measure of intercultural adjustment potential in Japanese sojourners: The Intercultural Adjustment Potential Scale (ICAPS). International Journal of Intercultural Relations, 25, 483–510.
Messick, S. (1995).Validity of psychological assessment:Validation of inferences frompersons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Messner, W., & SchÙ†fer, N. (2012). The ICCA facilitator’s manual: Intercultural Communication and Collaboration Appraisal. London, England: GloBus Research
Miville, M. L., Gelso, C. J., Pannu, R., Liu, W., Touradji, P., Holloway, P., & Fuertes, J. (1999). Appreciating similarities and valuing differences: The Miville-Guzman Universality Diversity Scale. Journal of Counseling Psychology, 46, 291–307
.
Morrell,D. L., Ravlin, E. C., Ramsey, J. R.,&Ward,A.K. (2013). Past experience, cultural intelligence, and satisfaction with international business studies. Journal of Teaching in International Business, 24, 31– 43.
Nguyen, N. T., Biderman,M. D., &McNary, L. D. (2010). A validation study of the Cross-Cultural Adaptability Inventory. International Journal of Training and Development, 14(2), 112–129.
Olebe, M., & Koester, J. (1989). Exploring the cross-cultural equivalence of the Behavioral Assessment Scale for Intercultural Communication. International Journal of Intercultural Relations, 13(3), 333–347. doi:10.1016/0147-1767(89)90016-3
Pruegger, V. J., & Rogers, T. B. (1993). Development of a scale to measure cross-cultural sensitivity in the Canadian context. Canadian Journal of Behavioural Science/Revue Canadienne des Sciences du Comportement, 25(4), 615–6
Remhof, S., Gunkel,M., & Schlaegel, C. (2013).Working in the “Global Village”:The influence of cultural intelligence on the intention to work abroad. Zeitschrift für Personalforschung/German Journal of Human Resource Management, 27(3), 224–250.
Rios, J. A., &Wells, C. S. (2014). Validity evidence based on internal structure. Psicothema, 26(1), 108–116.
Ruben, B. D. (1976). Assessing communication competency for intercultural adaptation. Group and Organization Studies, 1, 334–354.
Sampson, D. L., & Smith, H. P. (1957. A scale to measure world-minded attitudes.The Journal of Social Psychology, 45(1), 99–106.
Schmitz, J., Tarter, L., & Sine, J. (2012). Understanding the cultural orientations approach: An overview of the development and updates to the COA. Retrieved from
https://www.culturalorientations.com/SiteData/docs/ArticleUnd/616d3a22b5d5d472/Article%20-
Shealy, C. N., Burdell, L. L., Sivo, S. A., Davino, D. F., & Hayes, D. J. (1999, August).Men, masculinity, and Scale 10 (Gender Stereotypes) of the “BEVI.” Poster session presented at the meeting of the American Psychological Association, Boston, MA.
Shealy, C. N., Sears, J. L., Sivo, S. A., Alessandria, K. P., & Isley, E. B. (1999, August). Intercultural psychology and Scale 3 (Sociocultural Closure) of the “BEVI.” Poster session presented at the meeting of the American Psychological Association, Boston, MA.
Shirts, G. R. (1977). BaFa’ BaFa’ – A cross-cultural simulation. La Jolla, CA: Simile II.
Shokef, E., & Erez, M. (2006). Global work culture and global identity, as a platform for a shared understanding in multicultural teams. National Culture and Groups, 9, 325–352.
Shokef, E.,&Erez, M. (2008). Cultural intelligence and global identity inmulticultural teams. In S.
Ang&L.VanDyne (Eds.), Handbook of cultural intelligence:Theory, measurement, and applications (pp. 177–191). Abington, England: M. E. Sharpe.
Skinner, M. (2002).The renaissance of unconventional warfare as an SF mission. SpecialWarfare, 15, 16–22.
Smith, D. E., & Mitry, D. J. (2008). Benefits of study abroad and creating opportunities:The case for short-term programs. Journal of Research in Innovative Teaching, 1(1), 236–246.
Stephan,W. G., & Stephan, C. (1985). Intergroup anxiety. Journal of Social Issues, 41, 157–176.
Stevens, M., Bird, A., Mendenhall, M. E., & Oddou, G. (2014). Measuring global leader intercultural competency: Development and validation of the Global Competencies Inventory (GCI). Advances in Global Leadership, 8, 115–154.
Strubler, D., Agarwal, A., Park, S., & Elmer,M. (2011). From cognition to behavior: A cross cultural study for global business effectiveness. Journal of International Business Research, 10, 35–46.
Thomas,D. C., & Lazarova,M. B. (2006). Expatriate adjustment and performance: A critical review. In G. K. Stahl & I. Bjorkman (Eds.), Handbook of research in international human resource management (pp. 247–265). Northampton, MA: Edward Elgar.
Thomas, D. C., Stahl, G., Ravlin, S., Poelmans, A., Pekerti, M., Maznevski, M., ”¦ Brislin, R. (2015). Development of the Cultural Intelligence Assessment. Advances in Global Leadership, 8, 155–178.
Van der Zee, K. I., Atsma, N., & Brodbeck, F. (2004).The influence of social identity and personality on outcomes of cultural diversity in teams. Journal of Cross-Cultural Psychology, 35(3), 283–303.
Van der Zee, K. I., & Van Oudenhoven, J. P. (2000).The Multicultural Personality Questionnaire: A multidimensional instrument of multicultural effectiveness. European Journal of Personality, 14(4),
291–309
Varela, O., & Gatlin-Watts, R. (2013).The development of the globalmanager: An empirical study on the role of academic international sojourns. Academy of Management Learning & Education, 13, 187–207
Wang, Y. W., Davidson, M. M., Yakushko, O. F., Savoy, H. B., Tan, J. A., & Bleier, J. K. (2003). The scale of ethnocultural empathy: Development, validation, and reliability. Journal of Counseling Psychology, 50(2), 221–234.
Ward, C., & Kennedy, A. (1999).The measurement of sociocultural adaptation. International Journal of Intercultural Relations, 23(4), 659–677.
Ward, C.,Wilson, J., & Fischer, R. (2011). Assessing the predictive validity of cultural intelligence over time. Personality and Individual Differences, 51, 138–142.
Whetzel, D. L., &McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188–202.
Wiseman, R. L., Hammer, M. R., & Nishida, H. (1989). Predictors of intercultural communication competence. International Journal of Intercultural Relations, 13(3), 349–370.