Designing Culturally Responsive Online Assessments for Equity-Deserving Students

Kim Koh

doi:10.18733/cpi29712

Authors

Kim Koh University of Calgary https://orcid.org/0000-0001-8315-4299

DOI:

https://doi.org/10.18733/cpi29712

Abstract

In this article, the author offers a brief review of the theoretical foundations, historical origins, and development of culturally responsive assessment (CRA). The concept of authentic performance assessment is discussed with its crucial role in serving as a form of CRA to support students’ learning in online learning environments. Drawing on the fields of measurement, assessment, and technology, the author highlights some key considerations in the design and implementation of culturally responsive online assessments for equity-deserving students.

Author Biography

Kim Koh, University of Calgary

Dr. Kim Koh is a Professor and Chair of Learning Sciences from the Werklund School of Education at the University of Calgary. Her areas of expertise include classroom assessment, teacher professional learning, and educational measurement. Dr. Koh has been actively involved internationally in conducting research and publishing in the field of assessment.

References

Adie, L., & Klenowski, V. (2016). Moderation and assessment. In M. A. Peters (Ed.), Encyclopedia of educational philosophy and theory. Springer. https://doi.org/10.1007/978-981-287-532-7_393-1

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

Archbald, D. A., & Newmann, F. M. (1988). Beyond standardized testing: Assessing authentic academic achievement in the secondary school. Office of Educational Research and Improvement.

Bahar, A. K., & Maker, C. J. (2020). Culturally responsive assessments of mathematical skills and abilities: Development, field testing, and implementation. Journal of Advanced Academics, 31(3), 211-233. https://doi.org/10.1177/1932202X20906130

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102

Bond, L. (1995). Unintended consequences of performance assessments: Issues of bias and fairness. Educational Measurement: Issues and Practices, 14(4), 21-24. https://doi.org/10.1111/J.1745-3992.1995.TB00885.X

Brown, G. T. L., Kennedy, K. J., Fok, P. K., Chan, J. K. S., & Yu, W. M. (2009). Assessment for student improvement: Understanding Hong Kong teachers’ conceptions and practices of assessment. Assessment in Education: Principles, Policy & Practice, 16(3), 347-363. https://psycnet.apa.org/doi/10.1080/09695940903319737

Bryan, M., & Lewis, A. (2019). Culturally responsive evaluation as a form of critical qualitative inquiry. In G. W. Noblit (Ed.), Oxford research encyclopedia of education. Oxford University Press. https://doi.org/10.1093/acrefore/9780190264093.013.545

Cumming, J. J., & Maxwell, G. S. (1999). Contextualizing authentic assessment. Assessment in Education: Principles, Policy & Practice, 6(2), 177-194. https://doi.org/10.1080/09695949992865

Ercikan, K. (1998). Translation effects in international assessments. International Journal of Educational Research, 29(6), 543-553. https://doi.org/10.1016/S0883-0355(98)00047-0

Ercikan, K., Gierl, M., McCreith, T., Puhan, G., & Koh, K. (2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests. Applied Measurement in Education, 17(3), 301-321. https://doi.org/10.1207/s15324818ame1703_4

Ercikan, K., Guo, H., & He, Q. (2020). Use of response process data to inform group comparisons and fairness research. Educational Assessment, 25(3), 179-197. https://doi.org/10.1080/10627197.2020.1804353

Ercikan, K., & Koh, K. (2005). Examining the construct comparability of the English and French versions of TIMSS. International Journal of Testing, 5(1), 23-35. https://doi.org/10.1207/s15327574ijt0501_3

Ercikan, K., & Pellegrino, J. W. (2017). Validation of score meaning for the next generation of assessments: The use of response processes. Routledge.

Hansen, M. J., & Renguette, C. C. (2021). Assessment institute diversity, equity, and inclusion track: Highlights from 2020 and sneak preview of 2021. Assessment Update, 33(3), 12-13. https://doi.org/10.1002/au.30258

Henning, G., & Lundquist, A. E. (2022). Using assessment to advance equity. New Directions for Student Services, 2022 (178-179), 185-194. https://doi.org/10.1002/ss.20439

Holland, R. (2007). Portfolios: A backward step in school accountability. Lexington Institute. https://www.lexingtoninstitute.org/portfolios-a-backward-step-in-school-accountability/

Hood, S. (1998). Culturally responsive performance-based assessment: Conceptual and psychometric considerations. Journal of Negro Education, 67(3), 187-196. https://doi.org/10.2307/2668188

Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally responsive evaluation: Theory, practice, and future implications. In K. B. Newcomer, H. P. Hairy & J. S. Wholey (Eds.), Handbook of practical program evaluation (4th ed., pp. 281-317). John Wiley.

Hopson, R. K. (2009). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment. In K. Ryan & B. Cousins (Eds.), International handbook of educational evaluation (pp. 429-446). SAGE.

Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103(2), 219-230. https://doi.org/10.1007/s11205-011-9843-4

Johnson, S. T. (1998). Editor’s comments: The importance of culture for improving assessment and pedagogy. Journal of Negro Education, 67(3), 181-183. https://doi.org/10.2307/2668186

Joo, Y. J., Lim, K. Y., & Kim, N. H. (2016). The effects of secondary teachers’ technostress on the intention to use technology in South Korea. Computers and Education, 95, 114-122. https://doi.org/10.1016/j.compedu.2015.12.004

Kirkhart, K. E. (1995). 1994 Conference Theme: Evaluation and social justice seeking multicultural validity: A postcard from the road. Evaluation Practice,16(1), 1-12. https://doi.org/10.1177/109821409501600101

Koh, K. (2011a). Improving teachers’ assessment literacy. Pearson Education South Asia.

Koh, K. (2011b). Improving teachers’ assessment literacy through professional development. Teaching Education, 22(3), 255-276. https://doi.org/10.1080/10476210.2011.593164

Koh, K. (2014). Authentic assessment, teacher judgment and moderation in a context of high accountability. In C. Wyatt-Smith, V. Klenowski & P. Colbert (Eds.), Designing assessment for quality learning (pp. 249-264). Springer.

Koh, K. (2017). Authentic assessment. In G. W. Noblit (Ed.), Oxford research encyclopedia of education. Oxford University Press. https://doi.org/10.1093/acrefore/9780190264093.013.22

Koh, K., Chapman, O., & Lam, L. (2022). Exploring the integration of virtual reality into the design of authentic assessments for STEM learning. In S. Keengwe (Ed.), Handbook of research on transformative and innovative pedagogies in education (pp. 18-35). IGI Global.

Koh, K., Chapman, O., & Liu, S. (2022). Building elementary school teachers’ capacity in the design and implementation of authentic STEM assessments for girls. International Journal of Gender, Science and Technology,14(2), 247-279. https://genderandset.open.ac.uk/index.php/genderandset/article/view/1369

Koh, K., & Luke, A. (2009). Authentic and conventional assessment in Singapore schools: An empirical study of teacher assignments and student work. Assessment in Education: Principles, Policy & Practice, 16(3), 291-318. https://doi.org/10.1080/09695940903319703

Koh, K., & Zumbo, B. (2008). Multi-group confirmatory factor analysis for testing measurement invariance in mixed item format data. Journal of Modern Applied Statistical Methods, 7(2), 471-477. https://doi.org/10.22237/jmasm/1225512660

Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. American Educational Research Journal, 32(3), 465-491. https://doi.org/10.2307/1163320

Ladson-Billings, G. (1998). Teaching in dangerous times: Culturally relevant approaches to teacher assessment. Journal of Negro Education, 67(3), 255-267. https://doi.org/10.2307/2668194

Lee, C. D. (1992). Literacy, cultural diversity, and instruction. Education and Urban Society, 24(2), 279-291. https://doi.org/10.1177/0013124592024002008

Lee, C. D. (1998). Culturally responsive pedagogy and performance-based assessment. Journal of Negro Education, 67(3), 268-279. https://doi.org/10.2307/2668195

Lewis, C., Perry, R., Hurd, J., & O’Connell, M. P. (2006). Lesson study comes of age in North America. Phi Delta Kappan, 88(4), 273-281. https://doi.org/10.1177/003172170608800406

Lockee, B. B. (2021). Online education in the post-COVID era. Nature Electronics, 4, 5-6. https://doi.org/10.1038/s41928-020-00534-0

Luke, A., Green, J., & Kelly, G. J. (2010). Introduction: What counts as evidence and equity? Review of Research in Education, 34(1), vii-xvi. https://doi.org/10.3102/0091732X09359038

Maddox, B. (2023). The uses of process data in large-scale educational assessments. OECD Education Working Paper No. 286. https://one.oecd.org/document/EDU/WKP(2023)1/en/pdf

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Macmillan.

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23. https://doi.org/10.3102/0013189X023002013

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741

Miranda, C. P., & Cherng, H. S. (2018). Accountability reform and responsive assessment for immigrant youth. Theory Into Practice, 57(2), 119-126. https://doi.org/10.1080/00405841.2018.1425817

Montenegro, E., & Jankowski, N. A. (2017). Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper #29). National Institute for Learning Outcomes Assessment. https://files.eric.ed.gov/fulltext/ED574461.pdf

Moss, P., Girard, B. J., & Haniford, L. C. (2006). Chapter 4: Validity in educational assessment. Review of Research in Education, 30(1), 109-162. https://doi.org/10.3102/0091732X030001109

Nasir, N. S., & de Royston, M. K. (2013). Power, identity, and mathematical practices outside and inside school. Journal for Research in Mathematics Education, 44(1), 264-287.

Nasir, N. S., Hand, V., & Taylor, E. V. (2008). Culture and mathematics in school: Boundaries between “cultural” and “domain” knowledge in the mathematics classroom and beyond. Review of Research in Education, 32(1), 187-240. https://doi.org/10.3102/0091732X07308962

National Center for Education Statistics. (2023). English learners in public schools. Condition of education. U.S. Department of Education, Institute of Education Sciences. https://nces.ed.gov/programs/coe/indicator/cgf Newmann, F. M., Marks, H. M., & Gamoran, A. (1996). Authentic pedagogy and student performance. American Journal of Education, 104(4), 280-312.

Pajares, M. F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62(3), 307-332. https://doi.org/10.3102/00346543062003307

Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. National Research Council.

Pew Research Center. (2015). Modern immigration wave brings 59 million to U.S., driving population growth and change through 2065. Report. https://www.pewresearch.org/hispanic/2015/09/28/modern-immigration-wave-brings-59-million-to-u-s-driving-population-growth-and-change-through-2065/

Popham, W. J. (2014). The right test for the wrong reason. Phi Delta Kappan, 96(1), 46-52. https://doi.org/10.1177/0031721714547862

Salazar-Fernández, C., Navarro, R. M., Schnettler, B., & Saiz, J. L. (2020). Life satisfaction among indigenous people from Chile: Evaluation of measurement invariance. Revista Latinoamericana de Psicología, 52, 33-41. https://psycnet.apa.org/doi/10.14349/rlp.2020.v52.4

Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14. https://doi.org/10.3102/0013189X029007004

Shepard, L., Hammerness, K., Darling-Hammond, L., Rust, F., Baratz-Snowden, J., Gordon, E., Gutierrez, C., & Pacheco, A.(2005). Assessment. InL. Darling-Hammond & J. Bransford (Eds.), Preparing teachers for a changing world (pp. 275-326). John Wiley.

Smaill, E. (2020). Using involvement in moderation to strengthen teachers’ assessment for learning capability. Assessment in Education: Principles, Policy & Practice, 27(5), 522-543. https://doi.org/10.1080/0969594X.2020.1777087

Statistics Canada. (2022). Immigrants make up the largest share of the population in over 150 years and continue to shape who we are as Canadians. https://www150.statcan.gc.ca/n1/daily-quotidien/221026/dq221026a-eng.htm

Stiggins, R. J. (1995). Assessment literacy for the 21st century. Phi Delta Kappan, 77(3), 238-245.

Stiggins, R. J. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta Kappan, 83(10), 758-765. https://doi.org/10.1177/003172170208301010

Trumbull, E., & Nelson-Barber, S. (2019). The ongoing quest for culturally-responsive assessment for indigenous students in the U.S. Frontiers in Education, 4(40), 1-11. https://doi.org/10.3389/feduc.2019.00040

Trumbull, E., & Solano-Flores, G. (2011). The role of language in assessment. In M. Basterra, E. Trumbull & G. Solano-Flores (Eds.), Cultural validity in assessment: Addressing linguistic and cultural diversity (pp. 22-45). Routledge.

Truth and Reconciliation Commission of Canada. (2015). Truth and reconciliation commission of Canada: Calls to action. https://www2.gov.bc.ca/assets/gov/british-columbians-our-governments/indigenous-people/aboriginal-peoples-documents/calls_to_action_english2.pdf

Walker, D., Mahon, E., & Dray, B. (2021). Can we prepare culturally and linguistically responsive teachers online? A cross-case analysis of online and on-campus courses. Urban Education, 56(7), 1015-1041. https://doi.org/10.1177/0042085917735970

Warschauer, M., & Matuchniak, T. (2010). New technology and digital worlds: Analyzing evidence of equity in access, use, and outcomes. Review of Research in Education, 34(1), 179-225. https://doi.org/10.3102/0091732X09349791

Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70(9), 703-713. https://doi.org/10.1177/003172171109200721

Wiliam, D. (2010). What counts as evidence of educational achievement? The role of constructs in the pursuit of equity in assessment? Review of Research in Education, 34(1), 254-284. https://doi.org/10.3102/0091732X09351544

Zheng, M., Asif, M., Tufail, M. S., Naseer, S., Khokhar, S. G., Chen, X., & Naveed, R. T. (2022). COVID academic pandemic: Techno stress faced by teaching staff for online academic activities. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.895371

Zumbo, B. D., & Koh, K. H. (2005). Manifestation of differences in item-level characteristics in scale-level measurement invariance tests of multi-group confirmatory factor analyses. Journal of Modern Applied Statistical Methods, 4(1), 275-282. doi:10.22237/jmasm/1114907040