Evaluating an Instructional Intervention for Research Data Management Training
DOI:
https://doi.org/10.18438/eblip30439Abstract
Objective – At a large research university in Canada, a research data management (RDM) specialist and two liaison librarians partnered to evaluate the effectiveness of an active learning component of their newly developed RDM training program. This empirical study aims to contribute a statistical analysis to evaluate an RDM instructional intervention.
Methods – This study relies on a pre- and post-test quasi-experimental intervention during introductory RDM workshops offered 12 times between February 2022 and January 2023. The intervention consists of instruction on best practices related to file-naming conventions. We developed a grading rubric differentiating levels of proficiency in naming a file according to a convention reflecting RDM best practices and international standards. We used manual content analysis to independently code each pre- and post-instruction file name according to the rubric.
Results – Comparing the overall average scores for each participant pre- and post-instruction intervention, we find that workshop participants, in general, improved in proficiency. The results of a Wilcoxon signed-rank test demonstrate that the difference between the pre- and post-test observations is statistically significant with a high effect size. In addition, a comparison of changes in pre- and post-test scores for each rubric element showed that participants grasped specific elements more easily (i.e., implementing an international standard for a date format) than others (i.e., applying information related to sequential versioning of files).
Conclusion – The results of this study indicate that developing short and targeted interventions in the context of RDM training is worthwhile. In addition, the findings demonstrate how quantitative evaluations of instructional interventions can pinpoint specific topics or activities requiring improvement or further investigation. Overall, RDM learning outcomes grounded in practical competencies may be achieved through applied exercises that demonstrate immediate improvement directly to participants.
Downloads
References
Agogo, D., & Anderson, J. (2019). “The data shuffle”: Using playing cards to illustrate data management concepts to a broad audience. Journal of Information Systems Education, 30(2), 84–96. http://jise.org/Volume30/n2/JISEv30n2p84.html
Bernard, H. R., Wutich, A., & Ryan, G. W. (2016). Analyzing qualitative data: Systematic approaches. SAGE Ppublications.
Briney, K. A., Coates, H., & Goben, A. (2020). Foundational practices of research data management. Research Ideas and Outcomes, 6, Article e56508. https://doi.org/10.3897/rio.6.e56508 DOI: https://doi.org/10.3897/rio.6.e56508
Cox, A. M., Kennan, M. A., Lyon, L., & Pinfield, S. (2017). Developments in research data management in academic libraries: Towards an understanding of research data service maturity. Journal of the Association for Information Science and Technology, 68(9), 2182–2200. https://doi.org/10.1002/asi.23781 DOI: https://doi.org/10.1002/asi.23781
Ducas, A., Michaud-Oystryk, N., & Speare, M. (2020). Reinventing ourselves: New and emerging roles of academic librarians in Canadian research-intensive universities. College & Research Libraries, 81(1), 43–65. https://doi.org/10.5860/crl.81.1.43 DOI: https://doi.org/10.5860/crl.81.1.43
Eaker, C. (2014). Planning data management education initiatives: Process, feedback, and future directions. Journal of eScience Librarianship, 3(1): 3–14. https://doi.org/10.7191/jeslib.2014.1054 DOI: https://doi.org/10.7191/jeslib.2014.1054
Fitzpatrick, M. J., & Meulemans, Y. N. (2011). Assessing an information literacy assignment and workshop using a quasi-experimental design. College Teaching, 59(4), 142–149. https://doi.org/10.1080/87567555.2011.591452 DOI: https://doi.org/10.1080/87567555.2011.591452
Gunderman, H. C. (2022). Building a research data management program through popular culture: A case study at the Carnegie Mellon University Libraries. In M. E. Johnson, T.C. Weeks, & J. Putnam Davis (Eds.), Integrating pop culture into the academic library (pp. 273–285). Rowman & Littlefield Publishers.
Hswe, P. (2012). Data management services in libraries. In N. Xiao & L. R. McEwen (Eds.), Special issues in data management (pp. 115–128). American Chemical Society. https://doi.org/10.1021/bk-2012-1110.ch007 DOI: https://doi.org/10.1021/bk-2012-1110.ch007
Knight, L. A. (2006). Using rubrics to assess information literacy. Reference Services Review, 34(1), 43–55. https://doi.org/10.1108/00907320610640752 DOI: https://doi.org/10.1108/00907320610640752
Krewer, D., & Wahl, M. (2018). What’s in a name? On ‘meaningfulness’ and best practices in filenaming within the LAM community. Code4Lib Journal, 40. https://journal.code4lib.org/articles/13438
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. https://doi.org/10.1111/j.1468-2958.2004.tb00738.x DOI: https://doi.org/10.1093/hcr/30.3.411
Krippendorff, K. (2018). Content analysis: An introduction to its methodology (4th ed.). Sage publications. DOI: https://doi.org/10.4135/9781071878781
Kurasaki, K. S. (2000). Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods, 12(3), 179–194. https://doi.org/10.1177/1525822X0001200301 DOI: https://doi.org/10.1177/1525822X0001200301
Lombard, M., Snyder‐Duch, J., & Bracken, C. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4), 587–604. https://doi.org/10.1111/j.1468-2958.2002.tb00826.x DOI: https://doi.org/10.1111/j.1468-2958.2002.tb00826.x
Matlatse, R., Pienaar, H., & van Deventer, M. (2017). Mobilising a nation: RDM training and education in South Africa. International Journal of Digital Curation, 12(2), 299–310. https://doi.org/10.2218/ijdc.v12i2.579 DOI: https://doi.org/10.2218/ijdc.v12i2.579
Oo, C. Z., Chew, A. W., Wong, A. L., Gladding, J., & Stenstrom, C. (2022). Delineating the successful features of research data management training: A systematic review. International Journal for Academic Development, 27(3), 249–264. https://doi.org/10.1080/1360144X.2021.1898399 DOI: https://doi.org/10.1080/1360144X.2021.1898399
Powell, S., & Kong, N. N. (2020). Beyond the one-shot: Intensive workshops as a platform for engaging the library in digital humanities. In C. Millson-Martula & K. B. Gunn (Eds.), The digital humanities: Implications for librarians, libraries, and librarianship (pp. 382–397). Routledge. DOI: https://doi.org/10.4324/9780429401909-26
Rod, A. B., Hervieux, S., & Lee, N. (2023a). We will meet you where you are: The development and evaluation of tailored training for the management of research data. In D. M. Mueller (Ed.), Forging the future: The proceedings of the ACRL 2023 Conference, March 15–18, 2023, Pittsburgh, Pennsylvania (pp. 461–469). Association of College and Research Libraries. https://www.ala.org/acrl/conferences/acrl2023/papers
Rod, A. B., Hervieux, S., & Lee, N. (2023b). Research data management [LibGuide]. McGill Library. https://libraryguides.mcgill.ca/researchdatamanagement
Stemler, S. (2000). An overview of content analysis. Practical Assessment, Research, and Evaluation, 7(1), Article 17. https://doi.org/10.7275/z6fm-2e34
Tang, R., & Hu, Z. (2019). Providing research data management (RDM) services in libraries: Preparedness, roles, challenges, and training for RDM practice. Data and Information Management, 3(2), 84–101. https://doi.org/10.2478/dim-2019-0009 DOI: https://doi.org/10.2478/dim-2019-0009
Thielen, J., & Hess, A. N. (2017). Advancing research data management in the social sciences: Implementing instruction for education graduate students into a doctoral curriculum. Behavioral & Social Sciences Librarian, 36(1), 16–30. https://doi.org/10.1080/01639269.2017.1387739 DOI: https://doi.org/10.1080/01639269.2017.1387739
Wuensch, K. (2020, July 19). Nonparametric effect size estimators. East Carolina University. https://core.ecu.edu/wuenschk/docs30/Nonparametric-EffectSize.pdf
Xu, Z., Zhou, X., Kogut, A., & Clough, M. (2022a). Effect of online research data management instruction on social science graduate students’ RDM skills. Library & Information Science Research, 44(4), Article 101190. https://doi.org/10.1016/j.lisr.2022.101190 DOI: https://doi.org/10.1016/j.lisr.2022.101190
Xu, Z., Zhou, X., Kogut, A., & Watts, J. (2022b). A scoping review: Synthesizing evidence on data management instruction in academic libraries. The Journal of Academic Librarianship, 48(3), Article 102508. https://doi.org/10.1016/j.acalib.2022.102508 DOI: https://doi.org/10.1016/j.acalib.2022.102508
Xu, Z., Zhou, X., Watts, J., & Kogut, A. (2023). The effect of student engagement strategies in online instruction for data management skills. Education and Information Technologies, 28, 10267–10284. https://doi.org/10.1007/s10639-022-11572-w DOI: https://doi.org/10.1007/s10639-022-11572-w
Zhou, X., Xu, Z., & Kogut, A. (2023). Research data management needs assessment for social sciences graduate students: A mixed methods study. PLoS ONE, 18(2), e0282152. https://doi.org/10.1371/journal.pone.0282152 DOI: https://doi.org/10.1371/journal.pone.0282152
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Alisa Rod, Sandy Hervieux, NuRee Lee
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The Creative Commons-Attribution-Noncommercial-Share Alike License 4.0 International applies to all works published by Evidence Based Library and Information Practice. Authors will retain copyright of the work.
