Abstract
Introduction. This paper (a) discusses the theoretical and methodological evidence for the equivalence of computerized and conventional versions of psychological tests, (b) analyzes the studies investigating psychometric parameters of computerized versions of conventional tests, and (c) examines contradictions in approaches to assessing the equivalence of the two test forms. This paper represents a first effort in structuring the main problems in establishing the equivalence of paper-and-pencil and computer-based testing, as well as in finding ways and means to overcome them. Much attention is devoted to minimal and sufficient mathematical and statistical tools for assessing the equivalence of the two test forms. Theoretical Basis. The main problems associated with assessing the equivalence of computerized and conventional tests include the following: (a) the level of cultural and informational competence of respondents, (b) anxiety, (c) social environment, (d) motivation for testing, (e) difficulties in creating the same conditions for paper-and-pencil and computer-based testing (L. N. Babanin, Y. P. Chua, M. Russell, P. Květon et al.). Researchers most often choose the following procedures for assessing the equivalence of the two versions of psychological tests: (a) comparison of the main statistical parameters (mean values, variances, etc.) and (b) assessment of construct validity and reliability of the computer-based versions of psychological tests. Results and Discussion. The analysis of research methodology for assessing the equivalence of computerized and conventional versions of psychological tests focuses on a variety of approaches to (a) the use of mathematical and statistical methods for assessing psychometric parameters of computer-based versions of conventional tests, (b) the choice of research design, (c) considering specific characteristics of the situation of computer-based testing. It is necessary to formulate specific and structured requirements for the procedure for assessing the equivalence of computerized and conventional versions of psychological tests. The author suggests recommendations for the main sections of such requirements related to (a) the procedure for conducting empirical research, (b) mathematical and statistical methods, and (c) control of the factors specific to computer-based testing that may have impact on the results of equivalence assessment.
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Anastazi, A., & Urbina S. (2009). Psychological testing. St. Petersburg: Piter. (in Russ.).
Andersson, G., Kaldo-Sandström, V., Ström, L., & Strömgren, T. (2003). Internet administration of the Hospital Anxiety and Depression Scale in a sample of tinnitus patients. Journal of Psychosomatic Research, 55(3), 259–262. doi: http://dx.doi.org/10.1016/S0022-3999(02)00575-5
APA (1986). Guidelines for computer-based tests and interpretations. Washington, DC: Author.
Babanin, L. N. (2010). Equivalence of conventional and computerized psychological techniques. Experimental psychology in Russia: Traditions and prospects. Retrieved from http://psyjournals.ru/exp_collection/issue/32974.shtml (in Russ.).
Bartram, D. (1994). Computer-based assessment. In C. L. Cooper (Ed.), International review of industrial and organizational psychology (pp. 31–69). London: Wiley.
Baturin, N. A., & Mel'nikova, N. N. (2009). Technology of test development: Part I. Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 6(30), 4–14. (in Russ.).
Baturin, N. A., & Mel'nikova, N. N. (2009). Technology of test development: Part II Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 7(42), 11–25. (in Russ.).
Baturin, N. A., & Mel'nikova, N. N. (2010). Technology of test development: Part III. Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 8(4), 4–18. (in Russ.).
Baturin, N. A., & Mel'nikova, N. N. (2010). Technology of test development: Part IV Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 11(40), 13–28. (in Russ.).
Baturin, N. A., & Mel'nikova, N. N. (2011a). Technology of test development: Part V. Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 12(5), 4–14. (in Russ.).
Baturin, N. A., & Mel'nikova, N. N. (2011b). Technology of test development: Part VI. Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series: Psychology), 13(18), 48–59. (in Russ.).
Budko, V. N., Mishin, D. Yu., & Tregubova, T. A. (2007). Testing software package for readiness of students and schoolchildren for studying at university. Infokommunikatsionnye tekhnologii (Infocommunication Technologies), 5(3), 167–169. (in Russ.).
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally & Company.
Chua, Y. P. (2012). Effects of computer-based testing on test performance and testing motivation. Computers in Human Behavior, 28(5), 1580–1586. doi: 10.1016/j.chb.2012.03.020
Ermakov, S. S. (2016). Computer diagnostics of intellectual abilities among 4th to 9th grades school students. Psikhologicheskaya nauka i obrazovanie (Psychological Science and Education), 8(4), 199–207. doi: http://dx.doi.org/10.17759/psyedu.2016080419 (in Russ.).
Ford, B. D., Vitelli, R., & Stuckless, N. (1996). The effects of computer versus paper-and-pencil administration on measures of anger and revenge with an inmate population. Computers in Human Behavior, 12(1), 159–166. doi: http://dx.doi.org/10.1016/0747-5632(95)00026-7
George, C. E., Lankford, J. S., & Wilson, S. E. (1992). The effects of computerized versus paper-and-pencil administration on measures of negative affect. Computers in Human Behavior, 8(2–3), 203–209. doi: http://dx.doi.org/10.1016/0747-5632(92)90004-X
Hays, S., & Mccallum, R. S. (2005). A comparison of the pencil-and-paper and computer-administered Minnesota Multiphasic Personality Inventory–Adolescent. Psychology in the Schools, 42(6), 605–613. doi: http://dx.doi.org/10.1002/pits.20106
Honaker, L. M. (1988). The equivalency of computerized and conventional MMPI administration: A critical review. Clinical Psychology Review, 8(6), 561–577. doi: http://dx.doi.org/10.1016/0272-7358(88)90081-5
Iovlev, B. V., Novozhilova, M. Yu., Chervinskaya, K. R., & Shchelkova, O. Yu. (2006). Studying the effectiveness of computer psychodiagnostics: methodological perspectives. Vestnik Sankt-Peterburgskogo universiteta. Ser. 6. Filosofiya, politologiya, sotsiologiya, psikhologiya, pravo, mezhdunarodnye otnosheniya (Vestnik of Saint-Petersburg University, Series 6: Philosophy. Political Science. Law. International Relations), 2, 115–124. (in Russ.).
Kibal'chenko, I. A., Ustinov, D. A., & Shapovalov, S. N. (2004). Computer psychodiagnostics as a condition for the development of giftedness. Izvestiya TRTU (Proceedings of TRTU), 6, 301–303. (in Russ.).
Kononova, V. N., & Nakhaeva, I. V. (2013). Psychometric equivalence of conventional (Color Mirror) and computerized (Colorimeter of Personality) versions of the projective psychodiagnostic technique. Vestnik Moskovskogo universiteta. Seriya 14. Psikhologiya (Moscow University Psychology Bulletin. Series 14: Psychology), 1, 136–151. (in Russ.).
Květon, P., Jelínek, M., Vobořil, D., & Klimusová, H. (2007). Computer-based tests: The impact of test design and problem of equivalency. Computers in Human Behavior, 23(1), 32–51.
Mazzeo, J., Druesne, B., Raffeld, P. C., Checketts, K. T., & Muhlstein, A. (1992). Compatibility of computer and paper-and-pencil scores for two CLEP® general examinations. ETS Research Report Series, 1. doi: http://dx.doi.org/10.1002/j.2333-8504.1992.tb01446.x
Meade, A. W., Michels, L. C., & Lautenschlager, G. J. (2004, April). Are online and paper-and-pencil personality tests truly comparable? Symposium presented at the 19th Annual Conference of the Society for Industrial and Organizational Psychology. Chicago, IL.
Mel'nichuk, A., & Sergeev, V. (n.d.). Modern computer systems for effective personnel assessment. Psikhologiya i biznes (Psychology and Business). Retrieved from https://psycho.ru/library/93 (in Russ.).
Mitina, O. V., & Sorokina, V. V. (2015). Values of senior pupils: Development of computer diagnostic tools. Vestnik Moskovskogo universiteta. Seriya 14. Psikhologiya (Moscow University Psychology Bulletin. Series 14: Psychology), 1, 42–59. (in Russ.).
Romek, V. G., & Satin, D. K (2000). Maintaining reliability of multifactor tests when using them via the Internet. Psikhologicheskii Zhurnal, 21(2), 70–75. (in Russ.).
Russell, M., Goldberg, A., & O’connor, K. (2003). Computer-based Testing and Validity: A look back and into the future. Assessment in Education: Principles, Policy & Practice, 10(3), 279–293. doi: http://dx.doi.org/10.1080/0969594032000148145
Solomon, R. L. (1949). An extension of control group design. Psychological Bulletin, 46(2), 137–150. doi: http://dx.doi.org/10.1037/h0062958
Sugonyaev, K. V., Radchenko, Yu. I., & Sokolov, A. A. (2018). Voluntary Internet-based testing as a source of valid group psychometric intelligence scores. Sibirskii psikhologicheskii zhurnal (Siberian Journal of Psychology), 69, 6–32. doi: http://dx.doi.org/10.17223/17267080/69/1 (in Russ.).
The International Testing Commission. (2006). International guidelines on computer-based and Internet-delivered testing. International Journal of Testing, 6(2), 143–171. doi: http://dx.doi.org/10.1207/s15327574ijt0602_4
van de Looij-Jansen, P. M., Goldschmeding, J. E. J., & Jan de Wilde, E. (2006). Comparison of anonymous versus confidential survey procedures: Effects on health indicators in Dutch adolescents. Journal of Youth and Adolescence, 35, 652–658. doi: http://dx.doi.org/10.1007/s10964-005-9027-0
Vasserman, L. I., Iovlev, B. V., & Chervinskaya, K. R. (2010). Computer psychodiagnostics in the theory and practice of medical psychology: Stages and development prospects. Sibirskii psikhologicheskii zhurnal (Siberian Journal of Psychology), 35, 20–24. (in Russ.).
Vecchione, M., Alessandri, G., & Barbaranelli, C. (2012). Paper-and-pencil and web-based testing: The measurement invariance of the Big Five Personality Tests in applied settings. Assessment, 19(2), 243–246. doi: http://dx.doi.org/10.1177/1073191111419091
Žitný, P., Halama, P., Jelínek, M., & Květon, P. (2012). Validity of cognitive ability tests – comparison of computerized adaptive testing with paper and pencil and computer-based forms of administrations. Studia Psychologica, 54(3), 181–194.