Abstract
Introduction. Chatbots based on generative artificial intelligence (AI) have rapidly gained popularity and are increasingly influencing various aspects of daily life. An important component of the user experience is engagement, which reflects the depth and intensity of a person’s interaction with AI systems. The aim of this study was to adapt the User Engagement Scale (UES) to assess engagement among Russian-speaking users interacting with generative AI–based chatbots.
Methods. The study involved 210 respondents aged 18–60. The linguistic adaptation procedure included forward and backward translation, expert review, and a focus group. To evaluate the psychometric properties of the scale, exploratory and confirmatory factor analyses were conducted, along with assessments of test–retest reliability, convergent and divergent validity.
Results. Exploratory factor analysis identified a four-factor structure—positive interaction experience, engagement (immersion), interface appeal, and interaction difficulties—explaining 74.3% of the variance. Confirmatory factor analysis supported the adequacy of the proposed model. Cronbach’s alpha (0.83) and test–retest reliability (r = 0.81) indicated high stability of the instrument. Convergent validity was demonstrated by strong correlations with perceived usability (r = 0.823) and absorption by activity (r = 0.834), while divergent validity indicated weak correlations with negative affect and life satisfaction.
Discussion. The adapted version of the scale retains its theoretical foundation and accurately measures key aspects of user engagement. Interaction with generative AI chatbots emphasizes positive experience and immersion, whereas interaction difficulties have a smaller impact due to intuitive interfaces and the ability of AI systems to imitate interpersonal communication. The Russian version of the instrument demonstrated high reliability and validity and can be used in further studies of human–AI interaction.
References
Валуева, Е. А., Панфилова, А.С., & Рафикова, A. С. (2024). Автоматическая оценка тестов на вербальную креативность: от лексических баз данных к большим языковым моделям. Психология. Журнал Высшей школы экономики, 21(1), 202–225. https://doi.org/10.17323/1813-8918-2024-1-202-225
Золотарева, А. А. (2020). Диагностика предрасположенности к скуке: адаптация русскоязычной версии BPS-SR. Национальный психологический журнал, 13(1), 40–49. https://doi.org/10.11621/npj.2020.0104
Осин, Е.Н. (2012). Измерение позитивных и негативных эмоций: разработка русскоязычного аналога методики PANAS. Психология. Журнал Высшей школы экономики, 9(4), 91–110.
Осин, Е. Н., & Леонтьев, Д. А. (2008). Апробация русскоязычных версий двух шкал экспресс-оценки субъективного благополучия. Сборник материалов III Всероссийского социологического конгресса [Электронный ресурс]. Институт социологии РАН, Российское общество социологов. https://www.isras.ru/abstract_bank/1210190841.pdf
Akbar, M. A., Khan, A. A., & Liang, P. (2024). Ethical Aspects of ChatGPT in Software Engineering Research. IEEE Transactions on Artificial Intelligence, 1–14. https://doi.org/10.1109/TAI.2023.3318183
Al Lily, A. E., Ismail, A. F., Abunaser, F. M., Al-Lami, F., & Abdullatif, A. K. A. (2023). ChatGPT and the rise of semi-humans. Humanities and Social Sciences Communications, 10(1), 626. https://doi.org/10.1057/s41599-023-02154-3
Almufarreh, A. (2024). Determinants of Students’ Satisfaction with AI Tools in Education: A PLS-SEM-ANN Approach. Sustainability, 16(13), 5354. https://doi.org/10.3390/su16135354
Andrade-Arenas, L., Yactayo-Arias, C., & Pucuhuayla-Revatta, F. (2024). Therapy and Emotional Support through a Chatbot. International Journal of Online and Biomedical Engineering (iJOE), 20(02), 114–130. https://doi.org/10.3991/ijoe.v20i02.45377
Arghashi, V., & Yuksel, C. A. (2022). Interactivity, Inspiration, and Perceived Usefulness! How retailers’ AR-apps improve consumer engagement through flow. Journal of Retailing and Consumer Services, 64, 102756. https://doi.org/10.1016/j.jretconser.2021.102756
Avila-Garzon, C., Bacca-Acosta, J., & Chaves-Rodríguez, J. (2023). Predictors of Engagement in Virtual Reality Storytelling Environments about Migration. Applied Sciences, 13(19), 10915. https://doi.org/10.3390/app131910915
Bin-Nashwan, S. A., Sadallah, M., & Bouteraa, M. (2023). Use of ChatGPT in academia: Academic integrity hangs in the balance. Technology in Society, 75, 102370. https://doi.org/10.1016/j.techsoc.2023.102370
Bragazzi, N. L., Crapanzano, A., Converti, M., Zerbetto, R., & Khamisy-Farah, R. (2023). The Impact of Generative Conversational Artificial Intelligence on the Lesbian, Gay, Bisexual, Transgender, and Queer Community: Scoping Review. Journal of Medical Internet Research, 25, e52091. https://doi.org/10.2196/52091
Cha, S., Kim, C. Y., & Tang, Y. (2024). Gamification in the Metaverse: Affordance, perceived value, flow state, and engagement. International Journal of Tourism Research, 26(2), e2635. https://doi.org/10.1002/jtr.2635
Engeser, S., & Rheinberg, F. (2008). Flow, performance and moderators of challenge-skill balance. Motivation and Emotion, 32(3), 158–172. https://doi.org/10.1007/s11031-008-9102-4
Fergencs, T., & Meier, F. (2021). Engagement and Usability of Conversational Search – A Study of a Medical Resource Center Chatbot. In K. Toeppe, H. Yan, & S. K. W. Chu (Eds.), Diversity, Divergence, Dialogue (Vol. 12645, pp. 328–345). Springer International Publishing. https://doi.org/10.1007/978-3-030-71292-1_26
Gupta, R., Nair, K., Mishra, M., Ibrahim, B., & Bhardwaj, S. (2024). Adoption and impacts of generative artificial intelligence: Theoretical underpinnings and research agenda. International Journal of Information Management Data Insights, 4(1), 100232. https://doi.org/10.1016/j.jjimei.2024.100232
Israfilzade, K. (2023). The Role of Generative AI and Anthropomorphism in Shaping Conversational Marketing: Creating a Matrix for Future Research. The Eurasia Proceedings of Educational and Social Sciences, 32, 132–142. https://doi.org/10.55549/epess.1412832
Kronemann, B., Kizgin, H., & Rana, N. (2022). The “Other” Agent: Interaction with AI and Its Implications on Social Presence Perceptions of Online Customer Experience. In S. Papagiannidis, E. Alamanos, S. Gupta, Y. K. Dwivedi, M. Mäntymäki, & I. O. Pappas (Eds.), The Role of Digital Technologies in Shaping the Post-Pandemic World (Vol. 13454, pp. 70–81). Springer International Publishing. https://doi.org/10.1007/978-3-031-15342-6_6
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: When there’s no time for the SUS. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2099–2102. https://doi.org/10.1145/2470654.2481287
O’Brien, H. (2016). Theoretical Perspectives on User Engagement. In H. O’Brien & P. Cairns (Eds.), Why Engagement Matters (pp. 1–26). Springer International Publishing. https://doi.org/10.1007/978-3-319-27446-1_1
O’Brien, H. L., Cairns, P., & Hall, M. (2018). A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies, 112, 28–39. https://doi.org/10.1016/j.ijhcs.2018.01.004
O’Brien, H. L., & Lebow, M. (2013). Mixed‐methods approach to measuring user experience in online news interactions. Journal of the American Society for Information Science and Technology, 64(8), 1543–1556. https://doi.org/10.1002/asi.22871
Orrù, G., Piarulli, A., Conversano, C., & Gemignani, A. (2023). Human-like problem-solving abilities in large language models using ChatGPT. Frontiers in Artificial Intelligence, 6, 1199350. https://doi.org/10.3389/frai.2023.1199350
Park, G., Chung, J., & Lee, S. (2023). Effect of AI chatbot emotional disclosure on user satisfaction and reuse intention for mental health counseling: A serial mediation model. Current Psychology, 42(32), 28663–28673. https://doi.org/10.1007/s12144-022-03932-z
Shete, S. (2023). AI in Cybersecurity and User Interface Design beyond Chatbots. Journal of Artificial Intelligence & Cloud Computing, 1–4. https://doi.org/10.47363/JAICC/2023(2)164
Tsai, W.-H.S., Liu, Y. and Chuan, C.-H. (2021), "How chatbots' social presence communication enhances consumer engagement: the mediating role of parasocial interaction and dialogue", Journal of Research in Interactive Marketing, Vol. 15 No. 3, pp. 460-482. https://doi.org/10.1108/JRIM-12-2019-0200
Van De Vijver, F., & Hambleton, R. K. (1996). Translating Tests. European Psychologist, 1(2), 89–99. https://doi.org/10.1027/1016-9040.1.2.89
Von Garrel, J., & Mayer, J. (2023). Artificial Intelligence in studies–Use of ChatGPT and AI-based tools among students in Germany. Humanities and Social Sciences Communications, 10(1), 799. https://doi.org/10.1057/s41599-023-02304-7
Wang, T., Wang, D., Li, B., Ma, J., Pang, X. S., & Wang, P. (2023). The Impact of Anthropomorphism on Chatgpt Actual Use: Roles of Interactivity, Perceived Enjoyment, and Extraversion. SSRN. https://doi.org/10.2139/ssrn.4547430
Zheng, Y., Li, Y., Shi, N., Sun, X., & Pan, Y. (2024). Neither more nor less: Understanding positive emotion of posts and user engagement on tourism social media. Asia Pacific Journal of Tourism Research, 29(6), 736–752. https://doi.org/10.1080/10941665.2024.2342366
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2025 Russian Psychological Journal