Systematically Searching Empirical Literature in the Social Sciences: Results from Two Meta-Analyses Within the Domain of Education

  • David I. Pickup Centre for the Study of Learning and Performance (CSLP), Concordia University
  • Robert M. Bernard Centre for the Study of Learning and Performance (CSLP), Concordia University
  • Eugene Borokhovski Centre for the Study of Learning and Performance (CSLP), Concordia University https://orcid.org/0000-0002-5615-0417
  • Anne C. Wade Centre for the Study of Learning and Performance (CSLP)
  • Rana M. Tamim College of Education, Zayed University

Аннотация

Introduction. This paper provides an overview of the information retrieval strategy employed for two meta-analyses, conducted by a systematic review team at Concordia University (Montreal, QC, Canada). Both papers draw on standards first articulated by H.M. Cooper and further developed by the Campbell Collaboration, which promote a comprehensive approach to systematically searching an extensive array of resources (bibliographic databases, print resources, citation indices, etc.) in order to locate both published and unpublished research. The goal is to verify if searching comprehensively through multiple resources retrieves studies that are unique, and hence, improve the overall representativeness of a diverse body of literature. We also analyze the sensitivity and specificity of the results by data source.

Methods. In order to determine the source sensitivity, we consider percentage of results from each source retrieved for full-text review. In order to determine the source specificity, we derive a percentage from the total number of studies included in the final meta-analysis compared against the overall number of initial results found.

Results. Results demonstrate the need to search beyond the subject-specific databases of a particular discipline as unique results can be found in many places. Databases for related disciplines provided 129 unique includes to each meta-analysis, and multidisciplinary databases provided 44 and 99 unique includes for the two meta-analyses in question respectively. Manual search techniques were much more sensitive and specific than electronic searches of databases and yield a higher percentage of final includes.

Discussion. The results demonstrate the utility of a comprehensive information retrieval methodology like that proposed by the Campbell Collaboration, which goes beyond the main subject databases to locate the full range of information sources, including grey literature.

Скачивания

Данные скачивания пока не доступны.

Биография автора

Eugene Borokhovski, Centre for the Study of Learning and Performance (CSLP), Concordia University

доктор психологических наук, доцент факультета психологии (исследовательское направление), университета Конкордия, Монреаль, Канада

Руководитель Мета-Аналитических проектов, научно-исследовательский центр изучения когнитивных процессов в образовании (CSLP) университета Конкордия:

1515 St. Catherine Street West

Room S-GA-2.126

Montreal, Quebec, H3G 1W1, Canada

Литература

Abrami P. C., Bernard R. M., Borokhovski E., Waddington D. I., Wade C. A., & Persson T. Strategies for Teaching Students to Think Critically: A Meta-Analysis. Review of Educational Research, 2015, V. 85, Issue 2, pp. 275–314. DOI: 10.3102/0034654314551063

Schmid R. F., Bernard R. M., Borokhovski E., Tamim R. M., Abrami P. C., Surkes M. A., Wade C. A., Woods J. The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers & Education, 2014, V. 72, pp. 271–291. DOI: 10.1016/j.compedu.2013.11.002

Cooper H. M. The integrative research review: A systematic approach. Thousand Oaks, CA, Sage Publications, 1984. 144 p.

Cooper H. M. Synthesizing research: A guide for literature reviews. 5th ed. Thousand Oaks, CA, Sage Publications, 2017.

Hammerstrøm K., Wade A., Jørgensen A.-M. K. Searching for studies: A guide to information retrieval for Campbell systematic reviews. Oslo, The Campbell Collaboration, 2010.

Kugley S., Wade A., Thomas J., Mahood Q., Jørgensen A.-M. K., Hammerstrøm K. T., & Sathe N. Searching for studies: A guide to information retrieval for Campbell systematic reviews. Oslo, The Campbell Collaboration, 2017. DOI: 10.4073/cmg.2016.1

Glass G. V. Primary, Secondary and Meta-Analysis of Research. Educational Researcher, 1976, V. 5, Issue 10, pp. 3–8. DOI: 10.3102/0013189X005010003

Glass G. V., McGaw B., & Smith M. L. Meta-analysis in social research. Beverly Hills, CA, Sage Publications, 1981.

Lipsey M. W., Wilson D. B. Practical meta-analysis. Thousand Oaks, CA, Sage, 1990.

Borenstein M., Hedges L., Higgins J., & Rothstein H. Introduction to meta-analysis. Chichester, UK, Wiley, 2009.

Lefebvre C., Manheimer E., & Glanville J. Searching for studies. In J. P. T. Higgins & S. Green (eds.) Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0, Chap. 6). The Cochrane Collaboration, 2011. Retrieved from http://handbook.cochrane.org

Cooper H., Hedges L. V., & Valentine J. C. (eds.) The Handbook of Research Synthesis and Meta-Analysis (2nd ed.). New York, Russell Sage Foundation, 2009. 632 p.

Pettigrew M., & Roberts H. Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA, John Wiley & Sons, 2008.

Auger C. P. Use of reports literature. London, Butterworths, 1975. 226 p.

Auger C. P. Information Sources in Grey Literature (4th ed.). Boston, MA, Walter de Gruyter GmbH, 2017. 177 p.

Conn V. S., Valentine J. C., Copper H. M., & Rantz M. J. Grey Literature in Meta-Analyses. Nursing Research, 2003, V. 52, Issue 4, pp. 256–261.

Polanin J. R., Tanner-Smith E. E., & Hennessy E. A. Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review. Review of Educational Research, 2016, V. 86, Issue 1, pp. 207–236. DOI: 10.3102/0034654315582067

Rothstein H. R., Sutton A. J., & Borenstein M. (eds.) Publication bias in meta-analysis – prevention, assessment and adjustments. Chichester, UK, Wiley, 2005. 356 p.

Swinkels A., Briddon J., & Hall J. Two physiotherapists, one librarian and a systematic review: collaboration in action. Health Information and Libraries Journal, 2006, V. 23, Issue 4, pp. 248–256. DOI: 10.1111/j.1471-1842.2006.00689.x

Harris M. R. The librarian’s roles in the systematic review process: a case study. Journal of the Medical Library Association, 2005, V. 93, Issue 1, pp. 81–87.

DeLuca J. B., Mullins M. M., Lyles C. M., Crepaz N., Kay L., & Thadiparthi S. Developing a Comprehensive Search Strategy for Evidence Based Systematic Reviews. Evidence Based Library and Information Practice, 2008, V. 3, № 1, pp. 3–32. DOI: 10.18438/B8KP66

Helmer D., Savoie I., Green C., & Kazanjian A. Evidence-based practice: extending the search to find material for the systematic review. Bulletin of the Medical Library Association, 2001, V. 89, Issue 4, pp. 346–352.

Booth A. “Brimful of STARLITE”: toward standards for reporting literature searches. Journal of the Medical Library Association, 2006, V. 94, Issue 4, pp. 421–429.

Zhang L., Sampson M., & McGowan J. Reporting of the Role of the Expert Searcher in Cochrane Reviews. Evidence Based Library and Information Practice, 2006, V. 1, № 4, pp. 3–16. DOI: 10.18438/B85K52

Golder S., Loke Y., & McIntosh H. M. Room for improvement? A survey of the methods used in systematic reviews of adverse effects. BMC Medical Research Methodology, 2006, 6 (3). DOI: 10.1186/1471-2288-6-3

Yoshii A., Plaut D. A., McGraw K. A., Anderson M. J., & Wellik K. E. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association, 2009, 97 (1), pp. 21–29. DOI: 10.3163/1536-5050.97.1.004

Wade C. A., Turner H. M., Rothstein H. R., & Lavenberg J. G. Information retrieval and the role of the information specialist in producing high-quality systematic reviews in the social, behavioural and education sciences. Evidence & Policy, 2006, V. 2, № 1, pp. 89–108. DOI: 10.1332/174426406775249705

Bernard R. M. Things I Have Learned about Meta-Analysis Since 1990: Reducing Bias in Search of “The Big Picture”. Canadian Journal of Learning and Technology, 2014, V. 40, № 3. DOI: 10.21432/T2MW29

Bernard R. M., Borokhovski E., Schmid R. F., & Tamim R. M. An exploration of bias in meta-analysis: The case of technology integration research in higher education. Journal of Computing in Higher Education, 2014, V. 26, Issue 3, pp. 183–209. DOI: 10.1007/s12528-014-9084-z

Bernard R. M., Borokhovski E., & Tamim R. M. Detecting bias in meta-analyses of distance education research: big pictures we can rely on. Distance Education, 2014, V. 35, Issue 3, pp. 271–293. DOI: 10.1080/01587919.2015.957433

Watson G., & Glaser E. M. Watson-Glaser critical thinking appraisal. San Antonio, TX, PsychCorp, 1980.

Ennis R. H., & Millman J. Cornell critical thinking test. Pacific Grove, CA, Critical Thinking Books & Software, 1985.
Опубликован
2019-01-31
Как цитировать
Pickup, D., Bernard, R., Borokhovski, E., Wade, A., & Tamim, R. (2019). Systematically Searching Empirical Literature in the Social Sciences: Results from Two Meta-Analyses Within the Domain of Education. Российский психологический журнал, 15(4), 245-265. https://doi.org/10.21702/rpj.2018.4.10
Раздел
Теория и методика профессионального образования