ZU Scholars ZU Scholars

Introduction . This paper provides an overview of the information retrieval strategy employed for two meta-analyses, conducted by a systematic review team at Concordia University (Montreal, QC, Canada). Both papers draw on standards first articulated by H.M. Cooper and further developed by the Campbell Collaboration, which promote a comprehensive approach to systematically searching an extensive array of resources (bibliographic databases, print resources, citation indices, etc.) in order to locate both published and unpublished research. The goal is to verify if searching comprehensively through multiple resources retrieves studies that are unique, and hence, improve the overall representativeness of a diverse body of literature. We also analyze the sensitivity and specificity of the results by data source. Methods . In order to determine the source sensitivity, we consider percentage of re-sults from each source retrieved for full-text review. In order to determine the source specificity, we derive a percentage from the total number of studies included in the final meta-analysis compared against the overall number of initial results found. Results . Results demonstrate the need to search beyond the subject-specific databases of a particular discipline as unique results can be found in many places. Databases for related disciplines provided 129 unique includes to each meta-analysis, and multidisciplinary databases provided 44 and 99 unique includes for the two meta-analyses in question respectively. Manual search techniques were much more sensitive and specific than electronic searches of databases and yield a higher percentage of final includes. Discussion . The results demonstrate the utility of a comprehensive information retrieval methodology like that proposed by the Campbell Collaboration


Introduction
This paper will provide an overview of the information retrieval strategy employed for two large-scale meta-analyses within the domain of Education, conducted by a systematic review team at Concordia University [1,2].The team, in consultation with library professionals, has drawn on standards first articulated by H.M. Cooper [3,4] and further developed by the Campbell Collaboration [e.g., 5,6], which promote a comprehensive approach -by systematically searching an extensive array of resources (bibliographic databases, search engines, print resources, citation indices, etc.), using detailed strategies tailored to make maximum use of the features of each resource, in an attempt to locate both published and unpublished research.The results demonstrate the utility of a comprehensive information retrieval methodology that goes beyond the main literature databases to locate the full range of information sources, including so-called 'grey literature' .

Theoretical Framework
When G.V. Glass [7] introduced the concept of meta-analysis, he conceived of the research paradigm as an "analysis of analyses" that would offer a statistical examination of a large collection of studies so that their results could be integrated and a clearer view of the overall picture properly understood and presented.Rather than "casual, narrative discussions" the resulting publication would be a genuine attempt to make sense of an ever-expanding and often conflicting information landscape.A meta-analysis therefore is a specific class of systematic review that relies on quantitative data from primary studies addressing a common core research question.Meta-analysis summarizes systematically collected effect sizes from individual studies to estimate either the average magnitude of the difference on a common dependent variable between a treatment group and an alternative group (d-family effect size) or degree of association between variables of interest (r-family effect size) in the entire population (or a large-scale sample), in question and then tries to explain the variability that surrounds the overall effect size by systematically coding and analyzing methodological, substantive, and demographic moderator variables.The main research question (or group of related questions) should be stated and substantiated a-priori to inform search strategies, to set up and describe inclusion criteria, and to meaningfully guide the review process through all its steps [e.g., 4] -from information retrieval to study selection, through effect size extraction, aggregation, and analyses toward interpretation and presentation of the findings.
This paper focuses on one step in Cooper's process, the literature search stage.When first articulating the methodology of meta-analysis, G.V. Glass [7] said little about the best practices and methods to employ in searching the literature, however as the methodology of meta-analysis became more formalized, more detailed standards were developed [3,8,9,10].The work of two international organizations, the Cochrane Collaboration, and the Campbell Collaboration, further helped develop standards for the conduct of systematic reviews and meta-analyses, including for searching the literature [6,11].Essential components of these standards are systematicity, replicability and comprehensiveness of literature searches.
Systematicity demands a well-planned strategy, determined as part of the research question formulation, for how to proceed to gather all existing evidence, or the entire population of studies [4].This includes such steps as keyword formulation, informed by discipline-specific reference sources such as dictionaries and encyclopedias, as well as database thesauri, as well as determining which selection of resources to use.Once carefully planned, the strategy can then be carried out in a systematic fashion.
H.M. Cooper [4] notes that researchers conducting reviews will have varying degrees of resources available (for example, the bibliographic databases that their institutions subscribe to), but that by searching broadly using various strategies, they will produce data sets that arrive at the same overall conclusions, and the CC BY 4.0 ТЕОРИЯ И МЕТОДИКА ПРОФЕССИОНАЛЬНОГО ОБРАЗОВАНИЯ standard of replicability can be met.The standard of replicability can further be enhanced if strategies are recorded and documented so that they can be reviewed and judgments made about their potential impact on the overall review or meta-analysis.This 'search history' should also note the date searches were conducted to make clear any gaps that have since developed in the literature.
When aiming for comprehensiveness, it is important to find a proper balance between recall and precision [12], or in other words, sensitivity and specificity [13].A strategy that emphasizes sensitivity will yield a larger quantity of results by searching a greater number of sources and targeting a greater array of synonymous terms that may also be relevant.A strategy that aims for specificity will result in fewer, but more likely to be relevant results.In truth, these two concepts are not dichotomous and the best strategies aim to find a good balance between both sensitivity and specificity, for example by employing a good mix of both the OR and AND operators in a Boolean logic-driven search.

Study Identification and Retrieval
H.M. Cooper [4] described a broad strategy to identify all relevant studies, which made use of various tactics using what he termed informal and formal channels.The former refers to personal contacts and approaches to research communities (what he termed 'invisible colleges') as well as browsing related websites using the WWW.The latter referred to searching library catalogues and databases, as well as conference proceedings, and included browsing reference lists of identified studies.
The search should attempt to locate both formally published research, usually in the form of journal articles, and research published in less traditional forms such as locally generated technical and evaluation studies (e.g. at the school or community level), government commissioned reports, theses, and unpublished manuscripts -what is generally classified as "grey literature" [e.g.14,15].Although grey literature has often not undergone any peer review process, it may be considered a necessary counterbalance [16] to formally published materials, which demonstrate a tendency show greater statistical significance and higher effect sizes, i.e. 'publication bias' [17,18].
Due to the complicated and variated requirements of a systematic and comprehensive search strategy that balances sensitivity and specificity, as well as the retrieval of identified studies, many systematic review teams seek outside help from an information specialist or search professional.

The Role of the Librarian
Many researchers have drawn on the expertise of librarians to assist with the search and information retrieval stage of systematic review and meta-analysis projects, as their expertise with both electronic and manual search techniques coincides nicely with the requirements of the methodology [e.g.19,20,21,22].Evidence suggests that librarian involvement produces a marked improvement on the quality of the review; A. Booth [23] examined the qualitative reviews found in Medline and noted that those explicitly involving a librarian in the process had the largest number of databases searched (thus improving the scope of outreach).L. Zhang, M. Sampson and J. McGowan [24] found that it was notably easier to assess the quality of search strategies in reviews where a librarian had been involved as they were more likely to include detailed reporting and take personal responsibility.Likewise, S. Golder, Y. Loke and H.M. McIntosh [25] report that while few of the searches they analyzed were reported with enough description for the search to be replicated, nearly half of those that could be had been conducted by a librarian.
Librarians have also been at the forefront of attempts to identify areas for improvement in systematic review standards [24,26] and to formulate best practices and guidelines [e.g., 5,27].These standards, if respected, will lead to better quality and less publication bias [18,28].
The systematic review team of the Centre for the Study of Learning & Performance (CSLP) has for many years included a dedicated librarian on staff.The librarians employed over the years have assisted with question formulation through scoping of the literature, conducted literature searches following methods closely aligned with those promulgated by the Campbell Collaboration [5,6], tracked and managed the retrieved studies, assisted with coding, and served as co-authors on final reports.The review team has always advocated for a high standard of methodological quality in meta-analysis [29,30].This report, in turn, is intended to test a high standard in information retrieval methodology.

Methods
The following section reports the information retrieval strategies and results from two recent publications that were each the culmination of projects that stretched over several years and synthesized a large body of research.The first investigated critical thinking interventions (CT) and the second looked at the effectiveness of classroom technology integration at the post-secondary level (PedTech).Both reviews investigated areas of research that spanned a variety of academic disciplines and sectors.
A comprehensive approach best articulated in the Campbell Collaboration's Information Retrieval Policy Brief [5,6] was adopted to search widely, using a diversity of resources and methods, namely: 1) Subject databases; 2) Multidisciplinary databases; CC BY 4.0 ТЕОРИЯ И МЕТОДИКА ПРОФЕССИОНАЛЬНОГО ОБРАЗОВАНИЯ 3) Citation Indexes; 4) Web searching; 5) Branching (Hand searching).These strategies were designed to locate publicly available published and unpublished literature of assorted publication types (i.e.articles, reports, conference papers, manuscripts, dissertations & theses, etc.) and provide the most representative (i.e., unbiased) picture of available research evidence.The goal of this paper is to verify if searching comprehensively through multiple resources in various fields retrieves studies that are unique (not be found anywhere else), and hence, improve the overall replicability [4] and representativeness of a diverse body of literature.We will also analyze the specificity and sensitivity of the results by data source (i.e. by database, hand searching, etc.) to see how well they performed in each review and if any conclusions might be drawn.To estimate the sensitivity of an information source, we will determine what percentage of the results found in the source were retrieved for closer full-text examination.To estimate the specificity of an information source we will divide the total number of studies from a source that are included in the final meta-analysis by the total number of initial results found in that source.
Before looking at the various data sources used and how they performed, we shall review the strategies employed for each review and provide a summary of the results.To build the search strategy, keywords used were divided into two main concepts (Domain and Method).In some searches, a third concept (Context) was added.Searches were not limited to a particular population.Search strategies were customized for each database using a combination of controlled vocabulary and natural language terms.

Domain:
Critical Thinking, Thinking Skills.Method: Experiment, Studies, Intervention, Treatment, Control Group, Post test.Context: Education, Student, Learning, Teaching.Terms were combined within sets using the Boolean operator OR, and the sets themselves were combined using the AND operator (Figure 1).The databases selected can be classified into three groups: main subject databases, related subject databases, and multidisciplinary databases.
Main Approximately sixty review articles and previous meta-analyses were used for "branching" (their bibliographies were scanned for other relevant studies).A citation search was also conducted on many of these same review articles using the Web of Science database to locate publications that had cited them; citations searches were also conducted on the main CT tests [e.g.31].
Pedagogical Technology (PedTech) R.F.Schmid, R.M. Bernard, E. Borokhovsi, R.M. Tamim, P.C. Abrami, M.A. Surkes, C.A. Wade and J. Woods [2] performed a meta-analysis that reviewed primary research addressing the impact of computer technology, whether face-to-face or blended, on students' achievement, performance, or attitudes.The population was limited to post-secondary formal education.Results were limited to post-1990 to capture modern Internet-era technologies.The review reports the overall weighted average effects of using technology on the academic achievement and attitudes of students, while exploring moderator variables in an attempt to offer an explanation for how the technology interventions lead to positive or negative effects.The search strategy was broadly based and retrieved a total set of nearly 12,000 abstracts for review.Of these, 1105 were chosen for further full-text review, and produced 879 achievement effect sizes and 181 attitudinal effect sizes.
Similar to the CT review, search strategies were customized for each database using a combination of controlled vocabulary and natural language terms.Key concepts used in search strategies were grouped into three sets -domain, population and outcome.

THEORY AND METHODS OF PROFESSIONAL EDUCATION
Terms were combined within sets using the Boolean operator OR, and the sets themselves were combined using the operator AND (Figure 2).The main subject domain (educational technology) was quite broad and synonyms such as 'electronic' and 'computer' can appear in many different contexts.Therefore, wherever possible database-specific descriptors were used.In the case of some databases the NOT operator was used to exclude studies pertaining to "distance education" in the descriptor field (see Figure 3 for an example from ERIC).
Review articles and previous meta-analyses were used for "branching" and the tables of content of recent issues of major journals in the field of educational technology were manually scanned for additional studies.Further, a number of online-only e-journals in the subject area of educational technology had been identified in the Google searches and these were also browsed.

Results
Tables 1 and 2 provide the raw totals from each source for the CT and PedTech reviews respectively (including duplicates found in more than one source), of those, how many were retrieved for full text review, how many were included in the final analysis, and how many of the includes were found uniquely in that source and no other.

Analyzing the data sources
Using the datasets provided by these two large-scale meta-analyses, we shall now take a closer look at the results breakdown to ascertain the overall sensitivity and specificity of the various information sources.In order to get an idea of the sensitivity of each source, it may be informative to consider what percentage of studies from each source was retrieved for full-text review.Tables 3 and 4 provide a breakdown for each project on how the various information sources used compare as a percentage of the total number of results found by the search.Next, in order to focus on the specificity of each data source, we consider how each source compares in terms of total included studies (both uniquely discovered and duplicates) compared against the overall number of initial results.

Discussion
The results from an analysis of the information retrieval methods used in the P.C. Abrami, R.M. Bernard, E. Borokhovski, D.I. Waddington, C.A. Wade and T. Persson [1] and R.F.Schmid, R.M. Bernard, E. Borokhovsi, R.M. Tamim, P.C. Abrami, M.A. Surkes, C.A. Wade and J. Woods [2] meta-analyses demonstrate the critical importance of using a comprehensive approach to information retrieval.To begin with, these results show the need to search beyond the subject-specific discipline of your research question as unique results can be found in many different places.For the CT review, databases of related fields (not Education) provided 129 studies included in the final analysis not found in the subject databases or elsewhere, and the multidisciplinary databases (excluding ProQuest Dissertations & Theses) provided an additional 44 unique includes.For PedTech, the related field databases coincidentally also yielded 129 unique included studies, and the multidisciplinary databases (excluding ProQuest Dissertations & Theses) provided a further 99.The strategy to 'cast a broad net' and search in many databases ultimately proves warranted by the inclusion of studies in the final meta-analysis not found anywhere else.In the CT meta-analysis, the ABI/Inform Global database did not produce very many relevant results, with only 23 of the original 219 warranting full-text review -however the two studies included in the final meta-analysis were not found in any other database.Likewise, the Index to Thesis database (a collection of mainly British dissertations) produced three uniquely found includes out of its original 81 results.The same can be seen in the PedTech review with the Australian Education Index, despite only 19 out of the original 492 results making their way into the final meta-analysis, these 19 were all uniquely found in that database.
Also of particular note was the success in both cases of the 'manual' strategies -"branching" reference lists of previous reviews and key studies, as well as scanning recent issues of important journals (in the case of PedTech these were also supplemented with browsing more obscure e-journals).In the CT review these manual strategies located a combined 30 studies included in the final analysis that were not found elsewhere, and in PedTech the combined result was 177 unique includes.Perhaps not surprisingly given the human element in selection of results, the manual searches also proved more sensitive and specific than electronic searches of databases, yielding a higher percentage of final includes.
Also of interest was the performance of searches conducted using the Google and Yahoo search engines; in both cases the searches of Yahoo resulted in more unique studies not found elsewhere -11 studies for CT and 36 for PedTech.Please note that, at the time the Yahoo searches were conducted, their search was managed in-house (prior to 2004 search results were provided by Google CC BY 4.0 ТЕОРИЯ И МЕТОДИКА ПРОФЕССИОНАЛЬНОГО ОБРАЗОВАНИЯ and since 2009 they have been provided by Bing).Although academics are sometimes biased against using popular search engines for research, they do yield results not found elsewhere.Lastly, dissertations have also proved to be a fruitful source of analyzable results, especially in the case of the CT review, and a search of ProQuest's flagship Dissertations & Theses database is essential.
Overall, the results demonstrate the utility of a comprehensive search and information retrieval methodology which goes beyond the main subject databases to locate the full range of information sources, including grey literature from sources like the Web, as well as manual searches of conference proceedings and specialist collections.Further, while this paper reports the results of the methodological approach to systematic searching within the domain of Education, its principles are applicable to the Social Sciences more broadly, including Psychology, Sociology and others.Indeed, critical thinking (the subject focus of one of the two described reviews) is a cross-disciplinary concept.This same rigorous and systematic approach may be extended into primary research as well, with the same methods employed when writing literature review sections of empirical papers or standard narrative reviews.Researchers working at institutions that do not have access to a great array of bibliographic databases can take some comfort in the performance of the more 'manual' strategies -web searching and browsing online resources, i.e.EdITLib (now LearnTechLib.com),and conference websites.They may also wish to pursue international partnerships and/or obtain the services of an Information Specialist to conduct searches for their review projects.

Conclusions
This analysis provides some validation of the information retrieval standards promoted by the Campbell Collaboration [5,6].The results demonstrate that many relevant studies may be found using a diversity of retrieval methods and resources, which has significance for primary research as well as for the conduct of meta-analyses and systematic reviews in Education and the Social Sciences more broadly.The systematic reviewer should make every possible effort to find the available studies in order to provide as unbiased a result as possible and increase replicability [4].Information retrieval within a systematic review or meta-analysis is not a one-shot deal; it requires considerable expertise, time, and resources and researches may wish to consider consulting with a librarian when formulating their strategy.Ultimately, drawing on the full body of research available on a given research topic, and not simply the easiest retrie vable information, will provide a solid foundation to ensure a high quality review of the evidence.
Critical Thinking (CT) P.C. Abrami, R.M. Bernard, E. Borokhovski, D.I. Waddington, C.A. Wade and T. Persson[1] conducted a review of studies on the development and enhancement of critical thinking skills and dispositions with a link to student achievement; it began with the research question, "What impact do instructional interventions have on the development of students' CT skills and dispositions?"The final dataset contained 341 effect sizes from experimental research (quasi-or true-experiments) that used a standardized test for critical thinking skills as an outcome such as the Watson-Glaser Critical Thinking Appraisal[31] or the Cornell Critical Thinking Test[32].In addition, the review examined how different types of instructional interventions affect CT skills and dispositions, what impact pedagogical background (e.g., instructor training) has, and how calculated effect sizes vary with age (educational level), subject matter, and treatment duration.

Figure 3 .
Figure 3. PedTech controlled vocabulary sampleThe following databases were searched: Subject databases: ► Australian Education Index (https://www.acer.org/au/library/australian-education-index-aei); CC BY 4.0259 THEORY AND METHODS OF PROFESSIONAL EDUCATION Web of Science (www.webofknowledge.com/).The primary tools used for retrieval of grey literature were Yahoo (http:// yahoo.com)and Google (http://www.google.com); a series of searches were run using different combinations of keywords and the first 200 results of each were browsed.As a further step, the 'open access' library OAIster (https://www.oclc.org/en/oaister.html)was searched, as was the Ed/ITLib digital library (https:// www.editlib.org).