Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough

Narcissism and Social Networking Behavior: A Meta-Analysis

OBJECTIVE

The increasing popularity of social networking sites (SNS) such as Facebook and Twitter has given rise to speculations that the intensity of using these platforms is associated with narcissistic tendencies. However, recent research on this issue has been all but conclusive.

METHOD

We present a three-level, random effects meta-analysis including 289 effect sizes from 57 studies (total N = 25,631) on the association between trait narcissism and social networking behavior.

RESULTS

The meta-analysis identified a small to moderate effect of ρ = .17 (τ = .11), 95% CI [.13, .21], for grandiose narcissism that replicated across different social networking platforms, respondent characteristics, and time. Moderator analyses revealed pronounced cultural differences, with stronger associations in power-distant cultures. Moreover, social networking behaviors geared toward self-presentation and the number of SNS friends exhibited stronger effects than usage durations.

CONCLUSIONS

Overall, the study not only supported but also refined the notion of a relationship between engaging in social networking sites and narcissistic personality traits.

Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?

Websites and online resources outside academic bibliographic databases can be significant sources for identifying literature, though there are challenges in searching and managing the results. These are pertinent to systematic reviews that are underpinned by principles of transparency, accountability and reproducibility. We consider how the conduct of searching these resources can be compatible with the principles of a systematic search. We present an approach to address some of the challenges. This is particularly relevant when websites are relied upon to identify important literature for a review. We recommend considering the process as three stages and having a considered rationale and sufficient recordkeeping at each stage that balances transparency with practicality of purpose. Advances in technology and recommendations for website providers are briefly discussed.

The Implementation of Internet Interventions for Depression: A Scoping Review

BACKGROUND

Depression is one of the most common mental health problems among adults, but effective treatments are not widely accessible. The Internet holds promise as a cost-effective and convenient delivery platform of interventions for depression. However, studies suggest that Internet interventions are not widely available in routine settings.

OBJECTIVE

The aim of this study was to review the literature and examine whether there are systematic differences in reporting of the various implementation components on Internet interventions for depression, and then to examine what is known about and is characteristic of the implementation of these Internet interventions in regular care settings.

METHODS

We performed a scoping review, drawing upon a broad range of the literature on Internet interventions for depression in regular care, and used the active implementation framework to extract data.

RESULTS

Overall, the results suggested that knowledge about the implementation of Internet interventions for depression in regular care is limited. However, guided support from health professionals emphasizing program adherence and recruitment of end users to the interventions emerged as 2 main themes. We identified 3 additional themes among practitioners, including their qualifications, training, and supervision, but these were scarcely described in the literature. The competency drivers (ie, staff and user selection, training, and supervision) have received the most attention, while little attention has been given to organizational (ie, decision support, administration, and system intervention) and leadership drivers.

CONCLUSIONS

Research has placed little emphasis on reporting on the implementation of interventions in practice. Leadership and organizational drivers, in particular, have been largely neglected. The results of this scoping review have implications for future research and efforts to successfully implement Internet interventions for depression in regular care.

Specialist Bibliographic Databases

Specialist bibliographic databases offer essential online tools for researchers and authors who work on specific subjects and perform comprehensive and systematic syntheses of evidence. This article presents examples of the established specialist databases, which may be of interest to those engaged in multidisciplinary science communication. Access to most specialist databases is through subscription schemes and membership in professional associations. Several aggregators of information and database vendors, such as EBSCOhost and ProQuest, facilitate advanced searches supported by specialist keyword thesauri. Searches of items through specialist databases are complementary to those through multidisciplinary research platforms, such as PubMed, Web of Science, and Google Scholar. Familiarizing with the functional characteristics of biomedical and nonbiomedical bibliographic search tools is mandatory for researchers, authors, editors, and publishers. The database users are offered updates of the indexed journal lists, abstracts, author profiles, and links to other metadata. Editors and publishers may find particularly useful source selection criteria and apply for coverage of their peer-reviewed journals and grey literature sources. These criteria are aimed at accepting relevant sources with established editorial policies and quality controls.

Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study

BACKGROUND

Previously, we reported on the low recall of Google Scholar (GS) for systematic review (SR) searching. Here, we test our conclusions further in a prospective study by comparing the coverage, recall, and precision of SR search strategies previously performed in Embase, MEDLINE, and GS.

METHODS

The original search results from Embase and MEDLINE and the first 1000 results of GS for librarian-mediated SR searches were recorded. Once the inclusion-exclusion process for the resulting SR was complete, search results from all three databases were screened for the SR's included references. All three databases were then searched post hoc for included references not found in the original search results.

RESULTS

We checked 4795 included references from 120 SRs against the original search results. Coverage of GS was high (97.2 %) but marginally lower than Embase and MEDLINE combined (97.5 %). MEDLINE on its own achieved 92.3 % coverage. Total recall of Embase/MEDLINE combined was 81.6 % for all included references, compared to GS at 72.8 % and MEDLINE alone at 72.6 %. However, only 46.4 % of the included references were among the downloadable first 1000 references in GS. When examining data for each SR, the traditional databases' recall was better than GS, even when taking into account included references listed beyond the first 1000 search results. Finally, precision of the first 1000 references of GS is comparable to searches in Embase and MEDLINE combined.

CONCLUSIONS

Although overall coverage and recall of GS are high for many searches, the database does not achieve full coverage as some researchers found in previous research. Further, being able to view only the first 1000 records in GS severely reduces its recall percentages. If GS would enable the browsing of records beyond the first 1000, its recall would increase but not sufficiently to be used alone in SR searching. Time needed to screen results would also increase considerably. These results support our assertion that neither GS nor one of the other databases investigated, is on its own, an acceptable database to support systematic review searching.

Socially Desirable Responding in Web-Based Questionnaires: A Meta-Analytic Review of the Candor Hypothesis

Unproctored, web-based assessments supposedly reduce social desirability distortions in self-report questionnaires because of an increased sense of privacy among participants. Three random-effects meta-analyses focusing either on social desirability ( k = 30, total N = 3,746), the Big Five of personality ( k = 66, total N = 2,951), or psychopathology ( k = 96, total N = 16,034) compared social desirability distortions of self-reports across computerized and paper-and-pencil administration modes. Overall, a near-zero effect, Δ = 0.01, was obtained that did not indicate less socially desirable responding in computerized assessments. Moreover, moderator analyses did not identify differential effects for proctored and unproctored procedures. Thus, paper-and-pencil and computerized administrations of self-report scales yield comparable mean scores. Unproctored web-based surveys do not offer an advantage with regard to socially desirable responding in self-report questionnaires.

The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10–67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations’ websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

Atrial fibrillation in the Indigenous populations of Australia, Canada, New Zealand, and the United States: a systematic scoping review

Background

The epidemiology of atrial fibrillation (AF) among Indigenous minorities in affluent countries is poorly delineated, despite the high cardiovascular disease burden in these populations. We undertook a systematic scoping review examining the epidemiology of AF in the Indigenous populations of Australia, Canada, New Zealand (NZ) and the United States (US).

Methods

PubMed, Scopus, EMBASE and CINAHL-Plus databases were systematically searched in May 2014. Supplementary full-text searches of Google Scholar and government website searches were also undertaken.

Results

Key findings from 27 publications with diverse aims and methods were included. Small studies from Canada and NZ suggest higher AF prevalence in Indigenous than other populations. However, this was not reflected in a large sample of US male military veterans. No data were identified on community-based incidence rates of AF in Indigenous populations. Australian and Canadian studies indicate higher first-ever and overall AF hospitalisation rates among Indigenous than other populations, at younger ages and with more comorbidity. Studies in stroke, heart failure and other clinical groups demonstrate AF as a common comorbidity, with AF possibly more prevalent at younger ages in Indigenous people. Indigenous patients have similar early post-hospitalisation adjusted mortality but higher 1-year risk-adjusted mortality than non-Indigenous patients.

Conclusions

No clear epidemiological pattern of AF frequency across the considered Indigenous populations emerges from the limited available evidence. AF should be included in key conditions reported in national surveillance reports, although Indigenous identifiers are required in administrative data from Canada and the US. Sufficiently powered, community-based studies of AF epidemiology in diverse Indigenous populations are needed.

Electronic supplementary material

The online version of this article (doi:10.1186/s12872-015-0081-6) contains supplementary material, which is available to authorized users.

Conducting systematic reviews of intervention questions I: Writing the review protocol, formulating the question and searching the literature

This article is the fourth of six articles addressing systematic reviews in animal agriculture and veterinary medicine. Previous articles in the series have introduced systematic reviews, discussed study designs and hierarchies of evidence, and provided details on conducting randomized controlled trials, a common design for use in systematic reviews. This article describes development of a review protocol and the first two steps in a systematic review: formulating a review question, and searching the literature for relevant research. The emphasis is on systematic reviews of questions related to interventions. The review protocol is developed prior to conducting the review and specifies the plan for the conduct of the review, identifies the roles and responsibilities of the review team and provides structured definitions related to the review question. For intervention questions, the review question should be defined by the PICO components: population, intervention, comparison and outcome(s). The literature search is designed to identify all potentially relevant original research that may address the question. Search terms related to some or all of the PICO components are entered into literature databases, and searches for unpublished literature also are conducted. All steps of the literature search are documented to provide transparent reporting of the process.

Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme

BACKGROUND

Identifying literature for a systematic review requires searching a variety of sources. The main sources are typically bibliographic databases. Web searching using search engines and websites may be used to identify grey literature. Searches should be reported in order to ensure transparency and reproducibility. This study assesses the reporting of web searching for systematic reviews carried out by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme (UK). The study also makes recommendations about reporting web searching for systematic reviews in order to achieve a reasonable level of transparency and reproducibility.

METHODS

Systematic reviews were identified by searching the HTA database via the Centre for Reviews and Dissemination (CRD) website. Systematic reviews were included in the study if they made reference to searching the web using either search engines or websites. A data-extraction checklist was designed to record how web searching was reported. The checklist recorded whether a systematic review reported: the names of search engines or websites; the dates they were searched; the search terms; the results of the searches; and, in the case of websites, whether a URL was reported.

RESULTS

554 HTA reports published between January 2004 and December 2013 were identified. 300 of these reports are systematic reviews, of which 108 report web searching using either a search engine or a website. Overall, the systematic reviews assessed in the study exhibit a low standard of web search reporting. In the majority of cases, the only details reported are the names of websites (n = 54) or search engines (n = 33). A small minority (n = 6) exhibit the highest standard of web search reporting.

CONCLUSIONS

Most web search reporting in systematic reviews carried out on the UK HTA programme is not detailed enough to ensure transparency and reproducibility. Transparency of reporting could be improved by adhering to a reporting standard such as the standard detailed in the CRD systematic reviews methods guidance. Reproducibility is harder to achieve due to the frequency of changes to websites and search engines.

Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments

OBJECTIVES

To compare the effectiveness of two search methods in identifying studies that used the Control Preferences Scale (CPS), a health care decision-making instrument commonly used in clinical settings.

STUDY DESIGN AND SETTING

We searched the literature using two methods: (1) keyword searching using variations of "Control Preferences Scale" and (2) cited reference searching using two seminal CPS publications. We searched three bibliographic databases [PubMed, Scopus, and Web of Science (WOS)] and one full-text database (Google Scholar). We report precision and sensitivity as measures of effectiveness.

RESULTS

Keyword searches in bibliographic databases yielded high average precision (90%) but low average sensitivity (16%). PubMed was the most precise, followed closely by Scopus and WOS. The Google Scholar keyword search had low precision (54%) but provided the highest sensitivity (70%). Cited reference searches in all databases yielded moderate sensitivity (45-54%), but precision ranged from 35% to 75% with Scopus being the most precise.

CONCLUSION

Cited reference searches were more sensitive than keyword searches, making it a more comprehensive strategy to identify all studies that use a particular instrument. Keyword searches provide a quick way of finding some but not all relevant articles. Goals, time, and resources should dictate the combination of which methods and databases are used.

Toward a formalization of the process to select IMIA Yearbook best papers

BACKGROUND

Each year, the International Medical Informatics Association Yearbook recognizes significant scientific papers, labelled as "best papers", published the previous year in the subfields of biomedical informatics that correspond to the different section topics of the journal. For each section, about fifteen pre-selected "candidate" best papers are externally peer-reviewed to select the actual best papers. Although based on the available literature, little is known about the pre-selection process.

OBJECTIVE

To move toward an explicit formalization of the candidate best papers selection process to reduce variability in the literature search across sections and over years.

METHODS

A methodological framework is proposed to build for each section topic specific queries tailored to PubMed and Web of Science citation databases. The two sets of returned papers are merged and reviewed by two independent section editors and citations are tagged as "discarded", "pending", and "kept". A protocolized consolidation step is then jointly conducted to resolve conflicts. A bibliographic software tool, BibReview, was developed to support the whole process.

RESULTS

The proposed search strategy was fully applied to the Decision Support section of the 2013 edition of the Yearbook. For this section, 1124 references were returned (689 PubMed-specific, 254 WoS-specific, 181 common to both databases) among which the 15 candidate best papers were selected.

CONCLUSIONS

The search strategy for determining candidate best papers for an IMIA Yearbook's section is now explicitly specified and allows for reproducibility. However, some aspects of the whole process remain reviewer-dependent, mostly because there is no characterization of a "best paper".

The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews

BACKGROUND

The usefulness of Google Scholar (GS) as a bibliographic database for biomedical systematic review (SR) searching is a subject of current interest and debate in research circles. Recent research has suggested GS might even be used alone in SR searching. This assertion is challenged here by testing whether GS can locate all studies included in 21 previously published SRs. Second, it examines the recall of GS, taking into account the maximum number of items that can be viewed, and tests whether more complete searches created by an information specialist will improve recall compared to the searches used in the 21 published SRs.

METHODS

The authors identified 21 biomedical SRs that had used GS and PubMed as information sources and reported their use of identical, reproducible search strategies in both databases. These search strategies were rerun in GS and PubMed, and analyzed as to their coverage and recall. Efforts were made to improve searches that underperformed in each database.

RESULTS

GS' overall coverage was higher than PubMed (98% versus 91%) and overall recall is higher in GS: 80% of the references included in the 21 SRs were returned by the original searches in GS versus 68% in PubMed. Only 72% of the included references could be used as they were listed among the first 1,000 hits (the maximum number shown). Practical precision (the number of included references retrieved in the first 1,000, divided by 1,000) was on average 1.9%, which is only slightly lower than in other published SRs. Improving searches with the lowest recall resulted in an increase in recall from 48% to 66% in GS and, in PubMed, from 60% to 85%.

CONCLUSIONS

Although its coverage and precision are acceptable, GS, because of its incomplete recall, should not be used as a single source in SR searching. A specialized, curated medical database such as PubMed provides experienced searchers with tools and functionality that help improve recall, and numerous options in order to optimize precision. Searches for SRs should be performed by experienced searchers creating searches that maximize recall for as many databases as deemed necessary by the search expert.