Purpose of Study
The objectives of this study were to compare the benefits and tradeoffs of searches with and without the use of text-mining tools (TMTs) for evidence synthesis products in real world settings. Specific questions included: (1) Do TMTs decrease the time spent developing search strategies? (2) How do TMTs affect the sensitivity and yield of searches? (3) Do TMTs identify groups of records that can be safely excluded in the search evaluation step? (4) Does the complexity of a systematic review topic affect TMTs performance? In addition to quantitative data, we collected librarians’ comments on their experiences using TMTs to explore when and how these new tools may be useful in systematic review search creation.
Key Messages
- TMTs appear to decrease the time required to develop keyword and subject terms compared to usual practice (UP) search strategy development in six out of seven reports, but the small sample size precludes significance.
- TMTs searches appear less sensitive than UP searches in all but one project, but the small sample size precludes significance.
- Number-needed-to-read (NNR) results were mixed; NNR was lower using TMTs compared with UP in four out of seven reports. Again, the small sample size precludes significance.
- TMTs neither affected search evaluation time nor improved identification of exclusion concepts (irrelevant records) that can be safely removed from the search set.
- Across “simple” review topics (i.e., single indication-single drug) TMTs yielded no unique additional relevant citations while missing only one relevant study in three of four reports and reduced time spent on creating searches compared to UP. Thus, TMTs may be useful in simple review search strategy development, and when timeliness is prioritized and comprehensiveness is not critical.
- Across “complex” review topics (e.g., multicomponent interventions) TMTs identified some unique includable citations and reduced time spent in search strategy development but missed more relevant citations compared to UP. TMTs may be more useful as an adjunct to usual practice for complex evidence synthesis reviews (e.g., evidence maps, scoping reviews, systematic reviews, health technology assessments, and update reviews, etc.) especially when comprehensiveness is critical and the review team has adequate time to handle the increased screening burden.
Structured Abstract
Background. In an era of explosive growth in biomedical evidence, improving systematic review (SR) search processes is increasingly critical. Text-mining tools (TMTs) are a potentially powerful resource to improve and streamline search strategy development. Two types of TMTs are especially of interest to searchers: word frequency (useful for identifying most used keyword terms, e.g., PubReminer) and clustering (visualizing common themes, e.g., Carrot2).
Objectives. The objectives of this study were to compare the benefits and trade-offs of searches with and without the use of TMTs for evidence synthesis products in real world settings. Specific questions included: (1) Do TMTs decrease the time spent developing search strategies? (2) How do TMTs affect the sensitivity and yield of searches? (3) Do TMTs identify groups of records that can be safely excluded in the search evaluation step? (4) Does the complexity of a systematic review topic affect TMT performance? In addition to quantitative data, we collected librarians' comments on their experiences using TMTs to explore when and how these new tools may be useful in systematic review search creation.
Methods. In this prospective comparative study, we included seven SR projects, and classified them into simple or complex topics. The project librarian used conventional “usual practice” (UP) methods to create the MEDLINE search strategy, while a paired TMT librarian simultaneously and independently created a search strategy using a variety of TMTs. TMT librarians could choose one or more freely available TMTs per category from a pre-selected list in each of three categories: (1) keyword/phrase tools: AntConc, PubReMiner; (2) subject term tools: MeSH on Demand, PubReMiner, Yale MeSH Analyzer; and (3) strategy evaluation tools: Carrot2, VOSviewer. We collected results from both MEDLINE searches (with and without TMTs), coded every citation’s origin (UP or TMT respectively), deduplicated them, and then sent the citation library to the review team for screening. When the draft report was submitted, we used the final list of included citations to calculate the sensitivity, precision, and number-needed-to-read for each search (with and without TMTs). Separately, we tracked the time spent on various aspects of search creation by each librarian. Simple and complex topics were analyzed separately to provide insight into whether TMTs could be more useful for one type of topic or another.
Results. Across all reviews, UP searches seemed to perform better than TMT, but because of the small sample size, none of these differences was statistically significant. UP searches were slightly more sensitive (92% [95% confidence intervals (CI) 85–99%]) than TMT searches (84.9% [95% CI 74.4–95.4%]). The mean number-needed-to-read was 83 (SD 34) for UP and 90 (SD 68) for TMT. Keyword and subject term development using TMTs generally took less time than those developed using UP alone. The average total time was 12 hours (SD 8) to create a complete search strategy by UP librarians, and 5 hours (SD 2) for the TMT librarians. TMTs neither affected search evaluation time nor improved identification of exclusion concepts (irrelevant records) that can be safely removed from the search set.
Conclusions. Across all reviews but one, TMT searches were less sensitive than UP searches. For simple SR topics (i.e., single indication–single drug), TMT searches were slightly less sensitive, but reduced time spent in search design. For complex SR topics (e.g., multicomponent interventions), TMT searches were less sensitive than UP searches; nevertheless, in complex reviews, they identified unique eligible citations not found by the UP searches. TMT searches also reduced time spent in search strategy development. For all evidence synthesis types, TMT searches may be more efficient in reviews where comprehensiveness is not paramount, or as an adjunct to UP for evidence syntheses, because they can identify unique includable citations. If TMTs were easier to learn and use, their utility would be increased.
Journal Citation
Paynter RA, Featherstone R, Stoeger E, et al. A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. J Clin Epidemiol. 2021 March 19 [Epub ahead of print]. DOI: 10.1016/j.jclinepi.2021.03.013.
Citation
Suggested citation: Paynter RA, Fiordalisi C, Stoeger E, Erinoff E, Featherstone R, Voisin C, Adam GP. A Prospective Comparison of Evidence Synthesis Search Strategies Developed With and Without Text-Mining Tools. Methods Research Report. (Prepared by the Scientific Resource Center under Contract No. 290-2017-00003C). AHRQ Publication No. 21-EHC008. Rockville, MD: Agency for Healthcare Research and Quality. March 2021. Posted final reports are located on the Effective Health Care Program search page. DOI: 10.23970/AHRQEPCMETHODSPROSPECTIVECOMPARISON.