Introduction

Assessing the risk of bias of studies included in the body of evidence is a foundational part of all systematic reviews.^1,2 It is distinct from other important and related activities of assessing the degree of the congruence of the research question with the study design and the applicability of the evidence. The specific use of risk-of-bias assessments can vary. Assessment of risk of bias (labeled as unclear, high, moderate, or low) are intended to help interpret findings and explain heterogeneity; in addition, EPC reviews use risk-of-bias assessments of individual studies in grading the strength of the body of evidence. Some EPC reviews may exclude studies assessed as high risk of bias.

Despite the importance of risk-of-bias assessments in systematic reviews, evidence on the validity of such assessments is available only for a few risk-of-bias categories.^3-5 Specifically, evidence suggests that effect sizes may be inaccurate when allocation is inappropriately concealed; random sequences are inadequately generated; and patients, clinicians, or outcome assessors (particularly for subjective outcomes) are not blinded.^4,6 The influence on estimates of effect can be inconsistent and difficult to predict for other bias categories such as confounding, fidelity to the protocol, and attrition bias, possibly because meta-epidemiological studies are inadequately powered.⁵ In addition to concerns regarding the validity of such assessments, methodological studies have raised concerns about the limited reliability of risk-of-bias judgments.^7,8

We do not attempt, in this document, to address the underlying and important sources of uncertainty related to the validity or reliability of risk-of-bias assessment. This document updates the existing Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Methods Guide for Effectiveness and Comparative Effectiveness Reviews on assessing the risk of bias of individual studies. This update adds areas of guidance (e.g., evaluating subgroup analyses and including systematic reviews as evidence), modifies guidance to reflect new thinking (e.g., risk-of-bias categories), and offers guidance to promote clarity and consistency. As with other AHRQ methodological guidance, our intent is to present standards that can be applied consistently across EPCs and review topics, promote transparency and reproducibility in processes, and account for methodological changes in the systematic review process. These standards are based on epidemiological study design principles, available empirical evidence, or workgroup consensus. As greater evidence accumulates in this methodological area, our standards will continue to evolve. When possible, our guidance offers flexibility to account for the wide range of AHRQ EPC review topics and included study designs, but also offers parameters within which this flexibility can be applied.

In this guidance document, we define terms as appropriate for the EPC program, explore the potential overlap in different steps of the systematic review, and offer recommendations on the inclusion and exclusion of constructs that may apply to multiple steps of the systematic review process. This guidance applies to systematic reviews exploring the link between an intervention or exposure and outcome. (Reviewers focusing on diagnostic tests,⁹ prognosis,^10-12 prevalence, or qualitative¹³ analysis should also consult guidance specific to these topics.) Later sections of this guidance document provide advice on minimum design-specific criteria to evaluate risk of bias and the stages involved in assessing risk of bias. We conclude with guidance on summarizing risk of bias.

Journal Publication

Viswanathan M, Patnode CD, Berkman ND, et al. Recommendations for Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions.  Journal of Clinical Epidemiology. ePub ahead of print. DOI: https://dx.doi.org/10.1016/j.jclinepi.2017.12.004.