Background and Objectives for the Systematic Review
The RTI International - University of North Carolina Evidence-based Practice Center (RTI-UNC EPC) will use an ongoing review, Strategies to Improve Mental Health Care for Children and Adolescents (SIMHC), to generate a report on the additional information gained by including the information from clinicaltrials.gov. To achieve this goal, we will explore the differences between information from published and unpublished sources included in the review and clinicaltrial.gov.
In addition to this primary goal, the use of the SIMHC review will allow us to achieve three additional goals. First, this report provides the Agency for Healthcare Research and Quality (AHRQ) the opportunity to further influence new research and reporting requirements on a topic of increasing importance: QI, implantation, and dissemination. The volume of evidence in a range of topics will continue to rise exponentially. Despite advances in the evidence base, national health outcomes remain suboptimal, in part because of the failure of systems and providers to adopt established QI strategies and interventions with proven efficacy. Given the gap between observed and achievable processes and outcomes, the next critical step is the adoption of QI strategies and the development of strategies to implement or disseminate these interventions.1-3 Closing the gap requires more information on not just outcomes of complex interventions: it requires information on study conduct and processes to allow interpretation of results and enable scale-up. To achieve this goal, we will reach out to authors to understand the utility of clinicaltrials.gov and other archives (e.g., the World Health Organization [WHO] International Clinical Trials Registry and NIHReporter) for information on implementation processes.
Second, the project will afford AHRQ the opportunity to investigate reporting lacunae in complex study designs, such as cluster randomized controlled trials (cRCTs). cRCTs require advanced analytic methods (hierarchical linear modeling, for example) that account for clustering at each level of recruitment. To date, our investigation has revealed that a substantial proportion of the included studies in the SIMHC review use cRCTs (9 cRCTs of 154-17 included studies). However, the published data on these trials have been woefully inadequate and do always not permit an independent assessment of the effects of the intervention. These inadequacies hinder not only higher order analyses, such as risk of bias assessment, but also basic calculations of effect size and precision because of poor reporting of retention at the multiple levels of recruitment in a cRCT. To achieve this goal, we will seek information from clinicaltrials.gov on more design details, and if they are not available, seek to understand the impediments to reporting through outreach to study authors.
Third, the will allow AHRQ to understand the impediments to publication for pragmatic trials and systems interventions. As noted above, we will seek to understand the impediments to publication through outreach to study authors.
Key Questions
As noted in the RFTO, we will address the following questions below. We will also pose some questions to address the additional issues (in italics) described above that are specific to this review and complex interventions and study designs:
- Which studies were in the EPC report alone, clinicaltrials.gov alone or in both?
- For the completed studies which were in both:
- What were the differences, if any, in pre-specified outcome measures, statistical plan and size of the study reported, retention, study conduct, and other details of study design in the peer reviewed literature vs. clinicaltrials.gov?
- Were results reported in clinicaltrials.gov for any of the studies? If they were, what were the differences, if any, in the results reported in the peer reviewed literature vs. clinicaltrials.gov?
- For studies in clinicaltrials.gov that were not completed or discontinued:
- For the discontinued studies, were there reasons given for discontinuation? If so, what were they?
- For studies that are ongoing but not completed, what was the date of initiation of the studies? Are the studies proceeding according to the original schedule or is there information in clinicaltrials.gov indicating a delay in completion? If there is a delay in completion, what is the reason given?
- For studies that are completed but not published, what are the reasons for delay in or lack of publication?
- For included studies with limited or no information on study processes and conduct in clinicaltrials.gov, what, if any, publicly available sources provide or can provide information on implementation processes? What are the constraints to producing and disseminating this information? What is the perceived utility of clinicaltrials.gov as an archive for such information?
- What is the impact on the conclusions of the EPC report with and without the information from clinicaltrials.gov? What would be the impact on the strength of evidence (including impact of knowledge of outcomes measured in studies but not reported in the peer reviewed literature)?
Methods
KQ 1
We update our searches for SIMHC and then compare the yield with clinicaltrials.gov, using a dual independent review process.
KQ 2
- For studies with information in both peer-reviewed literature and clinicaltrials.gov, we will extract and compare the results, using a dual review process, with a second reviewer checking the first abstractions.
- For studies with differences in reporting by source, we will reach out to study authors via email and phone interview, if necessary, to understand the reasons for the differences.
KQ 3
- For discontinued studies, we will reach out to authors via email and phone interview, if necessary, to identify reasons for discontinuation.
- For ongoing incomplete studies, we will supplement information in clinicaltrials.gov with additional information from study authors via email and phone interview, if necessary.
- For completed and unpublished studies, we will reach out to authors of discontinued studies via email and phone interview, if necessary, to identify reasons for lack of publication
KQ 4
We will reach out to authors of included studies on the reasons for use or non-use of clinicaltrials.gov or other archive sites for information on study conduct and processes.
KQ 5
We will integrate the information for KQs 1-4, using data from searches; abstraction from clinicaltrials.gov; and email, personal interviews, and any additional information provided by authors. We will also update the strength of evidence and conclusion of the SIMHC report if relevant.
Table 1 provides a draft list of questions for email or personal interview. These are general questions, to be tailored for each interviewee. We do not anticipate requiring Office of Management and Budget clearance because no group exceeds 9 members. Once the protocol is approved, we will seek IRB review before conducting email or in-person interviews.
Item | Question |
---|---|
Introduction | The RTI-UNC Evidence-based Center is conducting a systematic review of strategies to improve mental health for children and adolescents. In addition, our funder, the Agency for Healthcare Research and Quality, has requested an additional investigation of the validity and reliability of clinicaltrials.gov as a potential additional source of information on study conduct, processes and results. Your study [xxx, has been included/is eligible for inclusion] in this review. We are reaching out to you to obtain some additional details about the reporting of your study. Thank you for agreeing to answer our questions. |
For authors of included clinical trials included in the report that do not have a clinicaltrials.gov listing, N=74,8,10,12,13,18,19 |
|
For authors of included clinical trials included in the report that are clinical trials who have a listing for that study in clinicaltrials.gov, with no results reported in clinicaltrials.gov, subset of N=46,7,9,14 |
|
For authors of included clinical trials that report differ results in clinicaltrials.gov and published studies, subset of N=46,7,9,14 |
|
For authors of studies included in the report that are NOT clinical trials, N=44,11,16,17 |
|
For authors of eligible discontinued studies identified via clinicaltrials.gov but not included in the SIMHC review, N to be determined |
|
For authors of ongoing incomplete studies identified via clinicaltrials.gov, not included in the SIMHC review, N=3 |
|
For authors of complete but unpublished studies identified via clinicaltrials.gov, but not included in the SIMHC review, N to be determined |
|
References
- Balas EA, Boron SA. Managing clinical knowledege for healthcare improvements. Stuttgart, Germany: Schattauer Verlagsgesellschaft; 2000.
- Shojania KG, McDonald KM, Wachter RM, et al. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies, Vol. 1: Series Overview and Methodology Technical Review 9 (Contract No. 290-02-0017 to the Stanford University-UCSF Evidence-based Practices Center). AHRQ Publication No. 04-0051-1. Rockville, MD: Agency for Healthcare Research and Quality; Aug 2004.
- Institute of Medicine. Crossing the Quality Chasm. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
- Gully KJ, Price BL, Johnson MK. Increasing abused children's access to evidence-based treatment: diffusion via parents as consumers. Child Maltreat. 2008 Aug;13(3):280-8. PMID: 18359928.
- Beidas RS, Edmunds JM, Marcus SC, et al. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv. 2012 Jul;63(7):660-5. PMID: 22549401.
- Carroll AE, Bauer NS, Dugan TM, et al. Use of a computerized decision aid for ADHD diagnosis: a randomized controlled trial. Pediatrics. 2013 Sep;132(3):e623-9. PMID: 23958768.
- Epstein JN, Langberg JM, Lichtenstein PK, et al. Use of an Internet portal to improve community-based pediatric ADHD care: a cluster randomized trial. Pediatrics. 2011 Nov;128(5):e1201-8. PMID: 22007005.
- Epstein JN, Rabiner D, Johnson DE, et al. Improving attention-deficit/hyperactivity disorder treatment outcomes through use of a collaborative consultation treatment service by community-based pediatricians: a cluster randomized trial. Arch Pediatr Adolesc Med. 2007 Sep;161(9):835-40. PMID: 17768282.
- Garner BR, Godley SH, Dennis ML, et al. Using pay for performance to improve treatment implementation for adolescent substance use disorders: results from a cluster randomized trial. Arch Pediatr Adolesc Med. 2012 Oct;166(10):938-44. PMID: 22893231.
- Glisson C, Hemmelgarn A, Green P, et al. Randomized trial of the Availability, Responsiveness, and Continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. J Am Acad Child Adolesc Psychiatry. 2012 Aug;51(8):780-7. PMID: 22840549.
- Henggeler SW, Sheidow AJ, Cunningham PB, et al. Promoting the implementation of an evidence-based intervention for adolescent marijuana abuse in community settings: testing the use of intensive quality assurance. J Clin Child Adolesc Psychol. 2008 Jul;37(3):682-9. PMID: 18645758.
- Lester H, Birchwood M, Freemantle N, et al. REDIRECT: cluster randomised controlled trial of GP training in first-episode psychosis. Br J Gen Pract. 2009 Jun;59(563):e183-90. PMID: 19520016.
- Lochman JE, Boxmeyer C, Powell N, et al. Dissemination of the Coping Power program: importance of intensity of counselor training. J Consult Clin Psychol. 2009 Jun;77(3):397-409. PMID: 19485582.
- Bickman L, Kelley SD, Breda C, et al. Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv. 2011 Dec;62(12):1423-9. PMID: 22193788.
- Glisson C, Schoenwald SK, Hemmelgarn A, et al. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010 Aug;78(4):537-50. PMID: 20658810.
- Wildman BG, Langkamp DL. Impact of location and availability of behavioral health services for children. J Clin Psychol Med Settings. 2012 Dec;19(4):393-400. PMID: 23053830.
- Ronsley R, Rayter M, Smith D, et al. Metabolic monitoring training program implementation in the community setting was associated with improved monitoring in second-generation antipsychotic-treated children. Can J Psychiatry. 2012 May;57(5):292-9. PMID: 22546061.
- Glisson C, Schoenwald SK, Hemmelgarn A, et al. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010 Aug;78(4):537-50. PMID: 20658810.
- Beidas RS, Edmunds JM, Marcus SC, et al. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv. 2012;63(7):660-5. PMID: 22549401.