Advertisement

Tackling poorly selected, collected, and reported outcomes in obstetrics and gynecology research

Published:September 28, 2018DOI:https://doi.org/10.1016/j.ajog.2018.09.023
      Clinical research should ultimately improve patient care. To enable this, randomized controlled trials must select, collect, and report outcomes that are both relevant to clinical practice and genuinely reflect the perspectives of key stakeholders including health care professionals, researchers, and patients. Unfortunately, many randomized controlled trials fall short of this requirement. Complex issues, including a failure to take into account the perspectives of key stakeholders when selecting outcomes, variations in outcome definitions and measurement instruments, and outcome reporting bias make research evidence difficult to interpret, undermining the translation of research into clinical practice. Problems with poor outcome selection, measurement, and reporting can be addressed by developing, disseminating, and implementing core outcome sets. A core outcome set represents a minimum data set of outcomes developed using robust consensus science methods engaging diverse stakeholders including health care professionals, researchers, and patients. Core outcomes should be routinely utilized by researchers, collected in a standardized manner, and reported consistently in the final publication. They are currently being developed across our specialty including infertility, endometriosis, and preeclampsia. Recognizing poorly selected, collected, and reported outcomes as serious hindrances to progress in our specialty, more than 80 journals including the Journal, have come together to support the Core Outcomes in Women’s and Newborn Health (CROWN) initiative. The consortium supports researchers to develop, disseminate, and implement core outcome sets. Implementing core outcome sets could make a profound contribution to addressing poorly selected, collected, and reported outcomes. Implementation should ensure future randomized controlled trials hold the necessary reach and relevance to inform clinical practice, enhance patient care, and improve patient outcomes.

      Key words

      THE PROBLEM: Research should ultimately improve patient care. For this to be possible, randomized controlled trials must select, collect, and report outcomes that are relevant to clinical practice and genuinely reflect the perspectives of diverse stakeholders including professionals, researchers, and patients.
      THE SOLUTION: Developing, disseminating, and implementing core outcome sets should standardize outcome selection, collection, and reporting to ensure future randomized controlled trials inform clinical practice, enhances patient care, and improves patient outcomes.

      Poorly selected: outcomes lacking relevance to clinical practice and key stakeholders

      Selecting appropriate outcomes is a critical step in designing randomized trials. However, the relative importance of individual outcomes may become a secondary consideration for researchers as they accommodate sample size requirements, costs, and time constraints. The primary outcome should be the outcome of greatest therapeutic importance to the study’s prospective hypothesis.
      When considering randomized trials evaluating potential treatments for infertility, there is a consensus regarding the selection of live birth as the primary outcome. Unfortunately, a recent systematic review of 142 infertility trials published in 2013 and 2014 found that only 52 included trials (37%) reported live birth.
      • Wilkinson J.
      • Roberts S.A.
      • Showell M.
      • Brison D.R.
      • Vail A.
      No common denominator: a review of outcome measures in IVF RCTs.
      Poorly selected outcomes extend beyond the primary outcome. Researchers should select secondary outcomes that reflect the complex pathophysiology of the health care condition of interest. For example, when considering preeclampsia, secondary outcomes should include blood pressure thresholds, clinical symptoms, maternal morbidity, and neonatal morbidity. Many preeclampsia trials fall short of this requirement. A systematic review of 79 preeclampsia trials demonstrated widespread variation in the reporting of outcomes. A minority of randomized trials reported outcomes concerning maternal morbidity including eclampsia (39 trials, 49%), pulmonary edema (15 trials, 19%), and renal failure (ten trials, 13%).
      • Duffy J.M.N.
      • Hirsch M.
      • Kawsar A.
      • et al.
      Outcome reporting across randomised controlled trials evaluating therapeutic interventions for pre-eclampsia.
      Over the past three decades, the scope of obstetric research has widened, with long-term childhood follow-up becoming increasingly prioritized by patients, health care professionals, and researchers. As the importance of assessing long-term outcomes gains momentum, challenging decisions regarding the selection of long-term outcomes, follow-up duration, and methods will need to be made.

      Poorly collected: different definitions and measurement instruments

      Evidence synthesis can be limited by variation in methods of measurement or definition, even when outcomes have been consistently collected and reported across trials. In the absence of a standardized approach, researchers can choose from a variety of different definitions and measurement instruments. For example, researchers have defined stillbirth using different combinations of gestational ages, birthweights, and crown-heel lengths. When considering endometriosis trials, dysmenorrhea has been measured by ten different measurement instruments including visual analogue scales, Likert scales, and questionnaires.
      • Hirsch M.
      • Duffy J.M.N.
      • Kusznir J.O.
      • et al.
      Variation in outcome reporting in endometriosis trials: a systematic review.
      Such variation can make it difficult to synthesize the results of individual randomized trials using secondary research, including individual patient data meta-analysis, pair-wise meta-analysis, and network meta-analysis.

      Poorly reported: outcome reporting bias

      Outcome reporting bias is defined as the selection for publication of a subset of the original recorded outcome variables on the basis of the results. Several systematic reviews have confirmed outcome reporting bias and quantified its impact when pooling data from individual trials in a meta-analysis. A systematic review of 519 randomized controlled trials identified substantial deficiencies in outcome reporting: when compared with the trial registry record, 360 trials (75%) did not fully report efficacy outcomes and 196 (64%) did not fully report harm outcomes in the final trial report.
      • Williamson P.R.
      • Altman D.G.
      • Bagley H.
      • et al.
      The COMET handbook: version 1.0.
      Statistically significant outcomes had higher odds of being fully reported when compared with nonsignificant outcomes (odds ratio, 2.0, 95% confidence interval, 1.6–2.7). Further problems arise with outcome reporting when the results of individual trials are combined within a meta-analysis. A systematic review of 157 Cochrane systematic reviews published in 2007 identified more than a third of systematic reviews contained at least 1 randomized trial at high risk of outcome reporting bias.
      • Williamson P.R.
      • Altman D.G.
      • Bagley H.
      • et al.
      The COMET handbook: version 1.0.
      A sensitivity analysis, taking into account outcome reporting bias, demonstrated a reduction of greater than 20% in the treatment effect of the primary outcome.
      Problems with poor outcome selection, measurement, and reporting can be addressed by developing, disseminating, and implementing a core outcome set to standardize outcome selection, collection, and reporting across future randomized controlled trials within individual health care conditions.

      Call to action: developing, disseminating, and implementing core outcome sets

      A core outcome set represents a minimum data set of outcomes developed using robust consensus science methods. Core outcomes should be consistently selected, collected in a standardized manner, and reported consistently in the final publication. The existence of a core outcome set does not imply that outcomes in a particular trial should be restricted. Rather, there is an expectation that the core outcomes will be collected and reported as a minimum with other outcomes included as appropriate for a given study. This would make it easier for the core outcomes from individual trials to be compared, contrasted, and combined as appropriate. More than 50 core outcome sets are currently being developed across our specialty including infertility, endometriosis, and preeclampsia.
      • Duffy J.M.N.
      • Rolph R.
      • Gale C.
      • et al.
      Core outcome sets in women’s and newborn health: a systematic review.
      When developing a core outcome set, a diverse range of stakeholders, including health care professionals, researchers, and patients, representing various disciplines, geographical regions, and methodological expertise, should be recruited. A high number of varied participants is desirable to secure the generalizability of the final core outcome set and increase its credibility with other researchers.
      The first step in core outcome set development is to develop a long list of potential core outcomes by undertaking a systematic review of published randomized controlled trials (Figure 1). Outcomes identified through systematic reviews of published trials largely reflect outcomes health care professionals, and researchers have considered important to collect and measure, particularly in which research predates the recent emphasis on patient and public involvement. Outcomes reported in historic trial reports may not hold the same relevance for other stakeholders including patients, and therefore, the evidence base to support shared decision making is likely to be suboptimal to inform clinical practice.
      Figure thumbnail gr1
      Figure 1Core outcome set development
      Duffy. Tackling poorly selected, collected, and reported outcomes in research. Am J Obstet Gynecol 2019.
      Core outcome set developers should use qualitative methods to provide reassurance that the long list of potential core outcomes includes those with relevance to patients. Unfortunately, only a few core outcome set developers have used in-depth patient interviews to identify relevant outcomes.
      • Duffy J.M.N.
      • Rolph R.
      • Gale C.
      • et al.
      Core outcome sets in women’s and newborn health: a systematic review.
      Future core outcome set developers should engage with methodological research to evaluate different qualitative methods, including secondary analysis of archive interviews, focus groups, and free text questionnaires, to identify treatment outcomes that hold relevance to patients.
      The next step is to reduce the long list of potential core outcomes to a core outcome set using consensus science methods. Core outcome set developers have used the modified Delphi method enabling individuals to participate in an iterative process that assesses the extent of agreement (consensus measurement) and then resolves disagreement (consensus development).
      • Duffy J.M.N.
      • Rolph R.
      • Gale C.
      • et al.
      Core outcome sets in women’s and newborn health: a systematic review.
      Potential core outcomes have been entered into sequential online surveys through which participants score the importance of individual outcomes. Repeated reflection and rescoring encourage convergence toward consensus core outcomes.
      The final step is to determine how the core outcomes should be defined and measured. Potential definitions and measurement instruments have been inventoried across formal definition development initiatives, international and national guidelines, Cochrane systematic reviews, and randomized controlled trials.
      • Duffy J.M.N.
      • Rolph R.
      • Gale C.
      • et al.
      Core outcome sets in women’s and newborn health: a systematic review.
      Such definitions can be entered into a consensus development workshop to prioritize standardized definitions for individual core outcomes. Potential measurement instruments have been quality assessed using the Consensus-Based Standards for the Selection of Health Measurement Instruments initiative quality assessment framework to associate high-quality measurement instruments with individual core outcomes.
      • Williamson P.R.
      • Altman D.G.
      • Bagley H.
      • et al.
      The COMET handbook: version 1.0.
      When a core outcome set has been developed, it should be regularly reviewed to confirm its ongoing validity. Future core outcome set developers should carefully design a strategy to ensure a core outcome set remains fit for purpose. There is uncertainty regarding the frequency of reviews and whether specific triggers should prompt a review. Methods including assessments of the uptake of an individual core outcome set, interviews with the intended users of the core outcome set, and monitoring outcome reporting in future trials could provide a framework when considering review triggers.

      Core outcome sets supported by women’s health journals

      Embedding core outcome sets within randomized controlled trials could make a profound contribution to advancing the usefulness of research to inform clinical practice, enhance patient care, and improve patient outcomes. Recognizing that poorly selected, collected, and reported outcomes are a serious hindrance to progress in women’s health, more than 80 journals, including the Journal, have come together to support the Core Outcomes in Women’s and Newborn Health (CROWN) initiative. The consortium supports researchers to develop, disseminate, and implement core outcome sets (Figure 2).
      Figure thumbnail gr2
      Figure 2Core Outcomes in Women’s and Newborn Health (CROWN) initiative’s aims
      Duffy. Tackling poorly selected, collected, and reported outcomes in research. Am J Obstet Gynecol 2019.
      Implementing the aims of the CROWN initiative across participating journals should encourage researchers committed to creating harmony in outcome selection, collection, and reporting. Over time, research consumers including health care professionals, researchers, and patients should become accustomed to reviewing complete core outcome sets in research reports. As a result, the temptation for selective reporting based on statistical significance should be substantially reduced.
      Implementation of core outcome sets is supported by other key stakeholders including opinion leaders, health research funders, and health research regulators. For example, the Standard Protocol Items: Recommendations for Interventional Trials statement, implemented by funders of health research including the National Institutes of Health, European Commission, and the National Institute of Health Research, recommend the use of core outcome sets wherever possible.
      • Williamson P.R.
      • Altman D.G.
      • Bagley H.
      • et al.
      The COMET handbook: version 1.0.
      Developing collaborative relationships with other stakeholders would be useful; for example, clinical trial registries, such as clinicaltrials.gov, could mandate the prespecified collection of individual core outcome sets.

      Conclusions

      Developing, disseminating, and implementing core outcome sets could make a profound contribution to addressing poorly selected, collected, and reported outcomes. Implementation should ensure that future randomized controlled trials hold the reach and relevance to inform clinical practice, enhance patient care, and improve patient outcomes.

      References

        • Wilkinson J.
        • Roberts S.A.
        • Showell M.
        • Brison D.R.
        • Vail A.
        No common denominator: a review of outcome measures in IVF RCTs.
        Hum Reprod. 2016; 31: 2714-2722
        • Duffy J.M.N.
        • Hirsch M.
        • Kawsar A.
        • et al.
        Outcome reporting across randomised controlled trials evaluating therapeutic interventions for pre-eclampsia.
        BJOG. 2017; 124: 1829-1839
        • Hirsch M.
        • Duffy J.M.N.
        • Kusznir J.O.
        • et al.
        Variation in outcome reporting in endometriosis trials: a systematic review.
        Am J Obstet Gynecol. 2016; 214: 452-464
        • Williamson P.R.
        • Altman D.G.
        • Bagley H.
        • et al.
        The COMET handbook: version 1.0.
        Trials. 2017; 18: 280
        • Duffy J.M.N.
        • Rolph R.
        • Gale C.
        • et al.
        Core outcome sets in women’s and newborn health: a systematic review.
        BJOG. 2017; 124: 1481-1489