Research Article: Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals

Date Published: July 20, 2017

Publisher: Public Library of Science

Author(s): Benjamin Howard, Jared T. Scott, Mark Blubaugh, Brie Roepke, Caleb Scheckel, Matt Vassar, Damir Janigro.


Selective outcome reporting is a significant methodological concern. Comparisons between the outcomes reported in clinical trial registrations and those later published allow investigators to understand the extent of selection bias among trialists. We examined the possibility of selective outcome reporting in randomized controlled trials (RCTs) published in neurology journals.

We searched PubMed for randomized controlled trials from Jan 1, 2010 –Dec 31, 2015 published in the top 3 impact factor neurology journals. These articles were screened according to specific inclusion criteria. Each author individually extracted data from trials following a standardized protocol. A second author verified each extracted element and discrepancies were resolved. Consistency between registered and published outcomes was evaluated and correlations between discrepancies and funding, journal, and temporal trends were examined.

180 trials were included for analysis. 10 (6%) primary outcomes were demoted, 38 (21%) primary outcomes were omitted from the publication, and 61 (34%) unregistered primary outcomes were added to the published report. There were 18 (10%) cases of secondary outcomes being upgraded to primary outcomes in the publication, and there were 53 (29%) changes in timing of assessment. Of 82 (46%) major discrepancies with reported p-values, 54 (66.0%) favored publication of statistically significant results.

Across trials, we found 180 major discrepancies. 66% of major discrepancies with a reported p-value (n = 82) favored statistically significant results. These results suggest a need within neurology to provide more consistent and timely registration of outcomes.

Partial Text

Bias plays a significant role in clinical research because it distorts the true effectiveness of interventions. One particular form of bias, known as outcome reporting bias, occurs when trialists fail to report prespecified outcomes, report primary outcomes that were not prespecified, report statistically significant secondary outcomes as primary outcomes, or report nonsignificant primary outcomes as secondary outcomes [1,2]. This form of bias is well documented in the medical literature [3,4,5,6,7,8,9]. Dwan et al. found that 40 to 62% of reviewed trials had at least one primary outcome that was changed, omitted, or newly introduced [10]. Because such reporting potentially affects clinical decision-making, growing interest exists for reviewing the medical literature, particularly randomized controlled trials (RCTs), for evidence of outcome reporting bias.

The primary goal of this study was to assess potential discrepancies between the primary and secondary outcomes in registered RCTs and the associated reports published in high impact factor neurology journals. Secondary outcomes were to highlight whether outcome reporting discrepancies favor statistically significant outcomes, whether there was any correlation between funding source and likelihood of outcome reporting bias, and whether any temporal trends in outcome reporting bias occurred during the time examined. We also catalogued any incidental findings during data extraction and analysis that warranted further examination. To accomplish these aims, we performed a methodological systematic review of the three highest impact factor neurology journals from 2011 to 2015. This study did not meet the regulatory definition of human subjects research according to 45 CFR 46.102(d) and (f) of the Department of Health and Human Services’ Code of Federal Regulations [16] and was not subject to Institutional Review Board oversight. Li et al. [17], the Cochrane Handbook for Systematic Reviews of Interventions[18], and the National Academies of Science, Engineering, and Medicine’s (previously the Institute of Medicine) Standards for Systematic Reviews [19] were consulted to ensure best practices regarding data extraction and management. PRISMA guideline [20] items 1, 3, 5–11, 13, 16–18, and 24–27 were applied to ensure reporting quality for systematic reviews in addition to SAMPL guidelines [21] for reporting descriptive statistics. Prior to initiation of the study, we registered it with the University hospital Medical Information Network Clinical Trials Registry (UMIN-CTR) with registry number: R000025976 UMIN000022541. All extracted data for this study are publicly available on figshare (

Our initial search yielded 424 articles. Two hundred twenty-eight RCTs were included, and of these, 211 were registered prior to patient enrollment. RCTs for which a primary or secondary outcome was not mentioned in their publication or RCTs that did not define a primary outcome in their registry were excluded, leaving us with a final sample size of 180 for data analysis. Other exclusion criteria are detailed in the Prisma diagram (Fig 1).

Our results indicate that trial registration and selective outcome reporting are problems that need to be addressed in neurology literature. Overall, only about 40% of trials were prospectively registered. The remaining 60% were improperly registered: they were either registered during patient enrollment or after study completion. Furthermore, we found evidence of 180 outcome inconsistencies across 180 RCTs. In many cases, these inconsistencies favored changes in accordance with statistically significant results. Given that outcome inconsistencies may have significant implications for clinical practice, solutions should be implemented to directly address this problem.




0 0 vote
Article Rating
Notify of
Inline Feedbacks
View all comments