Focus on registered studies, adequately powered analyses and true data: filter unreliable evidence from systematic reviews!
Background:
Information on the effectiveness and safety of health care interventions should be valid, precise, up-to-date, comprehensible and freely available. Currently none of these criteria is adequately satisfied. Many systematic reviews contain a biased sample of underpowered trials (some include fabricated trials) and reviews are often long and unreadable.
Methods:
In response to these concerns, the Cochrane Injuries Group developed a new policy1. In addition to following standard Cochrane methods, authors of new and updated reviews should:
- Only include a trial if it was prospectively registered (unless the trial report was published before 2010).
- Provide a sample size estimate showing how many participants need to be included in a meta-analysis for reliable results. Trial sequential methods should be used to explore all “significant” treatment effects obtained before reaching the required sample size.
- The editorial team will conduct statistical checks on data included in a review. Authors of selected trials will be asked to provide the original trial data for checking. If authors decline to provide it, the trial will be removed from the review.
Discussion:
We invite critical discussion of our policy and urge other Cochrane review groups to consider adopting it. Please write to us at emma.sydenham@Lshtm.ac.uk with feedback.
1 Roberts I, Ker K, Edwards P, Beecher D, Manno D, Sydenham E. The knowledge system underpinning health care is not fit for purpose and must change. BMJ 2015;350:h2463.
Rationale for the policy:
These five reviews used to include studies which were later retracted. The publication history for the review is given at the end of the main text in the Published Notes section, and also in History and What's New on the last page of the review.
- Lewis SR, Pritchard MW, Evans DJW, Butler AR, Alderson P, Smith AF, Roberts I. Colloids versus crystalloids for fluid resuscitation in critically ill people. Cochrane Database of Systematic Reviews 2018, Issue 8. Art. No.: CD000567.
- Bunn F, Trivedi D. Colloid solutions for fluid resuscitation. Cochrane Database of Systematic Reviews 2012, Issue 7. Art. No.: CD001319.
- Henry DA, Carless PA, Moxey AJ, O'Connell D, Stokes BJ, Fergusson DA, Ker K. Anti-fibrinolytic use for minimising perioperative allogeneic blood transfusion. Cochrane Database of Systematic Reviews 2011, Issue 3. Art. No.: CD001886.
- Carless PA, Rubens FD, Anthony DM, O'Connell D, Henry DA. Platelet-rich-plasmapheresis for minimising peri-operative allogeneic blood transfusion. Cochrane Database of Systematic Reviews 2011, Issue 3. Art. No.: CD004172.
- Wakai A, McCabe A, Roberts I, Schierhout G. Mannitol for acute traumatic brain injury. Cochrane Database of Systematic Reviews 2013, Issue 8. Art. No.: CD001049.
To download the full-text of the review from the Cochrane Library, search for the title of the review. On the right-hand side of the review's page select 'Full' to download a full-text copy of the review.
If you wish to submit feedback (comments) on a review, select Submit Comments under Article Tools on the Review's page.
To compare published versions of a review over time, select the Other Versions tab on the review's page. The editorial team can provide a copy of the review showing the edits with tracked changes (email: Emma.Sydenham@Lshtm.ac.uk).
New publications incorporating the 2015 guidance:
Updated reviews
Desmopressin use for minimising perioperative blood transfusion
Progesterone for acute traumatic brain injury
Protocols for new reviews
Sedation for moderate-to-severe traumatic brain injury in adults: a network meta-analysis
Antivenom for snake venom-induced neuromuscular paralysis
Strategies for use of blood products for major bleeding in trauma
Interventions for frostbite injuries
Dopamine agonists for traumatic brain injury
Intravenous iron therapy for non-anaemic iron deficient adults