Cochraneã¢â‚¬â„¢s Handbook for Systematic Reviews of Interventions Bias

• Research commodity
• Open Access
• Published: 11 April 2019

The judgement of biases included in the category "other bias" in Cochrane systematic reviews of interventions: a systematic survey

• Andela Pijukⁱⁱ,
• Lucie Brázdilová³,
• Yuliyana Georgieva^iv,
• Marco António Raposo Pereira⁵,
• Tina Poklepovic Pericic² &
• Livia Puljak ORCID: orcid.org/0000-0002-8467-6061 ^vi

BMC Medical Inquiry Methodology volume 19, Article number:77 (2019) Cite this article

• 4115 Accesses

• 15 Citations

• v Altmetric

• Metrics details

Abstruse

Background

Clinical decisions are fabricated based on Cochrane reviews, just the implementation of results of bear witness syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should appraise the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done past using Cochrane RoB tool. However, the tool is non necessarily practical according to the instructions. In this study, we aimed to make up one's mind the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool.

Methods

We analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized dissimilar types of other bias.

Results

We analyzed 768 Cochrane reviews that included 11,369 RCTs. In that location were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB tabular array of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged every bit low, unclear or loftier RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We establish numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews.

Conclusion

Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently guess the same supporting explanations. Inconsistency in appraising gamble of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce conviction in otherwise trustworthy sources of information. These results tin can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that tin be found in trials, and as well to help them avoid mistakes that were recognized in published Cochrane reviews.

Peer Review reports

Background

Assessment of the risk of bias (RoB) in included studies is an integral part of preparing Cochrane systematic reviews. Bias is whatsoever systematic error that can negatively affect the estimated effects of interventions and lead authors to wrong conclusions nigh efficacy and condom of analyzed interventions [1].

Cochrane reviews utilise Cochrane's RoB tool, whose aim is to enable better appraisal of bear witness and ultimately lead to amend healthcare [2]. Cochrane'due south standard RoB tool has seven domains. Starting time domain addresses random sequence generation as a potential source of pick bias, assessing potentially biased allocation to interventions due to inadequate generation of a randomized sequence. Second domain analyzes resource allotment concealment, which can also lead to option bias. The third domain is devoted to blinding of participants and personnel; it is associated with operation bias due to the cognition of the allocated interventions by participants and personnel during the written report. Quaternary domain addresses blinding of outcome assessment; if done inadequately, it can lead to detection bias due to the knowledge of the allocated interventions by result assessors. Fifth domain analyzes the presence of incomplete issue data, which can yield compunction bias due to amount, nature or handling of incomplete upshot information. The sixth domain is devoted to selective reporting, which tin crusade reporting bias due to selective outcome reporting. And finally, there is the seventh domain of Cochrane RoB assessment called "other bias", which is used to annotation bias occurring due to whatever additional problems that were not covered by the beginning six domains [3].

The Cochrane Handbook provides some examples of other potential threats to validity, such as design-specific risk of bias in not-randomized trials, baseline imbalance between groups of participants, blocked randomization in trials that are non blinded, differential diagnostic activity, written report changes due to acting results, deviations from the study protocol, giving intervention earlier randomization, inappropriate administration of an intervention or having co-intervention(s), contamination due to drug pooling amid participants, insufficient commitment of intervention, inappropriate inclusion criteria, using instruments that are not sensitive for specific outcomes, selective reporting of subgroups and fraud [3].

This list of potential other sources of bias mentioned in the Cochrane Handbook is limited, and it would, therefore, be useful to explore potential additional sources of 'other bias'. By consulting a more than comprehensive list of potential other biases, the systematic review might recognize certain problems in included studies that might not otherwise consider a potential source of bias.

The aim of this study was to define which issues authors of Cochrane reviews describe as "other bias", to decide the prevalence of various categories of other bias and to quantify qualitative data which support the assessment of other bias.

Methods

Nosotros conducted a retrospective analysis of published Cochrane reviews.

Inclusion and exclusion criteria

We retrieved Cochrane reviews that included RCTs about interventions published from July 2015 to June 2016 (N = 955) by using Advanced search in The Cochrane Library. Diagnostic Cochrane reviews, empty reviews, overviews of systematic reviews and reviews withdrawn in this period were excluded. Cochrane reviews that included both RCTs and non-randomized trials were included, but only RoB of RCTs were analyzed.

Screening

Ane author assessed all titles/abstracts to establish the eligibility of Cochrane reviews for inclusion (LP). Another writer verified all the assessments of the outset author (AB). At that place were no disagreements.

Data extraction and categorization

Data extraction table was developed and piloted using v Cochrane reviews. Initially, one author manually extracted the data by copy-pasting from included Cochrane reviews and another author verified 10% of extractions. Of the 77 verified Cochrane reviews, we found 3 Cochrane reviews which were partially extracted (3.nine%), which we consider to be a negligible percentage of the discrepancy. We extracted judgments (loftier, low or unclear risk) and supporting explanations for judgments (qualitative information which support the assessment to determine the reasons for the judgment) from the 'other bias' section of RoB table in Cochrane reviews. We besides extracted judgments and support for judgments from additional non-standard domains (domains which are not covered by seven standard RoB domains in RoB tabular array mentioned in the Background department) if Cochrane authors used them. For Cochrane reviews that did not use the 'other bias' domain in the RoB table or any other additional non-standard domains, we analyzed the text of results to see whether Cochrane authors mentioned any potential sources of other bias in the text of the review simply. Each supporting explanations for judgments of take a chance of bias in the analyzed trials were categorized by ii authors (AB and LP), via consensus. In 2018 nosotros enlisted a help of information specialist who used software for data extraction, and compared manually extracted data with software-extracted information; we found 12 farther discrepancies in extracted judgments.

Outcomes

We analyzed number, type, judgments and inconsistencies in judgments for certain comments nearly other risk of bias. These inconsistencies were judged equally follows: we analyzed whether Cochrane authors used different RoB judgments for the same supporting comment. We quantified Cochrane reviews in which authors did not use 'other bias' domain for any of the included RCTs to decide whether they used some non-standard additional RoB domain instead of 'other bias'. We conducted a quantitative and qualitative analysis of these not-standard domains.

Statistics

Nosotros performed descriptive statistics using Microsoft Excel (Microsoft Inc., Redmond, WA, United states). Nosotros presented information as frequencies and percentages. In the main analysis, nosotros analyzed Cochrane reviews that had the 'other bias' domain in the RoB table. In the secondary analysis, we analyzed Cochrane reviews that did not have the 'other bias' domain or had dissimilar non-standard variations of RoB assessment that were not mentioned in the Cochrane Handbook.

Results

Primary analysis

We analyzed 768 Cochrane reviews that included eleven,369 RCTs. Among those 768 Cochrane reviews, nosotros included in the primary analysis 602 Cochrane reviews that had 'other bias' domain in the RoB tables. Those 602 Cochrane reviews included a total of 7811 RCTs. We analyzed 166 Cochrane reviews in the secondary analysis because they either did not have 'other bias' domain in RoB Tables (N = 149), or those Cochrane reviews had both 'other bias' domain and additional not-standard domains in the RoB Tables (North = 17). The catamenia diagram showing inclusion of Cochrane reviews is shown in Fig. 1.

Fig. ane

Menstruum diagram presenting the inclusion of Cochrane systematic reviews in the study. We retrieved 955 Cochrane systematic reviews from the Cochrane Database of Systematic Reviews that were published from July 2015 to July 2016. Nosotros excluded 187 Cochrane reviews because they were either empty (without a single study included), diagnostic accuracy reviews, overviews of systematic reviews or they were withdrawn. Nosotros included 768 Cochrane reviews in our analysis; of those, 602 were included in our principal analysis because they had other bias domain in the Cochrane risk of bias tool, while 166 Cochrane reviews were included in our secondary analysis considering they either did non have other bias domain in the Cochrane take a chance of bias tool, or they had this domain, simply also other non-standard domains in the tool

Total size paradigm

Out of 602 Cochrane reviews in the primary analysis, there were 524 (87%) Cochrane reviews that described diverse sources of bias in the 'other bias' domain, while in 78 (13%) Cochrane reviews not a single source of other bias was reported. Furthermore, among 602 Cochrane reviews from the master assay, there were 337 (56%) Cochrane reviews in which at least 1 included trial indicated that no other bias was institute. Terminology for comments about non-real other bias varied, fifty-fifty within individual Cochrane reviews. In 268 (80%) Cochrane reviews only 1 version of the comment that no other bias was found was used, while in 69 (xx%) reviews Cochrane authors used unlike expressions in comments to indicate that no other sources of bias were found. Some examples of this varied terminology are shown in Boosted file 1: Tabular array S1.

In 40 (12%) out of 337 Cochrane reviews that indicated that no other bias was plant, we observed discrepancies in judgment for this domain. Namely, Cochrane authors in these 40 Cochrane reviews sometimes indicated that lack of other bias was associated with low RoB, and sometimes they marked it as unclear or high RoB. In 59 (18%) of these 337 Cochrane reviews at least i support for judgment that indicated that no other bias was identified Cochrane authors judged equally not being the low risk of bias (either loftier or unclear); in 278 Cochrane reviews this was judged as low RoB.

In 19 Cochrane reviews, all comments that referred to no other bias beingness identified were judged as unclear. In one review comment, 'no other bias' was judged as both low and high. References to Cochrane reviews for these specific examples are in Boosted file 2: Tabular array S2. In one review the aforementioned comment was judged in unlike RCTs equally either depression or high. In one review the same comment was judged in different RCTs as either low or unclear or high.

Of the 7811 trials that were included in the 602 Cochrane reviews from the main analysis, in 3703 (47%) trials domain for other bias indicated in the back up for judgment that other bias was not identified. Of those 3703 trials, there were 288 (7.eight%) that were judged as unclear RoB, iv (0.1%) that were judged as high RoB, while the others (N = 3411, 92.1%) were judged equally low RoB.

Sources of other bias

In the 524 analyzed Cochrane reviews that described various sources of other bias, there were 5762 unlike supporting explanations for judgments of other bias that we categorized into 31 categories. In 535 trials it was indicated only that it was not possible to assess other bias. For 24 (4%) of those 535 trials it was non indicated why this was not possible, while the well-nigh mutual reasons for not being able to assess other bias were that there was 'insufficient information' (N = 392, 73%), the trial was published every bit a conference abstruse only (N = 78, 15%) and that the trial was published in a foreign language so at that place were issues with translation (North = 11, 2%). Cochrane authors were not consistent in judging this type of supporting caption; for xi (2%) trials it was judged as high RoB, for 520 (94%) as unclear RoB and for 4 (0.7%) equally low RoB.

There were 236 trials for which Cochrane authors simply wrote that issues related to other bias were not described or unclear. This type of supporting explanation was also inconsistently judged by the Cochrane authors; 7 (3%) judged it as low RoB and 229 (97%) as unclear RoB.

The remaining 4991 explanations for judgments of other bias were divided into 29 categories that are shown in Tabular array 1. The most frequently used categories of explanations for other bias were related to baseline characteristics of participants, funding of a trial, reporting, sample size and conflict of interest (Tabular array 2). Cochrane authors used the domain for other bias to indicate positive, negative and unclear aspects of a trial. For example, iii virtually common types of explanations in the category related to baseline characteristic of participants indicated that either baseline characteristics were like, or that in that location was the imbalance in baseline characteristics, or that in that location was insufficient data near it (Additional file 3: Tabular array S3). Amidst 4991 explanations, we were unable to categorize 85 of them because they were uninformative, including explanations such as 'Adequate' or 'N/A' or 'Other risk of bias was possible'. Finally, there were 112 explanations that were used only once or twice in RoB tables we analyzed and so we categorized that group as 'Other explanations'. A tabular array with all the types of explanations is presented in Boosted file 3: Tabular array S3.

Table 1 Unlike categories of other bias (based on 4991 explanations) in Cochrane systematic reviews

Full size table

Table 2 Judgments for the xx about common explanations of other bias

Total size table

Partial studies included in the primary assay

We plant 34 Cochrane reviews with specific partial data regarding other bias, i.e. whose 'other bias' domains in RoB tables were not complete. Nosotros divided them into four distinct groups: the outset group with 28 reviews that had judgments for 'other bias', but non all had accompanying comments, second group with iv reviews where only one included RCT did not have the 'other bias' domain, 3rd grouping with ane review with included RCT without 'other bias' domain and included RCT with only judgment without comment, and fourth grouping with one review where RoB table was completely missing for 6 included RCTs. References to Cochrane reviews and RCTs for these specific examples are in Additional file ii: Table S2. Some Cochrane reviews had boosted non-standard RoB domains, separately or in addition to the 'other bias' domain. Categories of additional non-standard RoB domains in Cochrane reviews are shown in Tabular array 3.

Tabular array iii Categories of additional non-standard RoB domains in Cochrane systematic reviews

Full size table

Cochrane authors' judgments of dissimilar explanations for 'other bias'

At that place were 3033 trials for which only 1 category of explanation was written by Cochrane authors. When the explanation had merely one category of comment nosotros could exist sure that the judgment referred only to that specific comment so nosotros analyzed those in detail to see how the Cochrane authors judge dissimilar explanatory comments. There were 259 types of dissimilar explanations among those 3033 trials. We analyzed in more item those judgments for twenty near common explanations of other bias and found very high inconsistency in how Cochrane authors judge the same explanations (Table 2).

Secondary analysis

Reviews without 'other bias' domain in the RoB table

Among 149 Cochrane reviews that did not have 'other bias' domain in the RoB table, there were 102 reviews that did not accept any other replacement domain for 'other bias'. These 102 reviews used the varied number of standard RoB domains. In those 102 reviews, the number of standard RoB domains that were used varied, with one standard RoB domain in 4 reviews, iii RoB domains in 7 reviews, four RoB domains in 15 reviews, five domains in 51 reviews and 6 domains in 25 reviews.

For this group of Cochrane reviews, that did not have the 'other bias' domain in the RoB table, nosotros analyzed texts of results to see whether they mentioned whatsoever other sources of bias, beyond the standard six domains, in the department 'Risk of bias in included studies'. We constitute that 68/102 (67%) did not mention any sources of other bias in the results of the review. Even so, the remaining 34 (33%) did have comments about the other bias. Three of those 34 stated that they had non found any other risk of bias, while 31 reviews out of those 34 reported in the text of results that the included studies had had from 1 to 6 unlike categories of other bias.

Reviews with both 'other bias' domain and additional non-standard domain(due south) for other bias in RoB tables

9 Cochrane reviews had both 'other bias' domain and additional non-standard domain(s) for other bias in RoB tables (References in Additional file ii: Table S2). Those reviews used from i to 4 boosted not-standard domains; xviii in total. Those additional non-standard RoB domains are listed in Tabular array 3 and marked with the asterisk.

Reviews without 'other bias' domain but with the additional non-standard domain(southward)

There were 57 Cochrane reviews that did not accept the 'other bias' domain, but they did have additional not-standard RoB domains apart from the standard domains in the Cochrane RoB tabular array. Nigh of the reviews had but ane additional non-standard domain (N = 24), while others had 2–8 additional domains per each RCT. Table 3 shows non-standard domains that were used in those reviews without 'other bias' domain.

Reviews that consistently did not use support for judgment or they used not-standard judgments

We found 9 Cochrane reviews that consistently did not use supporting explanations for judgment or they used non-standard judgments. In 5 reviews authors used judgments low, high or unclear RoB, just without comments as support for judgment. In one review all trials were marked with the unclear take chances of other bias without any comment as back up for judgment. In four reviews all trials were marked with low risk of other bias without any comment as support for judgment. We also found iv reviews that did non accept judgments depression-loftier-unclear, but different kinds of judgments. One review had judgments yes/no without supporting comments; 2 reviews had judgments yes, no or unclear, with supporting comments and there was i review with judgments A-adequate and B-unclear (References in Boosted file two: Table S2).

Discussion

In this report, we analyzed 768 Cochrane systematic reviews, with 11,369 included trials. We found that Cochrane authors used numerous unlike categories of sources of other bias and that they were non judging them consistently. We categorized different types of supporting explanations into 31 categories, and we found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. Findings of this written report are disconcerting considering consistency in secondary enquiry is very of import to ensure comparability of studies.

Insufficient and unclear reporting of the 'other bias' domain was very mutual in the Cochrane reviews nosotros analyzed. Among the nigh common support for judgment were comments that we categorized equally 'not described/unclear', which is puzzling because 'other bias' domain is not specific like the other half-dozen domains of the RoB tool, and information technology is, therefore, difficult to fathom what it means that other bias was not described or that it was unclear. If the authors did not detect sources of other bias, or if they thought that they could not appraise other bias because of the brevity of report or language problems, they should take stated that. Likewise, for some trials, the only supporting explanation was that other bias was 'Adequate'. Without whatever further explanations, readers cannot know what exactly the Cochrane authors plant to be adequate in terms of other potential sources of bias. Many systematic reviews had a loftier number of included studies, and therefore some comments were repeated multiple times in the aforementioned systematic review.

The near commonly used specific category of other bias referred to baseline characteristics of participants. In RCTs, randomization should ensure resource allotment of participants into groups that differ only in intervention they received. Randomization should ensure that the characteristics of participants that may influence the outcome volition exist distributed equally across trial artillery so that any difference in outcomes can exist assumed to exist a consequence of intervention [4]. Baseline imbalances betwixt the groups may indicate that at that place was something wrong with the randomization process, or that they might be due to chance [v]. Severe baseline imbalances can occur because of deliberate actions of trialists if they aim to intentionally subvert the randomization process [6] or due to unintentional errors.

Chance imbalances should not be considered a source of bias, simply it may be hard to distinguish whether baseline imbalances are caused by risk or intentional actions. If there are multiple studies included in a meta-assay, it could be expected that hazard imbalances will act in reverse directions. Merely the problem may occur if there is a pattern of imbalances across several trials that may favor ane intervention over some other, suggesting imbalance due to bias and non due to chance [vii]. Cochrane is now developing a second generation of the RoB tool, titled RoB 2.0, and one of the signaling questions in the RoB domain virtually randomization process asks "Were there baseline imbalances that advise a problem with the randomization procedure" [vii]. The fact that then many Cochrane authors used comments nigh baseline imbalance as a domain of other bias, and not in the RoB domain about random sequence generation (selection bias) indicate that many Cochrane authors consider that this aspect should be emphasized separately from the pick bias domain.

The second virtually ordinarily used category of supporting explanations was related to funding of a trial, and comments about conflicts of interest were the fifth most common category. This is in direct dissimilarity with the recommendations from the Cochrane Handbook, where it is best-selling that information virtually vested interests should be collected and presented when relevant, but non in the RoB table; such data should be reported in the tabular array chosen 'Characteristics of included studies' [8]. RoB tabular array should be used to describe specific methodological aspects that may accept been influenced past the vested involvement and direct atomic number 82 to RoB [8]. Therefore, it is obvious that the authors of the Cochrane Handbook assume that the influence of sponsors can be mediated via other domains of RoB tool such every bit selective reporting of favorable outcomes.

However, Lundh et al. accept published a Cochrane review in 2017 about industry sponsorship and research outcomes, in which they included 75 primary studies, which shows that commercial funding leads to more than favorable efficacy results and conclusions compared to not-turn a profit funding [9]. They ended that industry sponsorship introduces bias that cannot be explained by standard domains of Cochrane'due south RoB assessment [nine]. The debate nearly whether funding presents the source of bias or not is ongoing in the Cochrane, with some considering that commercial funding is a clear risk of bias, while others fence against such standpoint [10, 11]. This debate apparently reflects the current situation in which many Cochrane authors keep to use funding and conflict of interest as a source of other bias despite the official warning against such use of information about sponsorship from the Cochrane Handbook, as nosotros have demonstrated in this study.

The tertiary virtually frequent category of supporting explanations for other bias was related to poor reporting, where Cochrane authors indicated that relevant information was missing or were inadequately reported. Poor reporting hinders transparency, every bit it allows authors to avoid attention to weak aspects of their studies. For this reason, reporting guidelines should exist used [12].

Comments about sample size were the fourth most mutual category either in a sense that the trial did or did non study sample size calculation, or that sample size was "small-scale" without whatever farther caption of what the Cochrane authors considered to be a minor sample. At that place were 21 trials for which Cochrane authors wrote that in that location were fewer than 50 participants in each arm. It is unclear where this cut-off is coming from, as there is no such guidance in the Cochrane Handbook in the chapter about the risk of bias. On the contrary, chapter 8.15.2. of the Cochrane Handbook specifically warns that "sample size or use of a sample size (or power) calculation" are examples of quality indicators that "should non exist assessed within this domain" [8].

The Cochrane Handbook also warns that authors should avoid double-counting, by non including potential sources of bias in the 'other bias' domain if they can be more appropriately covered by other domains in the tool [8]. Every bit can be seen by our study, Cochrane authors sometimes do double-counting because there were categories of comments supporting judgments that could have been addressed in the get-go six domains.

As we take shown, most Cochrane authors decided to use the other bias domain to describe potential additional biases that were not covered in the first six domains of the RoB tool. In the proposed RoB tool 2.0 there is no 'other bias' domain [seven]. The proposed RoB tool is much more complex, compared to the current version of the RoB tool, and many items that were specifically emphasized by Cochrane authors in the other bias domain, as shown in our study, are addressed in the RoB 2.0 tool. Yet, there are all the same potential biases from other sources that the RoB ii.0 may neglect past omitting the RoB domain for other bias. Relevant other bias that were identified in our report include, for example, bug with inclusion and exclusion criteria, data analyses, outcome domains and upshot measures that were used, usage of co-interventions that are non accounted for, deviations from the protocol, report pattern, issues related to specific types of trials such as cantankerous-over trials and biases specific to other to certain topics. Therefore, nosotros believe that there is a rationale for including 'other bias' domain in revised RoB tool too.

We have already conducted a like analysis of Cochrane RoB domain related to other RoB domains, and we constitute that judgments and supports for judgments in those domains were very inconsistent in Cochrane reviews [13,14,15]. This analysis related to sources of other bias in Cochrane reviews contributes to the perception that Cochrane RoB tool is inconsistently used among Cochrane authors. The authors do not necessarily follow guidance from the Cochrane Handbook. In the support for judgment, they mention issues that the Cochrane Handbook explicitly warns against. Various comments that serve every bit supports for judgments were inconsistently judged across Cochrane reviews and trials included in those reviews. Cochrane authors also apply inconsistent terminology to draw the same concepts. Increasing complexity of the RoB tool, every bit proposed in the RoB tool 2.0 volition likely only increase this problem of bereft consistency in RoB appraisal and worsen this problem of bereft comparability of judgments of RoB across Cochrane reviews.

Furthermore, our study indicated that Cochrane authors extensively utilise the bachelor option to customize the RoB tabular array. Nosotros plant that at that place were equally many as 102 (13%) out of 768 analyzed Cochrane reviews that did not use the other bias domain in the RoB table at all. Cochrane reviews are produced using the software Review Director (RevMan). As soon as an author inserts a new study in the RevMan among included studies, an empty RoB table for the written report automatically appears, with vii pre-determined domains. Therefore, Cochrane authors demand to intentionally remove or add together some domains if they want to customize the RoB tabular array. Among 102 Cochrane reviews that did not have other bias domain, 33% of those reviews had comments about other potential sources of bias in the body of the manuscript. It is unclear why some Cochrane authors apply only text for comments about other bias instead of using RoB table for this purpose. Additionally, we observed that in many Cochrane reviews without other bias domain there were other customizations of the RoB table, which had from i to six other, standard RoB domains included. Exactly half of those reviews without other bias domain in the RoB table had less than half-dozen standard domains in the RoB table.

Results of this study can contribute to better reporting of time to come systematic reviews and aid authors of systematic reviews to avoid mistakes. Firstly, results of this manuscript will provide more comprehensive information for Cochrane authors regarding 'other bias' domain – we present many sources of other bias that Cochrane authors recognize, and that are not mentioned in the Cochrane Handbook. Secondly, we showed mistakes that Cochrane authors are doing when they mention in 'other bias' domain issues that actually vest to other six domains of Cochrane RoB tool. Thirdly, nosotros are also pointing out mistakes that Cochrane authors are doing despite explicit instructions from the Handbook, i.e. authors apply sample size and funding to annotate about potential bias, even though the Handbook explicitly warns against this. Although our report was focused only on Cochrane reviews, our results are relevant also for non-Cochrane reviews that use Cochrane's risk of bias tool. Therefore, our manuscript can assistance authors of Cochrane and non-Cochrane reviews to create better and more than consistent reviews, to recognize additional potential sources of bias in trials they analyze, and to avoid mistakes that we have observed.

Limitation of our study is that nosotros included in our analysis a express number of analyzed Cochrane reviews, which were published in 2015 and 2016. We chose this convenience sample of Cochrane reviews considering we were interested in the state of the 'other bias' domain in recent times; we did not aim to clarify the change of this domain over the very long time menstruum. However, considering the number of Cochrane reviews analyzed, and the number of inconsistencies we observed, we have no reason to suspect that the results would be significantly unlike if a bigger cohort of published Cochrane reviews would have been used. It takes a long time to manually extract, check, analyze and categorize more than x thousands of RoB domains, and therefore using the same methodology on a larger sample might non be feasible. It is possible that some unintentional errors in categorizations may have been made, and therefore, for transparency, we decided to nowadays all categories and sub-categories of the supporting explanations we encountered in the Additional files 1, 2 and iii. Additionally, all systematic reviews are not the aforementioned and our findings cannot exist generalized to all systematic reviews – we analyzed only Cochrane systematic reviews of RCTs because Cochrane RoB tool was adult for these types of studies. However, we believe that our findings can be very useful also for authors of non-Cochrane reviews who will use Cochrane RoB tool in their methodology.

Finally, information technology is worth emphasizing that it is possible that some trials from our cohort were included in more than one review, and that Cochrane authors could give them different judgments for 'other bias'. It has been shown before that authors of unlike reviews can brand different RoB judgments of the same trials [16]. However, such assay was not the aim of our study.

Conclusion

Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical do and reduce confidence of practitioners in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to proceeds insight into various sources of other bias that can be found in trials, and likewise to help them avoid mistakes that were recognized in published Cochrane reviews. Potential remedies include more than attention to writer preparation, better resource for Cochrane authors, better peer-review and editorial consistency in the product of Cochrane systematic reviews.

Abbreviations

RCT:

Randomized controlled trial

RCTs:

Randomized controlled trials

RoB:

Take chances of bias

References

1. Gluud LL. Bias in clinical intervention enquiry. Am J Epidemiol. 2006;163(vi):493–501.

   Article  Google Scholar

2. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing gamble of bias in randomised trials. BMJ. 2011;343:d5928.

   Article  Google Scholar

3. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version five.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. Available from https://preparation.cochrane.org/handbook. Accessed 2 April 2019.

4. Roberts C, Torgerson DJ. Understanding controlled trials - baseline imbalance in randomised controlled trials. Br Med J. 1999;319(7203):185.

   CAS  Commodity  Google Scholar

5. Fu R, Vandermeer BW, Shamliyan TA, O'Neil ME, Yazdi F, Flim-flam SH. AHRQ methods for constructive health care: handling continuous outcomes in quantitative synthesis. Methods guide for effectiveness and comparative effectiveness reviews. Rockville: Agency for Healthcare Research and Quality (US); 2008.

   Google Scholar

6. Schulz KF. Subverting randomization in controlled trials. JAMA. 1995;274(18):1456–eight.

   CAS  Commodity  Google Scholar

7. A revised tool to assess risk of bias in randomized trials (RoB ii.0). Available at: https://sites.google.com/site/riskofbiastool//welcome/rob-2-0-tool. Accessed 2 Apr 2019.

8. Higgins J. Chapter 8: assessing risk of bias in included studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 (updated March 2011): The Cochrane Collaboration; 2011. [Available from https://training.cochrane.org/handbook]. Accessed 2 Apr 2019.

9. Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2017;2:MR000033.

   PubMed  Google Scholar

10. Bero LA. Why the cochrane hazard of bias tool should include funding source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000075.

11. Sterne JAC. Why the cochrane adventure of bias tool should not include funding source as a standard item. Cochrane Database Syst Rev. 2013;12:ED000076.

12. Schulz KF, Altman DG, Moher D. Espoused 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:eighteen.

    Commodity  Google Scholar

13. Propadalo I, Tranfic M, Vuka I, Barcot O, Poklepovic Pericic T, Puljak Fifty. In Cochrane reviews risk of bias assessments for resource allotment concealment was often not in line with cochrane's handbook guidance. J Clin Epidemiol. 2018. https://doi.org/ten.1016/j.jclinepi.2018.ten.002 In press.

    Commodity  Google Scholar

14. Babic A, Tokalic R, Silva Cunha JA, Novak I, Suto J, Vidak M, Miosic I, Vuka I, Poklepovic Pericic T, Puljak L. Risk of bias in cochrane systematic reviews: assessments of take chances related to attrition bias are highly inconsistent. bioRxiv. 2018:366658. https://doi.org/10.1101/366658.

15. Barcot O, Boric M, Poklepovic Pericic T, Cavar M, Dosenovic S, Vuka I, Puljak L. Judgments of risk of bias associated with random sequence generation in trials included in Cochrane systematic reviews are frequently erroneous. BioRxiv. 2018:366674. https://doi.org/x.1101/366674.

16. Hashemite kingdom of jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than than one systematic review. J Clin Epidemiol. 2017;81:72–6.

    Article  Google Scholar

Download references

Acknowledgments

We are grateful to Ms. Dalibora Behmen for linguistic communication editing. Nosotros are grateful to Dr. Ognjen Barcot for programming software information extraction, which was used to verify our initial manual data extraction.

Funding

No extramural funding.

Availability of information and materials

The datasets used and/or analyzed during the electric current study are available from the corresponding author on reasonable request.

Author information

Affiliations

1. Establish of Emergency Medicine in Split-Dalmatia County, Split, Republic of croatia

   Andrija Babic

2. Medical pupil, Academy of Split School of Medicine, Dissever, Croatia

   Andela Pijuk & Tina Poklepovic Pericic

3. Faculty of Medicine and Dentistry, Palacký University Olomouc, Olomouc, Czech Republic

   Lucie Brázdilová

4. Medical University of Plovdiv, Plovdiv, Republic of bulgaria

   Yuliyana Georgieva

5. Faculty of Health Sciences - University of Beira Interior, Covilhã, Portugal

   Marco António Raposo Pereira

6. Catholic University of Croatia, Zagreb, Croatia

   Livia Puljak

Contributions

Written report design: LP. Data analysis and interpretation: AB, AP, LB, YG, MARP, TPP, LP. Drafting the first version of the manuscript: AB, LP. Revisions of the manuscript: AB, AP, LB, YG, MARP, TPP, LP. All authors read and canonical the final manuscript. All authors concord to be answerable for this work.

Corresponding author

Correspondence to Livia Puljak.

Ethics declarations

Ethics approval and consent to participate

Not applicative; in this study we analyzed only published manuscripts.

Consent for publication

Not applicable.

Competing interests

Tina Poklepovic Pericic is a volunteer co-director of Cochrane Republic of croatia. Livia Puljak and Andrija Babic are volunteer members of Cochrane Republic of croatia. Livia Puljak is a Department Editor at the BMC Medical Research Methodology, only she was not involved in whatever way with editorial handling of this manuscript. Other authors have no competing interests to declare.

Publisher'due south Notation

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file ane:

Table S1. Some examples of different versions of support for judgment indicating that no other bias was found. In 268 (80%) Cochrane reviews only one version of the comment that no other bias was constitute was used, while in 69 (twenty%) reviews Cochrane authors used different expressions in comments to indicate that no other sources of bias were found. Some examples of this varied terminology are shown in Table S1. (DOCX thirteen kb)

Additional file 2:

Table S2. Cochrane systematic reviews and randomized controlled trials specifically mentioned in the results equally those that had different judgment for having no bias, partial information about other bias, or were included in secondary analyses. In 19 Cochrane reviews, all comments that referred to no other bias being identified were judged equally unclear. In i review comment, 'no other bias' was judged as both low and loftier. References to Cochrane reviews for these specific examples are shown in this Additional file. (DOCX 87 kb)

Additional file three:

Table S3. Categories of explanations of other bias in analyzed Cochrane risk of bias tables. In the 524 analyzed Cochrane reviews that described various sources of other bias, at that place were 5762 different supporting explanations for judgments of other bias that we categorized into 31 categories. The main text describes the near mutual categories of explanations, while all the types of explanations is presented in Tabular array S3. (XLSX 37 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/past/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give advisable credit to the original author(s) and the source, provide a link to the Creative Eatables license, and point if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/one.0/) applies to the information fabricated available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Cite this article

Babic, A., Pijuk, A., Brázdilová, Fifty. et al. The judgement of biases included in the category "other bias" in Cochrane systematic reviews of interventions: a systematic survey. BMC Med Res Methodol 19, 77 (2019). https://doi.org/10.1186/s12874-019-0718-8

Download citation

• Received: 04 May 2018

• Accepted: 26 March 2019

• Published: 11 April 2019

• DOI : https://doi.org/10.1186/s12874-019-0718-viii

Keywords

• Systematic review
• Cochrane
• Risk of bias
• Other bias inconsistency

stodolayousitel.blogspot.com

Source: https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-019-0718-8

Share this post