Browse the corpus

Introduction Network meta-analysis (also called multiple or mixed treatment comparison meta-analysis, MTC) permits the evaluation of the comparative effectiveness of multiple interventions.1 2 This approach has an inherent appeal for clinicians and decision makers as new or existing interventions must be placed within the context of all available evidence.3 4 5 6 Often, those undertaking an MTC will selectively choose interventions to include in the analysis. For example, some MTCs exclude placebo or no treatment from consideration because it is sometimes believed that placebo trials vary over time or are set in favourable conditions to appease regulatory authorities.7 Other MTCs may include only the treatments available in particular settings (for example, a specific country), only those of perceived dose relevance, or (often in the case of industry submissions to health technology assessment bodies) only specific competing treatments.8 To obtain empirical evidence on whether these choices make a difference in the results such as treatment effect estimates and treatment rankings, we examined a sample of complex networks and reanalysed their data after excluding specific treatment nodes.

echnology assessment bodies) only specific competing treatments.8 To obtain empirical evidence on whether these choices make a difference in the results such as treatment effect estimates and treatment rankings, we examined a sample of complex networks and reanalysed their data after excluding specific treatment nodes. Methods Eligibility criteria and retrieval of data from existing networks We considered networks that had five or more treatments, contained at least two closed loops, had at least twice as many studies as nodes, and had individual trial level data or estimates available. The eligibility criteria aimed to generate a sample of networks that had many treatments and studies and sufficient data to explore the impact of exclusions. We used a systematic literature search that has been published previously that identified potentially eligible networks.9 We also attempted to contact study authors for missing individual data at trial level. We included an additional network from an MTC conducted by our team, where we had direct access to the primary data at trial level. In studies that considered more than one outcome using MTCs, we favoured the efficacy outcome over safety outcomes.

ted to contact study authors for missing individual data at trial level. We included an additional network from an MTC conducted by our team, where we had direct access to the primary data at trial level. In studies that considered more than one outcome using MTCs, we favoured the efficacy outcome over safety outcomes. Data abstraction For each eligible network with available trial level data, we recorded whether the eligibility criteria excluded specific types of active or inactive/control (placebo, no treatment, best supportive care) treatment comparators, and the rationale for such exclusions. We recorded for each network the number of studies, treatments, and loops; the geometry of the network (the distribution of treatments and comparisons thereof in each network); the condition being treated; the primary outcome measure and the statistical effect measure used; and the range of node connectivity (the number of direct comparisons connected to each node). The supplementary figure displays the concepts of loops and connectivity. Statistical analysis Regardless of the analyses chosen in the original publications, we analysed each network using random effects Bayesian MTCs with uninformed priors, the most common analytical approach used for network meta-analysis.8 Details on code and specific analysis are available from the authors.

Data abstraction For each eligible network with available trial level data, we recorded whether the eligibility criteria excluded specific types of active or inactive/control (placebo, no treatment, best supportive care) treatment comparators, and the rationale for such exclusions. We recorded for each network the number of studies, treatments, and loops; the geometry of the network (the distribution of treatments and comparisons thereof in each network); the condition being treated; the primary outcome measure and the statistical effect measure used; and the range of node connectivity (the number of direct comparisons connected to each node). The supplementary figure displays the concepts of loops and connectivity. Statistical analysis Regardless of the analyses chosen in the original publications, we analysed each network using random effects Bayesian MTCs with uninformed priors, the most common analytical approach used for network meta-analysis.8 Details on code and specific analysis are available from the authors. For each network we analysed the complete available data (full model) and also performed analyses excluding one or multiple treatment nodes—that is, disregarding in the calculations data from trials where the excluded nodes were comparators. Firstly, we investigated the effect of excluding the treatment node with the largest expected impact from each network. We used the Brier score to identify the treatment node with the largest expected impact on results.10 The Brier score is the average of the squared differences between the log ratios (odds, relative risk or hazard) estimated with the full treatment network data versus the treatment network data where one or more treatment nodes are excluded. Secondly, we investigated the effect of excluding other single treatment nodes that could be classified as active interventions (that is, not placebo/no treatment). Thirdly, we investigated the impact of excluding placebo/no treatment from the treatment network. Lastly, we focused on selected examples of situation specific exclusions that were chosen owing to perceived relevance for clinical practise or decision making. Specifically, for the network of thrombolytic therapy we excluded data from trials involving anisoylated plasminogen streptokinase activated complex (ASPAC) and urokinase because these treatments were not available in the United Kingdom and one previous UK based network meta-analysis had excluded them11; and for the networks of breast, colorectal, and ovarian cancer, we excluded nodes of treatment regimens that are currently uncommonly used (all those deemed miscellaneous older agents alone or in combination for breast cancer; fluorouracil monotherapy and older drug monotherapy for colorectal cancer; platinum monotherapy and non-platinum/non-taxane drug regimens for ovarian cancer).

e excluded nodes of treatment regimens that are currently uncommonly used (all those deemed miscellaneous older agents alone or in combination for breast cancer; fluorouracil monotherapy and older drug monotherapy for colorectal cancer; platinum monotherapy and non-platinum/non-taxane drug regimens for ovarian cancer). We used odds ratios for dichotomous outcomes, rate ratios for Poisson outcomes, and hazard ratios for survival outcomes. From each network we estimated treatment effects for each treatment comparison and the probability of each treatment being the best treatment.12

e excluded nodes of treatment regimens that are currently uncommonly used (all those deemed miscellaneous older agents alone or in combination for breast cancer; fluorouracil monotherapy and older drug monotherapy for colorectal cancer; platinum monotherapy and non-platinum/non-taxane drug regimens for ovarian cancer). We used odds ratios for dichotomous outcomes, rate ratios for Poisson outcomes, and hazard ratios for survival outcomes. From each network we estimated treatment effects for each treatment comparison and the probability of each treatment being the best treatment.12 To evaluate the impact of excluding treatment nodes we calculated the relative change in the estimated treatment effects of the remaining treatments; changes in the top three ranked treatments, according to the probability of being best; and changes in the probability of being the best treatment for the top treatment of the full model. Relative changes in the estimated treatment effects of remaining nodes are always expressed as a value greater than 1.00—for example, if a specific odds ratio is 0.80 in the full network and 0.88 in the network with one node excluded, then the relative change is 0.88/0.80=1.10-fold; if these odds ratios are 0.80 and 0.60, then the relative change is 0.80/0.60=1.25-fold. Given that most treatment effects are relatively small or modest,13 relative changes of 1.10-fold can be considered large and of 1.20-fold can be considered substantial. Choice of reference group does not affect the values of the fold change. As most networks have a natural choice for the reference group we chose to keep this node as a reference group to best mimic changes that would be observed by researchers. For each network with excluded nodes, we noted the maximum and geometric average of the relative change.

s not affect the values of the fold change. As most networks have a natural choice for the reference group we chose to keep this node as a reference group to best mimic changes that would be observed by researchers. For each network with excluded nodes, we noted the maximum and geometric average of the relative change. Analyses were conducted using WinBUGS version 1.4 (Medical Research Council Biostatistics Unit, Cambridge) and R version 2.15.1 (www.r-project.org/). Results Analysed networks The search identified 890 relevant abstracts, of which 276 were assessed as potentially eligible and their full articles were screened. After adding an eligible network from our own available data, 18 networks with individual trial data met the inclusion criteria,14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 involving 757 randomised controlled trials with 750 possible treatment comparisons and 1 036 701 patients. Table 1 shows the characteristics of the individual networks. Networks ranged in size from 11 studies on five treatments to 128 studies on 22 treatments (figure). Networks included in analysis. The line thickness is proportional to the number of trials comparing the treatments along the edge. Nodes leading to changes in ranks among the top ranked treatments are in large font Table 1  Study characteristics

Results Analysed networks The search identified 890 relevant abstracts, of which 276 were assessed as potentially eligible and their full articles were screened. After adding an eligible network from our own available data, 18 networks with individual trial data met the inclusion criteria,14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 involving 757 randomised controlled trials with 750 possible treatment comparisons and 1 036 701 patients. Table 1 shows the characteristics of the individual networks. Networks ranged in size from 11 studies on five treatments to 128 studies on 22 treatments (figure). Networks included in analysis. The line thickness is proportional to the number of trials comparing the treatments along the edge. Nodes leading to changes in ranks among the top ranked treatments are in large font Table 1  Study characteristics Network Type of treatments Condition/outcome Consideration/exclusion of specific treatment comparators No of studies (nodes) No of loops Effect measure Connectivity (min/max)† Ara 200914 Statins Hypercholesterolaemia/discontinuation due to adverse events Only trials involving high dose statins or low dose statins 12 (5) 5 Odds ratio 2/4 Bansback 200915 Biologicals Moderate to severe plaque psoriasis All options considered 22 (8) 2 Relative risk* 1/3 Brown 200616 Pharmacological interventions NSAID induced gastrointestinal toxicity/serious gastrointestinal complications Only trials involving specific strategies were considered 40 (6) 6 Relative risk* 2/4 Cipriani 200917 Antidepressants Unipolar major depression in adults/response to treatment Only trials involving second generation antidepressants were considered (not placebo) 111 (12) 70 Odds ratio 3/11 Cooper 200618 Anticoagulants and antiplatelet therapy Non-rheumatic atrial fibrillation/ischaemic stroke prevention All options considered 19 (9) 11 Rate ratio 1/7 Dias 201019 Thrombolytic drugs and angioplasty Acute myocardial infarction/death All options considered (previous analysis excluded two treatments not licensed in the UK) 50 (9) 11 Odds ratio 1/7 Elliot 200720 Antihypertensive drugs Hypertension, high risk patients/patients who developed diabetes All options considered 22 (6) 16 Odds ratio 4/5 Golfinopoulos 200721 Chemotherapy regimens Colorectal cancer Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy 40 (11) 7 Hazard ratio 1/7 Imamura 201022 Non-surgical Stress urinary incontinence/cure Only non-surgical options 38 (13) 26 Odds ratio 1/7 Kyrgiou 200623 Chemotherapy regimens Ovarian cancer Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy 61 (9) 10 Hazard ratio 1/6 Lam 200824 Implantable defibrillator Left ventricular dysfunction All options considered 11 (5) 3 Odds ratio 2/4 Mauri 200825 Chemotherapy regimens Breast cancer Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy 128 (22) 27 Hazard ratio 1/10 Mills 201126 COPD drugs COPD/exacerbations All options considered 25 (10) 7 Rate ratio 1/6 Mills 201227 Nicotine

n All options considered 11 (5) 3 Odds ratio 2/4 Mauri 200825 Chemotherapy regimens Breast cancer Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy 128 (22) 27 Hazard ratio 1/10 Mills 201126 COPD drugs COPD/exacerbations All options considered 25 (10) 7 Rate ratio 1/6 Mills 201227 Nicotine replacement therapy Smoking addiction/cessation All options considered 100 (6) 4 Odds ratio 1/4 Psaty 199728 Antihypertensive therapy Coronary heart disease/fatal and non-fatal events Only the six most commonly used antihypertensives 28 (7) 10 Relative risk* 1/5 Sciarretta 201129 Antihypertensive therapy Heart failure/prevention of heart failure Excluded trials before 1997 26 (8) 13 Odds ratio 1/6 Thijs 200830 Antiplatelets Transient ischaemic attack or stroke Excluded oral anticoagulants, trifusal, cilostazol, dipyridamole alone, and glycoprotein llb/lla inhibitors 24 (5) 3 Odds ratio 2/4 Wang 201031 Central venous catheters Catheter related infections Excluded self preparing coated catheters 43 (9) 4 Odds ratio 1/4 NSAID=non-steroidal anti-inflammatory drug; COPD=chronic obstructive pulmonary disease. *Our analyses use odds ratios to avoid issues of non-transitivity with risk ratios. †Connectivity reports the smallest and largest number of connections in each network.

replacement therapy Smoking addiction/cessation All options considered 100 (6) 4 Odds ratio 1/4 Psaty 199728 Antihypertensive therapy Coronary heart disease/fatal and non-fatal events Only the six most commonly used antihypertensives 28 (7) 10 Relative risk* 1/5 Sciarretta 201129 Antihypertensive therapy Heart failure/prevention of heart failure Excluded trials before 1997 26 (8) 13 Odds ratio 1/6 Thijs 200830 Antiplatelets Transient ischaemic attack or stroke Excluded oral anticoagulants, trifusal, cilostazol, dipyridamole alone, and glycoprotein llb/lla inhibitors 24 (5) 3 Odds ratio 2/4 Wang 201031 Central venous catheters Catheter related infections Excluded self preparing coated catheters 43 (9) 4 Odds ratio 1/4 NSAID=non-steroidal anti-inflammatory drug; COPD=chronic obstructive pulmonary disease. *Our analyses use odds ratios to avoid issues of non-transitivity with risk ratios. †Connectivity reports the smallest and largest number of connections in each network. Exclusion of treatment nodes in analysed networks Table 1 shows the eligibility criteria that had been set by the original authors of these meta-analyses and that resulted in exclusion of specific treatment comparators from the network. Eleven of the 18 networks upfront decided not to consider all treatment comparators. One network mentioned that a previous network meta-analysis had excluded data on two treatments (ASPAC and urokinase for thrombolysis) that were not licensed in the United Kingdom, but the full data were available in the current analysis. Several networks deliberately excluded placebos or no treatment options (for example, best supportive care for cancers). Several other networks focused on specific types of treatments that they considered to be of clinical interest and made statements that other treatments were not considered at all, because they were of different class, old (for example, trials published before 1997), or to be considered in a separate review.

cancers). Several other networks focused on specific types of treatments that they considered to be of clinical interest and made statements that other treatments were not considered at all, because they were of different class, old (for example, trials published before 1997), or to be considered in a separate review. Changes after removal of treatment node with largest expected impact When the node with the largest Brier score was removed (table 2) the average relative change of treatments effects exceeded 1.10-fold in seven of the 18 networks.14 16 18 22 27 29 31 The largest observed relative change exceeded 1.10-fold (maximum 3.64-fold) in all but four networks. In three networks some non-significant effects became significant, and in 12 networks the opposite change was seen. Switches in the ranking of the top three treatments were observed in nine of the 18 networks.16 17 18 21 22 25 27 29 31 Table 3 shows a worked example of the calculations and results for a single node removal in the Cooper 2006 network.18 Table 2  Nodes leading to greatest change in estimated effects and average fold changes in estimated treatment effects when single treatment nodes are removed

Changes after removal of treatment node with largest expected impact When the node with the largest Brier score was removed (table 2) the average relative change of treatments effects exceeded 1.10-fold in seven of the 18 networks.14 16 18 22 27 29 31 The largest observed relative change exceeded 1.10-fold (maximum 3.64-fold) in all but four networks. In three networks some non-significant effects became significant, and in 12 networks the opposite change was seen. Switches in the ranking of the top three treatments were observed in nine of the 18 networks.16 17 18 21 22 25 27 29 31 Table 3 shows a worked example of the calculations and results for a single node removal in the Cooper 2006 network.18 Table 2  Nodes leading to greatest change in estimated effects and average fold changes in estimated treatment effects when single treatment nodes are removed Network Node leading to largest Brier score* Connectivity of removed node Removal of largest expected impact node Removal of other active intervention nodes Average fold change Largest fold change Average fold change Largest fold change Change in top three ranked treatments: Brown 200616 Cox 2 preferentials (20%) 3 (max 4) 1.83 3.13 1.38 4.40 Cipriani 200917 Fluoxetine (0%) 11 (max) 1.06 1.18 1.03 1.16 Cooper 200618 Fixed low dose warfarin+aspirin (0%) 4 (max 7) 1.25 2.65 1.06 1.60 Golfinopoulos 200721 Fluorouracil (0%) 5 (max 7)† 1.03 1.05 1.01 1.09 Imamura 201022 Pelvic floor muscle training+biofeedback (5%) 6 (max 7) 1.22 1.40 1.07 2.18 Mauri 200825 Standard dose anthracycline (0%) 10 (max) 1.05 1.12 1.01 1.09 Mills 2012 (6 mth)27 Low dose nicotine replacement therapy (0%) 4 (max) 1.12 1.39 1.01 1.03 Sciarretta 201129 Calcium channel blockers (1%) 6 (max) 1.27 1.53 1.04 1.31 Wang 201031 Chlorhexidine and silver sulfadiazine (0%) 4 (max) 1.27 4.64 1.01 1.08 No changes in top three ranked treatments: Ara 200914 Simvastatin 40 mg (21%) 4 (max) 1.13 1.19 1.04 1.11 Bansback 200915 Adalimumab (0%) 2 (max 3) 1.08 1.32 1.01 1.02 Dias 201019 Accelerated alteplase (0%) 7 (max) 1.08 1.16 1.03 1.07 Elliot 200720 Angiotensin converting enzyme inhibitors (22%) 4 (max 5) 1.08 1.12 1.02 1.06 Kyrgiou 200623 Platinum monotherapy (0%) 6 (max) 1.08 1.25 1.02 1.13 Lam 200824 Implantable cardioverter defibrillator (9%) 3 (max 4)† 1.03 1.07 1.01 1.04 Mills 201126 Inhaled corticosteroids (0%) 3 (max 6) 1.01 1.02 1.00 1.01 Psaty 199728 Diuretics (10%) 5 (max) 1.06 1.14 1.03 1.08 Thijs 200830 Aspirin (0%) 4 (max) 1.03 1.04 1.01 1.03 *Probability of being best treatment.

.25 1.02 1.13 Lam 200824 Implantable cardioverter defibrillator (9%) 3 (max 4)† 1.03 1.07 1.01 1.04 Mills 201126 Inhaled corticosteroids (0%) 3 (max 6) 1.01 1.02 1.00 1.01 Psaty 199728 Diuretics (10%) 5 (max) 1.06 1.14 1.03 1.08 Thijs 200830 Aspirin (0%) 4 (max) 1.03 1.04 1.01 1.03 *Probability of being best treatment. †Most connected node that was removed. Table 3  Example of fold change calculation and change in rank probabilities for Cooper 200618 network Treatment Full model Reduced model Fold change* Relative rate v placebo (95% CI) Probability (%) Relative rate v placebo (95% CI) Probability (%) Warfarin: Adjusted standard dose 0.37 (0.26 to 0.53) 3 0.41 (0.27 to 0.7) 2 1.11 Adjusted low dose 0.34 (0.19 to 0.58) 13 0.34 (0.19 to 0.60) 41 1.00 Fixed low dose 0.76 (0.3 to 1.76) 1 0.93 (0.36 to 2.66) 1 1.22 Aspirin 0.62 (0.43 to 0.86) 0 0.66 (0.47 to 1.01) 0 1.06 Fixed low dose warfarin+aspirin 0.98 (0.6 to 1.67) 0 Removed Removed NA Ximelagatran 0.34 (0.18 to 0.62) 14 0.37 (0.21 to 0.83) 13 1.09 Alternate day aspirin 0.17 (0.01 to 1.15) 66 0.45 (0.03 to 18.48) 38 2.65 Indobufen 0.46 (0.19 to 1.14) 5 0.54 (0.23 to 1.57) 6 1.17 NA=not applicable. *Average fold change reported in table 2 is averaged on log scale. The average of these fold changes is 1.33, slightly higher than 1.25.

Treatment Full model Reduced model Fold change* Relative rate v placebo (95% CI) Probability (%) Relative rate v placebo (95% CI) Probability (%) Warfarin: Adjusted standard dose 0.37 (0.26 to 0.53) 3 0.41 (0.27 to 0.7) 2 1.11 Adjusted low dose 0.34 (0.19 to 0.58) 13 0.34 (0.19 to 0.60) 41 1.00 Fixed low dose 0.76 (0.3 to 1.76) 1 0.93 (0.36 to 2.66) 1 1.22 Aspirin 0.62 (0.43 to 0.86) 0 0.66 (0.47 to 1.01) 0 1.06 Fixed low dose warfarin+aspirin 0.98 (0.6 to 1.67) 0 Removed Removed NA Ximelagatran 0.34 (0.18 to 0.62) 14 0.37 (0.21 to 0.83) 13 1.09 Alternate day aspirin 0.17 (0.01 to 1.15) 66 0.45 (0.03 to 18.48) 38 2.65 Indobufen 0.46 (0.19 to 1.14) 5 0.54 (0.23 to 1.57) 6 1.17 NA=not applicable. *Average fold change reported in table 2 is averaged on log scale. The average of these fold changes is 1.33, slightly higher than 1.25. The most influential nodes were typically highly connected. In 10 of the 18 networks they actually represented the most connected node. However, the most influential nodes were never the top ranked treatment of the networks and almost always (15/18) had a 10% or less probability of being the best treatment (0% probability in 11/18 networks). Thus standards of care, placebo, or other treatments that may easily be overlooked represent nodes of which the exclusion could be influential to the network results.

ranked treatment of the networks and almost always (15/18) had a 10% or less probability of being the best treatment (0% probability in 11/18 networks). Thus standards of care, placebo, or other treatments that may easily be overlooked represent nodes of which the exclusion could be influential to the network results. Changes of removal after other active intervention nodes Removal of the remaining, less influential nodes caused minor changes (average relative change ≤1.03-fold) in 13 of the 18 networks. None the less, seven networks contained a node the exclusion of which caused a relative change greater than 1.10-fold. Of the nine networks with changes in the top ranks by the exclusion of the most influential node, three had an additional node whose removal affected the top ranks.16 18 29

d) in 13 of the 18 networks. None the less, seven networks contained a node the exclusion of which caused a relative change greater than 1.10-fold. Of the nine networks with changes in the top ranks by the exclusion of the most influential node, three had an additional node whose removal affected the top ranks.16 18 29 Changes after placebo node removals or context specific node removals Table 4 presents the results after removal of placebo/no treatment nodes or after removal of context specific sets of treatment nodes. Placebos and no treatment nodes were removed from 10 networks that had originally included such nodes. This resulted in important changes in the results of four of these 10 networks,16 18 22 27 with average relative changes ranging from 1.16-fold to 3.10-fold, whereas in the other six networks the average relative changes were minor (1.01-1.03-fold). In one network, exclusion of the placebo node resulted in a switch in position between the first and second best treatment. In the other cases, the ranking of the top three treatments did not change, but modest to large changes were seen in the probability of being the best treatment, such as in Mills 2011 in which the probability of high dose nicotine replacement being the best treatment went from 58% to 90% (table 4).26 Table 4  Average fold changes in treatment effects and impact on treatment ranks and probability of best treatment with clinically relevant removals

Changes after placebo node removals or context specific node removals Table 4 presents the results after removal of placebo/no treatment nodes or after removal of context specific sets of treatment nodes. Placebos and no treatment nodes were removed from 10 networks that had originally included such nodes. This resulted in important changes in the results of four of these 10 networks,16 18 22 27 with average relative changes ranging from 1.16-fold to 3.10-fold, whereas in the other six networks the average relative changes were minor (1.01-1.03-fold). In one network, exclusion of the placebo node resulted in a switch in position between the first and second best treatment. In the other cases, the ranking of the top three treatments did not change, but modest to large changes were seen in the probability of being the best treatment, such as in Mills 2011 in which the probability of high dose nicotine replacement being the best treatment went from 58% to 90% (table 4).26 Table 4  Average fold changes in treatment effects and impact on treatment ranks and probability of best treatment with clinically relevant removals Network Nodes deleted Average fold change Largest fold change Change in top three treatment ranks Change in probability of best treatment Full model Reduced model Changes after removal of placebo/no treatment nodes: Ara 200914 Placebo 1.03 1.05 None Rosuvastatin (53%) Rosuvastatin (56%) Brown 200616 Placebo 3.10 4.44 None Cox 2 inhibitors (41%) Cox 2 inhibitors (43%) Cooper 200618 Placebo 1.16 1.50 None Alternate day aspirin (66%) Alternate day aspirin (79%) Elliot 200720 Placebo 1.02 1.06 None Angiotensin receptor blocker (77%) Angiotensin receptor blocker (83%) Imamura 201022 No treatment 1.21 1.77 None Pelvic floor muscle training+biofeedback+bladder training (61%) Pelvic floor muscle training+biofeedback+BT (56%) Mills 201126 Placebo 1.01 1.02 None Roflumilast+long acting muscarinic antagonists (49%) Roflumilast+long acting muscarinic antagonists (56%) Mills 201227 Placebo 1.22 1.40 None High dose nicotine replacement therapy (58%) High dose nicotine replacement therapy (90%) Psaty 199728 Placebo 1.01 1.03 None Angiotensin converting enzyme inhibitors (55%) Angiotensin converting enzyme inhibitors (43%) Sciarretta 201129 Placebo 1.03 1.06 Top two switch β blockers (29%) Angiotensin converting enzyme inhibitors (32%) Thijs 200830 Placebo 1.01 1.03 None Aspirin+dipyridamole (80%) Aspirin+dipyridamol (70%) Changes after removal of situation specific relevant treatment nodes Dias 201019 ASPAC and urokinase 1.04 1.06 None Percutaneous transluminal coronary angioplasty (83%) Percutaneous transluminal coronary angioplasty (96%) Golfinopoulos 200721 Monotherapies: fluorouracil, oxaliplatin, irinotecan 1.06 1.18 None Fluorouracil and leucovorin+irinotecan+bevacizumab (63%) Fluorouracil and leucovorin+irinotecan+bevacizumab (48%) Kyrgiou 200623 Platinum monotherapy and non-platinum/non-taxane drugs 1.21 1.68 None Platinum+taxane based combination (ip) (92%) Platinum+taxane based combination (ip) (89%) Mauri 200825 All drugs deemed miscellaneous 1.05 1.17 3rd becomes 1st Novel non-taxane agents+taxanes (24%) Taxanes (combination regimen) (21%) APSAC=anisoylated plasminogen streptokinase activated complex; ip=including at least one agent given intraperitoneally.

(ip) (92%) Platinum+taxane based combination (ip) (89%) Mauri 200825 All drugs deemed miscellaneous 1.05 1.17 3rd becomes 1st Novel non-taxane agents+taxanes (24%) Taxanes (combination regimen) (21%) APSAC=anisoylated plasminogen streptokinase activated complex; ip=including at least one agent given intraperitoneally. Removal of APSAC and urokinase from the network on thrombolysis and percutaneous intervention did not result in major changes. Removal of uncommonly used treatment regimens from the networks of treatments for malignancies resulted in a large change in estimated effects (on average by 1.21-fold) for ovarian cancer and small changes for colorectal and breast cancer. However, in the case of advanced breast cancer, these removals switched the ranks of novel non-taxane agents+taxanes and taxanes (combination regimen), the first and third ranked treatments, respectively.

estimated effects (on average by 1.21-fold) for ovarian cancer and small changes for colorectal and breast cancer. However, in the case of advanced breast cancer, these removals switched the ranks of novel non-taxane agents+taxanes and taxanes (combination regimen), the first and third ranked treatments, respectively. Discussion In this metaepidemiological study involving 18 large network meta-analyses, we found that it is common for networks to exclude trials and specific comparators based on widely diverse criteria. Furthermore, we documented that exclusions occasionally can have an important impact on the results. The comparators that had the largest potential to change the results were typically those used in many trials; and they were unlikely to be the best treatments. We also explored the impact of exclusion of each other active treatment and placebo/no treatment as well as the impact of excluding several treatments based on scenarios that may be clinically relevant. Our results show, moreover, that some of these exclusions could affect substantially the estimated treatment effects and occasionally even affect the ranking of the top treatments.

reatment and placebo/no treatment as well as the impact of excluding several treatments based on scenarios that may be clinically relevant. Our results show, moreover, that some of these exclusions could affect substantially the estimated treatment effects and occasionally even affect the ranking of the top treatments. These findings seem particularly relevant to network meta-analysis where analysts choose only to evaluate certain newer treatments or where they have chosen to exclude well established interventions or placebo, or both from a network. Readers of network meta-analysis should examine whether a network represents all available interventions. Excluding treatments from networks may reduce the applicability and usefulness of a multiple or mixed treatment comparison (MTC) if comparisons of interest to doctors are not considered. A substantial literature already exists on traditional pairwise meta-analyses, showing how eligibility criteria may result in differences in the results and conclusions of meta-analyses on the same topic.32 33 34 35 36 As meta-analyses have become popular, most of the topics of interest are addressed by two or more published meta-analyses and these may differ in their eligibility criteria.37 Some examples have started to accumulate where independent network analyses on the same topic may reach different conclusions.38 A key consideration may be which treatment nodes are considered eligible for analysis. For example, on evaluating the relative merits of second generation antidepressants, an MTC that did not consider placebo comparisons yielded different rankings from one that included only placebo controlled comparisons and one that considered both types of comparisons.17 39 40 41 We should also note that although treatment rankings and probabilities are arguably easy to interpret, their interpretability is limited by the fact that they are driven predominantly by the estimated effect sizes, and that standard errors play an unduly small role in determining their position.32 Readers are advised to observe the estimated effects first and use the rankings only as a supplementary measure.

interpret, their interpretability is limited by the fact that they are driven predominantly by the estimated effect sizes, and that standard errors play an unduly small role in determining their position.32 Readers are advised to observe the estimated effects first and use the rankings only as a supplementary measure. Limitations of this study There are some caveats to our analysis. Firstly, we excluded simple networks such as star networks (that cannot be analysed in an MTC)42 and poorly connected networks. Less well connected networks with less total evidence are likely to be more “fragile” when nodes are removed—that is, the results may change even more from what we documented here for well connected networks. Secondly, we applied a uniform approach—a random effects analysis using non-informative priors—to our reanalysis of each published network meta-analysis. Different analytical choices may introduce some further variability in the results. Thirdly, some of the examined networks had already excluded specific trials and treatments upfront, and it was not practical for us to try to retrieve the excluded trials and evaluate the impact of their inclusion. Conversely, we focused on the potential impact that further exclusions would have had on the results. Fourthly, we did not question the clustering of treatment regimens into specific nodes by the original meta-analysts. However, in some MTCs there may be some ambiguity on whether slightly or modestly different regimens (for example, different doses or schedules for administering drugs or drugs belonging to the same class) should be treated as a single node or separate nodes. This could introduce some additional variability in the results, depending on what choices are made. For example, in the MTC of systemic treatment for colorectal cancer,21 242 trials were identified but only 37 could be used, because the others were deemed to compare regimens that belonged to the same treatment node and thus would not be distinguishable. In all, given these reasons our estimates of the potential impact on the results of choices pertaining to the geometry of the network are probably conservative. To avoid potential bias, network meta-analysts should consider the inclusion of all relevant interventions for the condition of interest.

t be distinguishable. In all, given these reasons our estimates of the potential impact on the results of choices pertaining to the geometry of the network are probably conservative. To avoid potential bias, network meta-analysts should consider the inclusion of all relevant interventions for the condition of interest. We should acknowledge that the choice of including all possible interventions that have ever been evaluated in randomised controlled trials on a topic can be daunting. Sometimes specific exclusions are clearly justifiable, and many of these exclusions may not have any substantial impact on the estimated treatment effects of the remaining “relevant” interventions. However, exclusions of potential nodes and decisions about eligibility criteria need to be carefully justified—for example, it may be argued that alternative medicine or non-mainstream comparators should not be included in the same network as standard interventions. It may even be reasonable to perform sensitivity analyses to examine the impact of the removal of specific nodes. Moreover, our findings indicate that the largest impact on the results occurs when well connected nodes are removed. It seems reasonable to advocate that the most evaluated treatments available for a condition should be considered necessary to include for a network to be valid. Typically, this includes older established standards of care, placebo, and well-tested interventions. In particular, some industry sponsored meta-analyses increasingly focus on target competitor agents that compete for market share.43 However, the exclusion of other interventions in a network can importantly affect the results.

, this includes older established standards of care, placebo, and well-tested interventions. In particular, some industry sponsored meta-analyses increasingly focus on target competitor agents that compete for market share.43 However, the exclusion of other interventions in a network can importantly affect the results. What is already known on this topic Network meta-analysis is an increasingly popular method that allows the comparative effectiveness of multiple treatments to be evaluated Examples have started to accumulate where network analyses on the same topic have reached different conclusions based on the exclusion of treatment nodes What this study adds It is common for networks to exclude trials and specific comparators based on widely diverse criteria Excluding treatments from a network meta-analysis can importantly change effect estimates and the probability rankings of being the best treatment Well connected treatments that are unlikely to be the best treatment are the most likely to be influential The authors appreciate the contribution of Georgia Salanti for providing data and comments on protocol and analysis. Contributors: EJM, SK, KT, and JPAI conceived the study design. AC, A-AV, and SK acquired the data. SK, KT, and EJM conducted the analyses. EJM, SK, KT, and JPAI wrote the manuscript and all contributed to the writing. All authors approved the final manuscript. EJM is guarantor for the study. Funding: This study was supported by the Drug Safety and Effectiveness Network (DSEN) of the Canadian Institutes of Health Research. No funding agency has seen this study.

Contributors: EJM, SK, KT, and JPAI conceived the study design. AC, A-AV, and SK acquired the data. SK, KT, and EJM conducted the analyses. EJM, SK, KT, and JPAI wrote the manuscript and all contributed to the writing. All authors approved the final manuscript. EJM is guarantor for the study. Funding: This study was supported by the Drug Safety and Effectiveness Network (DSEN) of the Canadian Institutes of Health Research. No funding agency has seen this study. Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. Ethical approval: Not required. Data sharing: The technical appendix and statistical code are available from the corresponding author at jioannid@stanford.edu. Cite this as: BMJ 2013;347:f5195