Browse the corpus

Background Quantifying diagnostic accuracy is an important first step in assessing whether a new diagnostic device is suitable for implementation into clinical practice. Without initial evidence as to whether a device is able to improve diagnostic performance, it is difficult to justify larger studies to assess the impact on patient outcomes. To many clinicians and researchers, statistical measures of diagnostic accuracy (which we refer to in this paper as ‘technical accuracy’) may appear counterintuitive and may not adequately reflect how a test result should influence decisions about the treatment of the patient.1 This difficulty arises because many test accuracy study results are expressed in terms of sensitivity and specificity rather than measures of ‘clinical accuracy’; that is, the probability that the patient has the disease or condition under consideration after receiving a positive or a negative test result.2 3 There is also evidence that many clinicians find it difficult to extract usable probabilistic information from diagnostic test accuracy results in the way that they are typically reported.4 5 However, there are conflicting opinions on the extent to which this depends on the type of information provided.6

To many clinicians and researchers, statistical measures of diagnostic accuracy (which we refer to in this paper as ‘technical accuracy’) may appear counterintuitive and may not adequately reflect how a test result should influence decisions about the treatment of the patient.1 This difficulty arises because many test accuracy study results are expressed in terms of sensitivity and specificity rather than measures of ‘clinical accuracy’; that is, the probability that the patient has the disease or condition under consideration after receiving a positive or a negative test result.2 3 There is also evidence that many clinicians find it difficult to extract usable probabilistic information from diagnostic test accuracy results in the way that they are typically reported.4 5 However, there are conflicting opinions on the extent to which this depends on the type of information provided.6 The purpose of this article is twofold: to review the concepts of technical accuracy and clinical accuracy and highlight the measures of diagnostic performance that are particularly useful for statisticians, on the one hand, and patients and clinicians, on the other, and to demonstrate an interactive graphical interface to help medical educators and health professionals to teach, design and interpret the results of diagnostic accuracy studies.

asures of diagnostic performance that are particularly useful for statisticians, on the one hand, and patients and clinicians, on the other, and to demonstrate an interactive graphical interface to help medical educators and health professionals to teach, design and interpret the results of diagnostic accuracy studies. Example Serum C reactive protein (CRP) is indicated as a marker of acute and chronic inflammation and bacterial infection and is widely used to assist in the diagnosis of these conditions.7 For illustration, we consider here the study of Liu et al,8 conducted in an older patient group (age >70 years). Defining elevated CRP levels as those exceeding 60 mg/L, the article reports the results in table 1 to show CRP test performance in relation to diagnosing bacterial infection, as assessed using a reference test based on clinical and microbiological criteria. The number of patients in each cell of the table is labelled as the number of true positive (TP), false positive (FP), false negative (FN) and true negative (TN) test results. Table 1 Summary results table from a study of CRP and infection Reference test result Definite, probable or possible infection No infection Total CRP test result Positive: elevated CRP (>60 mg/L) TP=67 FP=6 73 Negative: non-elevated CRP (<60 mg/L) FN=16 TN=143 159 Total 83 149 232 CRP, C reactive protein; FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Reference test result Definite, probable or possible infection No infection Total CRP test result Positive: elevated CRP (>60 mg/L) TP=67 FP=6 73 Negative: non-elevated CRP (<60 mg/L) FN=16 TN=143 159 Total 83 149 232 CRP, C reactive protein; FN, false negative; FP, false positive; TN, true negative; TP, true positive. Assessing diagnostic performance Often, the diagnostic performance of the test is expressed using as summary statistics the sensitivity (proportion of infections correctly identified by the CRP test, TP/(TP+FN)=67/83=81%) and the specificity (proportion of non-infections correctly identified by the CRP test, TN/(FP+TN)=143/149=96%).9 Although widely used, these statistics do not by themselves enable the user to judge the probability that a patient who receives a particular CRP test result has infection. This probability depends additionally on the prevalence, or pre-test probability, of infection—how common bacterial infections are in the patient group under consideration. In this case, the estimated prevalence is 83/232=36%. In the context of a single study, the relevant post-test probabilities, or ‘predictive values’, can be calculated directly. The data in table 1 enable us to estimate the positive predictive value (TP/(TP+FP)=67/73=92%) and the negative predictive value (TN/(FN +TN)=143/159=90%).

Assessing diagnostic performance Often, the diagnostic performance of the test is expressed using as summary statistics the sensitivity (proportion of infections correctly identified by the CRP test, TP/(TP+FN)=67/83=81%) and the specificity (proportion of non-infections correctly identified by the CRP test, TN/(FP+TN)=143/149=96%).9 Although widely used, these statistics do not by themselves enable the user to judge the probability that a patient who receives a particular CRP test result has infection. This probability depends additionally on the prevalence, or pre-test probability, of infection—how common bacterial infections are in the patient group under consideration. In this case, the estimated prevalence is 83/232=36%. In the context of a single study, the relevant post-test probabilities, or ‘predictive values’, can be calculated directly. The data in table 1 enable us to estimate the positive predictive value (TP/(TP+FP)=67/73=92%) and the negative predictive value (TN/(FN +TN)=143/159=90%). Disease prevalences may vary considerably between patient groups and care settings, even those in which the same diagnostic test is used. This has a substantial impact on predictive values. For example, a Swiss prospective cohort study of 218 patients aged >75 years found a lower prevalence of infection of 23% (50/218).10 However, provided the pre-test probability of infection is available, predictive values in the new population can be calculated on the assumption that the performance of the test remains the same. The prevalence of infection is likely to be a plausible estimate of the pre-test probability in the absence of other patient-specific information such as symptoms, signs or previous test results. Box Calculation of post-test probabilities

e calculated on the assumption that the performance of the test remains the same. The prevalence of infection is likely to be a plausible estimate of the pre-test probability in the absence of other patient-specific information such as symptoms, signs or previous test results. Box Calculation of post-test probabilities Positive Diagnostic Likelihood Ratio(DLR+)=TP/(TP+FN)FP/(FP+TN)=67/836/149=20.05 Post-test odds(+ve result)=DLR+×Prevalence1−Prevalence=20.05×50/2181−50/218=5.97 Post-test probability (+ve result)=Post-test odds(+ve result)1+Post-test odds(+ve result)=5.976.97=86% Negative Diagnostic Likelihood Ratio (DLR−)=FN/(TP+FN)TN/(FP+TN)=16/83143/149=0.201 Post-test odds (−ve result)=DLR−×Prevalence1−Prevalence=0.201×50/2181−50/218=0.0598 Post-test probability (−ve result)=Post-test odds(−ve result)1+Post-test odds(−ve result)=0.05981.0598=5.6% Using the 23% prevalence from Stucker et al 10 gives estimated probabilities of infection of 86% following a positive CRP test result and 5.6% following a negative test result. The Box provides details of the calculations, which use likelihood ratios11 estimated using the data from Liu et al.8 Both post-test probabilities are somewhat lower than those found in the setting described by Liu et al,8 which is a reflection of the reduced prevalence of infection in the Swiss population.

est result. The Box provides details of the calculations, which use likelihood ratios11 estimated using the data from Liu et al.8 Both post-test probabilities are somewhat lower than those found in the setting described by Liu et al,8 which is a reflection of the reduced prevalence of infection in the Swiss population. Interactive graphical presentation To help visualise and interpret the results of probability calculations when assessing diagnostic tests, we have created two free interactive tools, titled ‘Test Accuracy’ (https://micncltools.shinyapps.io/TestAccuracy)12 and ‘Clinical Accuracy and Utility’ (https://micncltools.shinyapps.io/ClinicalAccuracyAndUtility).13 These were developed using the RStudio application ‘Shiny’.14 The first of these provides a clear interface for illustrating measures of diagnostic technical accuracy, that is, sensitivity and specificity. It does so by showing the natural frequencies of TP, TN, FP and FN that would result for a given prevalence and sample size. The screenshot in figure 1 displays in graphical form the same information that is shown in table 1 for the study of CRP and infection. Figure 1 Screenshot from the ‘Test Accuracy’ tool, giving a graphical representation of parameters relating to diagnostic performance. FN, false negative; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive.

The first of these provides a clear interface for illustrating measures of diagnostic technical accuracy, that is, sensitivity and specificity. It does so by showing the natural frequencies of TP, TN, FP and FN that would result for a given prevalence and sample size. The screenshot in figure 1 displays in graphical form the same information that is shown in table 1 for the study of CRP and infection. Figure 1 Screenshot from the ‘Test Accuracy’ tool, giving a graphical representation of parameters relating to diagnostic performance. FN, false negative; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive. The second tool is designed to help users to interpret pre-test and post-test probabilities of disease in relation to clinical decision thresholds.15 Figure 2 shows results based on the calculation described above, showing the hypothetical performance of the CRP test (the ‘Index Test’) in a population with 23% prevalence. Additionally, predictive probabilities are shown across the full range of possible prevalences from 0% to 100% to show the user the relationship between these two parameters. CIs are depicted as the coloured bands around each curve to aid communication of uncertainties associated with test accuracy on the resulting clinically relevant parameters. Figure 2 Screenshot from the ‘Clinical Accuracy and Utility’ tool, showing the relationship between disease prevalence (or pre-test probability) and post-test probability. CRP, C reactive protein.

The second tool is designed to help users to interpret pre-test and post-test probabilities of disease in relation to clinical decision thresholds.15 Figure 2 shows results based on the calculation described above, showing the hypothetical performance of the CRP test (the ‘Index Test’) in a population with 23% prevalence. Additionally, predictive probabilities are shown across the full range of possible prevalences from 0% to 100% to show the user the relationship between these two parameters. CIs are depicted as the coloured bands around each curve to aid communication of uncertainties associated with test accuracy on the resulting clinically relevant parameters. Figure 2 Screenshot from the ‘Clinical Accuracy and Utility’ tool, showing the relationship between disease prevalence (or pre-test probability) and post-test probability. CRP, C reactive protein. The resulting predictive probabilities can easily be compared directly to rule-in or rule-out thresholds for clinical decision-making. In further options, these thresholds can be varied by the user, perhaps as a first step in performing a full decision curve analysis, in which decision-making is based on a trade-off between the consequences of FP and FN predictions.16

be compared directly to rule-in or rule-out thresholds for clinical decision-making. In further options, these thresholds can be varied by the user, perhaps as a first step in performing a full decision curve analysis, in which decision-making is based on a trade-off between the consequences of FP and FN predictions.16 In practice, a range of decision thresholds has been proposed for CRP testing in different populations, as described in systematic reviews on the subject.7 17 For the purpose of illustration, suppose that a policy recommendation suggests that a particular treatment be initiated if the post-test probability of treatment exceeded 90%. Using the interactive tools, the user can change the available parameters to see the effect of improved or reduced performance of the test in a different setting, or the different prevalence of disease that might better reflect the characteristics of a new population. Varying the prevalence of disease (figure 2) shows that, given the performance of the diagnostic test, this threshold would be exceeded for individuals who receive a positive test result only in populations for which the disease prevalence is above around 30%. The threshold would therefore not be exceeded in the lower prevalence setting of the Swiss study described above.

t, given the performance of the diagnostic test, this threshold would be exceeded for individuals who receive a positive test result only in populations for which the disease prevalence is above around 30%. The threshold would therefore not be exceeded in the lower prevalence setting of the Swiss study described above. These tools are intended to help those involved in communicating information about diagnostic test performance and are likely to be of benefit when teaching these concepts. They may also be useful for manufacturers of clinical tests in planning product development, for authors of test evaluation studies to improve reporting and for users of test evaluations to facilitate interpretation and application of the results. Example scenarios include those in which predictive values are not provided directly, but can be inferred from sensitivity, specificity and prevalence information, and situations in which the prevalence of the condition varies. They could also be useful for authors of systematic reviews of diagnostic test accuracy studies to derive predictive values from sensitivity and specificity values. They have value in designing new studies, for which preliminary estimates of predictive values and their CIs are useful in helping to choose appropriate and ethical sample sizes. The tool quickly allows users to assess the impact of different sample size and prevalence assumptions on CIs, which can be compared directly against a decision-making threshold.

ies, for which preliminary estimates of predictive values and their CIs are useful in helping to choose appropriate and ethical sample sizes. The tool quickly allows users to assess the impact of different sample size and prevalence assumptions on CIs, which can be compared directly against a decision-making threshold. Conclusion In summary, the clinical accuracy of diagnostic tests, as expressed by post-test probabilities, may be used to guide treatment decisions. These probabilities may vary across different populations. We have created two free, interactive tools to help to visualise these concepts. Future work may include extending these tools to incorporate diagnostic results based on continuous measurements. The authors thank Ann Van den Bruel, Gail Hayward and Louise Johnston for helpful discussions. Contributors: TRF wrote the paper with assistance from all other authors. AJA, SG and MP developed the accompanying online interactive tools. All authors assessed the paper and the accompanying online interactive tools for intellectual content. Funding: TRF and JMO-M are supported by the NIHR Diagnostic Evidence Co- operative (DEC) Oxford. JMO-M is also supported by the NIHR Biomedical Research Centre, Oxford. AJA, SG and MP are supported by the NIHR Diagnostic Evidence Co-operative (DEC) Newcastle. Disclaimer: The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Commentary on: Crellin E, Mansfield KE, Leyrat C, et al. Trimethoprim use for urinary tract infection and risk of adverse outcomes in older patients: cohort study. BMJ 2018;360:k341. Context Urinary tract infection (UTI) is the most common bacterial infection in older adults.1 Trimethoprim is a first-line antibiotic prescribed in the UK for acuteuncomplicated UTI.2 Trimethoprim reduces potassium excretion in the distal nephron, which can cause elevated potassium levels.3 This is of particular importance in older adults, who are more likely to have comorbidities requiring prescription of additional medicines that may predispose them to hyperkalaemia, such as renin-angiotensin antagonists (RAA). Methods This was a cohort study using general practice data from the Clinical Practice Research Datalink and linked Hospital Episode Statistics data in adults aged 65 and above who had received a prescription for one of five different antibiotics (trimethoprim, nitrofurantoin, cefalexin, ciprofloxacin or amoxicillin) within three days of a diagnosis of UTI in primary care. The study spanned an 18-year period from April 1997 to September 2015. The primary outcomes were acute kidney injury (AKI), hyperkalaemia and death recorded within 14 days of antibiotic initiation for UTI. The odds ratio (OR) for each outcome was calculated for each antibiotic compared with amoxicillin (chosen by the study investigators as the reference antibiotic). The authors clearly stated the questions being addressed, inclusion and exclusion criteria and methods used for statistical analysis.

tic initiation for UTI. The odds ratio (OR) for each outcome was calculated for each antibiotic compared with amoxicillin (chosen by the study investigators as the reference antibiotic). The authors clearly stated the questions being addressed, inclusion and exclusion criteria and methods used for statistical analysis. Findings Among a cohort of 1, 191, 905 the authors identified 178, 238 individuals who received at least one antibiotic prescription for a UTI. Trimethoprim increased the odds of hyperkalaemia compared with amoxicillin (adjusted OR 2.27, 95% CI 1.49 to 3.45). Individuals prescribed trimethoprim (adjusted OR 1.72, 95% CI 1.31 to 2.24) and ciprofloxacin (adjusted OR 1.48, 95% CI 1.03 to 2.13) had increased odds of AKI compared with amoxicillin. The odds of death was not increased by prescription of any antibiotic compared with amoxicillin. There was no significant difference in the results for trimethoprim when analyses were restricted to users of RAAs. Commentary This review highlights the association of prescription of trimethoprim with hyperkalaemia and AKI, but not with death, in older adults being treated for a simple UTI.

Findings Among a cohort of 1, 191, 905 the authors identified 178, 238 individuals who received at least one antibiotic prescription for a UTI. Trimethoprim increased the odds of hyperkalaemia compared with amoxicillin (adjusted OR 2.27, 95% CI 1.49 to 3.45). Individuals prescribed trimethoprim (adjusted OR 1.72, 95% CI 1.31 to 2.24) and ciprofloxacin (adjusted OR 1.48, 95% CI 1.03 to 2.13) had increased odds of AKI compared with amoxicillin. The odds of death was not increased by prescription of any antibiotic compared with amoxicillin. There was no significant difference in the results for trimethoprim when analyses were restricted to users of RAAs. Commentary This review highlights the association of prescription of trimethoprim with hyperkalaemia and AKI, but not with death, in older adults being treated for a simple UTI. The results were adjusted for a wide range of appropriate confounders. The authors adjusted for baseline renal function, which was defined as ‘the most recent biochemical test results recorded in primary care’.4 Renal function declines with age.5 If there was a long interval between the most recent recorded renal function and renal function recorded post antibiotic prescription, it is possible that the perceived baseline renal function for some individuals may not have been a reflection of their true baseline renal function. This may have led to an overestimation of the number of cases of AKI.

een the most recent recorded renal function and renal function recorded post antibiotic prescription, it is possible that the perceived baseline renal function for some individuals may not have been a reflection of their true baseline renal function. This may have led to an overestimation of the number of cases of AKI. Death and hyperkalaemia were determined from read codes in either primary or secondary care. However, the authors defined AKI as hospital admission with AKI. By excluding cases of AKI occurring in general practice, this may have led to an underestimation of the number of episodes of AKI. The authors helpfully contextualise the results. While the odds of hyperkalaemia and AKI with trimethoprim prescription are increased by 127% and 72%, respectively, compared with amoxicillin prescription, the increase in the absolute risk is small. The authors state that for 1,000 UTI episodes treated with trimethoprim rather than amoxicillin, there would be one additional case of hyperkalaemia and two of AKI.4 These results would be very similar among those concurrently prescribed RAAs. However, among those individuals taking RAAs and potassium-sparing diuretics, a prescription of trimethoprim instead of amoxicillin would result in 18 additional cases of hyperkalaemia and 11 of AKI. The authors did not present ORs for the latter association in the main article.

mong those concurrently prescribed RAAs. However, among those individuals taking RAAs and potassium-sparing diuretics, a prescription of trimethoprim instead of amoxicillin would result in 18 additional cases of hyperkalaemia and 11 of AKI. The authors did not present ORs for the latter association in the main article. Implications for practice Clinicians should be vigilant for hyperkalaemia and AKI when prescribing trimethoprim to older adults with simple UTIs. Clinicians should avoid prescribing trimethoprim to individuals taking both RAAs and potassium-sparing diuretics; if prescribed, their renal function should be monitored diligently. Trimethoprim is not associated with an increased risk of death. Funding: Oghenekome Gbinigie is funded by the Wellcome Trust (Grant number: 203921/Z/16/Z). Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Commissioned; internally peer reviewed.

Introduction A perceived lack of transparency, including under-reporting of results, undermines the confidence of researchers, healthcare professionals and patients in conclusions drawn from clinical trials.1 All clinical trial sponsors, be they biopharmaceutical companies or non-industry bodies and triallists, such as government agencies, universities and research charities, have ethical obligations to register applicable trials before they start and to report their results in a timely fashion after they finish.2 3 In the USA, EU and elsewhere, it is required that certain types of clinical trial are registered and their results posted on dedicated registries (eg, EudraCT, the EU electronic Register of Post-Authorisation Studies and ClinicalTrials.gov) (online supplementary material, table S1).4–11 Other bodies, such as the WHO and the International Committee of Medical Journal Editors (ICMJE), have issued transparency standards and recommendations,2 12–14 and some biopharmaceutical companies have websites dedicated to their own trial results.15 16 This makes the clinical trial data transparency environment highly complex and diverse. 10.1136/bmjebm-2018-111145.supp1Supplementary data 10.1136/bmjebm-2018-111145.supp2Supplementary data

Introduction A perceived lack of transparency, including under-reporting of results, undermines the confidence of researchers, healthcare professionals and patients in conclusions drawn from clinical trials.1 All clinical trial sponsors, be they biopharmaceutical companies or non-industry bodies and triallists, such as government agencies, universities and research charities, have ethical obligations to register applicable trials before they start and to report their results in a timely fashion after they finish.2 3 In the USA, EU and elsewhere, it is required that certain types of clinical trial are registered and their results posted on dedicated registries (eg, EudraCT, the EU electronic Register of Post-Authorisation Studies and ClinicalTrials.gov) (online supplementary material, table S1).4–11 Other bodies, such as the WHO and the International Committee of Medical Journal Editors (ICMJE), have issued transparency standards and recommendations,2 12–14 and some biopharmaceutical companies have websites dedicated to their own trial results.15 16 This makes the clinical trial data transparency environment highly complex and diverse. 10.1136/bmjebm-2018-111145.supp1Supplementary data 10.1136/bmjebm-2018-111145.supp2Supplementary data Within the biopharmaceutical industry, which is responsible for approximately half of all clinical trials,17 18 two large associations, the European Federation of Pharmaceutical Industries and Associations (EFPIA) and the Pharmaceutical Research and Manufacturers of America (PhRMA), have developed joint ‘Principles for responsible clinical trial data sharing’.19 These joint principles, which became effective on 1 January 2014, make the following five commitments:To enhance data sharing with researchers.

and Associations (EFPIA) and the Pharmaceutical Research and Manufacturers of America (PhRMA), have developed joint ‘Principles for responsible clinical trial data sharing’.19 These joint principles, which became effective on 1 January 2014, make the following five commitments:To enhance data sharing with researchers. To enhance public access to clinical study information. To share results with patients who participate in clinical trials. To certify procedures for sharing clinical trial information. To reaffirm commitments to publish clinical trial results. In the present study, we aimed to evaluate the extent to which EFPIA/PhRMA members and non-members among the leading biopharmaceutical companies have committed to the responsible disclosure of clinical trial results. We also evaluated the reporting of results from clinical trials sponsored by biopharmaceutical companies compared with those from other sponsors.

the extent to which EFPIA/PhRMA members and non-members among the leading biopharmaceutical companies have committed to the responsible disclosure of clinical trial results. We also evaluated the reporting of results from clinical trials sponsored by biopharmaceutical companies compared with those from other sponsors. Methods Commitment to disclosure of clinical trial data by EFPIA/PhRMA member companies The global public websites of each EFPIA and/or PhRMA (‘EFPIA/PhRMA’) member and non-member company in the top 50 companies by 2015 worldwide prescription sales (‘top 50 companies’)20 were searched between December 2017 and January 2018 by one researcher (JP) for direct links to pages containing: (1) a general statement of commitment to disclosing clinical trial data; (2) a general statement of commitment to disclosing clinical trial data according to EFPIA/PhRMA joint principles; and (3) specific statements detailing commitments to upholding one or more of the five individual EFPIA/PhRMA joint principles for responsible disclosure of clinical trial data. If no direct links to such pages were found, the free-text search function of each website was used to search for statements relating to clinical trial data disclosure and implementation of the EFPIA/PhRMA disclosure principles using one or more the key words ‘EFPIA’, ‘PhRMA’, ‘data sharing’, ‘clinical trials’ and ‘transparency’. EFPIA/PhRMA membership was determined from the websites of these two organisations (www.efpia.eu/about-us/membership and http://www.phrma.org/about/members).

nd implementation of the EFPIA/PhRMA disclosure principles using one or more the key words ‘EFPIA’, ‘PhRMA’, ‘data sharing’, ‘clinical trials’ and ‘transparency’. EFPIA/PhRMA membership was determined from the websites of these two organisations (www.efpia.eu/about-us/membership and http://www.phrma.org/about/members). Subjective ease of access to relevant information was assessed: good access was rated as requiring either no more than four clicks from the homepage of the company website21 or a clear, direct link; poor access was rated as either needing more than four clicks or requiring navigation to satellite websites (eg, blogs). Clinical trial results reporting TrialsTracker is an independent, semiautomated, web-based tool that has been developed in an effort to incentivise sponsors of clinical trials to improve disclosure rates by highlighting the disclosure performance of individual sponsors (trials without results disclosed as a proportion of trials registered).22 For clinical trial sponsors to be included in TrialsTracker, they must have more than 30 phase II–IV clinical trials registered on ClinicalTrials.gov that were recorded as completed after 1 January 2006 and at least 24 months before the most recent TrialsTracker update. Because the most recent update to the database was in April 2017, the most recent studies to be included in this analysis were completed in April 2015.

linical trials registered on ClinicalTrials.gov that were recorded as completed after 1 January 2006 and at least 24 months before the most recent TrialsTracker update. Because the most recent update to the database was in April 2017, the most recent studies to be included in this analysis were completed in April 2015. Data detailing the number of trials registered on ClinicalTrials.gov by clinical trial sponsor, and the corresponding number of trials without results reported for each year from 2006 to 2015, were downloaded as a comma-separated values file from the TrialsTracker website (https://trialstracker.ebmdatalab.net). TrialsTracker identifies sponsors as industry (‘biopharmaceutical companies’ which we subcategorised as pharmaceutical/biotechnology, generics/biosimilars, medical devices, plasma products and nutraceuticals, using information on the company websites that was found during the research for our study) or non-industry (classified as National Institutes of Health, US Federal or other) institutions. For each industry and non-industry sponsor and for each category, the number of disclosed trials and the percentage of eligible studies with disclosed results were calculated in Microsoft Excel 2016. An analysis of disclosure rates was performed on subsets of the industry sponsors within TrialsTracker based on sales revenue (the top 50 companies)20 and membership of EFPIA/PhRMA. An arbitrary disclosure rate threshold of 80% was applied to sponsor subgroups.

Data detailing the number of trials registered on ClinicalTrials.gov by clinical trial sponsor, and the corresponding number of trials without results reported for each year from 2006 to 2015, were downloaded as a comma-separated values file from the TrialsTracker website (https://trialstracker.ebmdatalab.net). TrialsTracker identifies sponsors as industry (‘biopharmaceutical companies’ which we subcategorised as pharmaceutical/biotechnology, generics/biosimilars, medical devices, plasma products and nutraceuticals, using information on the company websites that was found during the research for our study) or non-industry (classified as National Institutes of Health, US Federal or other) institutions. For each industry and non-industry sponsor and for each category, the number of disclosed trials and the percentage of eligible studies with disclosed results were calculated in Microsoft Excel 2016. An analysis of disclosure rates was performed on subsets of the industry sponsors within TrialsTracker based on sales revenue (the top 50 companies)20 and membership of EFPIA/PhRMA. An arbitrary disclosure rate threshold of 80% was applied to sponsor subgroups. Exploratory analyses of results posted on websites other than ClinicalTrials.gov In an exploratory analysis, clinical trial results from locations other than ClinicalTrials.gov or from linked publications in PubMed were sought for three studies that were selected from four of the top 50 companies. ClinicalTrials.gov was searched by National Clinical Trial (NCT) identifier in order to establish the presence or absence of posted results and/or links to publications on PubMed. We made a separate search of PubMed, Google Scholar and Google using the NCT identifier and a search of EudraCT and the relevant company’s website based on NCT identifier and study title.

nical Trial (NCT) identifier in order to establish the presence or absence of posted results and/or links to publications on PubMed. We made a separate search of PubMed, Google Scholar and Google using the NCT identifier and a search of EudraCT and the relevant company’s website based on NCT identifier and study title. Exploratory analysis of commitments to disclosure of clinical trial data by non-industry sponsors In an exploratory analysis, we searched for statements relating to the disclosure of clinical trial data on the websites of 10 non-industry sponsors of clinical trials with results completed in the period 2006–2015. Data analysis Disclosure rates for all industry sponsors, the top 50 companies and EFPIA/PhRMA members in the top 50 companies were compared with those for non-industry sponsors. Patient involvement Patients were not directly involved in conducting this study, but patients’ perspectives were sought during the development of the manuscript. Results Commitment to disclosure of clinical trial data EFPIA/PhRMA membership Of the top 50 companies, 6 were EFPIA members only, two were PhRMA members only, 22 were both EFPIA and PhRMA members and 20 were neither EFPIA nor PhRMA members. There were more EFPIA/PhRMA members in the largest 25 companies (n=23) than in the second group of 25 largest companies (n=7) (online supplementary material, table S2).

50 companies, 6 were EFPIA members only, two were PhRMA members only, 22 were both EFPIA and PhRMA members and 20 were neither EFPIA nor PhRMA members. There were more EFPIA/PhRMA members in the largest 25 companies (n=23) than in the second group of 25 largest companies (n=7) (online supplementary material, table S2). All 30 EFPIA/PhRMA members in the top 50 companies were pharmaceutical/biotechnology companies, whereas the 20 non-members were more varied, comprising pharmaceutical/biotechnology (n=8), generics/biosimilars (n=6), medical devices (n=1), both generics and medical devices (n=2), intravenous products and medical devices (n=1), plasma products (n=1) and nutraceuticals (n=1) companies. Of the two EFPIA/PhRMA non-members in the top 25 companies, one was a biotechnology company and the other was a generics company. Access to a general disclosure statement A general statement committing to the disclosure of clinical trial information was found on 26 of the top 50 company websites (52%), all of which were EFPIA/PhRMA members (table 1). In 19 cases (38% of the top 50 companies; 63% of EFPIA/PhRMA members), the statement was found within four clicks of entering the website; in seven cases, access was rated as poor. An analysis of the proportion of EFPIA/PhRMA members versus non-members in the top 50 companies with statements committing to responsible data transparency is shown in online supplementary material, table S3.

, the statement was found within four clicks of entering the website; in seven cases, access was rated as poor. An analysis of the proportion of EFPIA/PhRMA members versus non-members in the top 50 companies with statements committing to responsible data transparency is shown in online supplementary material, table S3. Table 1 Number of EFPIA and PhRMA member and non-member companies in the top 50 biopharmaceutical companies as ranked by 2015 worldwide prescription sales and their public commitment to disclosing clinical trial data Top 50 companies: membership of EFPIA and/or PhRMA General data sharing statement, n (%) EFPIA/PhRMA principles Five EFPIA/PhRMA joint principles of responsible clinical data sharing Commitment to sharing clinical trial data with researchers Public availability of CSR synopsis as a minimum Availability of results for trial participants Public certification of adoption of EFPIA/PhRMA commitments Commitment to publish clinical trial data (phase III minimum) All five principles Member (n=30) 26 (86.7) 20 (66.7) 25 (83.3) 22 (73.3) 18 (60.0) 21 (70.0) 24 (80.0) 16 (53.3) Non-member  (n=20) 0 (0.0) 0 (0.0) 0 (0.0) 1 (5.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Total  (n=50) 26 (52.0) 20 (40.0) 25 (50.0) 23 (46.0) 18 (36.0) 21 (42.0) 24 (48.0) 16 (32.0) CSR, clinical study report; EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America.

0 (0.0) 0 (0.0) 0 (0.0) 1 (5.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Total  (n=50) 26 (52.0) 20 (40.0) 25 (50.0) 23 (46.0) 18 (36.0) 21 (42.0) 24 (48.0) 16 (32.0) CSR, clinical study report; EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America. Specific EFPIA/PhRMA principles An overview statement referring to the adoption of the joint principles (http://phrma-docs.phrma.org/sites/default/files/pdf/PhRMAPrinciplesForResponsibleClinicalTrialDataSharing.pdf) was found on 20/30 websites (67%) of EFPIA/PhRMA members. Reference to all five joint principles was found for 16/30 members (53%). Of non-member companies, only one company made a specific disclosure statement (to enhance public access to clinical study information by making synopses of clinical study reports publicly available) (table 1). The most frequently communicated individual commitments were to share clinical trial data with researchers (83% of EFPIA/PhRMA members; 50% of the top 50 companies) and to publish clinical trial data (80% of members; 48% of the top 50 companies).

making synopses of clinical study reports publicly available) (table 1). The most frequently communicated individual commitments were to share clinical trial data with researchers (83% of EFPIA/PhRMA members; 50% of the top 50 companies) and to publish clinical trial data (80% of members; 48% of the top 50 companies). Clinical trial results reporting Of 29 377 trials listed in TrialsTracker, 9511 (32%) were sponsored by 69 biopharmaceutical companies (a mean of 138 trials per company) and 19 866 (68%) were sponsored by 254 non-industry institutions (a mean of 78 trials per institution) (figure 1). Of all 13 266 undisclosed trials, 10 792 (81%) were sponsored by non-industry institutions and 2474 (19%) were sponsored by industry. The mean±SD disclosure rate for all trials was 55%±21.0%, with higher rates for industry (7037/9511; 74%±22.1%) than for non-industry sponsors (9074/19 866; 46%±15.7%) (figure 2; online supplementary material, table S3). Figure 1 Types and numbers of sponsors represented in TrialsTracker, with their disclosure rates. EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America. Figure 2 Disclosure of clinical trial results by sponsor type. EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America.

Figure 1 Types and numbers of sponsors represented in TrialsTracker, with their disclosure rates. EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America. Figure 2 Disclosure of clinical trial results by sponsor type. EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America. The overall disclosure rate for all clinical trials substantially increased during 2007 and 2008 before declining thereafter. The maximum mean disclosure rate for all clinical trials was observed in 2008 (2069/3135; 66%); for industry-sponsored and non-industry-sponsored trials, the maximum mean disclosure rates were in 2012 (761/914; 83%) and 2009 (1160/2063; 56%), respectively (figure 2). Disclosure rates for non-industry sponsors declined after 2009, whereas disclosure rates for industry sponsors were maintained at approximately 80% until 2014 (figure 2).

and non-industry-sponsored trials, the maximum mean disclosure rates were in 2012 (761/914; 83%) and 2009 (1160/2063; 56%), respectively (figure 2). Disclosure rates for non-industry sponsors declined after 2009, whereas disclosure rates for industry sponsors were maintained at approximately 80% until 2014 (figure 2). There was high variability in disclosure rate between sponsor type (figure 3A). The highest disclosure rate achieved by a non-industry sponsor was 353/418 (84%), whereas two biopharmaceutical industry sponsors achieved 100% disclosure. Of the top 50 companies, a mean±SD of 76%±19.1% of trials were disclosed by the 30 companies with data reported in TrialsTracker (all of which were pharmaceutical/biotechnology companies). The mean±SD disclosure rate was 77%±17.4% for EFPIA/PhRMA members (4434/5785 trials; 25 companies) and was 67%±25.3% for non-members (264/394; 5 companies) (figures 2 and 3B). An arbitrary disclosure rate threshold of 80% was reached by fewer than 1% (2/254) of non-industry sponsors compared with 39% (27/69) of industry sponsors. Of the 69 biopharmaceutical industry sponsors with results in TrialsTracker, the 80% threshold was met by 57% (13/23) of EFPIA/PhRMA members in the top 50 companies, by 20% (1/5) of EFPIA/PhRMA non-members in the top 50 companies and by 31% (12/39) of biopharmaceutical industry sponsors that were not in the top 50 companies (figure 3B).

aceutical industry sponsors with results in TrialsTracker, the 80% threshold was met by 57% (13/23) of EFPIA/PhRMA members in the top 50 companies, by 20% (1/5) of EFPIA/PhRMA non-members in the top 50 companies and by 31% (12/39) of biopharmaceutical industry sponsors that were not in the top 50 companies (figure 3B). Figure 3 Disclosure rates versus total eligible trials for (A) biopharmaceutical industry and non-industry clinical trial sponsors and (B) industry-only sponsors, highlighting EFPIA/PhRMA members and non-members in the top 50 companies, and industry sponsors not in the top 50 companies. Whiskers represent SD. EFPIA, European Federation of Pharmaceutical Industries and Associations; PhRMA, Pharmaceutical Research and Manufacturers of America. Exploratory analyses In the first exploratory analysis, commitments to disclosure of clinical trial data by non-industry sponsors were made by 1/10 institutions (online supplementary material, table S4). In the second exploratory analysis, clinical trial results from four industry sponsors that were not posted on ClinicalTrials.gov were disclosed on a variety of other websites (eg, EU Clinical Trials Register and company website) as summarised in online supplementary material, table S5.

pplementary material, table S4). In the second exploratory analysis, clinical trial results from four industry sponsors that were not posted on ClinicalTrials.gov were disclosed on a variety of other websites (eg, EU Clinical Trials Register and company website) as summarised in online supplementary material, table S5. Discussion This analysis of the disclosure environment of clinical trial sponsors began with a review of the publicly stated disclosure policies of the top 50 biopharmaceutical companies. Of these, 26 companies (52%; all of which were members of one or both of the two leading international industry bodies [EFPIA and PhRMA]) made their disclosure policies easily available on their websites. Most EFPIA/PhRMA members (87%) communicated that they had a commitment to disclose clinical trial results and two-thirds (67%) specifically referred to the EFPIA/PhRMA joint principles; approximately half (53%) described those principles in detail. To be useful, information on websites should be easy to find and have a logical flow. Research has shown that four clicks may be viewed as the minimum required to achieve any level of success for completing a search21; the ‘three-click rule’ is no longer regarded as the benchmark for website utility.21 23 Inherent in the principle of publicly committing to data disclosure should be that the statements have good accessibility.

four clicks may be viewed as the minimum required to achieve any level of success for completing a search21; the ‘three-click rule’ is no longer regarded as the benchmark for website utility.21 23 Inherent in the principle of publicly committing to data disclosure should be that the statements have good accessibility. The second phase of our study collated disclosure information from a large number of trials and sponsors over a 10-year period using data from TrialsTracker. This showed that the disclosure of clinical trial data remains suboptimal. By the end of April 2017, data were disclosed for approximately half of the phase II–IV trials registered on ClinicalTrials.gov and completed in the period 2006–2015. Over this period, results were disclosed by approximately three-quarters of the biopharmaceutical industry sponsors compared with less than half of the non-industry sponsors. Of undisclosed trials, more than 80% were originally funded by governmental, charitable or academic institutions compared with just under 20% by industry, even though approximately one-third of all trials in our data set were industry funded. Disclosure rates for both types of funder substantially increased between 2007 and 2008, coinciding with mandatory reporting as required by the Food and Drug Administration Amendments Act 801 (FDAAA 801). For industry sponsors, disclosure rates were maintained for the next 6 years before declining slightly in 2015. This decline may reflect delays in the publication process, which usually takes approximately 2 years from study completion.24 With the implementation of the ‘Final Rule’ in January 2017, there should no longer be delays in the posting of results from applicable clinical trials on ClinicalTrials.gov.24–26 By contrast, disclosure rates for non-industry studies steadily declined after 2009, possibly reflecting a lack of policing of FDAAA 801. It is noteworthy that those companies within the top 50, which as members of EFPIA/PhRMA, have committed to adhere to the EFPIA/PhRMA principles,27 have numerically the highest mean disclosure rate.

closure rates for non-industry studies steadily declined after 2009, possibly reflecting a lack of policing of FDAAA 801. It is noteworthy that those companies within the top 50, which as members of EFPIA/PhRMA, have committed to adhere to the EFPIA/PhRMA principles,27 have numerically the highest mean disclosure rate. The proportion of trials identified in the present study as sponsored by the biopharmaceutical industry (approximately one-third of all trials) was similar to that reported previously.17 26 In our analysis, the disclosure rate for industry-sponsored studies was similar to that previously observed using TrialsTracker and the EU Clinical Trials Register (EUCTR),22 28 although this rate is lower than the rates reported for newly approved drugs in the USA and Europe,29 30 and either lower than or similar to rates of publication that have been reported by single sponsors,24 25 and with a similar study design profile to a previously reported study.18 Similarly, for non-industry studies, the disclosure rate was similar to that previously seen with TrialsTracker,22 and either lower than or similar to those reported for academic medical centres in the USA and the UK but higher than for EUCTR.18 28 31 32

a similar study design profile to a previously reported study.18 Similarly, for non-industry studies, the disclosure rate was similar to that previously seen with TrialsTracker,22 and either lower than or similar to those reported for academic medical centres in the USA and the UK but higher than for EUCTR.18 28 31 32 Our assessment of biopharmaceutical company disclosure policies showed results similar to those from a recent EFPIA/PhRMA survey in which 77% of the 44 EFPIA/PhRMA members confirmed that they state on a publicly available website that they adhere to the joint principles.27 In a recent survey of the internal disclosure policies of 25 top biopharmaceutical companies, 96% reported that they had a policy committing to the sharing of summary results in academic articles or on a clinical trial registry.33 However, in the present study, we found such commitments on the websites of only about half of the top 50 companies, suggesting that many companies are missing the opportunity to inform the general public about their disclosure policies.

sharing of summary results in academic articles or on a clinical trial registry.33 However, in the present study, we found such commitments on the websites of only about half of the top 50 companies, suggesting that many companies are missing the opportunity to inform the general public about their disclosure policies. In contrast to the results for EFPIA/PhRMA members, but in line with those for non-member companies, a preliminary review of commitments to data transparency that was conducted for 10 non-industry sponsors of clinical trials completed in the period 2006–2015 demonstrated that only one institution referred to the disclosure of clinical trial data (online supplementary material, table S4). The requirement for EFPIA/PhRMA members to commit publicly to data disclosure is reflected in clear differences between these companies and non-members/non-industry sponsors. A summary of the findings from this study is presented in an infographic that can be found in online supplementary material.

material, table S4). The requirement for EFPIA/PhRMA members to commit publicly to data disclosure is reflected in clear differences between these companies and non-members/non-industry sponsors. A summary of the findings from this study is presented in an infographic that can be found in online supplementary material. Because we used data from TrialsTracker, which searches only on ClinicalTrials.gov, the largest available clinical trial registry, using only NCT identifiers, we made an exploratory search of alternative sources of clinical trial data for 12 clinical trials sponsored by four of the top 50 biopharmaceutical companies; although these analyses were not performed with the same rigour or level of detail as the main analyses, they show that some results that were missing from ClinicalTrials.gov were found on EudraCT and company websites, suggesting that TrialsTracker was underestimating the number of trials that had published results. As recommended by the ICMJE, the inclusion of the study, NCT and/or EudraCT numbers in the abstract of publications linked to clinical trials would help to improve assessments of the disclosure of clinical trial data.7

s, suggesting that TrialsTracker was underestimating the number of trials that had published results. As recommended by the ICMJE, the inclusion of the study, NCT and/or EudraCT numbers in the abstract of publications linked to clinical trials would help to improve assessments of the disclosure of clinical trial data.7 Conducting clinical trials requires a high level of trust between the patient, the medical team and the trial sponsor. Central to the trust placed in the sponsor by the patient is that the results of the trial will be made publicly available. Many patients enrol in clinical trials in the hope of improving their own health and in the expectation that their participation will contribute to a better understanding of their condition and to the development of potential new treatments. Participants must weigh these potential benefits against the risk of adverse reactions. For their involvement in clinical trials to have meaning to the participants, all trial sponsors should transparently report all results, both positive and negative.

heir condition and to the development of potential new treatments. Participants must weigh these potential benefits against the risk of adverse reactions. For their involvement in clinical trials to have meaning to the participants, all trial sponsors should transparently report all results, both positive and negative. Strengths and limitations Our study was based on an evaluation of a large number of phase II–IV clinical trials from industry and non-industry sponsors over a 10-year period; however, several caveats should be considered when interpreting the results. First, it should be noted that the results obtained from the automated data acquisition system used by TrialsTracker are subject to error in the reporting rate. In a previous article,22 Powell-Smith and Goldacre compared their results in TrialsTracker with those from a previous manual audit of the disclosure of results from 4347 trials (performed by Chen et al).31 Of the 2562 trials in both analyses, 1149 were found to be reported in both, 534 were unreported by both, 497 were reported by Chen et al but not by Powell-Smith and Goldacre and 382 were unreported by Chen et al but were reported by Powell-Smith and Goldacre. Thus, the total number of discordant trials (874 of 2562) represents 34.3% of the trials in both analyses. However, when analysing the results from Powell-Smith and Goldacre in comparison with those from Chen et al more closely, 14.9% of the trials were ‘overreported’ and 19.4% were ‘underreported’, which may be interpreted as a net underestimation of the reporting rate by 4.5%; underestimation of studies performed by industry sponsors may be a particular issue because many companies only disclosed their results on their own websites. Second, only two types of disclosure were included: publication in a journal and posting of results on ClinicalTrials.gov. Because publications were identified through automated searches of PubMed for NCT identifiers, identification and discoverability were limited to trials published with NCT identifiers included in the secondary source ID field of PubMed, title or abstract.34 35 Results disclosed elsewhere (eg, institutional websites and other registries) or published without reference to the NCT identifier could lead to the understating of disclosure rates.

bility were limited to trials published with NCT identifiers included in the secondary source ID field of PubMed, title or abstract.34 35 Results disclosed elsewhere (eg, institutional websites and other registries) or published without reference to the NCT identifier could lead to the understating of disclosure rates. Third, we may not be able to generalise our findings to all sponsors and clinical trials because our analysis included sponsors of only 30 or more trials (CCW has previously calculated that sponsors of 30 studies or fewer are responsible for approximately half of all registered trials).36 Fourth, our study looked only at the disclosure of registered studies but not all studies are registered; indeed, unregistered studies seem to be less likely than registered studies to be published.37 Finally, our analysis of publicly available disclosure policies used key word searches that focused on disclosure, so it is possible that specific publication policies were missed. Nevertheless, our findings suggest that statements related to the disclosure of results are difficult to find in many cases.

e published.37 Finally, our analysis of publicly available disclosure policies used key word searches that focused on disclosure, so it is possible that specific publication policies were missed. Nevertheless, our findings suggest that statements related to the disclosure of results are difficult to find in many cases. The problem of incomplete and inconsistent clinical trial disclosure remains despite public awareness campaigns and the introduction of various policies, legislation and fines. Company-sponsored trials have been the focus of many of these activities because of their perceived commercial influence. However, the present data demonstrate that results from trials sponsored by the biopharmaceutical industry are disclosed more often than those from non-industry funded studies. The results of our analysis agree with those from two recent studies that reported that industry funders disclose the results from a higher proportion of their trials than do non-industry funders.18 28 These findings may reflect the considerable resources that commercial organisations have dedicated to clinical trial disclosure. They also suggest that the focus of future efforts to improve trial disclosure should shift towards the development of a single set of globally applicable rules and harmonisation of clinical trial data transparency principles to make them more easily implemented by organisations without the resources of pharma companies. This could be achieved by active discussion between, and endorsement by, all stakeholders, including clinical trial sponsors, regulatory bodies and other public bodies (eg, WHO, ICMJE and EU Council), as well as those campaigning for increased transparency of clinical trial information. We recommend that investigators do not begin recruitment and patients do not participate in clinical trials unless they have been registered on an international registry. We believe that well-defined EFPIA/PhRMA joint principles could be used as a basis for the development of harmonised transparency and disclosure principles and meanwhile should be established as an example of best practice in order to encourage consensus. Simplification of the transparency rules and regulations, the implementation of a single study identifier that can be used across all registries, publications and results databases and improved scrutiny of compliance should extend across all aspects of clinical trials and sponsors.

tice in order to encourage consensus. Simplification of the transparency rules and regulations, the implementation of a single study identifier that can be used across all registries, publications and results databases and improved scrutiny of compliance should extend across all aspects of clinical trials and sponsors. Summary box What is already known on this subject? Clinical trial sponsors are ethically and legally bound to register every trial and report the results in a timely fashion. Not all clinical trials are reported via public registries and/or primary manuscripts. By joining the European Federation of Pharmaceutical Industries and Associations (EFPIA) and/or the Pharmaceutical Research and Manufacturers of America (PhRMA), biopharmaceutical companies make a commitment to reporting clinical trial data following EFPIA/PhRMA joint principles for responsible data sharing. What are the new findings? Publicly stated commitment to transparency in clinical trial data sharing and disclosure is more common among biopharmaceutical companies that are members of EFPIA and/or PhRMA than among non-members. Disclosure rates for studies registered on ClinicalTrials.gov for biopharmaceutical industry sponsors are higher than those for non-industry sponsors, which are declining. Preliminary data suggest that a reliance on studies registered on ClinicalTrials.gov and publications that quote the NCT number may lead to underestimation of clinical trial disclosure rates.

Disclosure rates for studies registered on ClinicalTrials.gov for biopharmaceutical industry sponsors are higher than those for non-industry sponsors, which are declining. Preliminary data suggest that a reliance on studies registered on ClinicalTrials.gov and publications that quote the NCT number may lead to underestimation of clinical trial disclosure rates. Most undisclosed clinical trials are sponsored by non-commercial organisations such as government agencies, research charities and universities rather than by the biopharmaceutical industry. How might these results change the focus of research or clinical practice? Further improving of clinical trial transparency and data disclosure may be achieved through simplification of the transparency rules and regulations, the implementation of a single study identifier that can be used across all registries, publications and other results databases and improving the scrutiny of compliance across all aspects of clinical trials. The authors would like to thank Ben Goldacre and Anna Powell-Smith for the use of the TrialsTracker tool, the patients involved in the clinical trials analysed in this article, Laura Schmidt and Simon Levy of Oxford PharmaGenesis for development of the infographic and Christopher Rains, Valerie Philippon and Borislava Pavlova of Shire (now part of Takeda) for reviewing the manuscript. The authors would also like to thank Alan Thomas and Elizabeth Kinder for their review of this article from the patient perspective.

Levy of Oxford PharmaGenesis for development of the infographic and Christopher Rains, Valerie Philippon and Borislava Pavlova of Shire (now part of Takeda) for reviewing the manuscript. The authors would also like to thank Alan Thomas and Elizabeth Kinder for their review of this article from the patient perspective. Contributors: All authors contributed to the planning, conduct and reporting of the work described in this article. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. Funding: Funding was provided by Oxford PharmaGenesis and Shire (now part of Takeda), employees of which reviewed and approved the draft text. Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: JP, ES, JB and CW are employees of Oxford PharmaGenesis, Oxford, UK, which receives funding from multiple pharmaceutical companies for the provision of medical communications support. CW owns shares in Oxford PharmaGenesis Holdings Ltd; AP and SB are employees Shire International GmbH (now part of Takeda). Provenance and peer review: Not commissioned; externally peer reviewed.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: JP, ES, JB and CW are employees of Oxford PharmaGenesis, Oxford, UK, which receives funding from multiple pharmaceutical companies for the provision of medical communications support. CW owns shares in Oxford PharmaGenesis Holdings Ltd; AP and SB are employees Shire International GmbH (now part of Takeda). Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: The data presented in this article are updated from the following presentations: SB, JP, JB, ES, CW, AP. Commitments to data sharing by pharmaceutical companies: the evolving environment. Presented at the 2017 European Meeting of the International Society for Medical Publication Professionals (ISMPP), London, UK, 17–18 January 2017 [poster presentation]. SB, JP, JB, ES, CW, AP. Commitments to data sharing by pharmaceutical companies: the evolving environment. Presented at the 2017 Annual Meeting of ISMPP, National Harbor, Maryland, USA, 1–3 May 2017. Curr Med Res Opin 2017;33(Suppl 1):7 [oral presentation]. SB, JP, JB, ES, CW, AP. Disclosure of results of clinical trials sponsored by pharmaceutical companies. Presented at the 2017 Meeting of the International Peer Review Congress (PRC), Chicago, IL, USA, 10–12 September 2017 [poster presentation]. SB, JP, CW, ES, JB, AP. Disclosure of results of clinical trials sponsored by pharmaceutical companies. Presented at the 2018 European Meeting of ISMPP, London, UK, 23–24 January 2018 [poster presentation]. This research was also presented at Evidence Live 2018, Oxford, UK, 18–19 June 2018. This article is available as a preprint at https://www.biorxiv.org: SB, JP, ES, JB, CW, AP. Commitments by the biopharmaceutical industry to clinical trials transparency: the evolving environment. 2017;bioRxiv 349902; doi: https://doi.org/10.1101/349902.

lso presented at Evidence Live 2018, Oxford, UK, 18–19 June 2018. This article is available as a preprint at https://www.biorxiv.org: SB, JP, ES, JB, CW, AP. Commitments by the biopharmaceutical industry to clinical trials transparency: the evolving environment. 2017;bioRxiv 349902; doi: https://doi.org/10.1101/349902. Patient consent for publication: Not required.

ii The null hypothesis concerns the threshold H 0:θ≤1.0, why Z=0.83≈(1.3−1.0)/0.36. A Z value of ±0.83 divides the probability density function into three areas: 0.20 and 0.60 and 0.20. In a one-sided test, the p value is 0.20, and in a double-sided test it is 0.20+0.20=0.40. A Z value of 1.96 divides the PDF into the following familiar areas: 2.5%, 95% and 2.5%. Contributors: All authors contributed to the planning. SA wrote the first draft. All other authors contributed equally. Funding: This work is supported by a grant from the Swedish Foundation for Humanities and Social Sciences, grant number M14-0138:1. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Cardiovascular diseases are the leading causes of death globally and are increasing.1 Risk factors can be effectively treated, but more people need to be treated more aggressively for the high-risk strategy to be effective.2 Proposed solutions include polypills3 and risk-based approaches4 5, the latter recognising the merits of basing treatment decisions for the prevention of cardiovascular disease on a person’s predicted absolute risk of disease, rather than on the level of a single risk factor.6 7 Both solutions need the answer to a specific, but hitherto overlooked question: are there any synergistic effects between the preventive drug treatments? When a high-risk person is identified as a candidate for primary or secondary prevention against cardiovascular disease, the potential drug regimen is likely to include blood pressure-lowering drugs and statins. Combining these two treatments may result in anything from less than an additive effect to more than a multiplicative effect on cardiovascular disease risk. Experimental studies have suggested positive8 9 as well as negative10 synergistic effects between blood pressure-lowering drugs and statins.

ing drugs and statins. Combining these two treatments may result in anything from less than an additive effect to more than a multiplicative effect on cardiovascular disease risk. Experimental studies have suggested positive8 9 as well as negative10 synergistic effects between blood pressure-lowering drugs and statins. Several factorial trials with these two treatments have been made in humans, but as a whole, their interaction is unknown. With the recent addition to that literature,11 the evidence base may now be sufficient to answer the question. We hypothesised that the combined effects of blood pressure-lowering drugs and statins are multiplicative. In order to test the hypothesis, we performed a systematic review of randomised factorial trials of the effects of blood pressure-lowering drugs and statins on cardiovascular outcomes in primary or secondary prevention.

othesised that the combined effects of blood pressure-lowering drugs and statins are multiplicative. In order to test the hypothesis, we performed a systematic review of randomised factorial trials of the effects of blood pressure-lowering drugs and statins on cardiovascular outcomes in primary or secondary prevention. Methods Data sources and searches Using a pre-specified systematic review protocol, we performed data searches in MEDLINE, SCOPUS, Cochrane Central Register of Controlled Trials and ClinicalTrials.gov, and Web of Knowledge conference abstracts, and limited the search from earliest available to December 2017. The MEDLINE search strategy was: (1) randomised controlled trial (pt); (2) controlled clinical trial (pt); (3) randomised (tiab); (4) clinical trial (all); (5) randomly (tiab); (6) trial (ti); (7) placebo (ti); (8) 1 or 2 or 3 or 4 or 5 or 6 or 7; (9) hypolipidaemic agents (mesh); (10) hydroxymethylglutaryl-CoA reductase inhibitors (mesh); (11) statin (tw); (12) 9 or 10 or 11; (13) antihypertensive agents (mesh); (14) cardiovascular (all); (15) cardiovascular diseases (majr); (16) 14 or 15; (17) 8 and 12 and 13 and 16. Similar search strategies were applied to the other sources. Reference lists of relevant publications were hand searched.

Empirical research has demonstrated that randomised controlled trials with positive findings are more likely to be submitted and published than trials with negative or null findings (OR of the likelihood of publication 3.90; 95% CI 2.68 to 5.68).1 Trials with positive findings are also published earlier (4–5 years vs 6–8 years)1 and are more likely to be published in English or in journals with higher impact.2 Thus, at any given time a meta-analysis is conducted, it will likely contain more trials with positive findings than those with negative findings and the estimated pooled effect size is likely to be exaggerated. This is a type of publication bias. When decision makers make a recommendation based on evidence affected by publication bias, this action is supported by likely exaggerated benefit. Therefore, the balance of benefit and harm that led to the recommended action is likely distorted. Decision makers acknowledge this distortion by placing lower certainty in the evidence.3 Hence, they rate down their certainty in the evidence, which is a construct also called quality or confidence in the evidence.4 Presence of publication bias can be determined using various statistical tests. These tests have major limitations, require several assumptions that are difficult to ascertain and are commonly underpowered.5 In addition, these tests provide a binary outcome (bias is absent or present), which by itself, is a major challenge.

nce of publication bias can be determined using various statistical tests. These tests have major limitations, require several assumptions that are difficult to ascertain and are commonly underpowered.5 In addition, these tests provide a binary outcome (bias is absent or present), which by itself, is a major challenge. In this guide, we propose further evaluation of publication bias and how it impacts certainty in the evidence. We draw attention to the issues of magnitude and direction of publication bias. Methods We propose a framework to determine the impact of publication bias on certainty in the evidence. This framework is based on the empirical literature on publication bias, published guidance from the GRADE Working Group, and various epidemiological and statistical principles. Necessary definitions are provided in table 1. Table 1 Definitions Publication bias The publication or non-publication of research findings, depending on the nature and direction of the results.22 Certainty in the evidence The certainty that a true effect lies on one side of a specified threshold or within a chosen range supporting a decision.4

Methods We propose a framework to determine the impact of publication bias on certainty in the evidence. This framework is based on the empirical literature on publication bias, published guidance from the GRADE Working Group, and various epidemiological and statistical principles. Necessary definitions are provided in table 1. Table 1 Definitions Publication bias The publication or non-publication of research findings, depending on the nature and direction of the results.22 Certainty in the evidence The certainty that a true effect lies on one side of a specified threshold or within a chosen range supporting a decision.4 Selection model A weight function of effect size or p value is used to model the probability of publication. This method highly depends on this weight and is usually recommended as a sensitivity analysis. Begg test A method that uses the rank test to examine the association between the observed effect sizes and their variances. However, it suffers from low statistical power. Egger test A method in which we regress the standardised effect size against the precision. The intercept is close to zero if no publication bias is present. This method may have inflated false-positive rates for ORs. Trim and fill method A method in which the missing studies are imputed to provide a bias-adjusted effect estimate. However, it requires the strong assumption that the missing studies have the most negative (or positive) effect sizes. Skewness A method that examines the asymmetry of residuals of the regression test. It has more statistical power than other tests. However, it may lose power if the available studies have a distribution that tends to have multiple modes. Direction of bias A meta-analysis of 25 randomised controlled trials that evaluated the effects of exercise on depression has shown that exercise reduced depression symptoms (standardised mean difference (SMD) 0.98, 95% CI 0.68 to 1.28).6 The Begg test and the Egger test indicated potential publication bias (p values 0.001 and 0.004; respectively). However, it was the unpublished trials that might have shown a larger magnitude of improvement. Adjusting SMD using the trim and fill method would show a larger pooled effect of improvement (SMD 1.11; 95% CI 0.79 to 1.43).6 In other words, publication bias has led to an underestimated SMD (opposite to the typical publication bias that exaggerates benefits of interventions). This phenomenon, although not likely to be common, should intuitively prevent decision makers from reducing their certainty in the evidence when recommending exercise as a treatment for depression.

as has led to an underestimated SMD (opposite to the typical publication bias that exaggerates benefits of interventions). This phenomenon, although not likely to be common, should intuitively prevent decision makers from reducing their certainty in the evidence when recommending exercise as a treatment for depression. This phenomenon is akin to when plausible bias and confounding suggests that the true association is stronger than the observed association.7 In the example of treating depression with exercise, the original rating of certainty in evidence would be reduced due to publication bias. However, evaluation of bias direction may prevent us from rating it down. Visualisation of the funnel plot, although a crude and subjective approach with shapes possibly influenced by the statistic used to measure the effect size,8 9 can give a sense about the direction of bias (observation of the empty area of the funnel). Magnitude of bias Several methods have been proposed to determine the magnitude of bias or adjust the pooled estimate for bias. In sensitivity analysis, one can test whether the magnitude of bias is sufficiently small to the extent that it remains unimportant (will not change the decision at hand).

Visualisation of the funnel plot, although a crude and subjective approach with shapes possibly influenced by the statistic used to measure the effect size,8 9 can give a sense about the direction of bias (observation of the empty area of the funnel). Magnitude of bias Several methods have been proposed to determine the magnitude of bias or adjust the pooled estimate for bias. In sensitivity analysis, one can test whether the magnitude of bias is sufficiently small to the extent that it remains unimportant (will not change the decision at hand). One of the approaches depends on selection models using the weighted distribution theory. In this approach, the probability a study is published can be modelled using certain functions of its p value or effect size, and this probability is incorporated in a meta-analysis model to adjust the effect size, possibly along with other study level covariates as regression predictors. Although they may provide some benefits in correcting publication bias,10 11 selection models are not widely used because they require a large number of studies and because of their complexity and unavailability in standard meta-analysis software.12

ibly along with other study level covariates as regression predictors. Although they may provide some benefits in correcting publication bias,10 11 selection models are not widely used because they require a large number of studies and because of their complexity and unavailability in standard meta-analysis software.12 The trim and fill method13 is based on the funnel plot in which missing studies are imputed by creating a mirror image of opposite corresponding studies. The adjusted effect size accounting for the missing studies can be used as a sensitivity analysis to determine the presence and magnitude of publication bias. It is important to note that this adjusted effect size is based on strong assumptions about the missing studies and should only be used for the purpose of sensitivity analysis (ie, should not be considered as a more accurate effect size to be used for decision making).14 The Begg rank test and the Egger regression test have been two popular methods for detecting publication bias.8 15 Both methods examine the association between the observed effect sizes and their precisions. However, the Begg test has been found to have very low power, and the Egger test may have inflated false-positive rates for binary outcomes.16 17

regression test have been two popular methods for detecting publication bias.8 15 Both methods examine the association between the observed effect sizes and their precisions. However, the Begg test has been found to have very low power, and the Egger test may have inflated false-positive rates for binary outcomes.16 17 A more recent measure, the skewness of the standardised deviates, quantifies publication bias by describing the asymmetry of the distribution of included trials (values range from −∞ to +∞ and a value of 0 suggests no skewness).18 The skewness is roughly considered mild, noticeable and substantial if it’s absolute value is smaller than 0.5, between 0.5 and 1, and greater than 1, respectively. Like other measures, the skewness is recommended to be reported along with its 95% CI to reflect its variability. In addition, a positive skewness measure indicates that some studies on the left side in the funnel plot (ie, those with OR or relative risk <1.0) might be missing due to publication bias. Therefore, this approach can suggest a direction and magnitude of publication bias and can be more powerful than other tests in some situations.18 For example, in a meta-analysis that evaluated nicotine replacement therapy for smoking cessation,19 three commonly used publication bias tests yielded p values greater than 0.10. The skewness measure was 0.91 (95% CI 0.14 to 1.68) suggesting substantial publication bias18 and possible unpublished studies with relative risk <0.0. In this example of nicotine replacement therapy, the original rating of certainty in evidence would not be reduced unless the magnitude of publication bias is considered.

e skewness measure was 0.91 (95% CI 0.14 to 1.68) suggesting substantial publication bias18 and possible unpublished studies with relative risk <0.0. In this example of nicotine replacement therapy, the original rating of certainty in evidence would not be reduced unless the magnitude of publication bias is considered. Proposed framework Considering the importance of the direction and magnitude of bias on certainty in the evidence supporting a particular action, we propose a framework in which certainty in evidence is compromised by publication bias only if the bias was not trivial and if it had a direction that shifts the balance of net benefit in a way that would make the recommended action less compelling (figure 1). In other words, if unpublished studies are unlikely to change the net impact of the intervention, we would not rate it down. If the magnitude or directions of the bias are unknown or are likely to affect the net impact of the intervention, then we would rate it down. Figure 1 A proposed framework for determining the impact of publication bias on certainty in the evidence (incorporating the magnitude and direction of publication bias).

Proposed framework Considering the importance of the direction and magnitude of bias on certainty in the evidence supporting a particular action, we propose a framework in which certainty in evidence is compromised by publication bias only if the bias was not trivial and if it had a direction that shifts the balance of net benefit in a way that would make the recommended action less compelling (figure 1). In other words, if unpublished studies are unlikely to change the net impact of the intervention, we would not rate it down. If the magnitude or directions of the bias are unknown or are likely to affect the net impact of the intervention, then we would rate it down. Figure 1 A proposed framework for determining the impact of publication bias on certainty in the evidence (incorporating the magnitude and direction of publication bias). Discussion Publication bias is one of the worst threats to the validity of scientific research. From an evidence synthesis perspective, we realise that the pooled estimates we produce may be based on studies exclusively showing that the treatment is effective (ie, we only see what works). This is a humbling fact. From a decision maker perspective, knowing that any decision we make is based on studies showing exaggerated benefit gives us very low certainty in our recommendations. Studies have shown that the likelihood of publication is often not associated with sample size, funding mechanism, investigator rank or gender.1 Therefore, publication bias remains unpredictable.

hat any decision we make is based on studies showing exaggerated benefit gives us very low certainty in our recommendations. Studies have shown that the likelihood of publication is often not associated with sample size, funding mechanism, investigator rank or gender.1 Therefore, publication bias remains unpredictable. The available tests for publication bias lack statistical power; particularly when heterogeneity is high or the number of studies is low.20 Their validity depends on the assumptions often unmet in practice. At the present time, no single test can be recommended. Systematic reviewers should consistently use multiple publication bias detection methods, and non-statistical approaches such as comparing published evidence with data available in clinical trials registries, records of drugs or device approving agencies such as the Food and Drug Administration, and scientific conference proceedings. Searching trial registries is an important step that can reveal registered trials that remained unpublished. Three commonly searched registries are ClinicalTrials.gov (http://www.clinicaltrials.gov), International Standard Randomised Controlled Trial Number Register (http://isrctn.org) and Australian New Zealand Clinical Trials Registry (http://www.anzctr.org.au).21 Despite the low power and unreliability of the current methods, information about the direction and magnitude of the bias may inform judgements about the certainty of evidence better than the binary decision of a statistical test.

The available tests for publication bias lack statistical power; particularly when heterogeneity is high or the number of studies is low.20 Their validity depends on the assumptions often unmet in practice. At the present time, no single test can be recommended. Systematic reviewers should consistently use multiple publication bias detection methods, and non-statistical approaches such as comparing published evidence with data available in clinical trials registries, records of drugs or device approving agencies such as the Food and Drug Administration, and scientific conference proceedings. Searching trial registries is an important step that can reveal registered trials that remained unpublished. Three commonly searched registries are ClinicalTrials.gov (http://www.clinicaltrials.gov), International Standard Randomised Controlled Trial Number Register (http://isrctn.org) and Australian New Zealand Clinical Trials Registry (http://www.anzctr.org.au).21 Despite the low power and unreliability of the current methods, information about the direction and magnitude of the bias may inform judgements about the certainty of evidence better than the binary decision of a statistical test. Contributors: MHM conceived the idea and drafted the manuscript. HC, LL and ZW critically revised the manuscript and approved the final submission. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Healthcare professionals (HCP) must apply their knowledge to their practice. Consequently, the implementation of research evidence largely depends on HCPs’ knowledge, acceptance of new evidence and choices. Improving patients’ lives and the quality of healthcare requires a strong emphasis on learning by HCP, teams and patients. Mastery of any area is never the result of a single inquiry, but is instead a continuum of inquiries, searches and reflections. Spiral learning is a teaching method in which the learner progressively gains knowledge on a subject with each encounter. Usually, complexity increases with each encounter and previous material is reinforced. Spiral learning, which is currently used in medical curricula, was initially justified because learners gained expertise as they revisited topics during practical applications. Therefore, spiral learning is suggested in this paper as an appropriate process to facilitate inquiry and effective learning.

al is reinforced. Spiral learning, which is currently used in medical curricula, was initially justified because learners gained expertise as they revisited topics during practical applications. Therefore, spiral learning is suggested in this paper as an appropriate process to facilitate inquiry and effective learning. Healthcare frameworks, such as the Deming Plan-Do-Study-Act problem-solving method (PDSA), evidence-based medicine (EBM) with its five steps and the five steps of patient self-management (SM), have been applied to healthcare to help learners implement best practices in healthcare delivery. Most of these frameworks are presented as a single inquiry. This paper suggests that these frameworks will be more effective than they are at present if they follow a spiral learning approach. The continuum in spiral learning is a factor among many that determines the success of these frameworks. Others include comprehensiveness of an inquiry, learner competencies and learning context. An additional determinant is the reflection on the elements of decision-making in healthcare, such as patient values, risks versus benefits, costs and evidential strength.

ong many that determines the success of these frameworks. Others include comprehensiveness of an inquiry, learner competencies and learning context. An additional determinant is the reflection on the elements of decision-making in healthcare, such as patient values, risks versus benefits, costs and evidential strength. Managing harms is important to safely care for patients. Every healthcare decision has a potential for harm, which could be active (eg, direct assault), come from errors in implementing care, neglecting recommended care or not preventing harm. Harm should be proactively managed or prevented via a similar spiral learning process. Finally, an important determinant is proactively targeting the learning environment to promote best evidence-based practices. The spiral learning approach in frameworks for research-based practice According to Donabedian, ‘quality’ is the science and technology of healthcare and the practical application of that science and technology.1 Learning to provide quality healthcare entails a series of cognitive processes that facilitate uptake of the science and its implementation, which integrates with learners’ existing skills and knowledge, both of which strengthen with time and experience. A continuum approach, spiral learning, will help learners’ to apply and integrate the best scientific and research evidence.2–6

nitive processes that facilitate uptake of the science and its implementation, which integrates with learners’ existing skills and knowledge, both of which strengthen with time and experience. A continuum approach, spiral learning, will help learners’ to apply and integrate the best scientific and research evidence.2–6 We can apply the spiral learning approach in frameworks used in improving healthcare, professional learning and patient empowerment. Some examples are the Deming PDSA cycle,3 evidence-based medicine and the five steps in applying it4 5 and the five steps in patients’ empowerment in self-management.6

nitive processes that facilitate uptake of the science and its implementation, which integrates with learners’ existing skills and knowledge, both of which strengthen with time and experience. A continuum approach, spiral learning, will help learners’ to apply and integrate the best scientific and research evidence.2–6 We can apply the spiral learning approach in frameworks used in improving healthcare, professional learning and patient empowerment. Some examples are the Deming PDSA cycle,3 evidence-based medicine and the five steps in applying it4 5 and the five steps in patients’ empowerment in self-management.6 Spiral learning supports decision-making at the team, HCP and patient levels. Learning always begins with a question about a practice, disease or patient care, which triggers data collection to search for evidence. Those data are assessed and aggregated to draw conclusions for implementation. Four aspects of the evidence are considered: its strength, cost, patient values and harms versus benefits. Ultimately, the results of an implementation should trigger further inquiry. Therefore, applying evidence to practice is like a spiral of learning and practice as opposed to a stepwise process, such as EBM key steps, or a cross-sectional approach, such as PDSA. This paper proposes spiral learning, which is a shared learning process, instead of PDSA, EBM and SM, all of which influence the quality of patient care through team, HCP or patient experiences. Figure 1 illustrates the spiral approach in repeated Ask, Search, Appraise or Assess, Aggregate, Agree, Act, Audit and re-Ask cycles as a continuous accumulative process.

a shared learning process, instead of PDSA, EBM and SM, all of which influence the quality of patient care through team, HCP or patient experiences. Figure 1 illustrates the spiral approach in repeated Ask, Search, Appraise or Assess, Aggregate, Agree, Act, Audit and re-Ask cycles as a continuous accumulative process. Figure 1 Spiral learning approach: (A) improving care and (B) prevention of harm. Determinants of success Comprehensive inquiry at each step This determinant concerns previous experience and the depth of the literature review. Learners with different experiences approach questions from different perspectives, and differences in skill level accordingly influence learning, which produces a spectrum of learning outcomes from novice to expert. Create a continuum of learning Learning continuously progresses and triggers more learning in an upward spiral in which questions arise at progressively higher levels. Spiral learning emphasises a continuum built on previous knowledge gained because EBM knowledge is not static and frequent reconsideration is needed. As learners revisit topics in practice, they gain expertise and their questions become deeper and more complex.7 Thus, previous advocates for empowering HCP to practice EBM were correct.

emphasises a continuum built on previous knowledge gained because EBM knowledge is not static and frequent reconsideration is needed. As learners revisit topics in practice, they gain expertise and their questions become deeper and more complex.7 Thus, previous advocates for empowering HCP to practice EBM were correct. Learner competencies Learners integrate knowledge, communication skills, professionalism, ethics and practical skills in decision-making. They build skills and knowledge by deeply questioning as their expertise increases. Competent HCP can be identified by assessing their work products as an expression of their extent of knowledge and the attention given to implementing the best possible healthcare. Regardless of the extent of their knowledge and skills, a HCP’s commitment to learning and determination to implement best evidence is essential. Learning context Questions might be the same at different points in the learning process, whereas search methods, data and, thus, conclusions might differ. An example is the use of interventions for waiting times: if a question were about business aspects, answers from managers and clinicians would likely differ.

Learner competencies Learners integrate knowledge, communication skills, professionalism, ethics and practical skills in decision-making. They build skills and knowledge by deeply questioning as their expertise increases. Competent HCP can be identified by assessing their work products as an expression of their extent of knowledge and the attention given to implementing the best possible healthcare. Regardless of the extent of their knowledge and skills, a HCP’s commitment to learning and determination to implement best evidence is essential. Learning context Questions might be the same at different points in the learning process, whereas search methods, data and, thus, conclusions might differ. An example is the use of interventions for waiting times: if a question were about business aspects, answers from managers and clinicians would likely differ. Fundamental reflection on the four essential elements of a conclusion (patient values, risks vs benefits, costs and strength of the evidence) These elements are clear in the aggregate steps of the EBM and SM, but not the PDSA, approaches. This reflection leads learners towards effective decisions on knowledge application and avoids blind implementations of research findings. Evidence is implemented in practice after decisions are made, which is followed by audits of outcomes. Then, the learner returns to the spiral, but at a higher level, with deeper questions based on previous knowledge, and the spiralling continues.

ledge application and avoids blind implementations of research findings. Evidence is implemented in practice after decisions are made, which is followed by audits of outcomes. Then, the learner returns to the spiral, but at a higher level, with deeper questions based on previous knowledge, and the spiralling continues. Special applications Learning to manage harms in healthcare is different because behavioural changes begin at the end of the spiral framework. The framework used to learn about harms is the reverse because it starts with an action, not a question, before evidence is collected to appraise or compare, which is similar to Costa and Kallick’s feedback spiral and recommendation for reflection on practice with a spiral approach.8 Data need to be collected to understand and compare actions and to understand causes before researching and experimenting the concluded evidence. The suggested spiral learning framework is the same as in figure 1, but reversed, so that the process is report/ask, collect data, assess, reflect/aggregate and implement evidence-based solutions. This approach to preventing harms is likely to facilitate improved efficiency in best-evidence practice. Additionally, figure 2 demonstrates the expression of harm, on a scale of low and high risk, and healthcare outcomes, on a scale of cured, improved or not cured, with different possible outcome combinations of both. The approach to improve quality or prevent harm should be targeted separately. Figure 2 Dimensions of harm and its expression with outcome of care.

Special applications Learning to manage harms in healthcare is different because behavioural changes begin at the end of the spiral framework. The framework used to learn about harms is the reverse because it starts with an action, not a question, before evidence is collected to appraise or compare, which is similar to Costa and Kallick’s feedback spiral and recommendation for reflection on practice with a spiral approach.8 Data need to be collected to understand and compare actions and to understand causes before researching and experimenting the concluded evidence. The suggested spiral learning framework is the same as in figure 1, but reversed, so that the process is report/ask, collect data, assess, reflect/aggregate and implement evidence-based solutions. This approach to preventing harms is likely to facilitate improved efficiency in best-evidence practice. Additionally, figure 2 demonstrates the expression of harm, on a scale of low and high risk, and healthcare outcomes, on a scale of cured, improved or not cured, with different possible outcome combinations of both. The approach to improve quality or prevent harm should be targeted separately. Figure 2 Dimensions of harm and its expression with outcome of care. Managing harms has four dimensions (figure 2): active or intentional, such as unprivileged practices; errors implementing care, such as wrong site surgeries; omitting care, such as a predetermined protocol for myocardial infarction and not proactively preventing harm.

Figure 2 Dimensions of harm and its expression with outcome of care. Managing harms has four dimensions (figure 2): active or intentional, such as unprivileged practices; errors implementing care, such as wrong site surgeries; omitting care, such as a predetermined protocol for myocardial infarction and not proactively preventing harm. Learning environment The learning environment must be directly targeted to promote spiral learning for evidence-based practice. It must be receptive to change. Learning in teams and through practice has been identified as important ways to develop HCP.9 Environments that promote a culture of continuous learning and quality improvements are equally important. Conclusion Teaching HCP how to make decisions and implement best practices requires a continuum of learning rather than one-time activities. Spiral learning should be applied to that continuum, and we should understand the crucial competencies throughout the process for reaching accurate answers and making appropriate decisions. Mastery of skills, knowledge and cognition that develop over time determines implementation of best evidence, which spiral learning emphasises. Employing this framework and learning the methodology will improve the implementation of best evidence in practice. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction: the challenge of knowledge inclusion in guidelines Evidence-based guidelines whether national, regional or developed by specialty groups, must search for, and explicitly consider, evidence from sources other than conventional clinical trials and their quantitative data. This need for appraising and including knowledge from a wide variety of sources in guideline development is well recognised.1–3 Although evidence on statistical association—usually from randomised controlled trials (RCTs)—is commonly thought to be the dominant type of knowledge appraised and included, guideline developers frequently use a range of other types of knowledge including the views and experiences of those using and providing health services, understanding of how interventions work (eg, from logic models or realist evaluations), and other information, such as aetiology and the context of care (online supplementary text box 1). 10.1136/bmjebm-2017-110844.supp1Supplementary file 1

Although evidence on statistical association—usually from randomised controlled trials (RCTs)—is commonly thought to be the dominant type of knowledge appraised and included, guideline developers frequently use a range of other types of knowledge including the views and experiences of those using and providing health services, understanding of how interventions work (eg, from logic models or realist evaluations), and other information, such as aetiology and the context of care (online supplementary text box 1). 10.1136/bmjebm-2017-110844.supp1Supplementary file 1 These different types of knowledge are used and needed in many situations, for example, when evidence from RCTs is not available, impossible to obtain, contradictory or inappropriate. They can also be used in conjunction with knowledge from RCTs to provide context, to assess relevance and to understand bias. Furthermore, explicit (written or spoken) knowledge and the more intricate forms of knowledge like experiential and contextual knowledge can help guideline makers to take an approach consistent with the intentions of early evidence-based medicine (EBM) proponents: namely, that best evidence is not restricted to evidence from RCTs and meta-analyses alone.4

ken) knowledge and the more intricate forms of knowledge like experiential and contextual knowledge can help guideline makers to take an approach consistent with the intentions of early evidence-based medicine (EBM) proponents: namely, that best evidence is not restricted to evidence from RCTs and meta-analyses alone.4 However, how to properly appraise (judge) and include (integrate) different kinds of knowledge remains unclear. Agreed methods are not yet available or are in the early stages of development and the need for and use of different kinds of knowledge is not always explicitly acknowledged, which affects the use of guidelines in practice.5 6 International and cultural differences in guideline production practices may further impede developments in appraising and including a broader range of types of knowledge (online supplementary text box 2). In this paper, we discuss four specific aspects of guideline development to highlight the main challenges identified by the AID Knowledge Working Group through discussions and workshops with guideline developers and users (online supplementary text box 3):the purpose of guideline development; the problem of induction; the dominance of frequency based reasoning; the challenge of integrating different sources of knowledge. In order to do this, we refer to some philosophical concepts around knowledge creation.

In this paper, we discuss four specific aspects of guideline development to highlight the main challenges identified by the AID Knowledge Working Group through discussions and workshops with guideline developers and users (online supplementary text box 3):the purpose of guideline development; the problem of induction; the dominance of frequency based reasoning; the challenge of integrating different sources of knowledge. In order to do this, we refer to some philosophical concepts around knowledge creation. The purpose of guideline development The efforts of the pioneers of the EBM movement were primarily in response to the discovery of the variation problem in population studies. Reducing variation of the care provided at a population level was considered to be an important way to achieve improved quality for individual patients.7 Hence, epidemiology, the science of studying populations, gained prominence in guidelines, the aims of which are to support decisions for individual patients. Classic epidemiology became clinical epidemiology when introduced to the bedside and the dominance of RCTs as the gold standard for intervention studies to assess causal relation between interventions and effect followed in this construct of epidemiology as used in EBM. The underlying—yet little explored—assumption is that guidelines based on population studies provide the best advice to inform clinical decisions for individual patients or situations.

ervention studies to assess causal relation between interventions and effect followed in this construct of epidemiology as used in EBM. The underlying—yet little explored—assumption is that guidelines based on population studies provide the best advice to inform clinical decisions for individual patients or situations. However, reducing variation is not the only reason for developing guidelines; they are developed for several reasons, of which the most important one is to improve the quality of care. In order to meet the range of needs, guidelines may need different approaches, such as summarising large quantities of knowledge for practising healthcare professionals, serving as an intermediate product for other tools or applications (such as clinical decision support software) or providing implementation guidance. Although not primarily developed for this purpose, guidelines can also serve as tools to legally shield both patients and professionals, to help governments and health insurers allocate scarce resources and to act as governance frameworks for practitioners and governments. There is also the role of guideline development as a discipline in itself; along with its associated practices and institutions, it provides employment and intellectual interest for many. There has been surprisingly little research into the purposes of guideline development. One mixed-method study found the purposes of guidelines were: defining norms, summarising evidence, formalising current consensus and/or describing current practices in a handbook-type format.5

There is also the role of guideline development as a discipline in itself; along with its associated practices and institutions, it provides employment and intellectual interest for many. There has been surprisingly little research into the purposes of guideline development. One mixed-method study found the purposes of guidelines were: defining norms, summarising evidence, formalising current consensus and/or describing current practices in a handbook-type format.5 Making the purposes of guidelines more explicit may help determine how different types of knowledge could and should be used. For instance, if the aim is to describe current good practice (eg, how services are organised to deliver care), this may be better achieved by drawing on qualitative or mixed-method evaluative research rather than RCTs. If the aim is to assess the effectiveness of a specific treatment or approach, evidence from RCTs or high-quality prospective cohort studies would usually be the primary source of knowledge, with qualitative or mixed-method studies serving to help understand the local context of implementation. It is important to note, however, that for most guideline developers, the primary purpose remains that of supporting decision making in the clinical encounter. This leads us to the next fundamental aspect of guideline development. The problem of induction How do different types of knowledge in guidelines development help to make clinical decisions? Some basic concepts from the philosophy of science may help to understand the problem.

It is important to note, however, that for most guideline developers, the primary purpose remains that of supporting decision making in the clinical encounter. This leads us to the next fundamental aspect of guideline development. The problem of induction How do different types of knowledge in guidelines development help to make clinical decisions? Some basic concepts from the philosophy of science may help to understand the problem. Inference, the problem of induction and evasions In logic, to infer means to conclude from evidence using reasoning.8 In everyday healthcare practice, care professionals and patients reason to reach conclusions about what has happened, to make predictions about what will happen and to decide what to do next. Because of uncertainty in medicine, we usually deal with a specific type of inference, called induction, where the conclusions of our reasoning are not always right even when based on true premises. In philosophy, there is a concern whether this is actually possible, called the problem of induction,8 as introduced by Hume in 1739.9 At its simplest, this means we cannot predict the future with certainty. Although this seems reasonable, we are in fact able to predict the future quite accurately on many occasions in clinical practice. How is this possible? Philosopher of science Ian Hacking8 argues that we never solve the problem of induction, but only evade it by applying different kinds of reasoning to reduce uncertainty and increase our chance of reaching the best possible outcome.

e future quite accurately on many occasions in clinical practice. How is this possible? Philosopher of science Ian Hacking8 argues that we never solve the problem of induction, but only evade it by applying different kinds of reasoning to reduce uncertainty and increase our chance of reaching the best possible outcome. The dominance of frequency-based reasoning The evasion most dominantly used in guideline development is frequency-type reasoning in the form of systematic reviews, RCTs and observational studies.5 This evades the problem of induction by recognising that ‘although we can’t predict the future for the individual case, we can be “usually” right (eg, 95% of the time)’8 as long as events or cases are frequent enough. Frequency-based reasoning relies on basic assumptions that have some drawbacks. First, this line of reasoning assumes that reality is dice like and that we—eg, scientists guideline developers and healthcare professionals—are rolling the same dice (online supplementary text box 4). Frequency-type reasoning presupposes adequate framing and defining of what is similar and what is not, which is always based on judgement and choice.

ssumes that reality is dice like and that we—eg, scientists guideline developers and healthcare professionals—are rolling the same dice (online supplementary text box 4). Frequency-type reasoning presupposes adequate framing and defining of what is similar and what is not, which is always based on judgement and choice. Second, frequency-based reasoning aims to find simple causal correlations, independent of context. The question is whether these simple correlations hold true in real life. Different understandings of causality exist that could help us address this drawback.10 For instance, a network of complex causal relationships may be more realistic. This drawback is described as the efficacy paradox, where the different interference from non-specific effects (different from those controlled for between groups in a trial), measurement artefacts (that mimicked therapeutic effects in the trial) and regression patterns (such as the self-limiting nature of a disease) in real life can outweigh the specific effect found in a trial. This paradox may become especially apparent when inferring in the context of multimorbidity.11

roups in a trial), measurement artefacts (that mimicked therapeutic effects in the trial) and regression patterns (such as the self-limiting nature of a disease) in real life can outweigh the specific effect found in a trial. This paradox may become especially apparent when inferring in the context of multimorbidity.11 Finally, and most importantly, although frequency-based reasoning works well for frequent events (large groups, many data points and long periods of time), such reasoning faces fundamental limitations when inferring in the single-case scenario: a single patient, a rare disease, a system intervention and an one-off event. This can be particularly challenging when recommendations based on frequency-type evidence alone are deployed to help decision making for individual patients or unique situations, such as a public health response to a disease outbreak.2 Given these drawbacks, it is worth noting that other types of reasoning to evade the epistemological problem of induction exist. In table 1, several alternative ways of reasoning are listed. They are mainly used in areas where frequency-based reasoning is particularly problematic, for instance in guidelines focusing on complex interventions, public and occupational health, rehabilitation, and social care and welfare.11–13 Table 1 Alternative types of reasoning to evade the problem of induction

In table 1, several alternative ways of reasoning are listed. They are mainly used in areas where frequency-based reasoning is particularly problematic, for instance in guidelines focusing on complex interventions, public and occupational health, rehabilitation, and social care and welfare.11–13 Table 1 Alternative types of reasoning to evade the problem of induction Type of reasoning (with examples of key scholars) Shorthand description Explanation Bayesian evasion (Bayes, Hacking) Learning from experience This type of inductive inference agrees with Hume that we cannot predict the future perfectly, but that we can learn from our experiences reasonably well. This allows us to do more and better predictions. This type of reasoning can update current beliefs with information from frequent events (informing prior probabilities and likelihood ratios). However, because we can learn from a single event too, this approach is suited for the individual case scenario.8 Abduction (Peirce) Reasoning to the best explanation Abduction makes inferences by updating beliefs leading to the best explanation.28 Where Bayesian evasion takes prior probabilities as a given (at least as some argue), which may not be the case, abduction does not. It introduces the consideration of theory and mechanism in the act of inferring.29

o the best explanation Abduction makes inferences by updating beliefs leading to the best explanation.28 Where Bayesian evasion takes prior probabilities as a given (at least as some argue), which may not be the case, abduction does not. It introduces the consideration of theory and mechanism in the act of inferring.29 Mechanistic/deterministic reasoning How things appear to work This type of reasoning makes an inference based on a mechanism. Illari et al 16 define a mechanism as consisting ‘of entities and activities organised in such a way that they are responsible for the phenomenon’. Falsification (Popper) Trial and error Popper30 agreed with Hume: we cannot say anything about the future, there are only theories that cannot even be proven. At best, we can only prove that they are wrong (falsifiable). This ‘anti-inductivist’ reasoning suggests to continue using a certain theory or practice and adjust if they fail. Precautionary principle In case of uncertainty about the future prevent harm The precautionary principle, often used in environmental decision making and occupational health, favours to take preventive action in the face of uncertainty when making an inference. It puts ‘the burden of proof to the proponents of an activity; exploring a wide range of alternatives to possibly harmful actions; and increasing public participation in decision making’.17 31

decision making and occupational health, favours to take preventive action in the face of uncertainty when making an inference. It puts ‘the burden of proof to the proponents of an activity; exploring a wide range of alternatives to possibly harmful actions; and increasing public participation in decision making’.17 31 Means-to-ends reasoning Find ways to reach a goal This type of reasoning asks the question what ways are there to reach a certain wanted outcome and which of those ways would be the more efficient? Often used in clinical consultations to make sure that something happens whatever the circumstances. The inference remains uncertain but less so by using multiple means that will lead to the same outcome.32 Logic of care (Mol) Taking care while the uncertain future unfolds In The Logic of Care Annemarie Mol15 suggests that healthcare is more like a ‘practice’ than it is about making choices. This approach puts emphasis on the importance of taking good care for the patient and the prevention of neglect. Inferring is a process that unfolds over time, while addressing many factors on the way. Non-analytical reasoning (Gigerenzer, Stolper) Using intuition Non-analytical reasoning such as heuristics and gut feelings (combination of heuristics and emotions33) used to make inferences. These types of reasoning are considered fast, intuitive and automatic thought processes. Gigerenzer showed that non-analytical reasoning can in certain environments outperform analytical reasoning in psychological, biological, sociological and economic inference tasks.34

ristics and emotions33) used to make inferences. These types of reasoning are considered fast, intuitive and automatic thought processes. Gigerenzer showed that non-analytical reasoning can in certain environments outperform analytical reasoning in psychological, biological, sociological and economic inference tasks.34 Types of reasoning are not exclusive and may overlap. These different types of reasoning try to help make valid inferences for the single-case scenario, when there is no frequency of events. Many of these are already recognised and stated by Bradford-Hill14 in his criteria for causation, but some are newer, such as Annemarie Mol’s logic of care,15 where a practitioner will try something, wait and see and let unfolding events guide the next step. Using this type of reasoning, the problem of induction is solved through ‘tinkering’, making incremental changes to improve a situation.

riteria for causation, but some are newer, such as Annemarie Mol’s logic of care,15 where a practitioner will try something, wait and see and let unfolding events guide the next step. Using this type of reasoning, the problem of induction is solved through ‘tinkering’, making incremental changes to improve a situation. Guidelines can and do support these kinds of evasions by including different types of knowledge. For instance, providing laboratory information about aetiology helps to make an inference based on mechanistic reasoning.16 A description of cases of harm can offer an inference based on the precautionary principle.17 Rethinking how inferences are made in practice may shift the dominance of frequency-based reasoning and its reliance on a restrictive type of knowledge to a broader spectrum of knowledge being used to support different reasoning approaches. The need for using different type of knowledge is shown by a large Dutch analysis showing that knowledge from RCTs far outweighed other knowledge types used, irrespective of the question at hand, thus ignoring important and relevant knowledge from other sources.5 6 The challenge of integration Making a recommendation for a specific healthcare problem in a specific healthcare system requires the assessment of knowledge not just on its own merits, but importantly its integration with other knowledge. Indeed, EBM is defined as integrating the best evidence with clinical expertise and patient preference.4

ration Making a recommendation for a specific healthcare problem in a specific healthcare system requires the assessment of knowledge not just on its own merits, but importantly its integration with other knowledge. Indeed, EBM is defined as integrating the best evidence with clinical expertise and patient preference.4 However, in the context of medicine, and even more so in that of guideline production, integration of different types of knowledge remains underexplored and undertheorised. Some areas of evidence synthesis have addressed integration. For example, statistical techniques such as meta-analysis can be used to combine data from different studies, and another range of techniques can be used to synthesise qualitative data.

However, in the context of medicine, and even more so in that of guideline production, integration of different types of knowledge remains underexplored and undertheorised. Some areas of evidence synthesis have addressed integration. For example, statistical techniques such as meta-analysis can be used to combine data from different studies, and another range of techniques can be used to synthesise qualitative data. In guideline development, most of the activities and tools to support high-quality evidence synthesis such as risk of bias assessment and quality assessment (such as GRADE) tend to focus primarily on frequency-based reasoning and knowledge. For the assessment of quality of qualitative evidence, there are limited but relevant initiatives for guideline development in progress, for example, the recently published Grading of Recommendations Assessment, Development, and Evaluation – Confidence in the Evidence from Reviews of Qualitative research (GRADE-CERQual)18 guidance. However, many of these efforts try to achieve integration by synthesizing studies that share the same questions and design (eg a set of qualitative or, more narrowly, ethnographic studies),19 at times appraising18 all such knowledge again in frequentist terms, like with some qualitative evidence synthesis methods20 that ‘emphasize frequencies of the qualitative data they present…undermin[ing] the uniqueness of the qualitative knowledge they proclaim by focusing on frequency and the general patterns’.21

),19 at times appraising18 all such knowledge again in frequentist terms, like with some qualitative evidence synthesis methods20 that ‘emphasize frequencies of the qualitative data they present…undermin[ing] the uniqueness of the qualitative knowledge they proclaim by focusing on frequency and the general patterns’.21 The main issue is that these tools, activities and initiatives aim to integrate similar knowledge, such as data from the same study designs, the same populations or the same outcomes. How different kinds of knowledge are valued, appraised and weighed in relation to each other, for example, regarding effectiveness, efficiency or ethical concerns, is not clearly articulated. Nonetheless, guideline developers do recognise that other types of knowledge are often used and somehow integrated in practice, particularly when discussing the evidence and formulating recommendations, often called ‘judgement’ or ‘considered judgement’.22 23 This is the traditionally less clearly described or analysed black box part of the process that new initiatives try to shed a light on, such as NICE’s structured tables linking evidence to recommendations and the GRADE Evidence to Decision frameworks.22

ndations, often called ‘judgement’ or ‘considered judgement’.22 23 This is the traditionally less clearly described or analysed black box part of the process that new initiatives try to shed a light on, such as NICE’s structured tables linking evidence to recommendations and the GRADE Evidence to Decision frameworks.22 They appear promising yet challenges remain. First, findings from ethnography question whether structured frameworks really influence or reflect guideline development processes.24 25 In an ethnographic study of guideline development meetings, Moreira showed that guideline developers formulate guidance by combining different ‘repertoires of evaluation, organised around four different epistemic criteria: robustness, usability, acceptability and adequacy’.24 Importantly, such criteria are deployed at each stage of evidence appraisal: usability, acceptability and adequacy are integral to evidence assessment, rather than being easily categorised as either ‘judgements’ or ‘additional considerations’ as current evidence to decision frameworks suggest.22 Acknowledging the importance of these epistemic skills in evidence appraisal become much more important when it is understood that recommendations nearly always draw on different types of knowledge.

y categorised as either ‘judgements’ or ‘additional considerations’ as current evidence to decision frameworks suggest.22 Acknowledging the importance of these epistemic skills in evidence appraisal become much more important when it is understood that recommendations nearly always draw on different types of knowledge. Second, bringing knowledge together is not just a process of integrating, triangulation and finding a single answer. Knowledge from many sources is often conflicting, and indeed the exploration of opposing ideas is often very important. In social sciences, methods for evidence synthesis of other kinds of data have been developed and assessed in research26 and in practice guideline development,13 but these have not yet been adopted routinely in healthcare guideline development. A process of integration is not just a technical, simple mechanistic process. Guideline development is a human, social process involving relevant stakeholders in discussion, debate and judgement. Therefore, the guideline development processes also relies on a balanced and representative guideline committee that functions well.27

of integration is not just a technical, simple mechanistic process. Guideline development is a human, social process involving relevant stakeholders in discussion, debate and judgement. Therefore, the guideline development processes also relies on a balanced and representative guideline committee that functions well.27 Finally, integrating many types of knowledge is not a process in which anything goes. Some integration processes are likely to be better than others. Guideline development needs to be transparent and consistent so that reality, be it physical or social, can limit the inferences and recommendations made. We need a range of integration approaches depending on our understanding of what is true and real; for example, integration of different knowledge could be based on combinations of coherence (what fits best in a network of other theories), on consensus (what people agree on) and/or on correspondence (what links best to what is believed to be real). Given the current state of evidence to decision frameworks, there is still little guidance on how to robustly and consistently combine knowledge of different types without using the frequentist understanding of knowledge. A broader discussion within the guideline community is needed about the frameworks used to integrate and include different kinds of knowledge. Considering theories from epistemology and findings from ethnography (online supplementary text box 5) could be instrumental to deepen our understanding of how other types of knowledge can be synthesised and integrated in guideline production.

t the frameworks used to integrate and include different kinds of knowledge. Considering theories from epistemology and findings from ethnography (online supplementary text box 5) could be instrumental to deepen our understanding of how other types of knowledge can be synthesised and integrated in guideline production. Conclusion The development of guideline recommendations is an interactive human process that requires a range of knowledge and experience including, but not exclusively, knowledge from frequency-based research, such as clinical trials. As in the clinical encounter, appraising and including different types of knowledge in guideline development should be used to make better inferences to guide decisions, but in practice, arguments are used to exclude some kinds of knowledge for a range of reasons, including concerns about introducing bias in frequentist reasoning. In this paper, we present important epistemological reasons to appraise and include a (wide) variety of different types of knowledge to highlight important aspects of guideline development that await further exploration and practical suggestions.

Conclusion The development of guideline recommendations is an interactive human process that requires a range of knowledge and experience including, but not exclusively, knowledge from frequency-based research, such as clinical trials. As in the clinical encounter, appraising and including different types of knowledge in guideline development should be used to make better inferences to guide decisions, but in practice, arguments are used to exclude some kinds of knowledge for a range of reasons, including concerns about introducing bias in frequentist reasoning. In this paper, we present important epistemological reasons to appraise and include a (wide) variety of different types of knowledge to highlight important aspects of guideline development that await further exploration and practical suggestions. We acknowledge that appraising and including knowledge from a different variety of sources is likely to be complex and ongoing. Discussions about purpose, reasoning and integration in guideline development will continue. A simple set of tools or methodological quick fixes are unlikely to suffice, and developing criteria for appraising and integrating different knowledge will remain a challenge. However, we believe that much can be done to help guideline developers improve this—now often implicit5practice that is central to their work. Capacity-building workshops that confront implicit forms of reasoning are one example. AID Knowledge runs such workshops annually at G-I-N conferences. They help to strengthen ties between guideline developers who are concerned about the increasingly rigid methodological constraints on guideline methods at the expense of fostering epistemic sensibilities. It is important for guideline developers to feel they are part of a community of practice that encourages epistemic skill development, rather than a hierarchical community where superior guideline methods are defined by a small group of experts. This will help to keep guideline development innovative and diverse.

ties. It is important for guideline developers to feel they are part of a community of practice that encourages epistemic skill development, rather than a hierarchical community where superior guideline methods are defined by a small group of experts. This will help to keep guideline development innovative and diverse. Acknowledging that dominant frequentist methods are excellent for some questions but do not fit all knowledge needs is the first step to implementing different kind of reasoning in guideline development. How to address the diversity in methods for different kinds of questions should be among the top guideline research priorities. Contributors: All authors substantially contributed to the conception or design of the work, or the acquisition, analysis or interpretation of data; drafted the work or revising it critically for important intellectual content; gave final approval of the version published; agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding: SW received funding from the European Union Seventh Framework Programme (FP7‐PEOPLE‐2013‐COFUND), Grant/Award Number: 609020. TZJ received funding from a fellowship in the Future Research Leaders program of Linköping University. The OA fee was paid for by the Guidelines International Network.

Contributors: All authors substantially contributed to the conception or design of the work, or the acquisition, analysis or interpretation of data; drafted the work or revising it critically for important intellectual content; gave final approval of the version published; agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding: SW received funding from the European Union Seventh Framework Programme (FP7‐PEOPLE‐2013‐COFUND), Grant/Award Number: 609020. TZJ received funding from a fellowship in the Future Research Leaders program of Linköping University. The OA fee was paid for by the Guidelines International Network. Competing interests: SW was a guidelines update standing committee member for NICE from 2014 until 2018. FF was a Board member of Guidelines International Network from 2009 - 2013. He led the Norwegian Guideline Secretariat from 2001 - 2006 and has been a member of several Guideline Working Groups nationally and internationally. CH is guideline co-ordinator of the Netherlands Society of Occupational Medicine. SL worked as a guideline methodologist and developer in Australia. FM worked with NICE on clinical guidelines since 2000 and was Director of the Centre for Clinical Practice (responsible for guidelines) at NICE from 2008-2011. BS worked for NICE from 2008 until 2017, advising on the methods of guideline development. She is also the chair of the GIN AID Knowledge Working Group. TZJ was founding co-chair of the AID Knowledge Working Group. He has studied guideline development for about a decade, through multiple externally funded studies.

011. BS worked for NICE from 2008 until 2017, advising on the methods of guideline development. She is also the chair of the GIN AID Knowledge Working Group. TZJ was founding co-chair of the AID Knowledge Working Group. He has studied guideline development for about a decade, through multiple externally funded studies. Provenance and peer review: Not commissioned; externally peer reviewed. Collaborators: Stephanie Chang, Pwee Keng Ho, Sonja Kersten, Miranda Langedam, Peter O’Neill, Sarah Richards, Rodrigo Pardo Turriago, Sue Phillips.

One of the most widely used tools for assessing and communicating scientific uncertainty is Grading of Recommendations Assessment, Development, and Evaluation (GRADE), a system for rating the quality of evidence and grading strength of recommendations in healthcare. More than 100 organisations around the world—WHO included1—are using GRADE or have endorsed it. In GRADE, a quantitative assessment of uncertainty is qualitatively communicated, so that a result obtained as a CI relative to a threshold is expressed as a finding in which assessors have low, moderate or high certainty, or certainty described with other such qualifiers. What these correspond to in quantitative terms, and how decision-makers interpret them, is our issue here. We confine our attention to GRADE’s decision rules for systematic reviews, and do not comment on the problem of multiple outcomes in guideline recommendations. In a recent guideline article,2 GRADE introduced an idea that appears to undermine sound statistical reasoning in systematic reviews: the idea is that a result that is statistically inconclusive because the null hypothesis cannot be ruled out3 is converted into ‘moderate certainty’. We fear that, applied as a principle, this GRADE guideline may jeopardise patient health.

dea that appears to undermine sound statistical reasoning in systematic reviews: the idea is that a result that is statistically inconclusive because the null hypothesis cannot be ruled out3 is converted into ‘moderate certainty’. We fear that, applied as a principle, this GRADE guideline may jeopardise patient health. What is a statistically inconclusive result? Suppose the potential harm of a treatment is tested. A threshold is set above which the harm is clinically relevant. A confidence level is chosen that reflects how the consequences of erroneous inferences are weighted. If the harm is serious, the level may be 99%, with 1% error risk. If the harm is less serious, a 95% or a 90% level might be chosen. Then, if the interval estimate includes the threshold, the possibility of harm cannot be excluded. The result is inconclusive given the research question and given the chosen confidence level. More generally, when a CI includes the clinically relevant threshold, the result is inconclusive3 (p 2596). GRADE presents as an example a hypothetical case4 concerning the reduction of incidents of ischaemic stroke2 (p 6). The choice of confidence level adopted by GRADE is 95%. The threshold of minimally relevant reduction is set at 1.0% absolute reduction in strokes to reflect the harm associated with the treatment. The resulting interval estimate is 0.6%–2.0%. This means that the threshold is clearly included in the GRADE example. Notwithstanding this, the conclusion of the Grade Working Group (p 7) is the following:

relevant reduction is set at 1.0% absolute reduction in strokes to reflect the harm associated with the treatment. The resulting interval estimate is 0.6%–2.0%. This means that the threshold is clearly included in the GRADE example. Notwithstanding this, the conclusion of the Grade Working Group (p 7) is the following: Because the point estimate of 1.3% meets the threshold criterion… the imprecision-generated uncertainty will result in… moderate certainty that the [‘true’] effect is above the threshold [1.0]. In effect, GRADE is downplaying the importance of a prespecified α-level in a protocol by applying the idea that any null hypothesis (threshold) will be rejected to some degree, provided that the point estimate lies on the preferred side of the null hypothesis. This flexibility might be appreciated by guideline developers as well as by stakeholders, but it may also undermine the transparency of the process of the systematic review.

y null hypothesis (threshold) will be rejected to some degree, provided that the point estimate lies on the preferred side of the null hypothesis. This flexibility might be appreciated by guideline developers as well as by stakeholders, but it may also undermine the transparency of the process of the systematic review. This means that ‘inconclusive’ is converted into ‘moderate certainty’ when GRADE is used. For this specific result to be conclusive, the confidence level must be lowered to less than 80%.i The corresponding p value ii is 0.20 in a one-sided test and 0.40 in a two-sided test. GRADE’s latest stipulation of the meaning of ‘moderate’5 is that the ‘true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different’ (p 404). However, in everyday language a common understanding of ‘moderate’ is ‘within reasonable limits’. If the idea of converting statistically inconclusive results into ‘moderate certainty’ is understood as a principle, some systematic reviews using GRADE may unintentionally mislead, since it cannot be assumed that users will interpret ‘moderate’ in accordance with GRADE’s stipulation (Figure 1). Figure 1 P values (modified from https://xkcd.com/1478/). i Given a Z distribution, the SE is approximated 0.36≈(1.3−0.6)/1.96.

This means that ‘inconclusive’ is converted into ‘moderate certainty’ when GRADE is used. For this specific result to be conclusive, the confidence level must be lowered to less than 80%.i The corresponding p value ii is 0.20 in a one-sided test and 0.40 in a two-sided test. GRADE’s latest stipulation of the meaning of ‘moderate’5 is that the ‘true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different’ (p 404). However, in everyday language a common understanding of ‘moderate’ is ‘within reasonable limits’. If the idea of converting statistically inconclusive results into ‘moderate certainty’ is understood as a principle, some systematic reviews using GRADE may unintentionally mislead, since it cannot be assumed that users will interpret ‘moderate’ in accordance with GRADE’s stipulation (Figure 1). Figure 1 P values (modified from https://xkcd.com/1478/). i Given a Z distribution, the SE is approximated 0.36≈(1.3−0.6)/1.96. ii The null hypothesis concerns the threshold H 0:θ≤1.0, why Z=0.83≈(1.3−1.0)/0.36. A Z value of ±0.83 divides the probability density function into three areas: 0.20 and 0.60 and 0.20. In a one-sided test, the p value is 0.20, and in a double-sided test it is 0.20+0.20=0.40. A Z value of 1.96 divides the PDF into the following familiar areas: 2.5%, 95% and 2.5%. Contributors: All authors contributed to the planning. SA wrote the first draft. All other authors contributed equally.

ibitors (mesh); (11) statin (tw); (12) 9 or 10 or 11; (13) antihypertensive agents (mesh); (14) cardiovascular (all); (15) cardiovascular diseases (majr); (16) 14 or 15; (17) 8 and 12 and 13 and 16. Similar search strategies were applied to the other sources. Reference lists of relevant publications were hand searched. Study selection Studies were included in the systematic review if they were factorial and had at least two randomised interventions, had a total duration of at least 100 patient-years, involved patients aged ≥18 years, had at least one statin and one blood pressure-lowering drug as active treatment, had placebo or a less intensive blood pressure-lowering drug regimen as control treatment, and reported clinical cardiovascular events or mortality as outcomes. Trials were excluded if they did not report clinical outcomes or if the intervention strategies were unclear. Due to scarcity of data, we included studies with different intensities of antihypertensive treatment in the blood pressure factor. We defined treatment groups in these studies as more versus less intensive antihypertensive treatment based on achieved blood-pressure differences on the group level. In case of several useful publications from the same study, the one with the longest duration of follow-up was used. Two independent reviewers screened all abstracts for eligibility, reviewed relevant articles in full text, included relevant articles in the systematic review and, if applicable, extracted data for the meta-analysis.

veral useful publications from the same study, the one with the longest duration of follow-up was used. Two independent reviewers screened all abstracts for eligibility, reviewed relevant articles in full text, included relevant articles in the systematic review and, if applicable, extracted data for the meta-analysis. Data extraction and bias assessments Data were extracted from the articles using a prespecified spreadsheet. In case of unclear reporting of results, we contacted the authors asking for supplementary information. Data items extracted included study identification variables, treatments, numbers of patients, age, sex, baseline variables (body mass index, smoking, known hypertension, known dyslipidaemia, previous cardiovascular disease, previous myocardial infarction, type 2 diabetes, systolic and diastolic blood pressures, total, high-density lipoprotein cholesterol and low-density lipoprotein cholesterol; described in the online supplementary table 1) and outcome variables (major adverse cardiovascular events (MACE, as defined by the studies, mainly including myocardial infarction, stroke and cardiovascular death, optionally also heart failure), MACE-plus (expanded MACE classifications, as defined by the studies, usually adding unstable angina and coronary revascularisation), myocardial infarction, stroke, cardiovascular death, and total death; described in the online supplementary table 2). Quality of the included trials was gauged by using the Cochrane Collaboration’s risk-of-bias tool. Publication bias was assessed using funnel plots and Egger’s tests.

na and coronary revascularisation), myocardial infarction, stroke, cardiovascular death, and total death; described in the online supplementary table 2). Quality of the included trials was gauged by using the Cochrane Collaboration’s risk-of-bias tool. Publication bias was assessed using funnel plots and Egger’s tests. 10.1136/bmjebm-2017-110888.supp1Supplementary file 1 Data synthesis and analysis Baseline data were summarised using inverse variance weights. Treatment effects were visualised in forest plots of tabular trial data, using fixed-effects inverse variance-weighted meta-analysis models to illustrate heterogeneity as the I2 statistic. In order to investigate synergistic effects, we used a one-step individual patient data meta-analysis approach12 with a two-level mixed-effects logistic regression model with patient as the unit of analysis and trial modelled on a second level with random intercept. Models with the addition of random coefficients for treatments did not have a better fit than models with only a random intercept and were not further pursued. This is the recommended one-stage model for assessing within-trial interactions.13 14 Because the aggregate data used in this study are all dichotomous variables, a complete representation of the individual participant data for the primary analyses could be achieved.14

y a random intercept and were not further pursued. This is the recommended one-stage model for assessing within-trial interactions.13 14 Because the aggregate data used in this study are all dichotomous variables, a complete representation of the individual participant data for the primary analyses could be achieved.14 Because there is no generally accepted statistical definition of pharmacological synergism, we tested for interactions in two ways: as departure from additivity and as departure from multiplicativity. Departure from additivity was investigated as the relative excess risk due to interaction (RERI), the attributable proportion due to interaction (AP) and as the synergy index (S),15 in the mixed-effects models described above. In the absence of interaction on the additive scale, RERI and AP are equal to 0 and S is equal to 1. Departure from multiplicativity was examined using a within-trial product of the variables for statins and blood pressure-lowering drugs, added to the otherwise same mixed-effects models. In the absence of interaction on the multiplicative scale, the OR for that product is equal to 1. We used two-sided 95% CIs for all hypothesis tests. Stata V.14.2 was used for all analyses. Patient involvement No patients were involved in the design, conduction, interpretation or reporting of the analyses.

Because there is no generally accepted statistical definition of pharmacological synergism, we tested for interactions in two ways: as departure from additivity and as departure from multiplicativity. Departure from additivity was investigated as the relative excess risk due to interaction (RERI), the attributable proportion due to interaction (AP) and as the synergy index (S),15 in the mixed-effects models described above. In the absence of interaction on the additive scale, RERI and AP are equal to 0 and S is equal to 1. Departure from multiplicativity was examined using a within-trial product of the variables for statins and blood pressure-lowering drugs, added to the otherwise same mixed-effects models. In the absence of interaction on the multiplicative scale, the OR for that product is equal to 1. We used two-sided 95% CIs for all hypothesis tests. Stata V.14.2 was used for all analyses. Patient involvement No patients were involved in the design, conduction, interpretation or reporting of the analyses. Results Out of 1017 studies screened, 39 were read in full text and 711 16–21 were eventually included in the overview (figure 1). Using funnel plots (online supplementary figure 1), no evidence of publication bias was observed (all Egger’s test P>0.12). Risk of bias within studies was generally low (online supplementary figures 1 and 2). Figure 1 Flow chart of literature review.

Results Out of 1017 studies screened, 39 were read in full text and 711 16–21 were eventually included in the overview (figure 1). Using funnel plots (online supplementary figure 1), no evidence of publication bias was observed (all Egger’s test P>0.12). Risk of bias within studies was generally low (online supplementary figures 1 and 2). Figure 1 Flow chart of literature review. The included trials contributed a total of 27 020 patients. Baseline characteristics are described by the study group in table 1, and by trial in the online supplementary table 1. The trials were heterogeneous in terms of target populations, with some recruiting from the general population and some only among patients with established cardiovascular disease or organ damage. Baseline characteristics were well balanced between the randomised groups. Table 1 Baseline characteristics

The included trials contributed a total of 27 020 patients. Baseline characteristics are described by the study group in table 1, and by trial in the online supplementary table 1. The trials were heterogeneous in terms of target populations, with some recruiting from the general population and some only among patients with established cardiovascular disease or organ damage. Baseline characteristics were well balanced between the randomised groups. Table 1 Baseline characteristics Group Statin+BPRx BPRx Placebo Statin Patients, n 7227 6539 6452 6802 Age, years 63.2 (7.7) 63.7 (7.6) 63.7 (7.6) 63.6 (7.5) Men, % 64.2 65.4 66.9 64.5 BMI, kg/m2 27.7 (4.8) 27.7 (4.7) 27.6 (4.6) 27.7 (4.7) Smoking, % 30.9 30.3 30.5 30.1 Previous CVD, % 9.2 7.3 7.0 8.1 Previous MI, % 1.3 1.4 1.3 1.4 Type 2 diabetes, % 14.3 14.4 14.0 14.2 Hypertension, % 65.9 62.1 61.9 63.5 SBP, mm Hg 148.3 (15.3) 148.4 (16.1) 147.8 (16.0) 148.2 (15.9) DBP, mm Hg 87.6 (9.2) 87.2 (9.6) 87.0 (9.5) 87.2 (9.5) Hyperlipidaemia, % 30.3 21.6 20.9 24.7 LDL cholesterol, mM 3.57 (0.85) 3.46 (0.84) 3.44 (0.84) 3.51 (0.83) HDL cholesterol, mM 1.22 (0.36) 1.22 (0.35) 1.22 (0.36) 1.22 (0.35) BMI, body mass index; BPRx, blood pressure-lowering treatment; CVD, cardiovascular disease; DBP, diastolic blood pressure, HDL, high-density lipoprotein; LDL, low-density lipoprotein; MI, myocardial infarction; SBP, systolic blood pressure.

3.51 (0.83) HDL cholesterol, mM 1.22 (0.36) 1.22 (0.35) 1.22 (0.36) 1.22 (0.35) BMI, body mass index; BPRx, blood pressure-lowering treatment; CVD, cardiovascular disease; DBP, diastolic blood pressure, HDL, high-density lipoprotein; LDL, low-density lipoprotein; MI, myocardial infarction; SBP, systolic blood pressure. The trials contributed 1560 MACE-plus events, 857 MACE events, 409 strokes, 144 myocardial infarctions, 725 deaths and 348 cardiovascular deaths. Relative risk reductions with blood pressure-lowering drugs/more intense blood pressure-lowering regimen and statins are presented per trial in the online supplementary figure 4. The relative risk reduction with blood pressure-lowering drugs/more intense blood pressure-lowering regimen was not materially different in subgroups randomised to statins or placebo (figure 2). Likewise, the relative risk reduction with statins was not substantially different in subgroups randomised to blood pressure-lowering drugs/more intense blood pressure-lowering regimen or placebo/less intense regimen (figure 3). Figure 2 Effects of blood pressure-lowering treatment by subgroups of statin treatment. BPRx, blood pressure-lowering treatment; MACE, major adverse cardiovascular events; RR, risk ratio. Figure 3 Effects of statin treatment by subgroups of blood pressure-lowering treatment. BPRx, blood pressure-lowering treatment; MACE, major adverse cardiovascular events; RR, risk ratio.

Figure 2 Effects of blood pressure-lowering treatment by subgroups of statin treatment. BPRx, blood pressure-lowering treatment; MACE, major adverse cardiovascular events; RR, risk ratio. Figure 3 Effects of statin treatment by subgroups of blood pressure-lowering treatment. BPRx, blood pressure-lowering treatment; MACE, major adverse cardiovascular events; RR, risk ratio. Interaction analyses did not reveal any departures from either additivity or multiplicativity (table 2). The analysis cannot exclude the possibility of a small synergistic effect between the randomised treatments on major cardiovascular events, but it is not statistically significant given the current evidence base. Table 2 Departures from additivity and multiplicativity of effects

Interaction analyses did not reveal any departures from either additivity or multiplicativity (table 2). The analysis cannot exclude the possibility of a small synergistic effect between the randomised treatments on major cardiovascular events, but it is not statistically significant given the current evidence base. Table 2 Departures from additivity and multiplicativity of effects Outcome Departure from additivity Departure from multiplicativity Numbers Measure Estimate Lower Upper Factor OR Lower Upper MACE-plus N trials 3 RERI −0.04 −0.22 0.14 Statin 0.80 0.69 0.92 N persons 23 874 AP −0.06 −0.32 0.20 BPRx 0.92 0.80 1.05 N events 1560 S 1.14 0.61 2.12 Statin × BPRx 0.92 0.75 1.14 MACE N trials 5 RERI −0.09 −0.32 0.15 Statin 0.79 0.66 0.96 N persons 24 496 AP −0.14 −0.51 0.23 BPRx 0.93 0.77 1.11 N events 857 S 1.31 0.56 3.05 Statin × BPRx 0.86 0.65 1.13 Stroke N trials 5 RERI 0.02 −0.26 0.31 Statin 0.76 0.59 0.98 N persons 25 127 AP 0.05 −0.55 0.65 BPRx 0.70 0.54 0.90 N events 409 S 0.96 0.57 1.61 Statin × BPRx 0.90 0.60 1.35 Myocardial infarction N trials 3 RERI 0.18 −0.28 0.64 Statin 0.66 0.42 1.03 N persons 13 958 AP 0.31 −0.49 1.11 BPRx 0.74 0.48 1.15 N events 144 S 0.70 0.31 1.56 Statin × BPRx 1.19 0.61 2.32 Death N trials 5 RERI −0.04 −0.33 0.25 Statin 0.97 0.79 1.19 N persons 15 851 AP −0.04 −0.36 0.28 BPRx 0.97 0.79 1.20 N events 725 S 1.63 0.01 290.53 Statin × BPRx 0.96 0.71 1.30 Cardiovascular death N trials 5 RERI 0.13 −0.23 0.49 Statin 0.85 0.63 1.14 N persons 15 055 AP 0.16 −0.28 0.59 BPRx 0.85 0.63 1.14 N events 348 S 0.57 0.14 2.27 Statin × BPRx 1.15 0.75 1.76 AP, attributable proportion due to interaction; BPRx, blood pressure-lowering treatment; Lower, lower 95% CI limit; MACE, major adverse cardiovascular events; N, number; RERI, relative excess risk due to interaction; S, synergy index15; Upper, upper 95% CI limit.

events 348 S 0.57 0.14 2.27 Statin × BPRx 1.15 0.75 1.76 AP, attributable proportion due to interaction; BPRx, blood pressure-lowering treatment; Lower, lower 95% CI limit; MACE, major adverse cardiovascular events; N, number; RERI, relative excess risk due to interaction; S, synergy index15; Upper, upper 95% CI limit. Heterogeneity was low overall; hence, subgroup analyses were not called for. Results were largely driven by findings in two large trials (online supplementary figure 4). Discussion In this systematic review of randomised factorial trials, the joint relative effects of blood pressure-lowering drugs and statins on major cardiovascular events appeared multiplicative. The analysis cannot exclude the possibility of a slightly more than multiplicative effect between the treatments on major cardiovascular events, but it is unlikely that the combined effect is less than multiplicative.

s of blood pressure-lowering drugs and statins on major cardiovascular events appeared multiplicative. The analysis cannot exclude the possibility of a slightly more than multiplicative effect between the treatments on major cardiovascular events, but it is unlikely that the combined effect is less than multiplicative. There is some experimental evidence regarding pharmacological synergism between blood pressure-lowering drugs and statins, with a few studies proposing potentiating synergistic effects8 9 and one suggesting diminished combined effects.10 In humans, observations in the Anglo-Scandinavian Cardiac Outcomes Trial (ASCOT)20 supported synergistic effects, but the recent Heart Outcomes Prevention Evaluation-3 (HOPE-3) study did not.11 While differences between the two studies may be explained by the different drug regimens (candesartan/hydrochlorothiazide vs placebo and rosuvastatin vs placebo in HOPE-3 and amlodipine/perindopril vs atenolol/bendroflumethiazide and atorvastatin vs placebo in ASCOT) or different populations (higher risk sample in ASCOT with markedly higher blood pressure), they may also be due to chance (the full-factorial combinations underpowered compared with the primary comparisons in the original studies). With the latest addition to the evidence base, we assumed it large enough now to answer the question.

opulations (higher risk sample in ASCOT with markedly higher blood pressure), they may also be due to chance (the full-factorial combinations underpowered compared with the primary comparisons in the original studies). With the latest addition to the evidence base, we assumed it large enough now to answer the question. The question is of substantial clinical importance. With the increasing recognition of similar relative effects of statins and blood pressure-lowering drugs across the whole spectra of cholesterol and blood pressure,6 7 and consequent merits of risk-based treatment decisions, the clinician needs to know how to treat a patient identified to be at high risk. Any potentiating synergistic effects should steer the treatment choice towards introducing both statins and blood pressure-lowering drugs in high-risk people in primary prevention, irrespectively of cholesterol and blood pressure levels. Likewise, any potentiating synergistic effects should also argue for fixed-combination polypills.3 In contrast, a less than additive combined effect should likely focus more on treating single risk factors aggressively. The present study shows that combination treatment with statins and blood pressure-lowering drugs on average gives a combination of at least the anticipated relative risk reductions of each of the treatments.

st, a less than additive combined effect should likely focus more on treating single risk factors aggressively. The present study shows that combination treatment with statins and blood pressure-lowering drugs on average gives a combination of at least the anticipated relative risk reductions of each of the treatments. Weaknesses of this study include the low power for analyses of some outcomes and heterogeneous treatment regimens in the included studies, which on the other hand all are representatives of drug classes available to answer the research question. Further, heterogeneity in results was fairly low in spite of the different treatments. Strengths of this study include the large sample relevant for prevention situations in primary care, stringent systematic review methods including transparent analysis of several biases, and analyses of interaction on both the additive and multiplicative scales. It should be noted that definitions and analytical operationalisations of synergy are debated.22 Using analogies to pharmacological analyses of drug combinations, the analyses of the present study reflect effect-based rather than dose–effect-based strategies, and most closely reflect the Bliss Independence model. Synergy can be assessed as departure from additivity on an absolute risk scale, and as departure from multiplicativity on a relative risk scale. A strength of the analysis framework in this study is that incorporates both analyses.22

n dose–effect-based strategies, and most closely reflect the Bliss Independence model. Synergy can be assessed as departure from additivity on an absolute risk scale, and as departure from multiplicativity on a relative risk scale. A strength of the analysis framework in this study is that incorporates both analyses.22 In sum, the combined relative effects of blood pressure-lowering drugs and statins on major cardiovascular events were multiplicative in this systematic review of factorial trials. The possibility of a slightly more than multiplicative effect between the treatments could not be excluded and future factorial trials may add relevant evidence. Contributors: MW performed literature review, collected the data, interpreted the data and contributed to the writing of the article. GG performed literature review, collected the data and interpreted the data. JS designed the study, handled funding, supervision, statistical analysis, interpreted the data and wrote the article. JS had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Competing interests: JS reports being on the advisory board for Itrim, outside of the study. GG and MW report no competing interests. Provenance and peer review: Not commissioned; externally peer reviewed.

In 1904, Dr James Herrick evaluated a 20-year-old patient from Grenada who was studying in Chicago and suffered from anaemia and a multisystem illness. The patient was found to have ‘freakish’ elongated red cells that resembled a crescent or a sickle. Dr Herrick concluded that the red cells were not artefacts because the appearance of the cells was maintained regardless of how the smear slide was prepared. He followed the patient who had subsequently received care from other physicians until 1907 and questioned whether this was syphilis or a parasite from the tropics. Then in 1910, in a published case report, he concluded that this presentation strongly suggested a previously unrecognised change in the composition of the corpuscle itself.1 Sickle cell disease became a diagnosis thereafter. Case reports and case series have profoundly influenced the medical literature and continue to advance our knowledge in the present time. In 1985, the American Medical Association reprinted 51 papers from its journal that had significantly changed the science and practice of medicine over the past 150 years, and five of these papers were case reports.2 However, concerns about weak inferences and the high likelihood of bias associated with such reports have resulted in minimal attention being devoted to developing frameworks for approaching, appraising, synthesising and applying evidence derived from case reports/series. Nevertheless, such observations remain the bread and butter of learning by pattern recognition and integral to advancing medical knowledge.

reports have resulted in minimal attention being devoted to developing frameworks for approaching, appraising, synthesising and applying evidence derived from case reports/series. Nevertheless, such observations remain the bread and butter of learning by pattern recognition and integral to advancing medical knowledge. Guidance on how to write a case report is available (ie, a reporting guideline). The Case Report (CARE) guidelines3 were developed following a three-phase consensus process and provide a 13-item checklist that can assist researchers in publishing complete and meaningful exposition of medical information. This checklist encourages the explicit presentation of patient information, clinical findings, timeline, diagnostic assessment, therapeutic interventions, follow-up and outcomes.3 Yet, systematic reviewers appraising the evidence for decision-makers require tools to assess the methodological quality (risk of bias assessment) of this evidence. In this guide, we present a framework to evaluate the methodological quality of case reports/series and synthesise their results, which is particularly important when conducting a systematic review of a body of evidence that consists primarily of uncontrolled clinical observations. Definitions In the biomedical published literature, a case report is the description of the clinical course of one individual, which may include particular exposures, symptoms, signs, interventions or outcomes. A case report is the smallest publishable unit in the literature, whereas case series report aggregates individual cases in one publication.4

blished literature, a case report is the description of the clinical course of one individual, which may include particular exposures, symptoms, signs, interventions or outcomes. A case report is the smallest publishable unit in the literature, whereas case series report aggregates individual cases in one publication.4 The median number of patients in articles with ‘case series’ in the title was found to be seven (range 1–6432). The median (range) of the number of cases of articles with ‘case report’ as a publication type was four (1–178).5 Case reports/case series are usually retrospective although can occasionally be prospective, such as the Herrick’s first case report of sickle cell disease.1 Case reports/series can also define their subject by exposure or outcome (analogous to a cohort study and case–control study). Therefore, a specific number of patients, the temporal direction of follow-up or even the definition by case/exposure are not differentiating characteristics of case report/series. One unique feature, however, is that case report/series are uncontrolled (non-comparative) and have a relatively small number of individuals.

erefore, a specific number of patients, the temporal direction of follow-up or even the definition by case/exposure are not differentiating characteristics of case report/series. One unique feature, however, is that case report/series are uncontrolled (non-comparative) and have a relatively small number of individuals. If a case series is prospective, differentiating it from a single-arm uncontrolled cohort study becomes difficult. In one clinical practice guideline, it was proposed that studies without internal comparisons can be labelled as case series unless they explicitly report having a protocol before commencement of data collection, a definition of inclusion and exclusion criteria, a standardised follow-up and clear reporting of the number of excluded patients and those lost to follow-up.6

that studies without internal comparisons can be labelled as case series unless they explicitly report having a protocol before commencement of data collection, a definition of inclusion and exclusion criteria, a standardised follow-up and clear reporting of the number of excluded patients and those lost to follow-up.6 Evaluating methodological quality Pierson7 provided an approach to evaluate the validity of a case report based on five components: documentation, uniqueness, objectivity, interpretation and educational value, resulting in a score with a maximum of 10 (a score above 5 was suggested indicate a valid case report). This approach, however, was rarely used in subsequent work and seems to conflate methodological quality with other constructs. For case reports of adverse drug reactions, other systems classify an association as definite, probable, possible or doubtful based on leading questions.8 9 These questions are derived from the causality criteria that was established in 1965 by the English epidemiologist Bradford Hills.10 Lastly, we have adapted the Newcastle Ottawa scale11 for cohort and case–control studies by removing items that relate to comparability and adjustment (which are not relevant to non-comparative studies) and retained items that focused on selection, representativeness of cases and ascertainment of outcomes and exposure. This tool was applied in several published systematic reviews with good inter-rater agreement.12–16

items that relate to comparability and adjustment (which are not relevant to non-comparative studies) and retained items that focused on selection, representativeness of cases and ascertainment of outcomes and exposure. This tool was applied in several published systematic reviews with good inter-rater agreement.12–16 Proposed tool The previous criteria from Pierson,7 Bradford Hills10 and Newcastle Ottawa scale modifications11 converge into eight items that can be categorised into four domains: selection, ascertainment, causality and reporting. The eight items with leading explanatory questions are summarised in table 1. Table 1 Tool for evaluating the methodological quality of case reports and case series Domains Leading explanatory questions Selection 1. Does the patient(s) represent(s) the whole experience of the investigator (centre) or is the selection method unclear to the extent that other patients with similar presentation may not have been reported? Ascertainment 2. Was the exposure adequately ascertained? 3. Was the outcome adequately ascertained? Causality 4. Were other alternative causes that may explain the observation ruled out? 5. Was there a challenge/rechallenge phenomenon? 6. Was there a dose–response effect? 7. Was follow-up long enough for outcomes to occur? Reporting 8. Is the case(s) described with sufficient details to allow other investigators to replicate the research or to allow practitioners make inferences related to their own practice? Questions 4, 5 and 6 are mostly relevant to cases of adverse drug events.

ect? 7. Was follow-up long enough for outcomes to occur? Reporting 8. Is the case(s) described with sufficient details to allow other investigators to replicate the research or to allow practitioners make inferences related to their own practice? Questions 4, 5 and 6 are mostly relevant to cases of adverse drug events. For example, a study that explicitly describes all the cases who have presented to a medical centre over a certain period of time would satisfy the selection domain. In contrast, a study that reports on several individuals with unclear selection approach leaves the reader with uncertainty to whether this is the whole experience of the researchers and suggests possible selection bias. For the domain of ascertainment, self-report (of the exposure or the outcome) is less reliable than ascertainment using administrative and billing codes, which in turn is less reliable than clinical records. For the domain of causality, we would have stronger inference in a case report of an adverse drug reaction that has resolved with cessation of the drug and reoccurred after reintroduction of the drug. Lastly, for the domain of reporting, a case report that is described with sufficient details may allow readers to apply the evidence derived from the report in their practice. On the other hand, an inadequately reported case will likely be unhelpful in the course of clinical care.

rred after reintroduction of the drug. Lastly, for the domain of reporting, a case report that is described with sufficient details may allow readers to apply the evidence derived from the report in their practice. On the other hand, an inadequately reported case will likely be unhelpful in the course of clinical care. We suggest using this tool in systematic reviews of case reports/series. One option to summarise the results of this tool is to sum the scores of the eight binary responses into an aggregate score. A better option is not to use an aggregate score because numeric representation of methodological quality may not be appropriate when one or two questions are deemed most critical to the validity of a report (compared with other questions). Therefore, we suggest making an overall judgement about methodological quality based on the questions deemed most critical in the specific clinical scenario. Synthesis of case reports/series A single patient case report does not allow the estimation of an effect size and would only provide descriptive or narrative results. Case series of more than one patient may allow narrative or quantitative synthesis. Narrative synthesis A systematic review of the cases with the rare syndrome of lipodystrophy was able to suggest core and supportive clinical features and narratively summarised data on available treatment approaches.17 Another systematic review of 172 cases of the infrequently encountered glycogenic hepatopathy was able to characterise for the first time patterns of liver enzymes and hepatic injury in this disease.18

to suggest core and supportive clinical features and narratively summarised data on available treatment approaches.17 Another systematic review of 172 cases of the infrequently encountered glycogenic hepatopathy was able to characterise for the first time patterns of liver enzymes and hepatic injury in this disease.18 Quantitative synthesis Quantitative analysis of non-comparative series does not produce relative association measures such as ORs or relative risks but can provide estimates of prevalence or event rates in the form of a proportion (with associated precision). Proportions can be pooled using fixed or random effects models by means of the various available meta-analysis software. For example, a meta-analysis of case series of patients presenting with aortic transection showed that mortality was significantly lower in patients who underwent endovascular repair, followed by open repair and non-operative management (9%, 19% and 46%, respectively, P<0.01).19 A common challenge, however, occurs when proportions are too large or too small (close to 0 or to 1). In this situation, the variance of the proportion becomes very small leading to an inappropriately large weight in meta-analysis. One way to overcome this challenge is to transform prevalence to a variable that is not constrained to the 0–1 range and has approximately normal distribution, conduct the meta-analysis and then transform the estimate back to a proportion.20 This is done using logit transformation or using the Freeman-Tukey double arcsine transformation,21 with the latter being often preferred.20

to a variable that is not constrained to the 0–1 range and has approximately normal distribution, conduct the meta-analysis and then transform the estimate back to a proportion.20 This is done using logit transformation or using the Freeman-Tukey double arcsine transformation,21 with the latter being often preferred.20 Another type of quantitative analysis that may be utilised is regression. A meta-analysis of 47 published cases of hypocalcaemia and cardiac dysfunction used univariate linear regression analysis to demonstrate that both QT interval and left ventricular ejection fraction were significantly correlated with corrected total serum calcium level.22 Meta-regression, which is a regression in which the unit of analysis is a study, not a patient, can also be used to synthesise case series and control for study-level confounders. A meta-regression analysis of uncontrolled series of patients with uveal melanoma treated with proton beam therapy has shown that this treatment was associated with better outcomes than brachytherapy.23 It is very important, however, to recognise that meta-regression results can be severely affected by ecological bias. From evidence to decision Several authors have described various important reasons to publish case reports/series (table 2).7 24 25 Table 2 Role of case reports/series in the medical literature Roles Examples Describe a new phenotype or genotype of disease The first case report of sickle cell disease.1 Recognise a known or common manifestation of a rare disease Liver cirrhosis as a result of Sitosterolaemia.33

From evidence to decision Several authors have described various important reasons to publish case reports/series (table 2).7 24 25 Table 2 Role of case reports/series in the medical literature Roles Examples Describe a new phenotype or genotype of disease The first case report of sickle cell disease.1 Recognise a known or common manifestation of a rare disease Liver cirrhosis as a result of Sitosterolaemia.33 Recognise a rare manifestation of a known or common disease Secretory diarrhoea and hypokalaemia in colonic pseudo-obstruction.15 Describe a new pathogen (microbe, virus or environmental exposure) Discovery of AIDS was an observation of a patient with immunodeficiency-related diseases who otherwise had no reason to be immunodeficient.34 Describe unknown adverse effect of an existing drug Reye syndrome and aspirin in children.35 Thalidomide and malformation of the limbs in pregnant women.36 Describe a novel treatment for a known condition Colchicine for the treatment of familial Mediterranean fever.37 Elucidate mechanisms of disease Functional imaging of the brain during auditory hallucinations.38

Describe unknown adverse effect of an existing drug Reye syndrome and aspirin in children.35 Thalidomide and malformation of the limbs in pregnant women.36 Describe a novel treatment for a known condition Colchicine for the treatment of familial Mediterranean fever.37 Elucidate mechanisms of disease Functional imaging of the brain during auditory hallucinations.38 To remind or educate Case presentations in clinicopathological conferences for postgraduate education. Quality improvement ‘Lesson of the week’ published in the British Medical Journal (do not make the same mistake as I did). It is paramount to recognise that a systematic review and meta-analysis of case reports/series should not be placed at the top of the hierarchy in a pyramid that depicts validity.26 The certainty of evidence derived from a meta-analysis is contingent on the design of included studies, their risk of bias, as well as other factors such as imprecision, indirectness, inconsistency and likelihood of publication bias.27 Commonly, certainty in evidence derived from case series/reports will be very low. Nevertheless, inferences from such reports can be used for decision-making. In the example of case series of aortic transection showing lower mortality with endovascular repair, a guideline recommendation was made stating ‘We suggest that endovascular repair be performed preferentially over open surgical repair or non-operative management’. This was graded as a weak recommendation based on low certainty evidence.28 The strength of this recommendation acknowledged that the recommendation might not universally apply to everyone and that variability in decision-making was expected. The certainty in evidence rating of this recommendation implied that future research would likely yield different results that may change the recommendation.28

rength of this recommendation acknowledged that the recommendation might not universally apply to everyone and that variability in decision-making was expected. The certainty in evidence rating of this recommendation implied that future research would likely yield different results that may change the recommendation.28 The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach clearly separates the certainty of evidence from the strength of recommendation. This separation allows decision-making based on lower levels of evidence. For example, despite low certainty evidence (derived from case series) regarding the association between aspirin and Reye’s syndrome in febrile children, a strong recommendation for using acetaminophen over aspirin is possible.29 GRADE literature also describes five paradigmatic situations in which a strong recommendation can be made based on low quality evidence.30 One of which is when the condition is life threatening. An example of which would be using hyperbaric oxygen therapy for purpura fulminans, which is only based on case reports.31

29 GRADE literature also describes five paradigmatic situations in which a strong recommendation can be made based on low quality evidence.30 One of which is when the condition is life threatening. An example of which would be using hyperbaric oxygen therapy for purpura fulminans, which is only based on case reports.31 Discussion Guideline developers and decision-makers often struggle when dealing with case reports/case series. On occasions, they ignore such evidence and focus the scope of guidelines on areas with higher quality evidence. Sometimes they label recommendations based on case reports as expert opinion.32 We propose an approach to evaluate the methodological quality of case reports/series based on the domains of selection, ascertainment, causality and reporting and provide signalling questions to aid evidence-based practitioners and systematic reviewers in their assessment. We suggest the incorporation of case reports/series in decision-making based on the GRADE approach when no other higher level of evidence is available. In this guide, we have made the case for publishing case reports/series and proposed synthesis of their results in systematic reviews to facilitate using this evidence in decision-making. We have proposed a tool that can be used to evaluate the methodological quality in systematic reviews that examine case reports and case series. Contributors: MHM drafted the paper and all coauthors critically revised the manuscript. All the authors contributed to conceive the idea and approved the final submitted version. Competing interests: None declared.

In this guide, we have made the case for publishing case reports/series and proposed synthesis of their results in systematic reviews to facilitate using this evidence in decision-making. We have proposed a tool that can be used to evaluate the methodological quality in systematic reviews that examine case reports and case series. Contributors: MHM drafted the paper and all coauthors critically revised the manuscript. All the authors contributed to conceive the idea and approved the final submitted version. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Tiny effect is among the issues of our time ‘that science has taken a turn towards darkness.’ 1 In the Catheter Ablation for Atrial Fibrillation with Heart Failure (CASTLE-AF) trial,2 the primary end point was death or hospitalisation for worsening heart failure. The authors concluded that: ‘after a median follow-up of 37.8 months, the primary composite end point occurred in significantly fewer patients in the ablation group than in the medical therapy group (51 patients (28.5%) vs 82 patients (44.6%); difference 16.1% (CI) 5%–25%. HR 0.62; 95% CI 0.43 to 0.87; p=0.007).’ That was statistically significant. Although the inferential statistical analysis provides information about the reliability of the result, the p value conveys little information on the significance of the clinically observed effect. This problem is solved by the concept of effect size, ‘which was developed to allow clinically meaningful comparisons of efficacy between treatment trials. Without using this concept, comparing two treatment trials can be difficult. as the name suggests, an effect-size estimate can place an easily interpretable value on the direction and magnitude of an effect of a treatment, a difference between two treatment groups, or any other numerical comparison or contrast.’3 The effect size measurement takes two factors into account: the difference between the mean values of the measures for the two groups and the variance. For convention, this is applied by the Cohen’s d principle. This means that the small effect size <0.2, medium 0.2–0.8 and larger >0.8.

erical comparison or contrast.’3 The effect size measurement takes two factors into account: the difference between the mean values of the measures for the two groups and the variance. For convention, this is applied by the Cohen’s d principle. This means that the small effect size <0.2, medium 0.2–0.8 and larger >0.8. For the CASTLE-AF trial, the effect size calculation4 was ф=2 arcsine √p. Thus, ф0.285=2 arcsine √0.285=1.32, ф0.446=2 arcsine √0.446=1.60 and h=ф0.285–ф0.446=0.28!! A tiny effect. What about the tiny effect? The tyranny of the statistical significance fills the literature with trivial and incorrect findings. Horton remarked recently, ‘the case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance.’1 In this context, a tiny effect is an effect without a clinical impact for management that adds no value to the patient care. In this way, although the study was able to find a more statistically significant difference between ablation group compared with the medical therapy group in the composite end point, the magnitude of the difference was tiny.5 We can conclude that this trial found a real clinical irrelevance with statistical significance. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; internally peer reviewed.

Global warming is the single biggest threat to the future of the planet according to World Health Organisation.1 Climate change has contributed to higher rates of weather-related disasters over the last 10 years, including exposure to heatwaves.2 An increased prevalence of chronic kidney disease (CKD)-related mortality in young people who do not have the usual risk factors for the same has raised concern. The problem is particularly concentrated in areas such as Central America, India, Sri Lanka and Egypt,3 where young working men are exposed to excessive heat on a regular basis. This is sometimes dubbed Mesoamerican nephropathy or CKD of unknown origin (CKDu). Working indoors, which affects more female workers, including kitchen work with ovens, smelters and steel manufacturing are also implicated.4 Approximately 40% of the world’s population inhabit areas subject to excessive heat throughout the year. Increasing prevalence of early-onset CKD has been noted worldwide, with El Salvador, in particular, recording the highest mortality rates.5

hen work with ovens, smelters and steel manufacturing are also implicated.4 Approximately 40% of the world’s population inhabit areas subject to excessive heat throughout the year. Increasing prevalence of early-onset CKD has been noted worldwide, with El Salvador, in particular, recording the highest mortality rates.5 Meanwhile, CKD is emerging as a leading cause of mortality particularly in patients suffering from hypertension, diabetes mellitus and obesity. Worldwide, kidney disease is now reported as the 12th most common cause of death.6 While debate persists regarding the diagnosis and actual prevalence of CKD in those of advanced age, the majority of these patients are derived from the older age groups.7 Moreover, age-related reductions in renal reserve, plasma renin and aldosterone responses, glomerular filtration rate, as well as a predisposition to hyperkalaemia and haemoconcentration with hypernatraemia render older patients particularly susceptible to kidney injury and death when exposed to excessive heat.8 9 Other particularly vulnerable demographic subgroups include young children, disabled, isolated individuals and those who are socioeconomically disadvantaged.10

kalaemia and haemoconcentration with hypernatraemia render older patients particularly susceptible to kidney injury and death when exposed to excessive heat.8 9 Other particularly vulnerable demographic subgroups include young children, disabled, isolated individuals and those who are socioeconomically disadvantaged.10 The possible mechanisms of the escalation in heat-related kidney failure-associated mortality in younger patients have not been fully elucidated. First, in a hot environment, increased insensible loss of body water and salt occurs. With prolonged heat exposure this leads to substantial fluid deficit which, if not replaced, may result in vasoconstriction and associated kidney injury.11 Moreover, with exertion in a state of chronic dehydration, subclinical rhabdomyolysis can occur due to chronic low-grade muscle injury, which can in turn also contribute to hyperuricaemia.12 13 This has been postulated to lead to glomerular hypertension and renal tubular injury. A pilot study performed on sugar cane workers in El Salvador demonstrated higher mean end-of-work serum levels of uric acid versus morning levels (428 μmol/L vs. 387 μmol/L), with the vast majority demonstrating concurrent glomerular filtration rates <60 mL/min.13 Resultant activation of the aldose reductase pathway within the kidney leads to increased levels of glucose and fructose. Fructose is then metabolised by fructokinase in the proximal tubule, which can lead to inflammation and fibrosis of the renal parenchyma as evidenced by Roncal-Jimenez et al., who demonstrated that mice without fructokinase were spared renal injury despite similar exposure to dehydration.14

els of glucose and fructose. Fructose is then metabolised by fructokinase in the proximal tubule, which can lead to inflammation and fibrosis of the renal parenchyma as evidenced by Roncal-Jimenez et al., who demonstrated that mice without fructokinase were spared renal injury despite similar exposure to dehydration.14 The aetiology of this enhanced susceptibility to CKD appears to be multifactorial.15 For example, water ‘hardness’ may play a role. With high levels of calcium, magnesium and other metal cations, endemic areas with harder water in Sri Lanka correlate positively with those areas where CKDu is most prevalent.16 It is suggested that these cations can combine with substances in herbicides such as glyphosate. Glyphosate is an aminophosphonic acid analogue of the natural amino acid, glycine, which can be absorbed by inhalation, transdermally, or ingested either in food sprayed by herbicide or contaminated water supplies.16 17 A study of 500 species of juvenile Clarias gariepinus fish which were fed with commercial pellets containing glyphosate demonstrated tissue evidence of renal necrosis and degenerated kidney tubules versus control fish without toxicant exposure.18

either in food sprayed by herbicide or contaminated water supplies.16 17 A study of 500 species of juvenile Clarias gariepinus fish which were fed with commercial pellets containing glyphosate demonstrated tissue evidence of renal necrosis and degenerated kidney tubules versus control fish without toxicant exposure.18 A recently published systematic review and meta-analysis of epidemiological studies on CKDu suggested positive associations with male gender, family history of CKD, water intake and lowland altitude.19 Meanwhile, in a cross-sectional study of sugar cane workers, Kupferman et al. reported that reduction of kidney function occurred commonly and almost half of those affected had CKD after 1 year.20 Moreover, in a longitudinal study examining rates of decline of kidney function in high-risk populations for CKDu, rapid decline occurred more commonly in men and was associated with outdoor agricultural work and lack of shade during work breaks.21 It is difficult to identify consistent risk factors isolated to affected areas but poorer economic areas generally do not provide equal access to healthcare, clean water, toilet facilities (with particularly female workers drinking less to avoid the need to urinate while working), and appropriate screening or diagnostics.4

s difficult to identify consistent risk factors isolated to affected areas but poorer economic areas generally do not provide equal access to healthcare, clean water, toilet facilities (with particularly female workers drinking less to avoid the need to urinate while working), and appropriate screening or diagnostics.4 Increased exposure of new populations to heat stress appears inevitable beyond the borders of currently identified countries, thus further enhancing the numbers at risk of CKD. Furthermore, in the presence of traditional risk factors for CKD, such as hypertension and diabetes mellitus, with associated increased rates of ischaemic heart disease, it is reasonable to assume that the increase in heat stress will result in further increases in mortality. Indeed, increased rates of heat-related myocardial infarction have already been reported in a cross-sectional study.22

ypertension and diabetes mellitus, with associated increased rates of ischaemic heart disease, it is reasonable to assume that the increase in heat stress will result in further increases in mortality. Indeed, increased rates of heat-related myocardial infarction have already been reported in a cross-sectional study.22 Prevention is the best approach to addressing heat-related CKD-associated mortality. In older patients, this includes anticipatory implementation of simple measures such as maintenance of adequate fluid intake combined with minimising insensible loss using tepid sponging and relocating affected people to a cooler environment. Adjusting doses of medications such as diuretics, monitoring electrolytes and renal function, followed by early intervention to optimise electrolyte abnormalities will minimise progression to acute kidney injury. Regional heat plans and robust social programmes are helpful.9 In previous heatwaves, a report indicates that those people who met in public places had less morbidity and mortality than those who retreated indoors.23

ly intervention to optimise electrolyte abnormalities will minimise progression to acute kidney injury. Regional heat plans and robust social programmes are helpful.9 In previous heatwaves, a report indicates that those people who met in public places had less morbidity and mortality than those who retreated indoors.23 Recent evidence indicates that implementing similar basic interventions in younger susceptible individuals, such as provision of portable water reservoirs, scheduled rest periods and mobile shaded tents to sugar cane workers, resulted in improvements in markers of dehydration and increased GFR.24 25 However these findings derived from small studies, with many methodological problems and thus lacks external validity. Guidelines recommend replacing fluid frequently when working in conditions that may lead to heat stress, for example, 250 mL water every 20 min with or without weight monitoring.26 Other interventions such as self-monitoring of early morning body weight, urine colour and thirst perception lack evidence. Education of workers regarding heat illness risks and preassessment of workers’ serum creatinine levels may highlight at-risk individuals, but workers may avoid such testing for fear of resultant loss of income. Employers need education to emphasise that time lost with regular hydration breaks should result in increased productivity, less illnesses, workplace accidents and absenteeism.

of workers’ serum creatinine levels may highlight at-risk individuals, but workers may avoid such testing for fear of resultant loss of income. Employers need education to emphasise that time lost with regular hydration breaks should result in increased productivity, less illnesses, workplace accidents and absenteeism. While the mechanisms of heat-related kidney disease may differ between young and older individuals, simple anticipatory interventions could ameliorate deleterious renal consequences for all age groups. Contributors: TÓF: first author, wrote the first, subsequent and final drafts, as well as background reading/review of prior research. AF: second author, reviewed all drafts and edits, made edits and further review of the subject. ECM: third author, formed the idea and theory, reviewed and edited the drafts, further reading into the subject. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction To highlight and advance clinical effectiveness and evidence-based practice (EBP) agendas, the Institute of Medicine set a goal that by 2020, 90% of clinical decisions will be supported by accurate, timely and up-to-date clinical information and will reflect the best available evidence to achieve the best patient outcomes.1 To ensure that future healthcare users can be assured of receiving such care, healthcare professions must effectively incorporate the necessary knowledge, skills and attitudes required for EBP into education programmes.

ical information and will reflect the best available evidence to achieve the best patient outcomes.1 To ensure that future healthcare users can be assured of receiving such care, healthcare professions must effectively incorporate the necessary knowledge, skills and attitudes required for EBP into education programmes. The promotion of EBP requires a healthcare infrastructure committed to supporting organisations to deliver EBP and an education system efficient in supporting healthcare professionals in acquiring EBP competencies.2 To this end, healthcare education programmes must effectively implement curricula that target these competencies.3 To facilitate this, the Sicily consensus statement on EBP provides a description of core knowledge and skills required to practise in an evidence-based manner and a curriculum that outlines the minimum requirements for educating health professionals in EBP.2 Initiatives such as the European Union Evidence-Based Medicine project4 and EBP teaching programmes for educators facilitated by Oxford (Centre for Evidence-Based Medicine) and McMaster Universities provide support in advancing the EBP agenda within healthcare education. Over the past two decades, more than 300 articles have been published on teaching evidence-based medicine alone and in excess of 30 experiments have been conducted to measure its effects.5 Recent reviews3 6 evaluating the adoption of evidence-based recommendations for teaching EBP however point to poor uptake of existing resources available to guide EBP education.

les have been published on teaching evidence-based medicine alone and in excess of 30 experiments have been conducted to measure its effects.5 Recent reviews3 6 evaluating the adoption of evidence-based recommendations for teaching EBP however point to poor uptake of existing resources available to guide EBP education. The application of EBP continues to be observed irregularly at the point of patient contact.2 5 7 The effective development and implementation of professional education to facilitate EBP remains a major and immediate challenge.2 3 6 8 Momentum for continued improvement in EBP education in the form of investigations which can provide direction and structure to developments in this field is recommended.6 As part of a larger national project looking at current practice and provision of EBP education across healthcare professions at undergraduate, postgraduate and continuing professional development programme levels, we sought key perspectives from international EBP education experts on the provision of EBP education for healthcare professionals. The two other components of this study, namely a rapid review synthesis of EBP literature and a descriptive, cross-sectional, national, online survey relating to the current provision and practice of EBP education to healthcare professionals at third-level institutions and professional training/regulatory bodies in Ireland, will be described in later publications.

a rapid review synthesis of EBP literature and a descriptive, cross-sectional, national, online survey relating to the current provision and practice of EBP education to healthcare professionals at third-level institutions and professional training/regulatory bodies in Ireland, will be described in later publications. Methods EBP expert interviews were conducted to ascertain current and nuanced information on EBP education from an international perspective. Experts from the UK, Canada, New Zealand and Australia were invited by email to participate based on their contribution to peer-reviewed literature on the subject area and recognised innovation in EBP education. Over a 2-month period, individual ‘Skype’ interviews were conducted and recorded. The interview guide (online supplementary appendix A) focused on current practice and provision of EBP education with specific attention given to EBP curricula, core EBP competencies, assessment methods, teaching initiatives and key challenges to EBP education within respective countries. Qualitative content analysis techniques as advised by Bogner et al 9 for examination of expert interviews were used. Specifically, a six-step process was applied, namely transcription, reading through/paraphrasing, coding, thematic comparison, sociological conceptualisation and theoretical generalisation. To ensure trustworthiness, a number of practices were undertaken, including explicit description of the methods undertaken, participant profile, extensive use of interview transcripts by way of representative quotations, peer review (PL-W) of the data analysis process and invited interviewees to feedback in relation to the overall findings.

ess, a number of practices were undertaken, including explicit description of the methods undertaken, participant profile, extensive use of interview transcripts by way of representative quotations, peer review (PL-W) of the data analysis process and invited interviewees to feedback in relation to the overall findings. 10.1136/bmjebm-2018-111019.supp1Supplementary file 1 Results Five EBP experts participated in the interviews (table 1). All experts waived their right to anonymity. Table 1 EBP education expert profile EBP expert Title Affiliation Country Professor Leanne Togher Professor of Communication Disorders Following Traumatic Brain Injury Faculty of Health Sciences, University of Sydney Australia Professor Gordon Guyatt Distinguished Professor Department of Health Research Methods, Evidence, and Impact, McMaster University Canada Professor Rodney Jackson Professor of Epidemiology School of Population Health, Faculty of Medical and Health Sciences, University of Auckland New Zealand Professor Bruce Arroll Professor of General Practice General Practice and Primary Healthcare, Faculty of Medical and Health Sciences, University of Auckland New Zealand Professor Carl Heneghan Professor of Evidence-Based Medicine and Director, Centre for Evidence-Based Medicine Department of Primary Care Health Sciences, University of Oxford UK EBP, evidence-based practice.

eral Practice and Primary Healthcare, Faculty of Medical and Health Sciences, University of Auckland New Zealand Professor Carl Heneghan Professor of Evidence-Based Medicine and Director, Centre for Evidence-Based Medicine Department of Primary Care Health Sciences, University of Oxford UK EBP, evidence-based practice. Three main categories emerged, namely (1) ‘EBP curriculum considerations’, (2) ‘Teaching EBP’ and (3) ‘Stakeholder engagement in EBP education’. These categories informed the overarching theme of ‘Improving healthcare through enhanced teaching and application of EBP’ (figure 1). Figure 1 Summary of data analysis findings from evidence-based practice (EBP) expert interviews—theme, categories and subcategories.

Three main categories emerged, namely (1) ‘EBP curriculum considerations’, (2) ‘Teaching EBP’ and (3) ‘Stakeholder engagement in EBP education’. These categories informed the overarching theme of ‘Improving healthcare through enhanced teaching and application of EBP’ (figure 1). Figure 1 Summary of data analysis findings from evidence-based practice (EBP) expert interviews—theme, categories and subcategories. EBP curriculum considerations Definitive advice in relation to curriculum considerations was provided with a clear emphasis on the need for EBP principles to be integrated throughout all elements of healthcare professions curricula. Educators, regardless of teaching setting, need to be able to ‘draw out evidence-based components’ from any and all aspects of curriculum content, including its incorporation into assessments and examinations. Integration of EBP into clinical curricula in particular was considered essential to successful learning and practice outcomes. If students perceive a dichotomy between EBP and actual clinical care, then “never the twain shall meet” (GG) requiring integration in such a way that it is “seen as part of the basics of optimal clinical care” (GG). Situating EBP as a core element within the professional curriculum and linking it to professional accreditation processes places further emphasis on the necessity of teaching EBP:

“never the twain shall meet” (GG) requiring integration in such a way that it is “seen as part of the basics of optimal clinical care” (GG). Situating EBP as a core element within the professional curriculum and linking it to professional accreditation processes places further emphasis on the necessity of teaching EBP: …it is also core in residency programmes. So every residency programme has a curriculum on evidence-based practice where again, the residency programmes are accredited…They have to show that they’re teaching evidence-based practice. (GG) In terms of the focus of curriculum content, all experts emphasised the oft-cited steps of asking questions, acquiring, appraising and applying evidence to patient care decisions. With regard to identifying and retrieving information, the following in particular was noted: …the key competencies would be to identify evidence-based sources of information, and one of the key things is there should be no expectation that clinicians are going to go to primary research and evaluate primary research. That is simply not a realistic expectation. In teaching it…they have to be able to identify the pre-processed sources and they have to be able to understand the evidence and they have to be able to use it… (GG) In addition to attaining proficiency in the fundamental EBP steps, developing competence in communicating evidence to others, including the patient, and facilitating shared decision-making were also highlighted: …So our ability to communicate risks, benefits, understand uncertainty is so poor…that’s a key area we could improve… (CH)

In addition to attaining proficiency in the fundamental EBP steps, developing competence in communicating evidence to others, including the patient, and facilitating shared decision-making were also highlighted: …So our ability to communicate risks, benefits, understand uncertainty is so poor…that’s a key area we could improve… (CH) …and a big emphasis [is needed] on the applicability of that information on patient care, how do you use and share the decision making, which is becoming a bigger and bigger deal. (GG) It was suggested that these EBP ‘basics’ can be taught “from the start in very similar ways” (GG), regardless of whether the student is at an undergraduate or postgraduate level. The concept of ‘developmental milestones’ was raised by one expert. This related to different levels of expectations in learning and assessing EBP skills and knowledge throughout a programme of study with an incremental approach to teaching and learning advocated over a course of study: …in terms of developmental milestones. So for the novice…it’s really trying to get them aware of what the structure of evidence-based practice is and knowing what the process of asking a question and the PICO process and learning about that…in their final year…they’re asked to do critically appraised topics and relate it to clinical cases…It’s a developmental process… (LT)

e…it’s really trying to get them aware of what the structure of evidence-based practice is and knowing what the process of asking a question and the PICO process and learning about that…in their final year…they’re asked to do critically appraised topics and relate it to clinical cases…It’s a developmental process… (LT) Teaching EBP Adoption of effective strategies and practical methods to realise successful student learning and understanding was emphasised. Of particular note was the grounding of teaching strategy and associated methods from a clinically relevant perspective with student exposure to EBP facilitated in a dynamic and interesting manner. The use of patient examples and clinical scenarios was repeatedly expressed as one of the most effective instructional practices: …ultimately trying to get people to teach in a way where they go, “Look, this is really relevant, dynamic and interesting"…so we teach them in loads of different ways…you’re teaching and feeding the ideas as opposed to “"Here’s a definitive course in this way”. (CH) …It’s pretty obscure stuff, but then I get them to do three examples…when they have done that they have pretty well got their heads around it…I build them lots of practical examples…clinical examples otherwise they think it’s all didactic garbage… (BA) EBP role models were emphasised as being integral to demonstrating the application of EBP in clinical decision-making and facilitating the contextualisation of EBP within a specific setting/organisation.

…It’s pretty obscure stuff, but then I get them to do three examples…when they have done that they have pretty well got their heads around it…I build them lots of practical examples…clinical examples otherwise they think it’s all didactic garbage… (BA) EBP role models were emphasised as being integral to demonstrating the application of EBP in clinical decision-making and facilitating the contextualisation of EBP within a specific setting/organisation. …where we’ve seen success is where organisations have said, “There’s going to be two or three people who are going to be the champions and lead where we’re going”…the issue about evidence, it’s complex, it needs to be contextualised and it’s different for each setting… (CH) It was further suggested that these healthcare professionals have the ‘X-factor’ required of EBP. The acquisition of such expertise which enables a practitioner to integrate individual EBP components culminating in evidence-based decisions was proposed as a definitive target for all healthcare professionals. And we call it the X factor…the idea is that the clinician who has the X factor is the good clinician. It’s actually integrating the evidence, the patient values, the patient’s pathophysiology, etc. It could be behavioural issues, systems issues…Those are the four quadrants and the clinical expertise is about integrating those together…You’re not actually adding clinical expertise. It seems to me that the clinical expertise is the ability to integrate those four quadrants. (RJ)

the patient’s pathophysiology, etc. It could be behavioural issues, systems issues…Those are the four quadrants and the clinical expertise is about integrating those together…You’re not actually adding clinical expertise. It seems to me that the clinical expertise is the ability to integrate those four quadrants. (RJ) The provision of training for educators to aid the further development of skills and use of resources necessary for effective EBP teaching was recommended: …so we choose the option to train people as really good teachers and give them really high level skills so that they can then seed it across their organisation… (CH) Attaining a critical mass of people who are ‘trained’ was also deemed important in making a sustained change: …and it requires getting the teachers trained and getting enough of them. You don’t need everybody to be doing it to make an impression, but you need enough of them really doing it. (GG) Stakeholder engagement in EBP education Engagement of national policy makers, healthcare professionals and patients with EBP was considered to have significant potential to advance its teaching and application in clinical care. The lack of a coherent government and national policy to EBP teaching was cited as a barrier to the implementation of the EBP agenda resulting in a somewhat ‘ad-hoc’ approach, dependent on individual educational or research institutions:

ve significant potential to advance its teaching and application in clinical care. The lack of a coherent government and national policy to EBP teaching was cited as a barrier to the implementation of the EBP agenda resulting in a somewhat ‘ad-hoc’ approach, dependent on individual educational or research institutions: …there’s no cohesive or coherent policy that exists…It’s not been a consistent approach. What we’ve tended to see is that people have started going around particular initiatives…but there’s never been any coordinated approach even from a college perspective, to say we are about improving the uptake and use of evidence in practice and/or generating evidence in practice. And so largely, it’s been left to research institutions… (CH) To further ingrain EBP within healthcare professional practice, it was suggested that EBP processes, whether related to developing, disseminating or implementing evidence, be embedded in a more structured way into everyday clinical care to promote active and consistent engagement with EBP on a continuous basis: …we think it should be embedded into care…we’ve got to have people being active in developing, disseminating and implementing evidence…developing can come in a number of formats. It can be an audit. It can be about a practice improvement. It can be about doing some aspect like a systematic review, but it’s very clearly close to healthcare. (CH) Enabling patients to engage with evidence with a view to informing healthcare professional/patient interactions and care decisions was also advocated:

…we think it should be embedded into care…we’ve got to have people being active in developing, disseminating and implementing evidence…developing can come in a number of formats. It can be an audit. It can be about a practice improvement. It can be about doing some aspect like a systematic review, but it’s very clearly close to healthcare. (CH) Enabling patients to engage with evidence with a view to informing healthcare professional/patient interactions and care decisions was also advocated: …I think we really need to put some energy into…this whole idea of patient-driven care, patient-led care and putting some of these tools in the hands of the consumers so that they’re enabled to be able to ask the right questions and to go into an interaction with some background knowledge about what treatments they should be expecting. (LT) If patients are considered as recipients of EBP rather than key stakeholders, the premise of shared decision-making for care cannot be achieved.

…I think we really need to put some energy into…this whole idea of patient-driven care, patient-led care and putting some of these tools in the hands of the consumers so that they’re enabled to be able to ask the right questions and to go into an interaction with some background knowledge about what treatments they should be expecting. (LT) If patients are considered as recipients of EBP rather than key stakeholders, the premise of shared decision-making for care cannot be achieved. The implementation of a successful EBP education is necessary so that learners not only understand the importance of EBP and be competent in the fundamental steps, but it ultimately serves to influence behaviour in terms of decision-making, through application of EBP in their professional practice. In essence, it serves the function of developing practitioners who value EBP and have the knowledge and skills to implement such practice. The ultimate goal of this agenda is to enhance the delivery of healthcare for improved patient outcomes. The overarching theme of ‘Improving healthcare through enhanced teaching and application of EBP’ represents the focus and purpose of the effort required to optimally structure healthcare professional (HCP) curricula, promote effective EBP teaching and learning strategies, and engage with key stakeholders for the overall advancement of EBP education as noted: …we think that everyone in training should be in the game of improving healthcare…It’s not just saying I want to do some evidence-based practice…it’s ultimately about…improving healthcare. (CH)

The implementation of a successful EBP education is necessary so that learners not only understand the importance of EBP and be competent in the fundamental steps, but it ultimately serves to influence behaviour in terms of decision-making, through application of EBP in their professional practice. In essence, it serves the function of developing practitioners who value EBP and have the knowledge and skills to implement such practice. The ultimate goal of this agenda is to enhance the delivery of healthcare for improved patient outcomes. The overarching theme of ‘Improving healthcare through enhanced teaching and application of EBP’ represents the focus and purpose of the effort required to optimally structure healthcare professional (HCP) curricula, promote effective EBP teaching and learning strategies, and engage with key stakeholders for the overall advancement of EBP education as noted: …we think that everyone in training should be in the game of improving healthcare…It’s not just saying I want to do some evidence-based practice…it’s ultimately about…improving healthcare. (CH) Discussion and recommendations Education programmes and associated curricula act as a key medium for shaping healthcare professional knowledge, skills and attitudes, and therefore play an essential role in determining the quality of care provided.10 Unequivocal recommendations were made in relation to the pervasive integration of EBP throughout the academic and clinical curricula. Such integration is facilitated by the explicit inclusion of EBP as a core competency within professional standards and requirements in addition to accreditation processes.11

provided.10 Unequivocal recommendations were made in relation to the pervasive integration of EBP throughout the academic and clinical curricula. Such integration is facilitated by the explicit inclusion of EBP as a core competency within professional standards and requirements in addition to accreditation processes.11 Further emphasis on communication skills was also noted as being key to enhancing EBP competency, particularly in relation to realising shared decision-making between patients and healthcare practitioners in making evidence-based decisions. A systematic review by Galbraith et al,12 which examined a ‘real-world’ approach to evidence-based medicine in general practice, corroborates this recommendation by calling for further attention to be given to communication skills of healthcare practitioners within the context of being an evidence-based practitioner. This resonates with recommendations by Gorgon et al 13 for the need to expose students to the intricacies of ‘real world’ contexts in which EBP is applied.

ndation by calling for further attention to be given to communication skills of healthcare practitioners within the context of being an evidence-based practitioner. This resonates with recommendations by Gorgon et al 13 for the need to expose students to the intricacies of ‘real world’ contexts in which EBP is applied. Experts in EBP, together with trends throughout empirical research and recognised educational theory repeatedly, make a number of recommendations for enhancing EBP teaching and learning strategies. These include (1) clinical integration of EBP teaching and learning, (2) a conscious effort on behalf of educators to embed EBP throughout all elements of healthcare professional programmes, (3) the use of multifaceted, dynamic teaching and assessment strategies which are context-specific and relevant to the individual learner/professional cohort, and (4) ‘scaffolding’ of learning.

ing, (2) a conscious effort on behalf of educators to embed EBP throughout all elements of healthcare professional programmes, (3) the use of multifaceted, dynamic teaching and assessment strategies which are context-specific and relevant to the individual learner/professional cohort, and (4) ‘scaffolding’ of learning. At a practical level this requires a more concerted effort to move away from a predominant reliance on stand-alone didactic teaching towards clinically integrative and interactive teaching.10 14–17 An example provided by one of the EBP experts represents such integrated teaching and experiential learning through the performance of GATE/CATs (Graphic Appraisal Tool for Epidemiological studies/Critically Appraised Topics) while on clinical rotation, with assessment conducted by a clinician in practice. Such an activity fulfils the criteria of being reflective of practice, facilitating the identification of gaps between current and desired levels of competence, identifying solutions for clinical issues and allowing re-evaluation and opportunity for reflection of decisions made with a practitioner. This level of interactivity facilitates ‘deeper’ learning, which is essential for knowledge transfer.8 Such practices are also essential to bridge the gap between academic and clinical worlds, enabling students to experience ‘real’ translation of EBP in the clinical context.6 ‘Scaffolding’ of learning, whereby EBP concepts and their application increase in complexity and are reinforced throughout a programme, was also highlighted as an essential instructional approach which is in keeping with recent literature specific both to EBP education and from a broader curriculum development perspective.3 6 18 19

of learning, whereby EBP concepts and their application increase in complexity and are reinforced throughout a programme, was also highlighted as an essential instructional approach which is in keeping with recent literature specific both to EBP education and from a broader curriculum development perspective.3 6 18 19 In addition to addressing challenges such as curriculum organisation and programme content/structure, identifying salient barriers to implementing optimal EBP education is recommended as an expedient approach to effecting positive change.20 Highlighted strategies to overcome such barriers included (1) ‘Training the trainers’, (2) development of and investment in a national coherent approach to EBP education, and (3) structural incorporation of EBP learning into workplace settings. National surveys of EBP education delivery21 22 found that a lack of academic and clinical staff knowledgeable in teaching EBP was a barrier to effective and efficient student learning. This was echoed by findings from EBP expert interviews, which correspond with assertions by Hitch and Nicola-Richmond6 that while recommended educational practices and resources are available, their uptake is somewhat limited. Effective teacher/leader education is required to improve EBP teaching quality.10 16 23 24 Such formal training should extend to academic and clinical educators. Supporting staff to have confidence and competence in teaching EBP and providing opportunities for learning throughout education programmes is necessary to facilitate tangible change in this area.

required to improve EBP teaching quality.10 16 23 24 Such formal training should extend to academic and clinical educators. Supporting staff to have confidence and competence in teaching EBP and providing opportunities for learning throughout education programmes is necessary to facilitate tangible change in this area. A national and coherent plan with associated investment in healthcare education specific to the integration of EBP was highlighted as having an important impact on educational outcomes. The lack of a coordinated and cohesive approach and perceived value of EBP in the midst of competing interests, particularly within the context of the healthcare agenda, was suggested to lead to an ‘ad-hoc’ approach to the implementation of and investment in EBP education and related core EBP resources. Findings from a systematic scoping review of recommendations for the implementation of EBP16 draw attention to a number of interventions at a national level that have potential to further promote and facilitate EBP education. Such interventions include government-level policy direction in relation to EBP education requirements across health profession programmes and the instalment and financing of a national institute for the development of evidence-based guidelines.

level that have potential to further promote and facilitate EBP education. Such interventions include government-level policy direction in relation to EBP education requirements across health profession programmes and the instalment and financing of a national institute for the development of evidence-based guidelines. Incorporating EBP activities into routine clinical practice has potential to promote the consistent participation and implementation of EBP. Such incorporation can be facilitated at various different levels and settings. At a health service level, the provision of computer and internet facilities at the point of care with associated content management/decision support systems allowing access to guidelines, protocols, critically appraised topics and condensed recommendations was endorsed. At a local workplace level, access to EBP mentors, implementation of consistent and regular journal clubs, grand rounds, audit and regular research meetings are important to embed EBP within the healthcare and education environments. This in turn can nurture a culture which practically supports the observation and actualisation of EBP in day-to-day practice16 and could in theory allow the coherent development of cohorts of EBP leaders.

, audit and regular research meetings are important to embed EBP within the healthcare and education environments. This in turn can nurture a culture which practically supports the observation and actualisation of EBP in day-to-day practice16 and could in theory allow the coherent development of cohorts of EBP leaders. There are study limitations which must be acknowledged. Four of the five interviewees were medical professionals. Further inclusion of allied healthcare professionals may have increased the representativeness of the findings. However, the primary selection criteria for participants were extensive and recognised expertise in relation to EBP education, the fundamental premises of which traverse specific professional boundaries.

als. Further inclusion of allied healthcare professionals may have increased the representativeness of the findings. However, the primary selection criteria for participants were extensive and recognised expertise in relation to EBP education, the fundamental premises of which traverse specific professional boundaries. Conclusion Despite positive attitudes towards EBP and a predominant recognition of its necessity for the delivery of quality and safe healthcare, its consistent translation at the point of care remains elusive. To this end, continued investigations which seek to provide further direction and structure to developments in EBP education are recommended.6 Although the quality of evidence has remained variable regarding the efficacy of individual EBP teaching interventions, consistent trends in relation to valuable andragogically sound educational approaches, fundamental curricular content and preferential instructional practices are evident within the literature in the past decade. The adoption of such trends is far from prevalent, which brings into question the extent of awareness that exists in relation to such recommendations and accompanying resources. There is a need to translate EBP into an active clinical resolution, which will have a positive impact on the delivery of patient care. In particular, an examination of current discourse between academic and clinical educators across healthcare professions is required to progress a ‘real world’ pragmatic approach to the integration of EBP education which has meaningful relevance to students and engenders active engagement from educators, clinicians and policy makers alike. Further attention is needed on strategies that not only focus on issues such as curricula structure, content and programme delivery but which support educators, education institutions, health services and clinicians to have the capacity and competence to meet the challenge of providing such EBP education.

makers alike. Further attention is needed on strategies that not only focus on issues such as curricula structure, content and programme delivery but which support educators, education institutions, health services and clinicians to have the capacity and competence to meet the challenge of providing such EBP education. Summary Box What is already known? Evidence-based practice (EBP) is established as a fundamental element and key indicator of high-quality patient care. Both achieving competency and delivering instruction in EBP are complex processes requiring a multimodal approach. Currently there exists only a modest utilisation of existing resources available to further develop EBP education. What are the new findings? In addition to developing competence in the fundamental EBP steps of ‘Ask’, ‘Acquire’, ‘Appraise’, ‘Apply’ and ‘Assess’, developing competence in effectively communicating evidence to others, in particular patients/service users, is an area newly emphasised as requiring additional attention by healthcare educators. The successful expansion of the assessment and evaluation of EBP requires a pragmatic amplification of the discourse between academic and clinical educators. How might it impact on clinical practice in the foreseeable future? Quality of care is improved through the integration of the best available evidence into decision-making as routine practice and not in the extemporised manner often currently practised.

The successful expansion of the assessment and evaluation of EBP requires a pragmatic amplification of the discourse between academic and clinical educators. How might it impact on clinical practice in the foreseeable future? Quality of care is improved through the integration of the best available evidence into decision-making as routine practice and not in the extemporised manner often currently practised. Special thanks to Professor Leanne Togher, Professor Carl Heneghan, Professor Bruce Arroll, Professor Rodney Jackson and Professor Gordon Guyatt, who provided key insights on EBP education from an international perspective. Thank you to Dr Niamh O’Rourke, Dr Eve O’Toole, Dr Sarah Condell and Professor Dermot Malone for their helpful direction throughout the project. Contributors: This project formed part of a national project on EBP education in Ireland of which all named authors are members. The authors named on this paper made substantial contributions to both the acquisition and analysis of data, in addition to reviewing the report and paper for submission. Funding: This research was funded by the Clinical Effectiveness Unit of the National Patient Safety Office (NPSO), Department of Health, Ireland. Competing interests: None declared. Patient consent: Not required.

Contributors: This project formed part of a national project on EBP education in Ireland of which all named authors are members. The authors named on this paper made substantial contributions to both the acquisition and analysis of data, in addition to reviewing the report and paper for submission. Funding: This research was funded by the Clinical Effectiveness Unit of the National Patient Safety Office (NPSO), Department of Health, Ireland. Competing interests: None declared. Patient consent: Not required. Ethics approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional ethical committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Ethical approval was granted by the Social Research Ethics Committee, University College Cork (Log 2016–140). Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: The full report entitled ’Research on Teaching EBP in Ireland to healthcare professionals and healthcare students' is available on the National Clinical Effectiveness, Department of Health website.

The tension between the benefits and harms of anticoagulation is finely balanced. Prophylactic aspirin in healthy, elderly patients provides no benefit and causes harm; clinicians should not use it for primary prevention in otherwise well patients aged over 70. As the world’s population ages, health in later life has become a public health priority. The prevention of disease is critical to these aims. As a chief cause of morbidity in the elderly,1 the prevention of cardiovascular disease is of particular focus. The use of aspirin unequivocally benefits patients who have already suffered a cardiovascular event.2 3 However, its role in primary prevention is much more contentious.4–6 The Aspirin in Reducing Events in the Elderly (ASPREE) trial7 set out to address the uncertainty surrounding the prophylactic use of aspirin in healthy, elderly patients. The ASPREE investigators randomised almost 20 000 people to receive 100 mg aspirin or placebo. Included participants were 70 years or older (or 65 if Hispanic or African–American) and had no history of cardiovascular disease, dementia, terminal illness or increased risk of bleeding. The rate of cardiovascular disease and haemorrhage were assessed over approximately 5 years (median follow-up 4.7).

rin or placebo. Included participants were 70 years or older (or 65 if Hispanic or African–American) and had no history of cardiovascular disease, dementia, terminal illness or increased risk of bleeding. The rate of cardiovascular disease and haemorrhage were assessed over approximately 5 years (median follow-up 4.7). This trial proved that the use of aspirin is harmful to healthy, community-dwelling elderly people. Participants that took aspirin derived no benefit but suffered more haemorrhagic events than those that took placebo. People in the aspirin group had 8.6 major haemorrhage events per 1000 person-years compared with 6.2 events per 1000 person-years in the placebo group (HR, 1.38; 95% CI 1.18 to 1.62). The rate of cardiovascular disease for people in the aspirin group was 10.7 events per 1000 person-years, compared with 11.3 in the placebo group (HR, 0.95; 95% CI 0.83 to 1.08). There are a few limitations to consider. Most significantly, only around two-thirds of the participants were still taking their aspirin by the end of the trial. This could have underestimated the effects of aspirin, but this is likely to have affected both the potential benefits and the harms equally. Furthermore, a small proportion of patients were taking aspirin before the trial started.

two-thirds of the participants were still taking their aspirin by the end of the trial. This could have underestimated the effects of aspirin, but this is likely to have affected both the potential benefits and the harms equally. Furthermore, a small proportion of patients were taking aspirin before the trial started. This trial has many important implications. Previous studies have provided heterogeneous estimates of the benefit of aspirin for primary prevention,4 but pointed to an increased benefit of aspirin in the elderly. The ASPREE trial has shown that elderly patients that take aspirin prophylactically will derive only harm. At this stage, clinical practice guidelines should not recommend the use of aspirin for otherwise well patients aged 70 or older (or 65 if African–American or Hispanic). Many guidelines already advise not to use aspirin for primary prevention, including the National Institute for Health and Clinical Excellence guidelines.8 However, other guideline organisations offer more circumspect guidance and acknowledge the previous lack of evidence. For instance, the United States Prevention Task Force currently states: ‘The current evidence is insufficient to assess the balance of benefits and harms of initiating aspirin use for the primary prevention of CVD and CRC in adults aged 70 years or older.’9 The ASPREE trial should lead to a change of guidance.

vidence. For instance, the United States Prevention Task Force currently states: ‘The current evidence is insufficient to assess the balance of benefits and harms of initiating aspirin use for the primary prevention of CVD and CRC in adults aged 70 years or older.’9 The ASPREE trial should lead to a change of guidance. The study authors of the ASPREE trial published two other linked randomised controlled trials investigating the effect of aspirin on all-cause mortality10 and disability free-survival.11 The results of these studies further support the recommendation not to use aspirin prophylactically in the elderly. Patients that took aspirin had higher rates of all-cause mortality (HR, 1.14; 95% CI 1.01 to 1.29); specifically they had higher rates of cancer-related death (HR, 1.31; 95% CI 1.10 to 1.56). Furthermore, the rate of disability (a composite of death, dementia or persistent physical disability) was no different between those that took aspirin and those that took a placebo (HR, 1.01; 95% CI 0.92 to 1.11; p=0.79). EBM verdict EBM Verdict on: Effect of Aspirin on Cardiovascular Events and Bleeding in the Healthy Elderly. McNeil JJ et al. N Engl J Med 2018;379:1509–1518. doi: 10.1056/NEJMoa1805819. [Epub 16 Sep 2018]. In healthy, community-dwelling elderly people aged 70 and older, aspirin does not prevent cardiovascular disease and does increase one’s risk of major haemorrhage. Clinicians should not offer aspirin as primary prevention to otherwise well elderly patients. Contributors: JWO’S is the sole author. Competing interests: None declared. Patient consent: Not required.

In healthy, community-dwelling elderly people aged 70 and older, aspirin does not prevent cardiovascular disease and does increase one’s risk of major haemorrhage. Clinicians should not offer aspirin as primary prevention to otherwise well elderly patients. Contributors: JWO’S is the sole author. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Commissioned; internally peer reviewed. Data sharing statement: This is a commentary piece with no original data.

Background and aims Attention deficit hyperactivity disorder (ADHD) is the most common psychiatric disorder in childhood and adolescents.1 2 ADHD is associated with failure in the attainment of many normal developmental milestones and can result in children experiencing school failure, poor family and peer relations, low self-esteem, as well as other emotional, behavioural and learning problems.3 Early recognition of this condition is critical as diagnosis and resultant intervention that addresses the wide range of personal, social, educational and occupational needs of children with ADHD can redirect the developmental trajectory of children with this condition.2 While current guidance from The National Institute for Health and Care Excellence (NICE) recommends that ‘primary care practitioners should not make the initial diagnosis or start medication in children or young people with suspected ADHD’,4 general practitioners (GPs) and other primary care professionals play an important role in identifying children and young people who have persistent behavioural and/or attention problems with at least moderate impairment to secondary care so that opportunities for assessment and referral are not missed.5

pected ADHD’,4 general practitioners (GPs) and other primary care professionals play an important role in identifying children and young people who have persistent behavioural and/or attention problems with at least moderate impairment to secondary care so that opportunities for assessment and referral are not missed.5 There is evidence that the threshold for ADHD diagnosis and the prescribing of stimulant medications may be influenced by demographic characteristics of the child,6 parental and family characteristics or other broader structural determinants such as socio-economic deprivation.7 8 However, the factors which are associated with variations in ADHD diagnosis and prescribing within primary care are largely unexamined.9 10 We carried out this study to describe variations in the average age of ADHD diagnosis and prescribing of stimulant medications among general practices who are members of the Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC) network11; and to identify child, parental, household and general practice factors that might account for these variations.

e age of ADHD diagnosis and prescribing of stimulant medications among general practices who are members of the Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC) network11; and to identify child, parental, household and general practice factors that might account for these variations. Methods Subjects and setting We used information from the RCGP RSC sentinel network database. At the time of this study it hosted a pseudonymised dataset from a nationally representative sample of just under 2 million people registered with a network practices.11 It has been used in surveillance of influenza and respiratory disease for over 50 years. Over this period practices have had feedback about their data quality around influenza and respiratory disease—in particular the differentiation of first or new (incident) from follow-up consultations. Feedback about data quality is also provided via a dashboard, that is refreshed weekly12 and is good for routine primary care.13 UK general practice is suitable for this type of study because it has a registration-based system with patients registered with a single practice. Practices have been computerised since the late 1990s, with pay-for-performance (P4P) introduced in 2004.14 Key data are coded,15 and this includes diagnoses, therapy, test results and other key data. Additionally, the RCGP RSC has a household key enabling people in the same household to be characterised.16

e practice. Practices have been computerised since the late 1990s, with pay-for-performance (P4P) introduced in 2004.14 Key data are coded,15 and this includes diagnoses, therapy, test results and other key data. Additionally, the RCGP RSC has a household key enabling people in the same household to be characterised.16 ADHD diagnosis and stimulant prescribing We used coded data reported in online appendix 1 as Read codes to extract information on children with a diagnosis of ADHD under 19 years of age between January 1st and December 31st 2016. The age at which the first diagnosis of ADHD was noted for each child, and the average age of ADHD diagnosis within each practice was calculated. 10.1136/bmjebm-2018-111133.supp1Supplementary data We used Read codes, and also some CMR brand specific proprietary drug dictionary codes, reported in online appendix 2 to extract information on stimulant prescribing for children with ADHD. A 2016 study found that 94% of prescriptions for children with ADHD was for the stimulant methylphenidate.17 The proportion of children with ADHD who were prescribed stimulant medications between January 1st and December 31st 2016 within each practice was calculated. 10.1136/bmjebm-2018-111133.supp2Supplementary data

We used Read codes, and also some CMR brand specific proprietary drug dictionary codes, reported in online appendix 2 to extract information on stimulant prescribing for children with ADHD. A 2016 study found that 94% of prescriptions for children with ADHD was for the stimulant methylphenidate.17 The proportion of children with ADHD who were prescribed stimulant medications between January 1st and December 31st 2016 within each practice was calculated. 10.1136/bmjebm-2018-111133.supp2Supplementary data Characteristics of children with ADHD within each practice We extracted demographic information about children with ADHD including their age, sex, ethnicity and socio-economic deprivation as indicated by their index of multiple deprivation (IMD) score.18 The IMD is a measure of relative deprivation for small areas of about 1500 people, termedLower Super Output Areas. It is a combined measure of deprivation based on a total of 37 separate indicators that have been grouped into seven domains, each of which reflects a different aspect of deprivation experienced by individuals living in an area. An average demographic profile of children with ADHD for each practice was compiled including their mean age, the proportion of female children with ADHD and the proportion of children from the lowest IMD quintile. Racial and ethnic disparities between non-white and white populations has also previously been shown in ADHD diagnosis and treatment,19 thus we also present the proportion of children who were non-white with ADHD in practices.

ion of female children with ADHD and the proportion of children from the lowest IMD quintile. Racial and ethnic disparities between non-white and white populations has also previously been shown in ADHD diagnosis and treatment,19 thus we also present the proportion of children who were non-white with ADHD in practices. The rate of hyperkinetic conduct disorders (HKCD) for each practice was also calculated separately. General practice characteristics Between August 1st and August 31st 2017 we extracted information about the general characteristics of RCGP RSC practices from summary statistics extracted for each practice, from individual practice websites, the NHS choices website and the General Medical Council (GMC) register.20 This includes the practice list size, the percentage of children in the lowest IMD quintile, the total number of children registered with the practice, the total number of GPs in the practice, the gender of GPs in the practice, the average number of years since their medical and specialist general practice qualifications, the average number of qualifications of GPs in the practice and whether the practice was in an urban or rural location. We collected information about the quality of care provided by each practice from the National General Practice Profiles published by Public Health England, including the total achievement score in the P4P scheme, called the Quality and Outcomes Framework (QOF), as well as the total scores in the clinical and public health management domains of the QOF.21 22

ty of care provided by each practice from the National General Practice Profiles published by Public Health England, including the total achievement score in the P4P scheme, called the Quality and Outcomes Framework (QOF), as well as the total scores in the clinical and public health management domains of the QOF.21 22 Characteristics of households within each practice We used a household identifier in each medical record to extract information about the average profile of households in each practice including the proportion of single adult households, the average number of children under 19 years in each household, and the average age difference between adults and children in each household.16 Statistical analysis We report descriptive statistics generally reporting mean for normally distributed data, but mode, median and mean where there is a skewed distribution of data. We used a Shapiro-Wilk test23 to show that the distribution of age of diagnosis of ADHD between practices was not normally distributed (p value=0.001465). The percentage of children with ADHD prescribed stimulant medications across RCGP RSC practices was normally distributed (p value=0.5807). We then undertook a logarithmic transformation of the mean age of ADHD diagnosis between practices.

gnosis of ADHD between practices was not normally distributed (p value=0.001465). The percentage of children with ADHD prescribed stimulant medications across RCGP RSC practices was normally distributed (p value=0.5807). We then undertook a logarithmic transformation of the mean age of ADHD diagnosis between practices. Multiple linear regression was then performed to examine the effect of child, general practice and household factors on the percentage of children with ADHD prescribed stimulant medications. The residuals from a multivariate regression of mean age of ADHD on study covariates was positively skewed, so we employed a log transformation of mean age of ADHD. The regression coefficients from this log-level regression was used to calculate an effect of a unit change in the study co-variates with a percentage change in the average ADHD diagnosis age between practices. All statistical analysis was undertaken using R release V.3.5,24 the Global Validation of Linear Models (V.1.0.0.2) R package by  Edsel Pena was used to test that linear model assumptions were met.

The regression coefficients from this log-level regression was used to calculate an effect of a unit change in the study co-variates with a percentage change in the average ADHD diagnosis age between practices. All statistical analysis was undertaken using R release V.3.5,24 the Global Validation of Linear Models (V.1.0.0.2) R package by  Edsel Pena was used to test that linear model assumptions were met. Results Characteristics of practices within the RCGP RSC network Data from 158 general practices and 353 774 children under 19 years of age in the RCGP RSC national sentinel practice network were included in the study. 82.9% (131/158) practices were located in an urban location. The mean number of GPs per practice was 7.5 (median=7, mode=2) and the mean number of years since GPs had qualified from medical school was 19.7 years. Each GP within the practice had a mean of more than two specialist qualifications in addition to their medical qualifications The GP practices within the network achieved a mean 97.7% (95% CI 97.1 to 98.4) total P4P/QOF score overall (median=99.0, mode=98.8425). This compares with a mean total P4P/QOF score of 95.5% (95% CI 95.3 to 95.6) median=97.7, mode=100.0 for all practices in England.

Results Characteristics of practices within the RCGP RSC network Data from 158 general practices and 353 774 children under 19 years of age in the RCGP RSC national sentinel practice network were included in the study. 82.9% (131/158) practices were located in an urban location. The mean number of GPs per practice was 7.5 (median=7, mode=2) and the mean number of years since GPs had qualified from medical school was 19.7 years. Each GP within the practice had a mean of more than two specialist qualifications in addition to their medical qualifications The GP practices within the network achieved a mean 97.7% (95% CI 97.1 to 98.4) total P4P/QOF score overall (median=99.0, mode=98.8425). This compares with a mean total P4P/QOF score of 95.5% (95% CI 95.3 to 95.6) median=97.7, mode=100.0 for all practices in England. Characteristics of households within the RCGP RSC network Information from a total of 803 218 households in the RCGP RSC network showed 23.2% included a child under 19 years of age between January 1st and December 31st 2016. This compares with national census data for 2015 of 29% of households with dependent children under 19 years of age. On average 31.0% (95% CI 29.6 to 32.4) of households with children were occupied by a single adult over 18 years of age. This compares with national census data for 2015 of 25% of single parent households. In households with children, there was an average of just under two children under 19 years of age. This compares with the national census data for 2015 that showed that 40% of families had two dependent children and 15% of families had three or more dependent children Within households in the RCGP RSC network the average age difference between adults and children was 36 years (95% CI 35.7 to 36.3).

19 years of age. This compares with the national census data for 2015 that showed that 40% of families had two dependent children and 15% of families had three or more dependent children Within households in the RCGP RSC network the average age difference between adults and children was 36 years (95% CI 35.7 to 36.3). Characteristics of children with ADHD in the RCGP RSC network 3470 children with a coded diagnosis of ADHD within the RCGP RSC network were included in the study. Of these, 649 (18.7%, 95% CI 17.4 to 20.0) were female and 316 (9.1%, 95% CI 8.1 to 10.1) were of non-white ethnicity. There was wide variation in the rate of ADHD diagnosis between practices within the RCGP RSC network from 0.01 to 3.2 per 100 children (inter-quartile range 0.62). See figure 1. The mean rate was 1.0 (95% CI 0.9 to 1.1) and the median was 0.9 per 100 children. Figure 1 Mean age of first diagnosis of ADHD amongst Royal College of General Practitioners Research and Surveillance Centre practices. ADHD, attention deficit hyperactivity disorder. Only 26 children with HKCD were recorded in the RCGP RSC network. Age of diagnosis of ADHD The mean age of first diagnosis with ADHD was 10.5 years (95% CI 10.1 to 10.9, median 10, IQR 9.0–11.9). There was also wide inter-practice variation in the age of first diagnosis of ADHD with the diagnosis being made anywhere between 5 and 17 years of age across RCGP RSC practices.

Only 26 children with HKCD were recorded in the RCGP RSC network. Age of diagnosis of ADHD The mean age of first diagnosis with ADHD was 10.5 years (95% CI 10.1 to 10.9, median 10, IQR 9.0–11.9). There was also wide inter-practice variation in the age of first diagnosis of ADHD with the diagnosis being made anywhere between 5 and 17 years of age across RCGP RSC practices. Prescribing of stimulant medications in children with ADHD The mean percentage of children with ADHD prescribed stimulant medications among RCGP RSC practices was 41.2% (95% CI 38.7 to 43.6). There was wide inter-practice variation in prescribing with RCGP practices prescribing stimulant medications for between 5.6% and 77.3% of their patients under 18 years with ADHD. Determinants of age of diagnosis and stimulant prescribing in children with ADHD Multiple regression analysis showed that characteristics of the households in which children with ADHD are raised was a strong predictor of the mean age that children are diagnosed in practices (see table 1). Table 1 Determinants of age of ADHD diagnosis

Determinants of age of diagnosis and stimulant prescribing in children with ADHD Multiple regression analysis showed that characteristics of the households in which children with ADHD are raised was a strong predictor of the mean age that children are diagnosed in practices (see table 1). Table 1 Determinants of age of ADHD diagnosis Regression estimate (95% CI) Effect size Interpretation Characteristics of households within the practice Single parent (>18 years) households Not statistically significant Not statistically significant Average number of children in the household 0.2759 (0.027774 to 0.524016) 27.59% For every unit increase in number of children in the household, there is on average 27.6% increase in ADHD diagnosis age within the practice Average age difference within the household 0.03755 (−0.00109 to 0.0761898) 3.755% For every unit increase in age difference within practice households, there is 3.8% increase in ADHD diagnosis age within the practice General practice characteristics General practice size 0.00002938 (0.00000730 to 0.00005147) 0.002938% For every increase in the practice size by 100, there is on average 0.3% increase in ADHD diagnosis age in the practice Total number of children −0.0001045 (−0.0001954 to −0.0000136) −0.001045% For every unit increase in the number of children in the practice by 100 there is on average 0.1% decrease in ADHD diagnosis age within the practice Proportion of eligible children—in lowest IMD quintile Not statistically significant Not statistically significant Mean years since medical qualification of GPs Not statistically significant Not statistically significant Mean number of specialist qualifications of GPs Not statistically significant Not statistically significant Urban/rural practice Not statistically significant Not statistically significant QOF achieved in 2015–2016— overall (%) Not statistically significant Not statistically significant NHS choices star rating Not statistically significant Not statistically significant Characteristics of children with ADHD within the practice ADHD prevalence rate Not statistically significant Not statistically significant Hyperkinetic conduct disorder prevalence rate Not statistically significant Not statistically significant ADHD of non-white ethnicity within the practice (%) Not statistically significant Not statistically significant Female children with ADHD (%) Not statistically significant Not statistically significant ADHD, attention deficit hyperactivity disorder;  GP, general practition

ally significant Not statistically significant ADHD of non-white ethnicity within the practice (%) Not statistically significant Not statistically significant Female children with ADHD (%) Not statistically significant Not statistically significant ADHD, attention deficit hyperactivity disorder;  GP, general practition er; IMD, index of multiple deprivation; QOF, Quality and Outcomes Framework. For every unit increase in the average number of children in the household, there was a 27.6% increase of in ADHD diagnosis age within the practice. Equally, for every unit increase in age difference within practice households, there was a 3.8% increase in ADHD diagnosis age within the practice. Larger practices with less registered children were also more likely to make ADHD diagnosis at an older age, although the magnitude of these effects was much less than the characteristics of the households. Other characteristics of the general practice including the urban/rural location, the proportion of children in the lowest socio-economic quintile, the qualification of GPs or indicators of the practice clinical care did not significantly affect the inter-practice variations in the age of ADHD diagnosis. The number of children with ADHD in a practice or the number of children with HKCD did not significantly affect the inter-practice variations in the age of ADHD diagnosis.

Other characteristics of the general practice including the urban/rural location, the proportion of children in the lowest socio-economic quintile, the qualification of GPs or indicators of the practice clinical care did not significantly affect the inter-practice variations in the age of ADHD diagnosis. The number of children with ADHD in a practice or the number of children with HKCD did not significantly affect the inter-practice variations in the age of ADHD diagnosis. Table 2 shows that stimulant prescribing for children with ADHD was significantly determined by the proportion of children in the practice in the lowest IMD quintile, with every unit increase in proportion of children in the lowest IMD quintile in the practice, resulting in 0.1575 less prescriptions for stimulant medication for children with ADHD. Table 2 Determinants of stimulant prescribing in children with ADHD

Table 2 shows that stimulant prescribing for children with ADHD was significantly determined by the proportion of children in the practice in the lowest IMD quintile, with every unit increase in proportion of children in the lowest IMD quintile in the practice, resulting in 0.1575 less prescriptions for stimulant medication for children with ADHD. Table 2 Determinants of stimulant prescribing in children with ADHD Regression estimate (95% CI) Interpretation Characteristics of households within the practice Single parent (>18 years) households Not statistically significant Not statistically significant Average number of children in the household Not statistically significant Not statistically significant Average age difference within the household Not statistically significant Not statistically significant General practice characteristics General practice size Not statistically significant Not statistically significant Total number of children Not statistically significant Not statistically significant Proportion of eligible children—in lowest IMD quintile −0.1575 (−0.29353 to −0.02157) For every unit increase in proportion of children in the lowest IMD quintile in the practice, there is 0.1575 less stimulant medication prescribed for children with ADHD Mean years since medical qualification of GPs Not statistically significant Not statistically significant Mean number of specialist qualifications of GPs Not statistically significant Not statistically significant Urban/rural practice Not statistically significant Not statistically significant QOF achieved in 2015–2016—overall Not statistically significant Not statistically significant NHS choices star rating Not statistically significant Not statistically significant Characteristics of children with ADHD within the practice ADHD prevalence rate Not statistically significant Not statistically significant Hyperkinetic conduct disorder prevalence rate Not statistically significant Not statistically significant ADHD of non-white ethnicity within the practice (%) Not statistically significant Not statistically significant Female children with ADHD (%) Not statistically significant Not statistically significant ADHD, attention deficit hyperactivity disorder;  GP, general practitioner; IMD, index of multiple deprivation; QOF, Quality and Outcomes Framework.

ithin the practice (%) Not statistically significant Not statistically significant Female children with ADHD (%) Not statistically significant Not statistically significant ADHD, attention deficit hyperactivity disorder;  GP, general practitioner; IMD, index of multiple deprivation; QOF, Quality and Outcomes Framework. Discussion Summary of principal findings Using data from a nationally representative network of general practices, we found wide inter-practice variation in the prevalence of recorded diagnosis of ADHD. While most children were diagnosed with ADHD by the age of 10, we also found widespread variations with practices making a diagnosis of ADHD anywhere between 5 and 17 years of age. ADHD diagnosis was more likely to be made later in households with a larger number of children and wider age gaps between adults and children in the households, suggesting that the diagnosis was more likely to be made in families with older parents. Larger practices with less registered children were also more likely to make diagnosis of ADHD at a later age. Stimulant prescribing for children with ADHD also showed widespread inter-practice variation, with more affluent practices more likely to prescribe stimulant medications for their children with ADHD.

While most children were diagnosed with ADHD by the age of 10, we also found widespread variations with practices making a diagnosis of ADHD anywhere between 5 and 17 years of age. ADHD diagnosis was more likely to be made later in households with a larger number of children and wider age gaps between adults and children in the households, suggesting that the diagnosis was more likely to be made in families with older parents. Larger practices with less registered children were also more likely to make diagnosis of ADHD at a later age. Stimulant prescribing for children with ADHD also showed widespread inter-practice variation, with more affluent practices more likely to prescribe stimulant medications for their children with ADHD. Strengths and weaknesses of the study There are a number of strengths of our study including its large sample size of 158 practices from a nationally representative sentinel network, which allows us to examine the impact of a key primary care determinants on the age of ADHD diagnosis, a relatively under-researched area.9 In addition, our sample included almost 3500 children with ADHD in primary care which is one of the largest studies to examine factors that affect ADHD diagnosis. This allows us to separately take into take into account the characteristics of the children with ADHD such as sex and ethnicity.

ely under-researched area.9 In addition, our sample included almost 3500 children with ADHD in primary care which is one of the largest studies to examine factors that affect ADHD diagnosis. This allows us to separately take into take into account the characteristics of the children with ADHD such as sex and ethnicity. However, our study has a number of limitations including the observational and cross-sectional nature of the study design which does not allow us to draw causal inferences from our results.25 In addition as our study focused on practices as the unit of analysis, our results cannot be used to make conclusions about the care of individual patients with ADHD. There are a number of limitations to our study related to the use of routinely collected data, including the quality of data in primary care records and data about practice characteristics in publicly available sources. Data from the RCGP RSC sentinel network has been previously appraised and published.11 We would expect that our study is limited by the accuracy of the age of ADHD diagnosis which is commonly under-diagnosed, misdiagnosed and undertreated.26 It may also be poorly or inaccurately recorded in the patient’s medical record, especially if they change practices. However, we have shown that the prevalence of ADHD among practices in the RCGP RSC network is equivalent to other studies in equivalent populations in the UK.27

iagnosed, misdiagnosed and undertreated.26 It may also be poorly or inaccurately recorded in the patient’s medical record, especially if they change practices. However, we have shown that the prevalence of ADHD among practices in the RCGP RSC network is equivalent to other studies in equivalent populations in the UK.27 Assessment of exposure to stimulant medications was not undertaken in this study, thus while we found inter-practice variations in stimulant prescribing practices, we cannot make conclusions about variation in stimulant exposure between practices. This is important as it has been suggested that exposure to stimulant medication, both early initiation of therapy and chronic exposure are important adverse childhood outcomes.28 29 We know that the RCGP RSC household key under-estimates household size.16 If some people in the same household are registered with a non-RCGP RSC practice they will not be included. Hence RCGP RSC has a greater proportion of single person households compared with census data. When households move, they generally register at the same practice. These factors may affect our interpretation of the impact of households in our study. Additionally, we have no separate information on the impact of families which has been shown to be important in the consequences of ADHD.26

compared with census data. When households move, they generally register at the same practice. These factors may affect our interpretation of the impact of households in our study. Additionally, we have no separate information on the impact of families which has been shown to be important in the consequences of ADHD.26 The enumeration of practice structural variables for this study relied on the extraction of publicly available data about practices from their own websites, NHS choices and the medical register held by the GMC. The quality of this information, especially its completeness, consistency, timeliness and accuracy has not been validated in published data, and given the number of practices involved in this study we did not attempt to validate the accuracy with individual practice staff. The outcomes of age of ADHD diagnosis and stimulant prescribing may also be affected by differences in behaviours of individual GPs in practices, however it was not possible to test any clustering of outcomes by individual GPs as there is insufficient details to consistently document activity by GPs in our dataset. Lastly, we have not included data from child and adolescent mental health services where the majority of diagnoses of ADHD are made. Thus, there is a risk that the diagnosis from secondary care may be missed or there may be a delay in the recording of diagnosis in the primary care record.

The outcomes of age of ADHD diagnosis and stimulant prescribing may also be affected by differences in behaviours of individual GPs in practices, however it was not possible to test any clustering of outcomes by individual GPs as there is insufficient details to consistently document activity by GPs in our dataset. Lastly, we have not included data from child and adolescent mental health services where the majority of diagnoses of ADHD are made. Thus, there is a risk that the diagnosis from secondary care may be missed or there may be a delay in the recording of diagnosis in the primary care record. Strengths and weaknesses in relation to other studies, discussing important differences in results The ADHD prevalence from this study of 1.0%, (0.98 per 100 children, 95% CI 0.95 to 1.01) is similar to previous studies in the UK,27 30 although it is lower than figures of global community prevalence published in a recent systematic review of 2%–7%.31

tion to other studies, discussing important differences in results The ADHD prevalence from this study of 1.0%, (0.98 per 100 children, 95% CI 0.95 to 1.01) is similar to previous studies in the UK,27 30 although it is lower than figures of global community prevalence published in a recent systematic review of 2%–7%.31 Previous studies have found variations in the prevalence of ADHD in community samples and variations in the age of ADHD diagnosis.31 32 Using data from a network of general practices we have shown that there also are significant inter-practice variations in ADHD diagnosis, prevalence and stimulant medication prescribing. We have shown that household factors are significant predictors of ADHD diagnosis. This suggests that the wider impact of ADHD symptoms on adults and children within in the same family, rather than the isolated impact on the child, are predictive of recognition of problems in ADHD. This is similar to findings from other studies conducted in the UK.33 Sayal and colleagues suggested that parental recognition of problems and views that their child has hyperactivity were associated with greater severity of symptoms. In their sample they found that the majority of parents discuss their concerns with professionals based in education services and few had consulted primary care for these problems.33 This highlights a weakness of our study which did not include information from education services.

ted with greater severity of symptoms. In their sample they found that the majority of parents discuss their concerns with professionals based in education services and few had consulted primary care for these problems.33 This highlights a weakness of our study which did not include information from education services. In contrast stimulant prescribing for children with ADHD was significantly determined by the proportion of children in the practice in the lowest IMD quintile. This is similar to other studies which have found that socio-economic status is an important determinant of stimulant prescribing,34–38 although there remains controversy about the direction of this relationship. Meaning of the study: possible explanations and implications for clinicians and policy-makers We have shown that the composition of households is important in the recognition of ADHD. Households with a greater number of children may hinder the parental recognition of an over-active child. The larger age gap between adults and children within households hints at another possible explanation which revolves around the greater experience and situation handling skills of the caregivers, which may result in greater absorption of over-active behaviours and a later ADHD diagnosis in these households. This would suggest that older parents with more children would benefit from greater support to recognise the symptoms of ADHD and present their children to healthcare services for earlier assessment.

Meaning of the study: possible explanations and implications for clinicians and policy-makers We have shown that the composition of households is important in the recognition of ADHD. Households with a greater number of children may hinder the parental recognition of an over-active child. The larger age gap between adults and children within households hints at another possible explanation which revolves around the greater experience and situation handling skills of the caregivers, which may result in greater absorption of over-active behaviours and a later ADHD diagnosis in these households. This would suggest that older parents with more children would benefit from greater support to recognise the symptoms of ADHD and present their children to healthcare services for earlier assessment. The finding that larger practices with fewer registered children diagnosed children with ADHD at a later age may suggest that GPs in these practices have less confidence in making the diagnosis early and a higher threshold for symptoms to be initially presented in these practices. It could also reflect poorer links with specialist education services or child mental health services for definitive diagnosis in some practices and is an area for further study. Unanswered questions and future research Previous research suggests that parental factors are critical to recognition of ADHD, and our research on households hints at how parental recognition may be influenced by the age and experience of the caregiver.

The finding that larger practices with fewer registered children diagnosed children with ADHD at a later age may suggest that GPs in these practices have less confidence in making the diagnosis early and a higher threshold for symptoms to be initially presented in these practices. It could also reflect poorer links with specialist education services or child mental health services for definitive diagnosis in some practices and is an area for further study. Unanswered questions and future research Previous research suggests that parental factors are critical to recognition of ADHD, and our research on households hints at how parental recognition may be influenced by the age and experience of the caregiver. However, further research is required to understand the barriers and facilitators to early parental recognition, especially the role of statutory services including school health services which are often the first point of call for parents with symptomatic children. Summary box What is already known about this subject? Early recognition, identification  and treatment of attention deficit hyperactivity disorder (ADHD) can alter the trajectory of this condition. However, the factors which influence variations in ADHD diagnosis within primary care in the UK are not described. What are the new findings? This study describes the variation in age of ADHD diagnosis and stimulant prescribing for children with ADHD among a nationally representative sample of English general practices.

Summary box What is already known about this subject? Early recognition, identification  and treatment of attention deficit hyperactivity disorder (ADHD) can alter the trajectory of this condition. However, the factors which influence variations in ADHD diagnosis within primary care in the UK are not described. What are the new findings? This study describes the variation in age of ADHD diagnosis and stimulant prescribing for children with ADHD among a nationally representative sample of English general practices. Family factors were significantly predictive of age of ADHD diagnosis in the practice. For every unit increase in the average number of children in the household, there was a 27.6% increase in ADHD diagnosis age within the practice. Equally, for every percentage increase in age difference within practice households, there was a 3.8% increase in ADHD diagnosis age within the practice. General practice factors were also significantly predictors of an older age of ADHD diagnosis in general practices, although the magnitude of these effects was much less than the characteristics of the households. General practice factors such as a decrease in the proportion of children from the lowest index of multiple deprivation (IMD) quintile in the practice were also significantly associated with increased stimulant prescribing for children with ADHD. For every unit increase in proportion of children in the lowest IMD quintile in the practice, there was 0.1575 less prescriptions for stimulant medication for children with ADHD.

le deprivation (IMD) quintile in the practice were also significantly associated with increased stimulant prescribing for children with ADHD. For every unit increase in proportion of children in the lowest IMD quintile in the practice, there was 0.1575 less prescriptions for stimulant medication for children with ADHD. Patient factors were not significantly predictive of age of ADHD diagnosis or stimulant prescribing for children with ADHD. How might it impact on clinical practice in the foreseeable future? Health system factors resulting in wide inter-practice variation in the age of ADHD diagnosis and stimulant prescribing could be identified and addressed to facilitate earlier diagnosis of these children. Families with older parents and parents with more children could be offered greater support to recognise the symptoms of ADHD and present their children to healthcare services for earlier assessment. Patients and practices who are members of the RCGP RSC network who allow their data to be shared for surveillance, quality improvement and research. Contributors: UH, ACJ, SJ, MJ and SL were responsible for the conception or design of the work; UH was responsible for data analysis and writing the initial draft manuscript. All authors were responsible for interpretation of data, revising the manuscript critically for important intellectual content and final approval of the version to be published.

ns are drivers of ‘too much medicine’. As mentioned above, there are typical and atypical symptoms that trigger an active search for healthcare. Patients/the family expect something from their GP. It can include a check-up, reassurance, a medical certificate, antibiotic therapy or simply guidance on symptom management. GPs who perceive high patient expectations for antibiotics are more likely to prescribe antibiotics (ie, overtreatment) unnecessarily. A recent study in Australia found that more than half of the GPs included in the study self-reported that they would prescribe antibiotics for an upper respiratory tract infection to meet patient expectations.11 However, while all patients are believed to expect something, only some patients expect antibiotics, and among these patients, a subset demands antibiotics. Notwithstanding, patient expectations may be based on false assumptions, with many overestimating the effectiveness of antibiotic treatment.12

d SL were responsible for the conception or design of the work; UH was responsible for data analysis and writing the initial draft manuscript. All authors were responsible for interpretation of data, revising the manuscript critically for important intellectual content and final approval of the version to be published. Funding: This study was derived from work undertaken as part of the Models of Child Health Appraised (MOCHA) project. MOCHA is funded by the European Commission through the Horizon 2020 Framework under the grant agreement number: 634201. Competing interests: None declared. Patient consent: Not required. Ethics approval: The study received ethical approval from the South West - Central Bristol Research Ethics Committee (REC reference number: 17/SW/0137, approval granted 16th June 2017). Permission to use RCGP RSC data was provided given (Reference RSC_3017, Date 27/09/2017). Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: Additional data are available by emailing UH (u.hoang@surrey.ac.uk).

Preparing for my decennial recertification exam in internal medicine opened my eyes to a startling disconnect in how evidence-based medicine (EBM) is communicated to practising clinicians compared with how it is implemented by expert panels and EBM authorities. Indeed, sources of continuing clinical education often omit basic tenets of EBM, in particular how to integrate the best available evidence with clinical expertise.1–3 This disconnect is mystifying: why should education experts well versed in EBM use communication templates disjointed from EBM to communicate evidence for practising medicine? For example, a common source of preparation for the American Board of Internal Medicine recertification exam is the Medical Knowledge Self-Assessment Program (MKSAP) published by the American College of Physicians.4 Even though the American College of Physicians prioritises the mission of teaching and implementing EBM,5 two advisements from the 2017 MKSAP illustrate how the curriculum they sponsor deviates from this mission. First, Advisement 1: ‘Routine screening for skin cancer using a total body skin examination is not recommended. The United States Preventive Services Task Force (USPSTF) found insufficient evidence that routine skin examination was effective at reducing the morbidity and mortality from cutaneous melanoma, basal cell carcinoma, or squamous cell carcinoma. Patients with high cumulative levels of sun exposure should be encouraged to wear sunscreen and protective clothing, although the benefit of such counselling is unknown.’

ine skin examination was effective at reducing the morbidity and mortality from cutaneous melanoma, basal cell carcinoma, or squamous cell carcinoma. Patients with high cumulative levels of sun exposure should be encouraged to wear sunscreen and protective clothing, although the benefit of such counselling is unknown.’ Here, clinicians are not told whether the advisement’s basis in EBM is because there is no direct evidence or because there is direct evidence, but it is unfavourable. Yet, this distinction is of key importance for integrating evidence with expertise to implement EBM and guiding decisions. No direct evidence could just as easily lead to screening or no screening, whereas direct unfavourable evidence should lead to no screening in the absence of a compelling reason to do otherwise. Second, Advisement 2: ‘Routine screening for CAD is not recommended. Resting electrocardiography, exercise treadmill testing, and electron-beam CT may identify some patients with asymptomatic disease, but these strategies lack supportive evidence.’

Here, clinicians are not told whether the advisement’s basis in EBM is because there is no direct evidence or because there is direct evidence, but it is unfavourable. Yet, this distinction is of key importance for integrating evidence with expertise to implement EBM and guiding decisions. No direct evidence could just as easily lead to screening or no screening, whereas direct unfavourable evidence should lead to no screening in the absence of a compelling reason to do otherwise. Second, Advisement 2: ‘Routine screening for CAD is not recommended. Resting electrocardiography, exercise treadmill testing, and electron-beam CT may identify some patients with asymptomatic disease, but these strategies lack supportive evidence.’ Again, clinicians are not told whether the advisement’s basis is because there is no direct evidence or because there is direct evidence, but it is unfavourable, thereby muddying inferences for decision-making. For example, should a clinician perform an ECG on an asymptomatic patient with an expected 10-year incidence of cardiac disease greater than 10%? One could argue that such a patient would not be ‘routine’ and therefore should be excluded from the prescribed inference by the ‘routine screening’ clause in the recommendation, but an improved method of disseminating EBM would remove this ambiguity.

h an expected 10-year incidence of cardiac disease greater than 10%? One could argue that such a patient would not be ‘routine’ and therefore should be excluded from the prescribed inference by the ‘routine screening’ clause in the recommendation, but an improved method of disseminating EBM would remove this ambiguity. Clinical resources that deviate from EBM principles are not unique to MKSAP, but rather are endemic. A well-regarded reference, UpToDate, routinely issues advisements with oblique relationships to EBM; for example, advising melanoma screening for high-risk persons after describing the uncertainty that screening lowers mortality and the biases in supporting studies.6 Indeed, failing to discriminate between no evidence and unfavourable evidence is similar to indiscriminate use of the phrase ‘there is no evidence to suggest’,7 which is also endemic and bears the imprint of contexts removed from medicine where there is an implicit desire for a contrary bias (eg, legal proceedings, where innocence is presumed in the absence of evidence of guilt). How might clinician resources become more aligned with EBM? It is not an easy question—media like those cited above are valued precisely because they transform difficult, often grey-shaded questions into digestible and actionable maxims. Nonetheless, I believe that substantial improvement is possible.

Indeed, failing to discriminate between no evidence and unfavourable evidence is similar to indiscriminate use of the phrase ‘there is no evidence to suggest’,7 which is also endemic and bears the imprint of contexts removed from medicine where there is an implicit desire for a contrary bias (eg, legal proceedings, where innocence is presumed in the absence of evidence of guilt). How might clinician resources become more aligned with EBM? It is not an easy question—media like those cited above are valued precisely because they transform difficult, often grey-shaded questions into digestible and actionable maxims. Nonetheless, I believe that substantial improvement is possible. One approach to improve the fidelity of disseminating EBM It is possible to design a taxonomy focused on EBM implementation that could be adopted by sources of continuing clinical education and other clinician resources. The key is to make it simple but not stupid. I will suggest one possible taxonomy which differs from other EBM-related taxonomies8–12 because it is designed around distinct inferences for clinical decisions rather than around distinct tiers of evidence quality.

ntinuing clinical education and other clinician resources. The key is to make it simple but not stupid. I will suggest one possible taxonomy which differs from other EBM-related taxonomies8–12 because it is designed around distinct inferences for clinical decisions rather than around distinct tiers of evidence quality. First, define a ‘benefit harm expectancy’ (BHE) as an informal qualitative assessment of balance of benefits to harms that may be based on clinical expertise or other subjective factors in the absence of scientific evidence. Categorical evaluations of BHE should be rank ordered and qualitative, such as ‘favourable’, ‘unfavourable’ or ‘unclear’. The notion of BHE is then a keystone that makes possible the next step, iterating an easily understood taxonomy of EBM-based inferences that is mutually exclusive and collectively exhaustive (table 1). Note that the quantitative analogue of BHE is a Bayesian ‘prior probability distribution’, but because this quantitative formalisation is difficult to perform in practice and is fraught with estimation error, I will not consider it further. The table 1 illustrates how the evidence summaries in advisements 1 and 2 are compatible with at least three very different EBM-based inferences. Rather than being ambiguous, advisements should clarify which inference(s) are sound, for example, employing phrases in the second column of the table 1 such as ‘probably indicated until better evidence emerges’. Table 1 Taxonomy of EBM-based inferences

First, define a ‘benefit harm expectancy’ (BHE) as an informal qualitative assessment of balance of benefits to harms that may be based on clinical expertise or other subjective factors in the absence of scientific evidence. Categorical evaluations of BHE should be rank ordered and qualitative, such as ‘favourable’, ‘unfavourable’ or ‘unclear’. The notion of BHE is then a keystone that makes possible the next step, iterating an easily understood taxonomy of EBM-based inferences that is mutually exclusive and collectively exhaustive (table 1). Note that the quantitative analogue of BHE is a Bayesian ‘prior probability distribution’, but because this quantitative formalisation is difficult to perform in practice and is fraught with estimation error, I will not consider it further. The table 1 illustrates how the evidence summaries in advisements 1 and 2 are compatible with at least three very different EBM-based inferences. Rather than being ambiguous, advisements should clarify which inference(s) are sound, for example, employing phrases in the second column of the table 1 such as ‘probably indicated until better evidence emerges’. Table 1 Taxonomy of EBM-based inferences Evidence classification Inference Relevant to advisements in text? No direct evidence—but BHE favourable Probably indicated until better evidence emerges. No No direct evidence—but BHE unfavourable Probably not indicated until better evidence emerges. Possibly 1 and 2 Direct evidence favourable but uncertain (not tested with sufficient statistical power to rule-out a clinically significant difference and/or unexplained differences in direction of effect). If BHE is favourable, indicated until better evidence emerges. If BHE is unfavourable, then there may be some indications for its use but caution is necessary. No Direct evidence unfavourable but uncertain (not tested with sufficient statistical power to rule-out a clinically significant difference and/or unexplained differences in direction of effect). If BHE is favourable, there may be some indications for its use but caution is necessary until better evidence emerges. If BHE is unfavourable, it is probably not indicated until better evidence emerges. Possibly 1 and 2 Direct evidence favourable (tested with sufficient statistical power to rule-out a clinically significant difference). BHE not relevant unless encompasses reasons unconsidered during evidence collection. Indicated except for those rationales. No Direct evidence unfavourable (tested with sufficient statistical power to rule-out a clinically significant difference). BHE not relevant unless encompasses reasons unconsidered during evidence collection. Not indicated except for those rationales. Possibly 1 and 2 See text for description.

ept for those rationales. No Direct evidence unfavourable (tested with sufficient statistical power to rule-out a clinically significant difference). BHE not relevant unless encompasses reasons unconsidered during evidence collection. Not indicated except for those rationales. Possibly 1 and 2 See text for description. BHE, benefit to harm expectancy; EBM, evidence-based medicine. While every clinical situation will never be ‘map-able’ to a specified set of inferences, higher fidelity dissemination of EBM would improve the quality of inferences in a large variety of scenarios. Prescribe an inference to a clinician, help her to make a decision for that day; teach a clinician how to make inferences, teach her to make EBM-based decisions for her whole career. It may be observed that the strategy advocated here simply embodies a ‘poor man’s’ use of Bayesian logic in the absence of estimable probability distributions. It also may be observed that similar roads have been travelled by EBM investigators that seek to use formal methods to synthesise a body of evidence. But it is long-past due for the conceptual underpinnings of EBM to be disseminated more broadly and with greater fidelity and to be implemented by a wider cross-section of practising physicians.

that similar roads have been travelled by EBM investigators that seek to use formal methods to synthesise a body of evidence. But it is long-past due for the conceptual underpinnings of EBM to be disseminated more broadly and with greater fidelity and to be implemented by a wider cross-section of practising physicians. Limitations Some may contend that because much clinical decision-making is pattern recognition and/or subject to many well-described biases,13 there is no point in trying to impose on it or embed within it an explicitly rational structure. But all too often, these biases are invoked a specious effort to counter any endeavour to improve the transparency or clarity of clinical decision-making. Just because clinicians are not explicitly rational thinkers all the time (nor should they be), it does not mean they should not be rational thinkers sometimes, particularly when scientific evidence is robust. After all, what patient would choose a clinician that never thinks rationally? Additionally, it is important to note that my perspective is not universally held. Some would consider BHEs no more easy to formulate than quantitative specifications of prior probability distributions, especially if any new evidence is viewed sceptically.14 Others would omit experiential data from EBM on the grounds that it is not really evidence. Summary EBM should be more than just a slogan. The taxonomy described here could facilitate its dissemination and anchor its role in routine clinical practice. Contributors: RSB conceived the project and drafted the manuscript.

Limitations Some may contend that because much clinical decision-making is pattern recognition and/or subject to many well-described biases,13 there is no point in trying to impose on it or embed within it an explicitly rational structure. But all too often, these biases are invoked a specious effort to counter any endeavour to improve the transparency or clarity of clinical decision-making. Just because clinicians are not explicitly rational thinkers all the time (nor should they be), it does not mean they should not be rational thinkers sometimes, particularly when scientific evidence is robust. After all, what patient would choose a clinician that never thinks rationally? Additionally, it is important to note that my perspective is not universally held. Some would consider BHEs no more easy to formulate than quantitative specifications of prior probability distributions, especially if any new evidence is viewed sceptically.14 Others would omit experiential data from EBM on the grounds that it is not really evidence. Summary EBM should be more than just a slogan. The taxonomy described here could facilitate its dissemination and anchor its role in routine clinical practice. Contributors: RSB conceived the project and drafted the manuscript. Competing interests: None declared. Patient consent: Not required. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Unnecessary and excessive use of antibiotics is the main driver of antimicrobial resistance.1 The prospect of patients dying from common infections caused by multiresistant bacteria in the long term is grim unless the situation is reversed by limiting the use of antibiotics only to patients in whom the benefits outweigh the harms. Since the publication of the book ‘Overdiagnosed: Making People Sick in the Pursuit of Health’ by Welch et al in 20112 and the launching of the first International conference on overdiagnosis in 2013, overdiagnosis has become a widely used term to address the drivers of ‘too much medicine’. Well-defined concepts are essential to advance understanding and to find concrete solutions to a problem. In that sense, Brodersen et al published a paper about what is and is not overdiagnosis.3 In this paper, overdiagnosis is defined as the diagnosis of a condition, that would otherwise not cause symptoms or harm to a patient during his or her lifetime’. They go further by arguing that terms such as overtesting, misdiagnosis and false-positives do not belong to the overdiagnosis paradigm.

not overdiagnosis.3 In this paper, overdiagnosis is defined as the diagnosis of a condition, that would otherwise not cause symptoms or harm to a patient during his or her lifetime’. They go further by arguing that terms such as overtesting, misdiagnosis and false-positives do not belong to the overdiagnosis paradigm. This assertion has raised more questions than answers within the clinical and research area regarding infectious diseases and the use of antibiotics. In this paper, we summarise the main points discussed during the workshop ‘Overdiagnosis and overtreatment of infectious diseases in general practice. How and where to break the endless loop?’.4 During the workshop, participants from all over the world reflected on the challenges of embracing the overdiagnosis paradigm as a tool to advance understanding and to find solutions to the unnecessary use of antibiotics in primary healthcare. It was concluded that the current definition of overdiagnosis is not suitable and falls short of addressing the complexity of the drivers and consequences of excessive and unnecessary use of antibiotics. On the contrary, the ‘too much medicine’ paradigm5 gives space for a broader understanding of overdiagnosis as errors during the diagnostic process, which are inherently connected to the error in the treatment decision called overtreatment.

f the drivers and consequences of excessive and unnecessary use of antibiotics. On the contrary, the ‘too much medicine’ paradigm5 gives space for a broader understanding of overdiagnosis as errors during the diagnostic process, which are inherently connected to the error in the treatment decision called overtreatment. Overdiagnosis versus ‘too much medicine’ The current definition of overdiagnosis is based on long-term or chronic conditions and asymptomatic patients. It markedly differs from the type of population managed in primary healthcare due to a suspected acute infectious disease. First of all, all the patients seeking care at primary healthcare have atypical or typical symptoms, a type of perceived change that triggers the need for seeking healthcare. Second, the focus of benefits against harms is mostly driven by outweighing the benefit of shortening the length of time with symptoms against the harms due to the development of adverse effects including the development of antimicrobial resistance.6 7 Therefore, neither the overdiagnosis of an asymptomatic patient nor the development of unforeseen long-term harms is the main problem regarding the diagnosis and subsequent management of patients with a suspected acute infection. The main problem is the interplay of multilevel factors that influence the diagnostic process and treatment decision, which leads to overtreatment.8 From this perspective, the ‘too much medicine’ paradigm is an umbrella term that gives room for recognising the singularities of each error while reminding us that they are part of a bigger picture.

erplay of multilevel factors that influence the diagnostic process and treatment decision, which leads to overtreatment.8 From this perspective, the ‘too much medicine’ paradigm is an umbrella term that gives room for recognising the singularities of each error while reminding us that they are part of a bigger picture. Looking for solutions to the unnecessary use of antibiotics is far from being an easy task. Determinants and actions targeting diagnostic errors influence the treatment decision and vice versa. Understanding the components of the bigger picture and the challenges of the diagnostic errors within the ‘too much medicine’ paradigm are crucial to bringing about effective solutions. The bigger picture Worldwide, the patient–doctor encounter in primary care faces similar challenges such as (a) short consultation time, (b) high uncertainty of the origin of the symptoms, (c) uncertainty of the evolution of symptoms and (d) organisational challenges for following the patient.9 10 During the patient–doctor encounter, patients’ expectations and general practitioners’ (GP) attitudes toward these expectations are drivers of ‘too much medicine’. As mentioned above, there are typical and atypical symptoms that trigger an active search for healthcare. Patients/the family expect something from their GP. It can include a check-up, reassurance, a medical certificate, antibiotic therapy or simply guidance on symptom management.

ient expectations.11 However, while all patients are believed to expect something, only some patients expect antibiotics, and among these patients, a subset demands antibiotics. Notwithstanding, patient expectations may be based on false assumptions, with many overestimating the effectiveness of antibiotic treatment.12 It is very important for GPs to meet patients’ expectations, as it is a way to preserve a good doctor–patient relationship. Prescribing increases patient satisfaction and reinforces the doctor–patient relationship as clinicians often feel prescribing as less stressful than an anxious patient or parent.13 Some clinicians do not feel that it is worth jeopardising their relationship with a patient or parent over a relatively minor matter of prescribing antibiotics. The consequences for the future relationship with patients are more of a concern for GPs prescribing antibiotics than antibiotic resistance.9 GPs are generally aware of and concerned about the threat that antimicrobial resistance poses and agree that this is a growing problem. However, antimicrobial resistance is considered more of a public health issue, whereas the GPs’ priority is to be responsive to the needs of the individual patient. This example shows the multidimensional factors that play a role in the decision whether to prescribe antibiotics. There is a patient expecting a solution to his/her problems. There is a GP expecting to be responsive to the needs of the patient. There is a system expecting a rational use of resources and contention of the antimicrobial resistance problem.

al factors that play a role in the decision whether to prescribe antibiotics. There is a patient expecting a solution to his/her problems. There is a GP expecting to be responsive to the needs of the patient. There is a system expecting a rational use of resources and contention of the antimicrobial resistance problem. The challenges of the diagnostic errors Mislabelling or misdefinition,14 not considered as overdiagnosis under the definition of overdiagnosis of Brodersen et al, is an error during the diagnostic process. However, it is difficult to know whether mislabelling lead to overtreatment or vice versa. Due to the short time of the consultation in primary care, it is difficult to determine the logical order of the decision taken by the GP. Sometimes a GP mislabels the patient with another diagnosis to better justify the administration of antibiotic therapy, for which treatment is unnecessary. A clear example is the differentiation of a sore throat into two different labels (pharyngitis and tonsillitis). Patients are diagnosed with tonsillitis if the clinicians intend to treat a sore throat with antibiotics and they use the milder category of pharyngitis if they think that the infection has a viral cause. In extreme cases, GPs decide to prescribe antibiotics even before examining the patient.15 It goes against logical thinking as treatment should be the subsequent step after a systematic diagnostic process.

th antibiotics and they use the milder category of pharyngitis if they think that the infection has a viral cause. In extreme cases, GPs decide to prescribe antibiotics even before examining the patient.15 It goes against logical thinking as treatment should be the subsequent step after a systematic diagnostic process. It implies that the current system to assess overtreatment is quite inaccurate. Future research should focus on finding better ways to assess overtreatment and the extent of mislabelling. For example, the assessment should be based on the presence of diagnostic information (eg, cluster of signs and symptoms, results of rapid tests) as well as prognostic criteria (eg, age and comorbidities) rather than the label given by the prescriber. Other examples of the broader vision of errors in the diagnostic process connected to the decision to treat are overtesting and misinterpretation of diagnostic information (ie, false positives). These terms are not considered as overdiagnosis under the Brodersen et al definition of overdiagnosis.

It implies that the current system to assess overtreatment is quite inaccurate. Future research should focus on finding better ways to assess overtreatment and the extent of mislabelling. For example, the assessment should be based on the presence of diagnostic information (eg, cluster of signs and symptoms, results of rapid tests) as well as prognostic criteria (eg, age and comorbidities) rather than the label given by the prescriber. Other examples of the broader vision of errors in the diagnostic process connected to the decision to treat are overtesting and misinterpretation of diagnostic information (ie, false positives). These terms are not considered as overdiagnosis under the Brodersen et al definition of overdiagnosis. Currently, there is no single item of the diagnostic information (eg, signs, symptoms and results of rapid tests) able to predict whether an infection has a viral or a bacterial origin with 100% certainty. Systematic reviews about the diagnostic accuracy of signs and symptoms for the most common infections managed in primary care16 17 have shown that the specificity of signs and symptoms is low. The cluster of symptoms like the Centor criteria—used to predict the bacterial origin of a sore throat18—or the cluster of symptoms to identify patients with influenza-like symptoms,19 who can benefit from early antiretroviral therapy can increase predictive accuracy, but it depends on the prevalence of the disease.20 Thus, a treatment decision only based on signs and symptoms is likely to yield a high proportion of false-positives, resulting in overtreatment.

patients with influenza-like symptoms,19 who can benefit from early antiretroviral therapy can increase predictive accuracy, but it depends on the prevalence of the disease.20 Thus, a treatment decision only based on signs and symptoms is likely to yield a high proportion of false-positives, resulting in overtreatment. The use of rapid tests has been advocated as a solution to increase diagnostic accuracy, thereby decreasing overtreatment.21 However, the introduction of this technology in primary care is in its infancy with a lot of controversy about the added value.22 23 Models to understand the factors influencing the proper use and added value of rapid testing in primary care have demonstrated a complex interaction between availability, interpretation and easy-to-use factors.24 Currently, rapid tests to support the diagnostic process in patients with a suspected infection seeking help in primary care are available in very few countries. In most of the world, the decision of whether to prescribe antibiotics is taken based only on signs and symptoms.

Models to understand the factors influencing the proper use and added value of rapid testing in primary care have demonstrated a complex interaction between availability, interpretation and easy-to-use factors.24 Currently, rapid tests to support the diagnostic process in patients with a suspected infection seeking help in primary care are available in very few countries. In most of the world, the decision of whether to prescribe antibiotics is taken based only on signs and symptoms. One of the few countries with the wide availability of rapid tests is Denmark. This scenario allowed us to assess the use of diagnostic tools in daily practice and its effect on treatment decision in patients with suspected urinary tract infection (UTI).25 We found that in 85% of the patients with a suspected uncomplicated UTI, urine culture was part of the diagnostic process. Many international guidelines recommend the use of urine culture only in patients with suspected complicated UTI; then this 85% could be regarded as overtesting. However, overtreatment was lower in the group of patients in which urine culture was performed in comparison with the group of patients in which antibiotics were prescribed based only on signs and symptoms. To disentangle the causal path between the use of rapid test and overtreatment is not straightforward. Thus, there is a growing awareness about the fact that the assessment of the causal pathway between the use of a rapid test and the effect on overtreatment needs re-structuring. The field of diagnostic test evaluation requires further development and operationalisation of variables within the ‘too much medicine’ paradigm if the rapid tests are to become systematically introduce in primary healthcare worldwide.

etween the use of a rapid test and the effect on overtreatment needs re-structuring. The field of diagnostic test evaluation requires further development and operationalisation of variables within the ‘too much medicine’ paradigm if the rapid tests are to become systematically introduce in primary healthcare worldwide. Conclusion Overall, it can be seen that the overdiagnosis paradigm is unrelated to these multidimensional factors. The type of population, as well as the dilemmas regarding benefits and harms in the management of patients with suspected infections, differ in crucial aspects of the population and dilemmas covered by the overdiagnosis paradigm. The umbrella term of too much medicine is broader and allows the use of theories and strategies without getting trapped into the dilemmas of trying to squeeze the multidimensional factors of diagnostic error and overtreatment into the narrow definition of what is and is not overdiagnosis. The authors would like to thank all the participants of the workshop ‘Overdiagnosis and overtreatment of infectious diseases in general practice. How and where to break the endless loop?’ for sharing their experience about the challenges in everyday practice. Contributors: GC: had the idea for the article. GC and CL: performed the literature search; wrote the article. Funding: This study was in part supported by læge Sofus Carl Emil Friis og Hustru Olga Doris Friis’ legat and University Institute in Primary Care Research Jordi Gol. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Contributors: GC: had the idea for the article. GC and CL: performed the literature search; wrote the article. Funding: This study was in part supported by læge Sofus Carl Emil Friis og Hustru Olga Doris Friis’ legat and University Institute in Primary Care Research Jordi Gol. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not required.

Introduction Natriuretic peptides (NP) are released by the myocardium in response to pressure or fluid overload and are raised in patients with heart failure (HF). NP testing is currently used in diagnosis, but its role in guiding HF treatment remains controversial. In August 2017, the long-awaited GUIDE-IT study was published by Felker and colleagues.1 It is the largest study of NP monitoring to date, recruiting 894 of the planned 1100 participants, and was stopped early due to futility. Lack of efficacy for the biomarker-guided treatment group compared with usual care was based on the primary end point, a composite outcome of HF admission and cardiovascular mortality. The conclusion that NP-guided treatment is not more effective than usual care has been rapidly disseminated with the paper viewed over 12 000 times in the first 4 months following publication and 22 000 to date. While the authors did not comment on the implications for practice, the emerging consensus favours an interpretation that ‘any plausible effects of an intervention are unlikely to be worthwhile’.2 In response to the publication of the GUIDE-IT study,1 Bajaj et al 3 published a research letter with a meta-analysis reporting all-cause mortality as significantly lower in the NP-guided group. This rapid analysis only collated evidence from six studies, obtained from one database for a single outcome. Although there tends to be good consistency between the single largest study and pooled estimates from a full systematic review,4 it is good practice to frame the newest evidence in context of the previous information.5 This helps validate new findings and if done by updating a previous meta-analysis, will allow for higher precision of the estimates, reduced variability and increased generalisability.

led estimates from a full systematic review,4 it is good practice to frame the newest evidence in context of the previous information.5 This helps validate new findings and if done by updating a previous meta-analysis, will allow for higher precision of the estimates, reduced variability and increased generalisability. Systematic reviews carried out in this area have proved inconclusive, providing conflicting results. Five out of 11 systematic reviews with meta-analyses have demonstrated that NP-guided treatment reduces all-cause mortality in patients with HF compared with usual clinical care,6–10 while the more recent systematic reviews, including two with individual patient data, have published meta-analyses contradicting this finding and showing no effect.11–14 One of the individual patient data analyses, Troughton et al,11 was able to adjust for patient characteristics and compared time to all-cause mortality between NP-guided and clinically-guided treatment groups and then reported a reduction in all-cause mortality. The most comprehensive meta-analysis12 to date, prior to the GUIDE-IT study,1 included 3660 participants from 18 studies. Data from GUIDE-IT1 increases the amount of evidence for all-cause mortality by over 25% (previously 3169 participants from 15 studies) and for HF hospital admission by 45% (previously 1928 participants from 10 studies). These two outcomes, along with adverse events, are the only outcomes where new evidence has been obtained in comparison to the most comprehensive meta-analysis to date. Therefore, this study aims to incorporate the GUIDE-IT study into an update of this meta-analysis12 to determine if the question of whether NP-guided treatment of patients with HF improves all-cause mortality, HF hospital admissions and adverse events, compared with clinical assessment alone, has been finally answered.

Therefore, this study aims to incorporate the GUIDE-IT study into an update of this meta-analysis12 to determine if the question of whether NP-guided treatment of patients with HF improves all-cause mortality, HF hospital admissions and adverse events, compared with clinical assessment alone, has been finally answered. Methods This systematic review and meta-analysis was conducted in accordance with the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) guidelines.15 The protocol was published in the Cochrane library.16 Data searches and study selection The same search strategy was used as in the original review13 (online supplementary appendix 1). The following databases were searched up to 27 November 2017: Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, Database of Abstracts of Reviews of Effects, NHS Economic Evaluation Database, Science Citation Index Expanded and the Conference Proceedings Citation Index on Web of Science, Clinical trials Registry and the WHO International Clinical Trials registry Platform. No restriction was made in terms of year published, language or publication status. All publications were independently double screened first by title and abstract and then by full text (JM, CB, AF). Discrepancies were resolved by consensus. 10.1136/bmjebm-2019-111208.supp1Supplementary data

Data searches and study selection The same search strategy was used as in the original review13 (online supplementary appendix 1). The following databases were searched up to 27 November 2017: Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, Database of Abstracts of Reviews of Effects, NHS Economic Evaluation Database, Science Citation Index Expanded and the Conference Proceedings Citation Index on Web of Science, Clinical trials Registry and the WHO International Clinical Trials registry Platform. No restriction was made in terms of year published, language or publication status. All publications were independently double screened first by title and abstract and then by full text (JM, CB, AF). Discrepancies were resolved by consensus. 10.1136/bmjebm-2019-111208.supp1Supplementary data All randomised controlled trials (RCT) of NP-guided treatment of HF, both in and out of hospital setting, reporting a clinical outcome were included. NP is a collective term for N-terminal pro B-type natriuretic peptide (NT-proBNP) and B-type natriuretic peptide (BNP). Studies compared NP-guided treatment to treatment guided by clinical assessment alone. There was no restriction on the length of follow-up. Participants less than 18 years old were excluded. The outcomes for the systematic review were prespecified. The primary outcome was all-cause mortality, while HF admission was the secondary outcome.

-guided treatment to treatment guided by clinical assessment alone. There was no restriction on the length of follow-up. Participants less than 18 years old were excluded. The outcomes for the systematic review were prespecified. The primary outcome was all-cause mortality, while HF admission was the secondary outcome. Data extraction and quality assessment For each included RCT, data were extracted independently by two people and any discrepancies resolved by consensus (JM, CB). Data extracted included baseline characteristics (gender, age, New York Heart Association (NYHA) classification and left ventricular ejection fraction (LVEF), setting, inclusion and exclusion criteria and descriptions of the intervention and clinical assessment arms including: NP biomarker used, target NP level and drug algorithms for uptitration of HF medication. Assessment of the methodological quality of the studies was made using the Cochrane Risk of bias tool.17 The outcomes were extracted as absolute number of events. If a study compared more than one type of control group, then the intervention group data were split equally between the control groups for both outcome events and sample size. Reporting bias was assessed using funnel plots.

ing the Cochrane Risk of bias tool.17 The outcomes were extracted as absolute number of events. If a study compared more than one type of control group, then the intervention group data were split equally between the control groups for both outcome events and sample size. Reporting bias was assessed using funnel plots. Data synthesis and analysis Trial level data were pooled using Review Manager V.5.3.18 For binary outcomes, data were combined using a Mantzel-Haenzel fixed-effect model to obtain a summary estimate of the risk ratio (RR) with 95% CI. The I2 statistic was used to assess the level of statistical heterogeneity.17 Where substantial heterogeneity (I2≥50%) was present, a random-effects model was used to test the robustness of the findings. Subgroup analyses were considered for age, NYHA class, baseline NP, target NP, achieved NP decrease, setting, gender, LVEF, duration of FU, control type and biomarker used. Blinding of outcome assessment and attrition were judged to potentially impact on the pooled results and therefore sensitivity analyses were conducted restricted to studies with low risk of bias in these two domains.

arget NP, achieved NP decrease, setting, gender, LVEF, duration of FU, control type and biomarker used. Blinding of outcome assessment and attrition were judged to potentially impact on the pooled results and therefore sensitivity analyses were conducted restricted to studies with low risk of bias in these two domains. Results From 4152 references (updated and original search), 357 full texts were assessed, and 19 studies involving a total of 4554 participants met the inclusion criteria (figure 1). This update to the Cochrane systematic review13 identified one new study: the GUIDE-IT study.1 The recently published PRIMA II study19 was excluded as this aimed to reduce NP level between hospital admission and discharge in patients with acute HF rather than ongoing NP-guided treatment. Baseline characteristics of included RCTS and patients are given in online supplementary appendices 2 and 3. Each of the 19 studies reported on one or more of the included outcomes. Two of the 19 studies had three comparison arms comparing NP-guided treatment both to clinical assessment and to usual care. Figure 1 Flowchart of study selection. BNP, B-type natriuretic peptide; HF, heart failure; NT-proBNP, N-terminal pro B-type natriuretic peptide.

Results From 4152 references (updated and original search), 357 full texts were assessed, and 19 studies involving a total of 4554 participants met the inclusion criteria (figure 1). This update to the Cochrane systematic review13 identified one new study: the GUIDE-IT study.1 The recently published PRIMA II study19 was excluded as this aimed to reduce NP level between hospital admission and discharge in patients with acute HF rather than ongoing NP-guided treatment. Baseline characteristics of included RCTS and patients are given in online supplementary appendices 2 and 3. Each of the 19 studies reported on one or more of the included outcomes. Two of the 19 studies had three comparison arms comparing NP-guided treatment both to clinical assessment and to usual care. Figure 1 Flowchart of study selection. BNP, B-type natriuretic peptide; HF, heart failure; NT-proBNP, N-terminal pro B-type natriuretic peptide. Outcomes Sixteen studies (n=4063) reported results that could be pooled for all-cause mortality. During follow-up, 331 out of 1929 participants (18%) died in the NP-guided treatment groups compared with 445 out of 2134 (22%) in the control groups. When pooled, the evidence favours the NP-guided treatment (RR 0.87, 95% CI 0.77 to 0.99). Heterogeneity was low indicating consistency across the included studies (I2=11%) (figure 2). Figure 2 Forest plot comparing NP-guided treatment versus control for all-cause mortality. NP, natriuretic peptide.

Outcomes Sixteen studies (n=4063) reported results that could be pooled for all-cause mortality. During follow-up, 331 out of 1929 participants (18%) died in the NP-guided treatment groups compared with 445 out of 2134 (22%) in the control groups. When pooled, the evidence favours the NP-guided treatment (RR 0.87, 95% CI 0.77 to 0.99). Heterogeneity was low indicating consistency across the included studies (I2=11%) (figure 2). Figure 2 Forest plot comparing NP-guided treatment versus control for all-cause mortality. NP, natriuretic peptide. Eleven studies with 2822 participants reported on HF admission. Out of 1304 participants, 366 (28%) experienced a HF admission in the NP-guided treatment groups compared with 544 out of 1518 (36%) participants in the control groups. Overall, the pooled evidence showed an effect favouring NP-guided treatment (RR 0.80, 95% CI 0.72 to 0.89) (figure 3). Heterogeneity was substantial (I2=67%). The robustness of this finding was tested by converting to a random-effects model; the effect remained consistent (RR 0.70, 95% CI 0.56 to 0.88). Figure 3 Forest plot comparing NP-guided treatment versus control for heart failure admission. NP, natriuretic peptide. Adverse events were reported as low and similar between groups in the GUIDE-IT study; the data were added to the tabulated data for all studies (online supplementary appendix 4). Data were not pooled for this outcome as it was possible to have multiple events per individual.

Figure 3 Forest plot comparing NP-guided treatment versus control for heart failure admission. NP, natriuretic peptide. Adverse events were reported as low and similar between groups in the GUIDE-IT study; the data were added to the tabulated data for all studies (online supplementary appendix 4). Data were not pooled for this outcome as it was possible to have multiple events per individual. Quality assessment of the studies is summarised in online supplementary appendices 5 and 6. The risk of bias was assessed to be high or unclear for the majority of studies across all domains, except selection bias. Blinding of outcome assessment was judged to be poor with only 36% of studies assessed as low risk of bias. Attrition bias was assessed as low risk in 42% of studies. Except for age, duration of follow-up and biomarker used, it was not possible to explore subgroups within the study populations. Data were reported as totals for whole study or in varying categories or averages for intervention and control groups. Although the GUIDE-IT study1 reported subgroup data for participants above and below 75, it was only for the study’s primary composite outcome. Therefore, the analysis in the original systematic review13 remained the same, which found uncertainty in the evidence for NP-guided treatment for all-cause mortality when examining under or over 75 years of age and a reduction in HF admission.

above and below 75, it was only for the study’s primary composite outcome. Therefore, the analysis in the original systematic review13 remained the same, which found uncertainty in the evidence for NP-guided treatment for all-cause mortality when examining under or over 75 years of age and a reduction in HF admission. Subgroup analyses were completed for duration of the intervention and NP biomarker used. For duration of the intervention, this shows that both at ≤1 and 1–2 years, there was a potential reduction for all-cause mortality, but the evidence showed uncertainty at>2 years. The effect of duration on HF admission showed a similar trend for each subgroup for the same durations (online supplementary appendices 7 and 8). Analyses completed for the biomarker used suggested no difference between the biomarker used and between the main findings for both all-cause mortality and HF admissions. However, the subgroups for both biomarkers for all-cause mortality were no longer statistically significant (online supplementary appendices 9 and 10). Sensitivity analyses were completed restricting studies to those with low risk of bias for blinding of outcome assessment and attrition. The GUIDE-IT study1 was assessed to be low risk of bias for blinding of the outcome assessment and therefore adding these data in the sensitivity analyses found for both all-cause mortality and HF admission the effect continued to favour NP-guided treatment, but was no longer statistically significant (all-cause mortality: RR 0.92, CI 0.79 to 1.06; HF admission: RR 0.92, CI 0.81 to 1.4).

f the outcome assessment and therefore adding these data in the sensitivity analyses found for both all-cause mortality and HF admission the effect continued to favour NP-guided treatment, but was no longer statistically significant (all-cause mortality: RR 0.92, CI 0.79 to 1.06; HF admission: RR 0.92, CI 0.81 to 1.4). For the risk of attrition bias, the GUIDE-IT study1 was assessed to be unclear and therefore provided no additional data to the original systematic review.13 Analyse restricted to studies with low risk of attrition bias showed similar results to blinding of outcome assessment for all-cause mortality and HF admissions, though it was no longer statistically significant for all-cause mortality.

lear and therefore provided no additional data to the original systematic review.13 Analyse restricted to studies with low risk of attrition bias showed similar results to blinding of outcome assessment for all-cause mortality and HF admissions, though it was no longer statistically significant for all-cause mortality. Discussion This meta-analysis, including all the available evidence, indicates that NP-guided treatment can improve all-cause mortality by 13% and reduce HF admissions by 20%. As expected, the addition of the GUIDE-IT study1 data, which increased the total number of participants included by 24%, substantially increased the precision of the estimates. It corroborates the conclusion of all previous meta-analyses, bar Pufulete et al,14 including the 2016 Cochrane systematic review13 for HF admissions which showed the intervention to be beneficial (online supplementary appendix 11). Perhaps unexpectedly, given that the study stopped early for futility, this meta-analysis contradicts recent previous reviews including the 2016 Cochrane systematic review13 and finds that the intervention is also beneficial for all-cause mortality (online supplementary appendix 11). Although the point estimate for the effect is the same as the 2016 Cochrace systematic review,13 with or without the GUIDE-IT study, the increase in precision means that the 95% CI is consistent with a significant reduction in all-cause mortality (change from CI (0.76 to 1.01) to CI (0.77 to 0.99)) on pooled data. A reduction in all-cause mortality of 13%, as shown in this meta-analysis, is a clinically important finding.

without the GUIDE-IT study, the increase in precision means that the 95% CI is consistent with a significant reduction in all-cause mortality (change from CI (0.76 to 1.01) to CI (0.77 to 0.99)) on pooled data. A reduction in all-cause mortality of 13%, as shown in this meta-analysis, is a clinically important finding. Clinical guidelines for HF management20–22 vary in their recommendations on NP-guided management for patients with HF reflecting the conflicting evidence in this area of research. The GUIDE-IT study1 had aimed to remove this uncertainty by recruiting an appropriate number of participants to detect a difference in clinically meaningful endpoints. Of relevance, the point estimate for all-cause mortality reported by the GUIDE-IT study1 showed the intervention to be marginally more effective (RR 0.86, CI 0.62 to 1.20) compared with that found in this meta-analysis. A sample size calculation indicates that in order to show an effect size of 13% reduction in all-cause mortality from a baseline risk of approximately 15% (as in the GUIDE-IT study1), a single study would need to include in excess of 10 000 participants to be sufficiently powered (≥80%). This meta-analysis currently includes just over 4000 participants. This is likely to have made all-cause mortality, as a single outcome, prohibitively expensive for the GUIDE-IT study justifying their use of a composite outcome. New studies are needed for even pooled evidence to have sufficient power.

y powered (≥80%). This meta-analysis currently includes just over 4000 participants. This is likely to have made all-cause mortality, as a single outcome, prohibitively expensive for the GUIDE-IT study justifying their use of a composite outcome. New studies are needed for even pooled evidence to have sufficient power. The GUIDE-IT authors suggest that the lack of effect may be due to a similar reduction in NP levels in both the intervention and control arms partly mediated by the control group having a higher number of clinic visits than usual as participants were managed in centres with HF expertise. The authors describe the study population as ‘high-risk’ and eligibility criteria included a HF event in the last 12 months (hospitalisation, A+E attendance or intravenous diuretic) and an NT-proBNP level over 2000 pg/mL in the prior 30 days. However, many of the other studies included in the meta-analysis recruited participants with NYHA class III-IV, recent hospital admission or a markedly raised NP level. It is therefore reasonable to consider the studies together.

ance or intravenous diuretic) and an NT-proBNP level over 2000 pg/mL in the prior 30 days. However, many of the other studies included in the meta-analysis recruited participants with NYHA class III-IV, recent hospital admission or a markedly raised NP level. It is therefore reasonable to consider the studies together. Nevertheless, the evidence for this effect is not yet robust, as indicated by the subgroup analysis comparing the type of NP biomarker and the sensitivity analyses restricting the evidence to studies assessed as having low risk of outcome assessment bias and low risk of attrition bias. There is still not sufficient evidence to conclude definitively on the effectiveness of NP-guided treatment in the management of HF, and particularly in less severely affected patients, such as those managed in primary care. Reporting of patient and study characteristics needs to be broader and more detailed to allow further exploration of study populations. Many of the interventions evaluated in the trials are multifaceted and as yet, we do not have clear evidence on the mechanism of action behind NP-guided treatment and which factors, if any, drive the change in salient health outcomes in NP-guided management of HF. Improved HF outcomes could be a result of clinical support indirectly intensified by NP-guided treatment rather than simply the monitoring of NP levels.23 Identification of key components within interventions linked to improving HF outcomes is needed. Research by Oke et al 24 attempted to provide information about the critical features of NP-guided monitoring of HF to reduce HF hospitalisation and suggested that interventions could be simplified. While unable to unequivocally determine an optimal intervention, they suggest that future intervention synthesis incorporating more data may provide clear procedural details of the essential elements of NP-guided monitoring.

toring of HF to reduce HF hospitalisation and suggested that interventions could be simplified. While unable to unequivocally determine an optimal intervention, they suggest that future intervention synthesis incorporating more data may provide clear procedural details of the essential elements of NP-guided monitoring. Limitations of the study While a thorough search with no date or language restrictions was performed, it is possible some studies may have been overlooked in searching and study selection. It was not possible to include data from one study for the primary outcome. Subgroups analyses of the data were limited due to the level of variation in how data were reported. In particular, it was not possible to complete subgroup analysis for LVEF which would have provided an indication of whether patients had HF with preserved ejection fraction (HFpEF) or HF with reduced ejection fraction (HFrEF). Studies did not define themselves as HFpEF or HFrEF, but all studies, at baseline, had either had an average LVEF<46% or inclusion criteria for the study of <40% (bar Maeder et al,25 though this study was not included in the meta-analyses). Subgroup analysis by age was possible, but only for three studies who reported data by age. However, since average ages were reported for each study, this analysis should be viewed with caution as it may mask different population distributions found across the included studies. To have an adequate exploration of the impact that age has on these outcomes, individual patient data arising from these studies would be required. Analysis of publication bias was limited due to the lack of studies. However, there is a suggestion of bias with a lack of smaller studies showing an effect favouring the control for the outcome of all-cause mortality (online supplementary appendix 12).

ividual patient data arising from these studies would be required. Analysis of publication bias was limited due to the lack of studies. However, there is a suggestion of bias with a lack of smaller studies showing an effect favouring the control for the outcome of all-cause mortality (online supplementary appendix 12). Conclusion The current pooled evidence indicates a beneficial effect of NP-guided therapy on all-cause mortality and on HF admissions. However, despite the publication of the GUIDE-IT study,1 the largest in this field, there is still insufficient evidence to reach a definitive conclusion on the effectiveness of NP-guided treatment in the management of HF. We would like to thank Nia Roberts (Information specialist, Bodleian Library, University of Oxford) for completing the library search and Alice Fuller (Research Officer, University of Oxford) for assisting with the title/abstract screening. Contributors: JM, RP and CB conceived and designed the study. JM and CB did the data screening, data extraction and quality assessment. JM, RP and CB did the statistical analyses. FDRH and CJT provided clinical expertise and input. All authors interpreted the data and drafted, critically appraised and approved the final manuscript.

JM, RP and CB conceived and designed the study. JM and CB did the data screening, data extraction and quality assessment. JM, RP and CB did the statistical analyses. FDRH and CJT provided clinical expertise and input. All authors interpreted the data and drafted, critically appraised and approved the final manuscript. Funding: This work was supported by funds for independent research funded by the National Institute for Health Research (NIHR) under the Programme Grants for Applied Research programme (RP-PG-1210-12003). RP and CB are supported by the NIHR Biomedical Research Centre, Oxford. FDRH acknowledges part-funding from the NIHR School for Primary Care Research, NIHR CLARHC Oxford, NIHR Oxford BRC and NIHR Oxford DEC. CJT is supported by an NIHR Academic Clinical Lectureship at the University of Oxford. Disclaimer: The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health). The study funders did not have any role in the study design, analysis or interpretation of data, in writing of the report or in the decision to submit the article for publication. All researchers were independent from funders. Competing interests: All authors have had financial support from the National Institute for Health Research (NIHR) for the submitted work. RH declared research support for a trial of HF diagnosis from Roche Diagnostics by supply of BNP assays in 2010/11. RP is an Associate Editor for BMJ Evidence-Based Medicine. Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not required.

Introduction A major challenge facing health professionals is the need to ensure that personal clinical knowledge is up-to-date to support effective patient care. This challenge is exacerbated by the ever-growing mass of clinical guidelines and available evidence,1 and further intensified in general practice given the broad role of clinicians in this setting. Not surprisingly, clinical questions at the point of care are common.2 Understanding the nature of clinical questions can help to ensure relevant and timely evidence for general practitioners (GPs); rather than ‘pushing’ evidence to GPs through disease-based guidelines, the information needs of clinicians can be used to drive the development of guidelines and resources that are relevant to their practice.

nature of clinical questions can help to ensure relevant and timely evidence for general practitioners (GPs); rather than ‘pushing’ evidence to GPs through disease-based guidelines, the information needs of clinicians can be used to drive the development of guidelines and resources that are relevant to their practice. A 2014 systematic review of questions raised by clinicians (physicians, medical residents, physician assistants, nurse practitioners, nurses, dentists and care managers) at the point of care identified 64 studies which classified clinical questions.2 The majority (75%) of included studies used ad hoc and informal classification approaches with poorly defined categories and methods, resulting in substantial variability between studies and precluding meaningful comparisons across time and geographical contexts.2 Five studies classified questions according to a formal taxonomy developed by Ely and colleagues3 and found that 34% of the questions asked were about drug treatments, and 24% were related to the potential causes of a symptom, physical finding or diagnostic test finding. Ely and colleagues’3 taxonomy is useful in providing an overview and comparison of the general nature of questions asked in clinical practice. However, as it is deliberately generic, it does not allow for a detailed exploration of the conditions, treatments or tests of interest to clinicians which may be more subject to change over time. In this way, its use may need to be complemented by other classification approaches.

questions asked in clinical practice. However, as it is deliberately generic, it does not allow for a detailed exploration of the conditions, treatments or tests of interest to clinicians which may be more subject to change over time. In this way, its use may need to be complemented by other classification approaches. Only three of the studies which classified clinical questions included in the systematic review2 were conducted in Australian settings,4–6 with the most recent article published over 10 years ago. Given that there have been major changes in Australian general practice over time including changes in GP and practice characteristics, reasons for encounters, problems managed and the management provided,7 clinical questions may have also changed over time. In addition, none of the 26 studies from the systematic review which analysed questions submitted to an information service used Ely’s taxonomy. This limits the comparisons which can be made between clinical questions simply arising from practice (without further action taken) and those which are prioritised for answering after the consultation. Given that clinicians pursue roughly half of their clinical questions2 and often do not pursue questions because they do not perceive them to be urgent or important,2 understanding the nature of those questions which are prioritised is a necessary step in facilitating evidence-based practice (EBP).

er the consultation. Given that clinicians pursue roughly half of their clinical questions2 and often do not pursue questions because they do not perceive them to be urgent or important,2 understanding the nature of those questions which are prioritised is a necessary step in facilitating evidence-based practice (EBP). Evidence-based information services are a way of supporting clinicians to answer their clinical questions, and seek to overcome some of the perceived barriers to EBP (see Sadeghi‐Bazargani et al,8 for an overview of barriers). Throughout 2017, we piloted an information service embedded within an EBP journal club9 in general practices in New South Wales and Queensland, Australia. This was part of a larger programme of work to support EBP and shared decision-making in general practice, and extends on previous efforts developing and evaluating ‘literature searching services’ for GPs.6 Five GP practices in New South Wales and Queensland were invited to participate in the information service, and to formulate and submit clinical questions arising from daily practice that they felt warranted answering and further discussion/critical appraisal in a subsequent journal club. These practices were purposively selected as early adopters of clinical advances and for comprising opinion leaders likely to influence national change. The aims of this study were to investigate the types of questions submitted by participating practices and characterise GPs’ information needs by classifying their questions using established and inductive coding systems.

pters of clinical advances and for comprising opinion leaders likely to influence national change. The aims of this study were to investigate the types of questions submitted by participating practices and characterise GPs’ information needs by classifying their questions using established and inductive coding systems. Methods In this study, we analysed clinical questions submitted by GPs participating in our information service and EBP journal club over the period of 1.5 years (between June 2016 and December 2017). Participating general practices submitted questions from clinical practice to the information service via an email from an assigned GP liaison. Questions were stored in an internal database and retrieved for coding in January 2018. Deductive and inductive coding methods were used to classify questions. Deductive coding: question type ICPC-2 PLUS The International Classification of Primary Care (ICPC) is a classification system for general practice. The first version (ICPC V.1)10 was published in 1987 with the second (ICPC-2) published by the World Organization of Family Doctors in 1997.11 ICPC-2 is accepted by the WHO in the WHO Family of International Classifications, and is the declared national standard in Australia for reporting of health data from general practice and patient self-reported health information.7 ICPC-2 PLUS is a clinical terminology and online interface classified to ICPC-2 and was used in this study.12

y the WHO in the WHO Family of International Classifications, and is the declared national standard in Australia for reporting of health data from general practice and patient self-reported health information.7 ICPC-2 PLUS is a clinical terminology and online interface classified to ICPC-2 and was used in this study.12 ICPC-2 is a multilevel coding system with 17 chapters: general (A), blood/blood forming (B), digestive (D), eye (F), ear (H), circulatory (K), musculoskeletal (L), neurological (N), psychological (P), respiratory (R), skin (S), metabolic/endocrine/nutritional (T), urinary (U), pregnancy/family planning (W), female genital (X), male genital (Y) and social (Z). Health data can also be classified with more granularity using ICPC-2 clinical terms. Examples of domains the terms may include are a disease label (eg, hypertension), a symptom (eg, cough) or a procedure (eg, dressing).12 Taxonomy of generic clinical questions Previous research by Ely and colleagues3 and Ebell and White13 developed and validated a taxonomy of generic clinical questions that classifies clinical questions into 1 of 64 generic types (eg, What is the cause of symptom x?; How good is test x in situation y?). Question types are broadly categorised as: diagnosis, treatment, management, epidemiology, non-clinical and non-classified.

oped and validated a taxonomy of generic clinical questions that classifies clinical questions into 1 of 64 generic types (eg, What is the cause of symptom x?; How good is test x in situation y?). Question types are broadly categorised as: diagnosis, treatment, management, epidemiology, non-clinical and non-classified. Analysis ICPC-2 PLUS was used by two blinded independent coders (DMM and PP), to code each clinical question at the level of chapters and clinical terms. If more than one chapter heading or clinical term could be applied to the question, all applicable were documented. Any conflicts and discrepancies in coding were resolved by a third coder with extensive clinical experience in general practice (LT). All clinical questions were also assigned a category from the taxonomy of generic clinical questions by LT and SR, with any discrepancies resolved through discussion. If more than one category could be applied to the question, all applicable were documented.

Analysis ICPC-2 PLUS was used by two blinded independent coders (DMM and PP), to code each clinical question at the level of chapters and clinical terms. If more than one chapter heading or clinical term could be applied to the question, all applicable were documented. Any conflicts and discrepancies in coding were resolved by a third coder with extensive clinical experience in general practice (LT). All clinical questions were also assigned a category from the taxonomy of generic clinical questions by LT and SR, with any discrepancies resolved through discussion. If more than one category could be applied to the question, all applicable were documented. Inductive coding: perceived knowledge gaps In addition to coding the questions deductively based on ICPC-2 PLUS and the taxonomy of generic clinical questions, we conducted a qualitative content analysis of questions to identify perceived knowledge gaps. Coding was performed inductively with categories derived from the data.14 Inductive codes were collected by senior author (LT) to form coding sheets and categories freely generated and grouped through the abstraction process.14 The coding scheme was revised over a two-round iterative process of discussion and revision involving LT and DMM. Two coders (LT and SR) then independently applied the final coding scheme to the full list of questions. Discrepancies were resolved by discussion between LT and SR.

ouped through the abstraction process.14 The coding scheme was revised over a two-round iterative process of discussion and revision involving LT and DMM. Two coders (LT and SR) then independently applied the final coding scheme to the full list of questions. Discrepancies were resolved by discussion between LT and SR. Results Demographic data The five participating general practice groups had a total of 53 GP members. The number of years practicing per GP ranged from 1 to 43 years. All practices were classified as ‘urban’ on the basis of population ranges according to the Section of State Structure of the Australian Statistical Geography Standard.15 One practice was a university health service. Description of the database A total of 126 questions from the five participating practice groups were entered into the database over the period of 1.5 years (June 2016 to December 2017). The number of questions submitted per practice per month varied, with a range of 0–10 questions. ICPC-2 PLUS ICPC-2 chapter headings and clinical terms for coded questions are presented in table 1. Nine clinical questions were not able to be coded using ICPC-2 PLUS and hence were not included (eg, What are patient views and emotional reactions to health professionals looking up resources during consultations?). Of those that were coded, 20 questions fell under two of the chapter headings, and one fell under three chapter headings. Regarding clinical terms, 38 questions fell under two of clinical terms, and 11 fell under three or more.

and emotional reactions to health professionals looking up resources during consultations?). Of those that were coded, 20 questions fell under two of the chapter headings, and one fell under three chapter headings. Regarding clinical terms, 38 questions fell under two of clinical terms, and 11 fell under three or more. Table 1 Number of questions under each chapter heading from the ICPC-2 PLUS classification system

and emotional reactions to health professionals looking up resources during consultations?). Of those that were coded, 20 questions fell under two of the chapter headings, and one fell under three chapter headings. Regarding clinical terms, 38 questions fell under two of clinical terms, and 11 fell under three or more. Table 1 Number of questions under each chapter heading from the ICPC-2 PLUS classification system ICPC-2 PLUS chapter heading n (%) ICPC-2 PLUS terms n (%) Endocrine/metabolic and nutritional 19 (13.7) Limited function/disability (t); endocrine/met/sympt/complt other; blood test; observe/educate/advice/diet; therapeutic counselling/listening; other referrals NEC; obesity; hypothyroidism/myxoedema; diabetes insulin dependent; diabetes non-insulin dependent; gout; endocrine/metab/nutrit. disorder other 30 (16.5) General and unspecified 19 (13.7) Weakness/tiredness general; limited function/disability NOS; microbiological/immunological test; blood test; urine test; other diagnostic procedure; diagnostic endoscopy; diagnostic radiology/imaging; preventive immunisation/medication; infectious disease other/NOS; malignancy NOS; complication of medical treatment; abnormal result investigation NOS; allergy/allergic reaction NOS 25 (13.7) Musculoskeletal 16 (11.5) Back symptom/complaint; muscle pain; sympt/comply. Musculoskeletal other; diagnostic radiology/imaging; consult with primary care provider; Medicat-script/Reqst/Renew/Inject; local injection/infiltration; infections musculoskeletal system; fracture: other; back syndrome with radiating pain; bursitis/tendinitis/synovitis NOS; osteoarthritis of knee; osteoarthritis other; shoulder syndrome; osteoporosis 21 (11.5) Digestive 12 (8.6) Abdominal pain/cramps general; dyspepsia/indigestion; flatulence/gas/belching; diarrhoea; rectal bleeding; digestive microbiological/immunological test; faeces test; diagnostic endoscopy (colonoscopy); gastrointestinal infection; congen.

osteoarthritis other; shoulder syndrome; osteoporosis 21 (11.5) Digestive 12 (8.6) Abdominal pain/cramps general; dyspepsia/indigestion; flatulence/gas/belching; diarrhoea; rectal bleeding; digestive microbiological/immunological test; faeces test; diagnostic endoscopy (colonoscopy); gastrointestinal infection; congen. Anomaly digestive system; oesophagus disease; irritable bowel syndrome; chronic enteritis/ulcerative colitis; disease digestive system, other 21 (11.5) Psychological 12 (8.6) Sleep disturbances; memory disturbances; specific learning problems; psychological symptom/complt other; dementia; affective psychosis; depressive disorder; psychological disorders, other 15 (8.2) Cardiovascular 12 (8.6) Medical examination/health evaluation-partial; physical function test; diagnostic radiology/imaging; electrical tracings; cardiac arrhythmias NOS; heart disease other; phlebitis/thrombophlebitis; cardiovascular disease other 14 (7.7) Skin 12 (8.6) Skin infection post-traumatic; excise/remove/biopsy/destruction/debride; repair/fixate-suture/cast/prosthetic; other therapeutic procedure NEC; malignant neoplasm of skin; solar keratosis/sunburn; psoriasis; acne 14 (7.7) Respiratory 12 (8.6) Cough; preventive immunisation/medications; influenza; pneumonia; respiratory infection other; asthma 12 (6.6) Neurological 8 (5.8) Headache; speech disorder; migraine; cluster headache; peripheral neuritis/neuropathy 9 (4.9) Female genital 7 (5.0) Menopausal symptom/complaint; breast symptom/complaint. Female other; malignant neoplasm breast female; fibromyoma uterus; abnormal cervix smear; premenstrual tension syndrome 6 (3.3) Pregnancy, childbearing, family planning 3 (2.2) Question of pregnancy; blood test; diagnostic radiology/imaging; abortion spontaneous; pregnancy high risk 5 (2.7) Male genital 3 (2.2) Cystitis/urinary infection other; abnormal urine test NOS; urinary disease other 4 (2.2) Blood, blood forming organs and immune mechanism 2 (1.4) Breast symptom/complaint. Male; B medical examination/health evaluation-partial; blood test 4 (2.2) Urological 2 (1.4) Blood test 2 (1.1) Eye 0 (0) N/A 0 (0) Ear 0 (0) N/A 0 (0) Social problems 0 (0) N/A 0 (0) Total 139 (100) 182 (100) ICPC, International Classification of Primary Care.

ans and immune mechanism 2 (1.4) Breast symptom/complaint. Male; B medical examination/health evaluation-partial; blood test 4 (2.2) Urological 2 (1.4) Blood test 2 (1.1) Eye 0 (0) N/A 0 (0) Ear 0 (0) N/A 0 (0) Social problems 0 (0) N/A 0 (0) Total 139 (100) 182 (100) ICPC, International Classification of Primary Care. Aside from the ‘general and unspecified’ chapter heading, the highest number of questions was coded to the ‘endocrine/metabolic and nutritional’ ICPC-2 chapter heading (n=19; 13.7%), followed by the ‘musculoskeletal’ chapter heading (n=16; 11.5%). Overall, the most common clinical terms were non-insulin-dependent diabetes (4%), blood test (endocrine/metabolic and nutritional (3%)); general and unspecified (3%)), and osteoporosis (3%). There did not appear to be large variations in the question types between GP practice groups although this was not checked for statistical significance. Taxonomy of generic clinical questions Using the taxonomy of generic clinical questions, the majority of questions were related to the broad category of treatment (71.3%), followed by diagnosis (14.5%), epidemiology (9.2%), management (2.3%), non-clinical (0.8%) and non-classified (0.8%). More specific generic question types are shown in table 2. Table 2 Number of questions coded by question type from the taxonomy of generic clinical questions

Taxonomy of generic clinical questions Using the taxonomy of generic clinical questions, the majority of questions were related to the broad category of treatment (71.3%), followed by diagnosis (14.5%), epidemiology (9.2%), management (2.3%), non-clinical (0.8%) and non-classified (0.8%). More specific generic question types are shown in table 2. Table 2 Number of questions coded by question type from the taxonomy of generic clinical questions Code Definition n % 2.1.2.1. Is drug x (or drug class x) indicated in situation y or for condition y? 28 21.7 2.2.1.1. How should I treat finding/condition y (given situation z)? 23 17.8 2.1.2.2. Should this kind of patient get prophylactic drug x to prevent condition y? 15 11.6 2.2.1.2. Should this kind of patient get prophylactic treatment (intervention) x to prevent condition y? 11 8.5 1.3.2.1. How good is test x in situation y? 10 7.8 4.2.1.1. Is x a risk factor for condition y? 10 7.8 2.1.1.3. When (timing, not indication) or how should I start/stop drug x? 8 6.2 1.3.1.1. Is test x indicated in situation y? 7 5.4 2.1.3.3. Is drug x safe to use in situation y? 3 2.3 3.1.1.1. How should I manage condition/finding/situation y? 2 1.6 1.3.3.1. When (timing, not indications) should I do test x? 1 0.8 1.7.1.1. What is the cost of test x? 1 0.8 2.1.2.1. Is drug x (or drug class x) indicated in situation y or for condition y? 1 0.8 2.1.3.1. Could finding y be caused by drug x? 1 0.8 2.1.3.2. How can drug x be administered without causing adverse effect y or minimising adverse effect y or in spite of adverse effect y? 1 0.8 2.1.4.1. Is it OK to use drug x with drug y? 1 0.8 2.2.1.1. How should I treat finding/condition y (given situation z)? 1 0.8 3.2.2.1. When should you refer in situation y? 1 0.8 4.3.1.1. What is the usual course (or natural history) of condition y? 1 0.8 4.4.1.1. Generic type varies (epidemiology not elsewhere classified) 1 0.8 5.1.1.3. How can I better teach this trainee (medical student, resident, other provider)? 1 0.8 6.1.1.1. Generic type varies. Unable to classify 1 0.8 Total 129 100 Perceived knowledge gaps Eleven categories of perceived personal knowledge gaps were identified from the inductive abstraction process (see table 3).

1 0.8 5.1.1.3. How can I better teach this trainee (medical student, resident, other provider)? 1 0.8 6.1.1.1. Generic type varies. Unable to classify 1 0.8 Total 129 100 Perceived knowledge gaps Eleven categories of perceived personal knowledge gaps were identified from the inductive abstraction process (see table 3). Table 3 Perceived knowledge gaps, as identified through inductive coding

1 0.8 5.1.1.3. How can I better teach this trainee (medical student, resident, other provider)? 1 0.8 6.1.1.1. Generic type varies. Unable to classify 1 0.8 Total 129 100 Perceived knowledge gaps Eleven categories of perceived personal knowledge gaps were identified from the inductive abstraction process (see table 3). Table 3 Perceived knowledge gaps, as identified through inductive coding Category name Example n (%) of questions 1 Checking more specifically on one aspect of current evidence on a common problem (staying up-to-date—focused) Is there any evidence to support the recommendation to avoid sex with a diagnosis of placenta praevia? 34 (27.0) 2 Checking claims/evidence about a relatively new test or treatment Evidence for FMT (faecal matter transplant) in treating ulcerative colitis 22 (17.5) 3 Evidence for complementary and alternative therapies Does fish oil prevent heart disease? 22 (17.5) 4 Checking broadly on guidelines and recommendations on a common topic (staying up-to-date—broad) Which migraine prophylaxis medication is most effective? 13 (10.3) 5 Application of evidence in a specific population subgroup or setting What is the effectiveness of antiviral medications in preventing complications in patients with influenza at low and high risk of complications? 13 (10.3) 6 Evidence about stopping treatments (including safety and minimising harms) Do we need to stop metformin for patients with diabetes? 7 (5.6) 7 Evidence for practice processes What are patients’ views and emotional reactions to health professionals looking up resources during consultations? 5 (4.0) 8 Checking on the safety or harms of a test or treatment Adverse effects of proton pump inhibitors 4 (3.2) 9 Evidence about lifestyle choices Does exercise reduce the risk of cancer? 3 (2.4) 10 Education (eg, clinical teaching, learning EBM skills) What personal qualities and behaviours do patients, medical students and trainees/registrars highly rate in their GP or their GP supervisor? 2 (1.6) 11 Checking the evidence about a less frequent condition Surgery versus other treatments for trigger thumb 1 (0.8) Total 125 (100%) GP, general practitioner.

g EBM skills) What personal qualities and behaviours do patients, medical students and trainees/registrars highly rate in their GP or their GP supervisor? 2 (1.6) 11 Checking the evidence about a less frequent condition Surgery versus other treatments for trigger thumb 1 (0.8) Total 125 (100%) GP, general practitioner. Discussion Summary of findings This study of the types of questions submitted by 53 Australian GPs participating in an information service revealed that the majority of questions prioritised for answering by GPs related to treatment, followed by diagnosis and epidemiology, respectively. Using the ICPC-2 PLUS classification system, the most common chapter headings were ‘endocrine/metabolic and nutritional’ and ‘general and unspecified’, followed by ‘musculoskeletal’ and ‘digestive’. Through an inductive coding process, we identified several knowledge gaps; however, over 70% of all questions related to checking more specifically on one aspect of current evidence on a common problem, or to be informed about new tests or treatments and complementary and alternative therapies.

eletal’ and ‘digestive’. Through an inductive coding process, we identified several knowledge gaps; however, over 70% of all questions related to checking more specifically on one aspect of current evidence on a common problem, or to be informed about new tests or treatments and complementary and alternative therapies. Several other studies have found questions concerning treatment to be the most common clinical question type.2 5 16 This may be unsurprising given the role of the GP to ‘treat all common medical conditions’,17 and patient expectations of (benefiting from) treatment.18 19 However, the percentage of questions pertaining to treatment was higher in our study compared with other Australian studies published 11 (35%)5 and 18 years ago (65%).6 This may be an artefact of slightly different classification systems across studies. For example, the taxonomy of generic clinical questions does not include a separate ‘prevention’ category, so may not reflect the expanding role of GPs in preventive healthcare activities. These include promoting smoking cessation, responsible alcohol consumption, weight control, physical activity and undertaking screening activities including blood pressure monitoring, cholesterol and blood glucose measurement and cancer screening. As well as generic question types, there are also some similarities between our findings regarding ICPC-2 PLUS classifications and other studies. Magrabi and colleagues, for example, found that gastrointestinal (13%), skin (12%) and musculoskeletal (11%) were the top three disease categories in a study of clinical questions from 227 Australian GPs,4 corresponding with two of three of our most common ICPC-2 PLUS chapter headings, and a study of questions submitted to a popular GP-restricted (Australia, New Zealand) Facebook group16 had two of the five major clinical topics (musculoskeletal and psychology) from our study. Future research may benefit from additional triangulation of data from different sources (eg, encounter observations; questions submitted to EBP question-answering services; questions submitted to public forums) to explore the role of different networks in answering clinical questions and to identify the most common question types across all forums.

m additional triangulation of data from different sources (eg, encounter observations; questions submitted to EBP question-answering services; questions submitted to public forums) to explore the role of different networks in answering clinical questions and to identify the most common question types across all forums. Through the use of a standard classification system (ICPC-2 PLUS), our results can also be compared with findings from the Bettering the Evaluation of Care and Health (BEACH) programme; a continuous 18-year national study of general practice activity in Australia.7 BEACH data from 2006–2007 to 2015–2016 indicates that general and unspecified problems were the most frequently managed type of problem in 2015–2016; their management rate increased from 16.2 per 100 encounters in 2006–2007 to 20.0 per 100 in 2015–2016, an increase of 23.5% over the decade.7 There were also substantial increases in the management rates of endocrine and metabolic problems in general practice encounters between 2006–2007 and 2015–2016, from 12.1 to 13.5 per 100 encounters.7 In all years from 2006–2007 to 2015–2016, the five most commonly managed problems were hypertension, immunisation, upper respiratory tract infection, depression and diabetes.20 These mirror our findings and highlight that GP’s clinical questions overlap with the common problems they see in practice. Similarly, the categories for which we did not code any questions (eg, ear, eye and social problems) were in the six least common reasons for an encounter by ICPC-2 chapter, accounting for 3.2%, 1.9% and 1.0% of clinical questions from 2015 to 2016 BEACH data.7

ons overlap with the common problems they see in practice. Similarly, the categories for which we did not code any questions (eg, ear, eye and social problems) were in the six least common reasons for an encounter by ICPC-2 chapter, accounting for 3.2%, 1.9% and 1.0% of clinical questions from 2015 to 2016 BEACH data.7 A novel contribution of this study is the use of inductive coding of GPs’ questions to identify perceived knowledge gaps. The need to stay up-to-date with the state of evidence and be informed about new tests or treatments were the most prevalent knowledge gaps. Other studies have similarly identified the need to achieve basic ‘currency’ as a driver of clinical questions.21 Based on our inductive coding scheme, staying up-to-date included checking broadly on guidelines and recommendations for a common topic (10%) as well as checking in a more focused ‘specific’ way on one aspect of current evidence for a common problem (27%). Aligned with this, Brassil and colleagues found that more than half of the respondents stated that their clinical questions arose from ‘unusual cases’.22 This reinforces that even though GPs often manage common problems, focused questions about specific nuisances of a problem remain an important knowledge gap. In addition to achieving currency, 18% of questions in this study were coded as knowledge gaps about complementary and alternative therapies/medicines (CAM). CAM is estimated to be used by up to two out of three Australians,23 and other studies have likewise found a perceived lack of knowledge about complementary and alternative therapies among Australian GPs.24 In a study to assess questions about CAM to the Regional Medicines Information and Pharmacovigilance Centres in Norway, Schjott and Erdak found that 7.7% of all questions regarded CAM.25 Together, these findings highlight the growing prevalence of questions about CAM from mainstream clinical practice and the knowledge gaps which remain in this area.

about CAM to the Regional Medicines Information and Pharmacovigilance Centres in Norway, Schjott and Erdak found that 7.7% of all questions regarded CAM.25 Together, these findings highlight the growing prevalence of questions about CAM from mainstream clinical practice and the knowledge gaps which remain in this area. Directions for research and practice Our results highlight that GPs’ clinical questions frequently relate to treatment and to commonly managed problems, but are often focused more narrowly on specific aspects of a problem or new tests and treatments. The submission of such questions to our information service suggests that some GPs may not feel well-equipped to manage some of the diverse nuances of common presentations. In line with this, previous studies have found that published research and clinical guidelines do not align with the problems most frequently encountered by GPs21 and continuing professional development programmes do not always correlate well with the spectrum of common clinical issues experienced on a daily basis by clinicians.26

this, previous studies have found that published research and clinical guidelines do not align with the problems most frequently encountered by GPs21 and continuing professional development programmes do not always correlate well with the spectrum of common clinical issues experienced on a daily basis by clinicians.26 In Australia, the National Health and Medical Council recognises that, if guidelines are to be relevant, those who are expected to use them should play a part in their conception.27 However, there are currently no system-wide approaches for obtaining GPs’ evidence and knowledge gaps, and clinical practice guidelines are often commissioned by different groups in a top-down manner (eg, based on government health priority topics). Our system for ‘pulling’ real-time questions from GPs may help to circumvent this, and can inform the ‘push’ of more relevant and timely evidence for use in the clinical encounter. Other work has also shown that sustained relationships between clinical practices and academic groups can help to raise and answer clinical questions of importance to GPs.28 To ensure sustainability of this method to ‘pull’ clinical questions into evidence ecosystems, work is needed to support linkages between academic departments, general practices and guideline developers, as well as sustained and sufficient funding for infrastructure including staff.29

questions of importance to GPs.28 To ensure sustainability of this method to ‘pull’ clinical questions into evidence ecosystems, work is needed to support linkages between academic departments, general practices and guideline developers, as well as sustained and sufficient funding for infrastructure including staff.29 Our results also support the increasing advocacy for ‘living guidelines’.30 A living guideline is one that remains under review on an ongoing basis, with updates published at set intervals (eg, annually).25 The use of living guidelines can support clinicians to find evidence about new tests and treatments more easily and, if hosted on searchable digital platforms, may better support them to search for specific information about aspects of a problem.30 This represents an important advance on the lengthy, static clinical practice guidelines currently in use throughout Australia. Finally, our findings also reinforce the call for better awareness and resources about CAM. Even where there is limited or uncertain evidence for CAM, this should be included in clinical guidelines to ensure that clinicians are informed about the evidence and can communicate this to patients to support informed, shared decision-making. This may help to increase the number of people who consult their GP prior to using CAM (currently only 20% in Australia31) and is particularly important given the competing sources (social networks; books; media; internet31) and quality of information available to patients.

to support informed, shared decision-making. This may help to increase the number of people who consult their GP prior to using CAM (currently only 20% in Australia31) and is particularly important given the competing sources (social networks; books; media; internet31) and quality of information available to patients. Strengths and limitations We analysed questions from a small and non-randomly selected sample of GPs, which limits the generalisability of our findings. In fact, we purposively selected lead practices recognised as opinion leaders and early adopters of clinical advances. The fact that even these lead GPs had clinical questions which they requested assistance in answering supports the utility of, and need for, better systems for pulling real-time questions from GPs to guide research and the development and revision of practice guidelines. As all questions were submitted via an appointed GP liaison, we do not have information about which GP submitted each question, so some GPs may be over-represented in the database, and it is possible that the GPs actually contributing questions may be the most motivated and evidence-aware practitioners within those practices.

questions were submitted via an appointed GP liaison, we do not have information about which GP submitted each question, so some GPs may be over-represented in the database, and it is possible that the GPs actually contributing questions may be the most motivated and evidence-aware practitioners within those practices. Direct observation studies such as that of Ely et al 3 have shown that GPs generate ~10–15 questions per day of practice. We, therefore, received a very select number of questions from participating GPs which is unlikely to be representative of the much larger number of questions that they generate. However, this study was designed to investigate the types of questions that GPs at the practice level have prioritised for answering to improve the quality of practice and uptake of evidence. The questions submitted to our service may also reflect those which GPs have particular difficulty answering through other means. The use of multiple coding systems and independent coders are strengths of our study. Standard classification systems such as ICPC-2 plus allow for meaningful comparison of clinical questions over time and contexts. The ICPC-2 is now used in >45 countries as the standard for data classification in primary care,7 which also allows us to compare clinical questions to clinical presentations. We coded all questions submitted by GP groups, not only those which were answered by our information service.

questions over time and contexts. The ICPC-2 is now used in >45 countries as the standard for data classification in primary care,7 which also allows us to compare clinical questions to clinical presentations. We coded all questions submitted by GP groups, not only those which were answered by our information service. Conclusion This study investigated the types of questions submitted by Australian GP groups over a period of 1.5 years and characterised GPs’ information needs by classifying their questions using established and inductive coding systems. Our findings highlight the diverse range of clinical questions which arise in general practice, but show that topic areas often overlap with common clinical presentations and often concern treatment. The questions submitted reinforce GPs’ need to stay up-to-date and desire to know more about the evidence for complementary and alternative medicines. Going forward, it is necessary to target research and resources towards meeting GPs’ specific information needs to support the dissemination of relevant and timely evidence for use in clinical encounters. Key messages What is already known about this subject? Clinical questions at the point of care are common; however, clinicians only pursue roughly half of these questions. Evidence-based information services are a way of supporting clinicians to answer prioritised clinical questions, and seek to overcome some of the perceived barriers to evidence-based practice.

Key messages What is already known about this subject? Clinical questions at the point of care are common; however, clinicians only pursue roughly half of these questions. Evidence-based information services are a way of supporting clinicians to answer prioritised clinical questions, and seek to overcome some of the perceived barriers to evidence-based practice. What are the new findings? Treatment (71%), diagnosis (15%) and epidemiology (9%) were the most common categories of questions submitted to a journal club support service by 53 Australian general practitioners (GPs) over a 1.5-year period. Using the International Classification of Primary Care 2 classification, the highest number of questions was coded to the endocrine/metabolic and nutritional chapter heading, followed by general and unspecified, digestive and musculoskeletal. Seventy per cent of all questions related to the need to stay up-to-date with the evidence, or be informed about new tests or treatments (including complementary and alternative therapies). Clinical questions prioritised for answering by Australian GPs overlapped with common clinical presentations. How might it impact on clinical practice in the foreseeable future? Understanding the nature of clinical questions prioritised for answering can help to target research and resources towards meeting GPs’ specific information needs. Methods and findings can support better systems for ‘pulling’ real-time questions from GPs to inform the ‘push’ of more relevant and timely evidence for use in the clinical encounter.

How might it impact on clinical practice in the foreseeable future? Understanding the nature of clinical questions prioritised for answering can help to target research and resources towards meeting GPs’ specific information needs. Methods and findings can support better systems for ‘pulling’ real-time questions from GPs to inform the ‘push’ of more relevant and timely evidence for use in the clinical encounter. Thanks to all of the GPs who participated in this study and contributed clinical questions. Contributors: LT, DMM, TH and PP conceived of this study and its design, and were responsible for data collection. LT, DMM, PP and SR performed the content analysis and coding, and all authors were involved in interpreting the results. All authors discussed the results and commented on the manuscript. Funding: This project was funded by a National Health and Medical Research Council Centre for Research Excellence (ID: 1106452). Competing interests: None declared. Ethics approval: The study was approved by the University of Sydney Human Research Ethics Committee (Protocol # 2016/1011). Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not required.