Browse the corpus

ersity, Canada); Naomi Dove (University of British Columbia, Canada); Ruramayi Rukini (University of Bristol, United Kingdom) and Shannon Gibson (University of Victoria, Canada). The authors are also grateful to all the experts and stakeholders from around the world who provided feedback on earlier drafts of this work. Competing interests: None. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction The World Alliance for Patient Safety recently identified poor test follow-up as one of the major processes contributing to unsafe patient care.1 Failure to follow up test results increases the risk of missed or delayed diagnoses. This may produce suboptimal clinical outcomes2–8 with potential medicolegal implications.9–12 Clinicians are concerned that their test management practices are not systematic,7 13 and considerable variation exists.10 14 15 It has been claimed that information technology can improve this process, making it safer, easier and more systematic, reducing the risk of results being missed.6 16–18 Yet, evidence of its effective application in practice is limited.1 5 19 Managing the follow-up of diagnostic and radiological test results is a complex process.20–22 It entails information exchange between patients, doctors, nurses and laboratories using a combination of information systems, including paper-based, telephone and electronic systems, and involving a variety of policies and procedures. Multiple steps, players and information systems create an environment which increases the risk of errors. There have been no published systematic reviews of the extent of the problem of failure to follow-up test results for hospital patients. Our aim was to review evidence which quantified the size of the problem and the impact on patient outcomes for hospital patients including patients attending the emergency department (ED).

have been no published systematic reviews of the extent of the problem of failure to follow-up test results for hospital patients. Our aim was to review evidence which quantified the size of the problem and the impact on patient outcomes for hospital patients including patients attending the emergency department (ED). Methods Data sources and searches A literature search of the following databases was undertaken for English-language publications from January 1990 to March 2010: Medline; CINAHL; Embase; Inspec and the Cochrane Database of Systematic Reviews (figure 1). Search terms were identified from keyword lists of core journal articles related to the research topic. Reference lists of articles which met the inclusion criteria were hand-searched. A web search using the Google Scholar search engine was completed to locate unindexed publications in press. Figure 1 Search flow for failure to follow-up test results literature, including keywords and Medical Subject Headings (MeSH) terms used in search process.

Methods Data sources and searches A literature search of the following databases was undertaken for English-language publications from January 1990 to March 2010: Medline; CINAHL; Embase; Inspec and the Cochrane Database of Systematic Reviews (figure 1). Search terms were identified from keyword lists of core journal articles related to the research topic. Reference lists of articles which met the inclusion criteria were hand-searched. A web search using the Google Scholar search engine was completed to locate unindexed publications in press. Figure 1 Search flow for failure to follow-up test results literature, including keywords and Medical Subject Headings (MeSH) terms used in search process. Study selection Four researchers (JC, AG, JL, JW) individually screened titles and abstracts to determine eligibility. Studies which quantified the extent of the failure to follow-up laboratory or radiology test results for hospital inpatients or patients treated in the ED were included in the review. Failure to follow-up was defined as neglecting to document a follow-up of test results by the ordering physician or another provider. Studies examining laboratory or radiology departments' failure to report critical results, uncommunicated pending results during handover or discharge summaries and physicians' failure to communicate results to patients were excluded. Studies reporting physicians'23–26 or patients'27 perceived rates of failure to follow-up were excluded, as were studies which measured time to treatment or how rapidly test results were acted upon.6 28–30

s during handover or discharge summaries and physicians' failure to communicate results to patients were excluded. Studies reporting physicians'23–26 or patients'27 perceived rates of failure to follow-up were excluded, as were studies which measured time to treatment or how rapidly test results were acted upon.6 28–30 Data extraction Eligible studies were independently reviewed (JC, AG, JL, JW), and discrepancies were resolved by further discussion until consensus was reached. Authors of papers were contacted when necessary for additional information. Results Search results Twelve studies3 4 20 31–39 met our inclusion criteria (table 1, available online only). Study characteristics Eight studies were conducted in the USA.3 4 20 33 34 36 38 39 Most study designs were medical record reviews, either retrospective4 20 32 33 36 37 39 or prospective.3 31 35 One study reviewed malpractice claims,38 while another retrospectively linked laboratory and pharmacy databases.34

Results Search results Twelve studies3 4 20 31–39 met our inclusion criteria (table 1, available online only). Study characteristics Eight studies were conducted in the USA.3 4 20 33 34 36 38 39 Most study designs were medical record reviews, either retrospective4 20 32 33 36 37 39 or prospective.3 31 35 One study reviewed malpractice claims,38 while another retrospectively linked laboratory and pharmacy databases.34 Hospital inpatients Extent of failure to follow-up results and impact on inpatient outcomes Seven studies examined the extent of failure to follow-up for hospital inpatients.3 4 20 32–34 39 Three reported aggregated results for inpatients and outpatients.4 34 39 The extent of follow-up failure was reported as a proportion of inpatients4 20 33 34 or of tests under study.3 32 39 The extent of failure to follow-up ranged from 1.0%33 to 22.9% of inpatients4 and from 20.04%39 to 61.9%3 when reported per test type. The range of test types included: urgent32 and critical20 laboratory results; abnormal actionable results pending at discharge3; diagnostic imaging4 33 39 and elevated Thyroid Stimulating Hormone levels.34 A study of radiology follow-up using an email alert system for important but not urgent imaging findings reported that 20.0% (10 598/52 883) of electronic reports were not viewed by the referring physician.39

ble results pending at discharge3; diagnostic imaging4 33 39 and elevated Thyroid Stimulating Hormone levels.34 A study of radiology follow-up using an email alert system for important but not urgent imaging findings reported that 20.0% (10 598/52 883) of electronic reports were not viewed by the referring physician.39 Four of the seven studies reported the impact of failure to follow-up which included missed diagnoses of malignancy,33 hypothyroidism,34 hyperthyroidism,3 osteoporosis,4 microbiological results which necessitated the starting or changing of antibiotic therapy,3 and positive serological test results for Helicobacter pylori.3

of the seven studies reported the impact of failure to follow-up which included missed diagnoses of malignancy,33 hypothyroidism,34 hyperthyroidism,3 osteoporosis,4 microbiological results which necessitated the starting or changing of antibiotic therapy,3 and positive serological test results for Helicobacter pylori.3 Follow-up of critical laboratory results and results pending at discharge There was wide variation in the three studies which examined follow-up of critical values which can be life-threatening if action is not taken promptly.40 Kilpatrick and Holding32 assessed the effects of replacing telephone notification of urgent laboratory results with computer terminal access. They found that 529/1836 (28.8%) urgent biochemistry results during a 6-month period were never accessed electronically. Interestingly, for 27 (5.1%) of the 529 results never accessed, the clinician had attempted access using the ward terminal before the results were available. Tate et al20 found that 15.3% (19/124) of medical records audited for a 2-month period contained no documentation for which either the nurse or physician was aware of the critical laboratory value or had taken corrective action. A study of critical radiology result follow-up found that in four of the 395 (1.0%) suspected malignancy cases, the provider was unaware of the findings.33 These four inpatients would have been lost to follow-up if the semiautomated coding and review process had not been instituted.33

ue or had taken corrective action. A study of critical radiology result follow-up found that in four of the 395 (1.0%) suspected malignancy cases, the provider was unaware of the findings.33 These four inpatients would have been lost to follow-up if the semiautomated coding and review process had not been instituted.33 Results which are pending at the time of discharge from hospital present a particular challenge to physicians. Roy et al3 found that hospital physicians were unaware of 65 results (61.6% (95% CI 51.3% to 70.9%)), and of these, 24 (37.1% (95% CI 25.7% to 50.2%)) were actionable, with eight (12.6% (95% CI 6.4% to 23.3%)) requiring urgent action. Although several limitations were reported in this study, they concluded that there was a need for better systems to follow up results that arrive after a patient is discharged.3

70.9%)), and of these, 24 (37.1% (95% CI 25.7% to 50.2%)) were actionable, with eight (12.6% (95% CI 6.4% to 23.3%)) requiring urgent action. Although several limitations were reported in this study, they concluded that there was a need for better systems to follow up results that arrive after a patient is discharged.3 Patients treated in the ED Extent of failure to follow-up results and impact on patient outcomes Seven studies quantified the extent of failure to follow-up in EDs.31–33 35–38 This ranged from 1.0%31 to 75% of tests36 and 0%33 to 16.5%38 of patients treated in the ED. Test types included: radiology with failure to follow-up ranging from none to 5.6%31 33 37; microbiology with failed follow-up ranging from 3.0% to 75%31 35 36; serum lead levels with 33.3% lost to follow-up36; and urgent biochemistry with 44.7% not followed up.32 One study examined 122 closed malpractice claims, for which test types were unknown, from four liability insurers for injuries which ED patients sustained between 1979 and 2001.38 The study found that 79 of the 122 claims (64.8%) involved missed ED diagnoses that harmed patients, and 13 of these 79 claims (16.5%) identified the breakdown to have occurred at the step of ‘test results transmitted to and received by the provider.’38

for injuries which ED patients sustained between 1979 and 2001.38 The study found that 79 of the 122 claims (64.8%) involved missed ED diagnoses that harmed patients, and 13 of these 79 claims (16.5%) identified the breakdown to have occurred at the step of ‘test results transmitted to and received by the provider.’38 All seven studies explored the impact on patient outcomes31–33 35–38 which included no negative effects,31 a delayed diagnosis from a missed x-ray report,37 one case of missed positive Chlamydia where the patient subsequently developed pelvic inflammatory disease,36 inappropriate or unnecessary antibiotics prescribed,35 missed cancer diagnoses33 and death.38 Discussion Extent of the problem and impact on patient outcomes There is evidence to suggest that the proportion of missed test results is a substantial problem which impacts on patient safety. However, there was enormous variability reported on the extent of the problem. Lack of follow-up of test results for patients treated in the ED ranged from 1.0%31 to 75%36 and for inpatients from 20.04%39 to 61.9%3 when calculated as a proportion of tests. The range when calculated as a proportion of patients was 0%33 to 16.5%38 for patients treated in the ED and 1.0%33 to 22.9%4 for hospital inpatients. Examples of serious patient outcomes were identified, including missed cancer diagnoses33 and positive Chlamydia with subsequent development of pelvic inflammatory disease.36

ange when calculated as a proportion of patients was 0%33 to 16.5%38 for patients treated in the ED and 1.0%33 to 22.9%4 for hospital inpatients. Examples of serious patient outcomes were identified, including missed cancer diagnoses33 and positive Chlamydia with subsequent development of pelvic inflammatory disease.36 The studies were heterogeneous in their approach limiting robust comparison. Medical record review was frequently used, which relied upon documentary evidence of follow-up. This may lead to an overestimation of the problem, since, in some cases, results may have been seen and acted upon but not documented. Publication bias may also be a factor whereby papers which reported high rates of missed test results are more likely to be published than those which did not. We found that factors associated with missed results included: the systems and practices used; reporting critical results, and test results for patients moving across care settings.

also be a factor whereby papers which reported high rates of missed test results are more likely to be published than those which did not. We found that factors associated with missed results included: the systems and practices used; reporting critical results, and test results for patients moving across care settings. Systems and practices used to follow-up test results Only two studies described complete electronic test management systems3 20 where tests were ordered on-line and results reported electronically with no paper used. The rate of missed results was high in both these studies, although it could be argued that the technology made the problem more explicit and easier to measure. Rates were also high in hospitals which used entirely paper-based systems4 36 and in those which used a mixture of paper and electronic systems.32 35 There was no evidence of any link between the system used and the extent of missed test results. Other studies have shown that the use of hybrid paper and electronic clinical information systems is associated with errors and duplications, with complete electronic systems showing fewer errors.13 41 A study of outpatient test results reported that the use of a partial electronic medical record (paper-based progress notes and electronic test results or vice versa) was associated with higher rates of failure to inform patients of clinically significant results compared with not having an electronic medical record (OR=1.92; p=0.03), or compared with having an electronic medical record that included both progress notes and test results (OR=2.37; p=0.007).42 A qualitative study which evaluated an electronic results management system in paediatric ambulatory care found that practices which had fully adopted the electronic system reported gains in efficiency, reliability, timeliness and provider satisfaction, whereas some partial adopters reported decreased efficiency and increased risk of lost test results.14

an electronic results management system in paediatric ambulatory care found that practices which had fully adopted the electronic system reported gains in efficiency, reliability, timeliness and provider satisfaction, whereas some partial adopters reported decreased efficiency and increased risk of lost test results.14 Despite many advocates of the use of information technologies to improve the management of test results,6 18 19 28 43–45 few studies have evaluated electronic test management systems, and results have been mixed.6 32 Existing electronic systems now provide the capacity for clinicians to acknowledge that they have viewed test results on-line and document their follow-up actions. Electronic test management systems also have the capability of reporting results to the ordering clinician and other members of the team to facilitate endorsement of results in team-based environments and shift handover situations. This on-line endorsement function would enable reports of rates of missed test results to be produced which could provide a continuous quality audit capability for use by management and physicians.

d other members of the team to facilitate endorsement of results in team-based environments and shift handover situations. This on-line endorsement function would enable reports of rates of missed test results to be produced which could provide a continuous quality audit capability for use by management and physicians. Advances in the functionality of test management systems are not sufficient to solve the problem. The complexity of the test management process and high volume of test results requires significant review and reform of work practices to allow electronic endorsement to occur easily. Management of test results can differ depending on the needs and work practices of physicians in different clinical settings, and so electronic test management systems need to be flexible to adapt to these divergent requirements.46–49

review and reform of work practices to allow electronic endorsement to occur easily. Management of test results can differ depending on the needs and work practices of physicians in different clinical settings, and so electronic test management systems need to be flexible to adapt to these divergent requirements.46–49 Reporting of critical test results A commonly cited problem in test result reporting was the breakdown in the communication process, including documentation of actions, between clinical units and the laboratories. This was particularly evident in studies which reported lack of follow-up for critical test results.20 32 33 Others have identified the follow-up of critical results as an area requiring attention.2 50 51 Despite established practice guidelines requiring critical values to be telephoned to the clinical team, compliance may be low, and information may not always go to the person involved in the patients' care.20 28 The traditional practice of laboratories telephoning results of urgent or critical tests is time-consuming with potential for errors.32 For the follow-up of critical results to occur without error, the information transfer between laboratory staff and clinicians must be examined to devise technological, work practice and policy solutions which take account of this cross-boundary communication process.38 Nurses have been shown to play an important role in test-result follow-up and should be included in any solutions devised.20 33

transfer between laboratory staff and clinicians must be examined to devise technological, work practice and policy solutions which take account of this cross-boundary communication process.38 Nurses have been shown to play an important role in test-result follow-up and should be included in any solutions devised.20 33 Electronic test management systems provide the potential to support notification of critical results. However, the study by Kilpatrick and Holding32 concluded that substituting the telephoning of urgent results with computer access could hinder, rather than promote, communication between laboratory and clinicians. Limitations of that study were that computer terminals had been in place for only 6 months, and it was not stated whether all clinicians were mandated to use the computers.32 Interestingly, Kilpatrick and Holding found that some results were never accessed: clinicians had attempted access via the ward terminals before the results were available.32 Thus, time may be wasted by clinicians continually checking if results are available. These passive retrieval systems rely on the clinician to ‘pull’ the information from the test management system rather than actively notifying clinicians of urgent results.18 Active notification of abnormal and critical results to clinicians using alerts has been shown to be effective.29 30 52–57

results are available. These passive retrieval systems rely on the clinician to ‘pull’ the information from the test management system rather than actively notifying clinicians of urgent results.18 Active notification of abnormal and critical results to clinicians using alerts has been shown to be effective.29 30 52–57 Test results for patients moving across settings Our review showed that patients moving across settings, for example, from inpatient to outpatient services, or patients treated in the ED discharged to the care of their general practitioner or to the ward, can cause problems with follow-up of results and continuity of care.3 35 36 The ED is particularly challenging for test-result follow-up due to the high patient throughput, team-based care, handoffs and lack of continuous relationships between patients and clinicians.36 38 Given the short length of stay for discharged ED patients, late-arriving results also increase the risk of certain test results being missed.36 37 One study reported a more comprehensive test follow-up for hospitalised than for discharged ED patients.35 Other studies have supported this finding, relating medical errors to discontinuity of care during handoffs between hospital teams or transfer of care between inpatient and outpatient settings.58–60 These studies highlight the need for systems, policies and practices which facilitate communication of information across different settings. The discharge summary is a standard means of communication between settings, but it may not always reach the intended recipient61–63 or be complete. One advantage of electronic discharge summaries is that family physicians are more likely to receive them in a timely fashion64–66 and hence seek to follow up any outstanding results. An electronic test management and discharge summary system which provided secure access for health professionals, both in hospital and in the community, could facilitate the follow-up of test results pending at discharge.

kely to receive them in a timely fashion64–66 and hence seek to follow up any outstanding results. An electronic test management and discharge summary system which provided secure access for health professionals, both in hospital and in the community, could facilitate the follow-up of test results pending at discharge. Conclusions The number of research studies in this area is limited, and the methods used prevent robust comparisons. The existing evidence suggests that the problem of missed test results is considerable and reported negative impacts on patients warrant the exploration of solutions. Further studies are urgently needed to test the effectiveness of interventions such as on-line endorsement of results. Attention must be paid to integration of solutions, particularly those which involve information technology, into clinical work practices. Funding: This study is part of an Australian Research Council Linkage Grant (LP0989144) funded project to investigate the use of information and communication technologies to support effective work practice innovation in the health sector. Competing interests: None. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Primary percutaneous coronary intervention (PCI) is superior to thrombolysis for acute ST-segment elevation myocardial infarction (STEMI).1 An important factor affecting outcome in primary PCI is delays to treatment,2–4 a particular concern after regular working hours, when facilities must be activated and staff brought in from home. Some institutions are therefore concerned that favourable outcomes may be difficult to achieve for patients presenting after hours. In a period of constrained resources, this would not lead to endorsement of routine after-hours procedures, and may in fact lead to scrutiny of how medical facilities operate at night, including more widespread adoption of night shifts. Recent attention has also been directed towards other causes of adverse patient outcomes occurring after hours, mostly related to the effects of sleep deprivation and fatigue on healthcare provider performance, process of care and medical error.5–13 While none of these data are specifically related to cardiac care, one can postulate that these important factors might be at play in the provision of primary PCI. We have developed a large, population-based, clinical registry capturing all patients undergoing cardiac catheterisation and revascularisation in Alberta, Canada, which provides a unique opportunity to evaluate outcomes in unselected patients. We sought to describe and compare crude and risk-adjusted survival for patients undergoing primary PCI for acute STEMI after-hours to those whose procedures occurred during regular working hours.

n and revascularisation in Alberta, Canada, which provides a unique opportunity to evaluate outcomes in unselected patients. We sought to describe and compare crude and risk-adjusted survival for patients undergoing primary PCI for acute STEMI after-hours to those whose procedures occurred during regular working hours. Methods Data sources The Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) is a clinical data-collection initiative capturing consecutive patients undergoing cardiac catheterisation in Alberta, Canada (population 3 290 350) since 1995.14 APPROACH contains detailed information including patients' age, sex, ejection fraction and multiple comorbidities as outlined in table 1. It tracks therapeutic interventions (previous thrombolytic therapy, revascularisation procedures). Coronary anatomy and procedural details are also recorded. Following data entry by catheterisation laboratory staff, an enhancement procedure verifies patient comorbidities and ensures that there are no missing data fields.15 Follow-up mortality for all patients is ascertained through semiannual linkage to the Alberta Bureau of Vital Statistics. Three hospitals in two large cities (Edmonton and Calgary) provide the only revascularisation services in Alberta, and primary PCI is the preferred treatment strategy for STEMI. APPROACH and this protocol were approved by the Institutional Review boards of the University of Alberta and the University of Calgary. Table 1 Baseline Characteristics

Methods Data sources The Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) is a clinical data-collection initiative capturing consecutive patients undergoing cardiac catheterisation in Alberta, Canada (population 3 290 350) since 1995.14 APPROACH contains detailed information including patients' age, sex, ejection fraction and multiple comorbidities as outlined in table 1. It tracks therapeutic interventions (previous thrombolytic therapy, revascularisation procedures). Coronary anatomy and procedural details are also recorded. Following data entry by catheterisation laboratory staff, an enhancement procedure verifies patient comorbidities and ensures that there are no missing data fields.15 Follow-up mortality for all patients is ascertained through semiannual linkage to the Alberta Bureau of Vital Statistics. Three hospitals in two large cities (Edmonton and Calgary) provide the only revascularisation services in Alberta, and primary PCI is the preferred treatment strategy for STEMI. APPROACH and this protocol were approved by the Institutional Review boards of the University of Alberta and the University of Calgary. Table 1 Baseline Characteristics After hours (n=906) Working hours (n=758) p Value Clinical characteristics Mean age (SD), years 60.3 (13.2) 61.5 (12.7) 0.07 Sex (% female) 22.2 26.0 0.07 Ejection fraction (%) 0.08 <35 6.9 6.9 35–50 29.9 29.2 >50 49.9 47.2 LV not done 10.7 14.6 Missing 2.5 2.1 Congestive heart failure (%) 11.9 10.6 0.38 Peripheral vascular disease (%) 4.4 4.0 0.64 Chronic pulmonary disease (%) 6.8 9.4 0.06 Cerebrovascular disease (%) 4.5 4.2 0.76 Creatinine >200 mmol/l (%) 4.0 2.5 0.09 Dialysis dependent (%) 1.0 0.5 0.29 Diabetes (%) 15.9 15.7 0.91 Hypertension (%) 50.0 47.9 0.39 Hyperlipidaemia (%) 45.8 45.4 0.86 Liver/gastrointestinal disease (%) 2.7 3.4 0.35 Malignancy (%) 3.0 2.9 0.92 Previous coronary artery bypass grafting (CABG) (%) 1.8 1.3 0.46 Previous myocardial infarction (MI) (%) 10.7 13.7 0.06 Previous percutaneous coronary intervention (PCI) (%) 3.5 3.7 0.86 Procedural characteristics Coronary anatomy (%) 0.79 One-vessel disease 38.5 38.9 Two-vessel disease 31.0 29.0 Three-vessel disease 26.6 28.4 Left main 3.9 3.7 Vessel intervened (%) 0.77 Right coronary 44.4 44.6 Circumflex 12.9 14.0 Left anterior descending 42.4 41.0 Left mainstem 0.3 0.3 Saphenous vein graft 0.0 0.1 Glycoprotein IIb/IIIa inhibitor (%) 73.2 74.7 0.49 Stent use (%) 92.3 90.5 0.20 Intra-aortic balloon pump (IABP, %) 6.1 11.6 <0.0001 Inotrope use (%) 2.4 1.1 0.036 IABP+inotrope (%) 7.6 12.0 0.003 The study population for this analysis consisted of STEMI patients undergoing primary PCI. Rescue PCI patients and those requiring hospital transfer were excluded. Door-to-balloon times for all patients were obtained through linkages to emergency room administrative data, and to the Calgary STEMI quality improvement data registry, which has prospectively collected time-interval data since 2004.

primary PCI. Rescue PCI patients and those requiring hospital transfer were excluded. Door-to-balloon times for all patients were obtained through linkages to emergency room administrative data, and to the Calgary STEMI quality improvement data registry, which has prospectively collected time-interval data since 2004. Timing of PCI procedures Data in APPROACH are entered in real-time, with a database ‘clock’ for regular working hours (weekdays 0700–1800) or after-hours (weeknights 1800–0700, weekends and holidays). In order to measure outcomes using currently available technology and adjunctive therapy, we limited our assessment to those patients undergoing primary PCI from 1 January 1999 to 31 March 2006. Outcome measures Our primary goal was to determine whether after-hours procedures were associated with higher crude and adjusted mortalities at 30 days. A secondary analysis assessed survival to 1 year, though we recognise a priori that many factors can intervene over this period to potentially dilute any influence of the timing of PCI on outcomes. Statistical analysis Patient characteristics were compared using χ2 tests. Kaplan–Meier plots and logrank tests were used to determine and compare crude mortalities. Multivariable Cox proportional hazards models were then used to adjust for the effects of baseline risk factors on group survival. The proportion hazards assumption was tested.16 The variables used for risk-adjustment analysis in these models are the baseline variables recorded in APPROACH (presented in table 1).14

es. Multivariable Cox proportional hazards models were then used to adjust for the effects of baseline risk factors on group survival. The proportion hazards assumption was tested.16 The variables used for risk-adjustment analysis in these models are the baseline variables recorded in APPROACH (presented in table 1).14 Additional analysis including door-to-balloon time The distributions of door-to-balloon times were described using simple box plots. These times were then entered as independent variables in the above-mentioned multivariable Cox proportional hazards models that included all of the baseline clinical variables, to determine whether adjustment for door-to-balloon times changed the point estimates of our adjusted HRs and were therefore a mediating factor of any potential associations of time of day with mortality. An additional sensitivity analysis using door-to-balloon time ≤90 min or >90 min (according to current guidelines for optimal performance of primary PCI) was performed, with door-to-balloon time first assessed as a potential confounding variable (through inclusion in the multivariable models), and then as an effect modifier (through stratification on door-to-balloon time).

n time ≤90 min or >90 min (according to current guidelines for optimal performance of primary PCI) was performed, with door-to-balloon time first assessed as a potential confounding variable (through inclusion in the multivariable models), and then as an effect modifier (through stratification on door-to-balloon time). We performed a meta-analysis of our study's RR for after-hours PCI along with other published studies, to place our findings in the context of what is already known about this important question. A detailed literature search identified all published manuscripts on this topic. The search strategy and study selection procedures are available from the authors upon request. Because of heterogeneity noted in the relative risks across studies (τ2 0.051, p=0.02), a random effects model was chosen for pooling of results across studies. Statistical analyses were performed using SAS Version 8.1. The meta-analysis was performed using Stata Version 8. Results Patient characteristics From 1 January 1999 to 31 March 2006, 1664 patients underwent primary PCI for acute MI in Alberta. Of these, 54.4% occurred after hours. Table 1 shows the baseline characteristics of regular working hours and after-hours cases. There were no significant differences between the groups in terms of cardiac risk factors, comorbidities, ejection fraction, extent of coronary disease or culprit vessel, with the exception of a higher use of intra-aortic balloon counterpulsation devices (alone or in combination with inotropes) during working hours, and higher use of inotropes alone after-hours.

n the groups in terms of cardiac risk factors, comorbidities, ejection fraction, extent of coronary disease or culprit vessel, with the exception of a higher use of intra-aortic balloon counterpulsation devices (alone or in combination with inotropes) during working hours, and higher use of inotropes alone after-hours. Crude and adjusted outcomes Mortalities at 30 days were 3.6% in the working hours group and 5.0% in the after-hours group (p=0.16). By 1 year, mortalities were 6.2% and 7.3% in the working hours and after-hours groups, respectively (p=0.35). Figure 1 shows Kaplan–Meier survival curves extending to 1 year of follow-up. After-hours patients do appear to have a poorer survival over time. Figure 1 Kaplan–Meier survival curves to 1 year for primary percutaneous coronary intervention (PCI) performed after hours and during regular working hours. Table 2 shows the HRs and 95% CIs of 1.34 (95% CI 0.85 to 2.12) for after-hours cases relative to working hours cases for survival extending to 30 days (our primary study outcome) and 1.18 (95% CI 0.81 to 1.72) for survival extending to 1 year. After adjusting for the variables in table 1, HRs (HR-1) changed slightly to 1.26 (95% CI 0.78 to 2.02) for survival to 30 days and 1.08 (95% CI 0.73 to 1.59) for survival to 1 year. Table 2 Crude and Adjusted Hazard Ratio for survival for after-hours relative to regular hours primary PCI

Table 2 shows the HRs and 95% CIs of 1.34 (95% CI 0.85 to 2.12) for after-hours cases relative to working hours cases for survival extending to 30 days (our primary study outcome) and 1.18 (95% CI 0.81 to 1.72) for survival extending to 1 year. After adjusting for the variables in table 1, HRs (HR-1) changed slightly to 1.26 (95% CI 0.78 to 2.02) for survival to 30 days and 1.08 (95% CI 0.73 to 1.59) for survival to 1 year. Table 2 Crude and Adjusted Hazard Ratio for survival for after-hours relative to regular hours primary PCI Crude HR (95% CI) HR-1 adjusted* (95% CI) Adjusted HR-2† (95% CI) 30-day survival 1.34 (0.85 to 2.12) 1.26 (0.78 to 2.02) 1.23 (0.77 to 1.99) 1-year survival 1.18 (0.81 to 1.72) 1.08 (0.73 to 1.59) 1.06 (0.71 to 1.56) * HR-1 adjusted for all variables in table 1. † HR-2 adjusted for variables in table 1 plus door-to-balloon time. Analysis controlling for door-to-balloon times The median door-to-balloon time was 72.0 min in the working hours group and 80.0 min in the after-hours group (p=0.007), as demonstrated by the box plots in figure 2. Figure 2 Boxplots illustrating door-to-balloon times for primary percutaneous coronary intervention performed after hours and during regular working hours. The median door-to-balloon time is indicated. The boundaries of the box plots refer to the 25th and 75th percentiles, with the whisker bars representing the 5th and 95th percentiles.

illustrating door-to-balloon times for primary percutaneous coronary intervention performed after hours and during regular working hours. The median door-to-balloon time is indicated. The boundaries of the box plots refer to the 25th and 75th percentiles, with the whisker bars representing the 5th and 95th percentiles. Table 2 also presents the HRs for survival associated with after-hours procedures, further adjusted for door-to-balloon times (HR-2, 1.23 (95% CI 0.77 to 1.99)). The sequential analysis for survival to 1 year also revealed a minimal effect of this additional adjustment. The full Cox regression model can be found in appendix 2. An additional sensitivity analysis performed using the door-to-balloon time cutpoints of ≤90 min and >90 min, treated as a confounding variable through inclusion in the multivariable models, yielded HRs that were essentially the same (30-day survival HR 1.23 (95% CI 0.77 to 1.99)). When considered as an effect modifier in stratified analyses, we found a stronger association for those with longer door-to-balloon times (≤90 min HR 1.23 (95% CI 0.63 to 2.42); >90 min HR 1.53 (95% CI 0.76 to 3.09)).

le models, yielded HRs that were essentially the same (30-day survival HR 1.23 (95% CI 0.77 to 1.99)). When considered as an effect modifier in stratified analyses, we found a stronger association for those with longer door-to-balloon times (≤90 min HR 1.23 (95% CI 0.63 to 2.42); >90 min HR 1.53 (95% CI 0.76 to 3.09)). Meta-analysis To present our study result more explicitly in the context of the existing literature, we performed a meta-analysis of studies examining outcomes in after-hours primary PCI (figure 3). The studies ranged from single centre experiences to large registries, and one clinical trial of PCI strategies (CADILLAC), conducted from 1994 to 2006. Several excluded cardiogenic shock, rescue PCI or transfer patients.17–21 A tabulated description of these studies is presented in appendix 1. Unadjusted risk ratios ranged from 0.61 to 6.54, with an overall random-effect pooled estimate of RR of 1.23 (95% CI 1.00 to 1.52). This pooled result across 12 studies, including ours, does suggest that there may still be a need to continue exploring the possibility of an association between after-hours procedures and poorer outcomes. Figure 3 Meta-analysis of studies examining outcomes of primary percutaneous coronary intervention performed after hours and during regular working hours.

Meta-analysis To present our study result more explicitly in the context of the existing literature, we performed a meta-analysis of studies examining outcomes in after-hours primary PCI (figure 3). The studies ranged from single centre experiences to large registries, and one clinical trial of PCI strategies (CADILLAC), conducted from 1994 to 2006. Several excluded cardiogenic shock, rescue PCI or transfer patients.17–21 A tabulated description of these studies is presented in appendix 1. Unadjusted risk ratios ranged from 0.61 to 6.54, with an overall random-effect pooled estimate of RR of 1.23 (95% CI 1.00 to 1.52). This pooled result across 12 studies, including ours, does suggest that there may still be a need to continue exploring the possibility of an association between after-hours procedures and poorer outcomes. Figure 3 Meta-analysis of studies examining outcomes of primary percutaneous coronary intervention performed after hours and during regular working hours. Discussion Our study adds to a growing body of literature on after-hours medical care. In an unselected patient population, outcomes for after-hours PCI cases did not differ significantly from those of working-hours cases. However, the point estimate from our study suggesting a 23% increased risk for adverse events early after PCI needs to be taken in the context of other studies, some of which have shown poorer outcomes in after-hours primary PCI. Further, the 23% increase seen in this study and our meta-analysis of prior studies is hardly negligible in that it is of similar magnitude to the benefits associated with beta-blocker and thrombolytic therapy for STEMI.22 23

context of other studies, some of which have shown poorer outcomes in after-hours primary PCI. Further, the 23% increase seen in this study and our meta-analysis of prior studies is hardly negligible in that it is of similar magnitude to the benefits associated with beta-blocker and thrombolytic therapy for STEMI.22 23 Interest in after-hours care has heightened with increased international focus on patient safety. Such issues have received considerable attention in relation to studies that have demonstrated increased mortality in patients with severe medical conditions admitted on weekends, as a direct result of delayed care.24 Kostis and colleagues also found that weekend admissions for patients with MI were associated with a higher mortality.25

eceived considerable attention in relation to studies that have demonstrated increased mortality in patients with severe medical conditions admitted on weekends, as a direct result of delayed care.24 Kostis and colleagues also found that weekend admissions for patients with MI were associated with a higher mortality.25 With primary PCI, concerns about outcomes are most important after-hours cases where the need to bring cardiac catheterisation laboratory staff from home may result in significant treatment delays. Previous investigations show conflicting results. Garot and colleagues assessed 288 primary PCI patients and found similar door-to-balloon times and no differences in in-hospital outcome.17 However, this study was conducted in a French centre which activates the cardiac catheterisation laboratory from the ambulance and is staffed after-hours by in-house nurses. It is difficult to apply the findings of this study to areas which lack these policies. Zahn et al examined the outcomes of 378 patients treated during regular working hours and 113 patients treated after-hours, where mortality was lower (5.3% vs 8.7%) in the after-hours group.26 However, eight facilities participated during working hours but fewer at night, raising the possibility of selection bias in after-hours cases. Data from the 2082 patients enrolled in the larger randomised CADILLAC primary PCI trial found that patients who presented after hours had similar 30-day and 1-year mortalities to those presenting during working hours.18 In contrast, in 1702 consecutive primary PCI cases, Henriques et al found that patients treated off-hours had a higher incidence of failed PCI and worse clinical outcomes, including increased 30-day mortality.27

presented after hours had similar 30-day and 1-year mortalities to those presenting during working hours.18 In contrast, in 1702 consecutive primary PCI cases, Henriques et al found that patients treated off-hours had a higher incidence of failed PCI and worse clinical outcomes, including increased 30-day mortality.27 We noted a lack of effect of controlling for door-to-balloon times on our point estimate of RR, even when using the accepted clinical cutpoint of ≤90 min as a confounding variable. When treated as an effect modifier in stratified analyses, we found a stronger association of hazard for those with longer door-to-balloon times (>90 min), suggesting that the impact of the after-hours construct is even greater when treatment is delayed. These findings, and the potential signal of harm suggested by the meta-analysis presented here, require us to consider other possible contributing explanations for increased mortality in after-hours patients. One possibility is that physician fatigue could influence procedural performance, well represented in the anaesthesia literature.6–8 In addition, in a study of the effect of heavy night call in residents, Arnedt et al found that postcall impairment was at least equivalent to the ingestion of 3–4 standard alcoholic drinks.9 Other investigators have found that manual dexterity and surgical skills may be specifically vulnerable to sleep deprivation.10–12 28

dition, in a study of the effect of heavy night call in residents, Arnedt et al found that postcall impairment was at least equivalent to the ingestion of 3–4 standard alcoholic drinks.9 Other investigators have found that manual dexterity and surgical skills may be specifically vulnerable to sleep deprivation.10–12 28 Staffing levels also tend to be lower on weekends and holidays than during working hours, despite often increased patient acuity, and are a potential contributor to suboptimal patient safety at such times.29–31 Another important concern relevant to our cohort relates to the fact that all revascularisation procedures in Alberta are performed in academic tertiary care centres, and overnight care outside the cardiac catheterisation laboratory is generally provided by junior housestaff. Serious medical errors and pronounced increases in after-hours mortality have both been demonstrated in major teaching hospitals, whereas after-hours admissions to tertiary care intensive care units with on-site attending physicians are not associated with increased mortality.13 31–33 Thus, the combination of relatively inexperienced housestaff, low staffing and fatigue among providers may be responsible for some of the suggestion of increased hazard associated with after-hours primary PCI.

re intensive care units with on-site attending physicians are not associated with increased mortality.13 31–33 Thus, the combination of relatively inexperienced housestaff, low staffing and fatigue among providers may be responsible for some of the suggestion of increased hazard associated with after-hours primary PCI. There are limitations to this study. Like other investigators studying acute MI care, we do not have any data regarding symptom onset-to-balloon time, which is difficult to characterise at night, as the perceived time of symptom onset may not reliably reflect actual ischaemic time, and patients who are at home when symptoms occur may be less likely to promptly seek medical attention. All PCI procedures were performed by experienced operators at high-volume academic centres, so our results may not be generalisable to patients in other settings, or to hospitals that do not rely upon trainees for major provision of after-hours care. Finally, our thoughts as to the other potential influences on after-hours outcomes remain speculative, as none of the above-mentioned studies are specific to cardiology. The above notwithstanding, our findings do not support abandoning after-hours primary PCI in favour of thrombolysis. Given that the major studies of primary PCI versus thrombolysis would have included at least some after-hours patients in both treatment arms, it is unlikely that the benefit of primary PCI would be entirely negated after-hours. In addition, potential factors influencing outcomes after-hours could also apply to patients receiving thrombolysis.

ies of primary PCI versus thrombolysis would have included at least some after-hours patients in both treatment arms, it is unlikely that the benefit of primary PCI would be entirely negated after-hours. In addition, potential factors influencing outcomes after-hours could also apply to patients receiving thrombolysis. In conclusion, our study findings suggest that primary PCI can be performed outside a clinical trial with acceptable short- and long-term mortalities, during working hours and after-hours. However, our findings taken in the context of other after-hours primary PCI studies, with an almost 25% increase in the risk for short-term mortality, do not provide complete reassurance; nor do they indicate complete equivalency of outcomes to working-hours procedures. This summary finding remains a concern and may be related to previously unexplored areas in after-hours care. Patient satisfaction will also need to be considered. Further research is thus still required to determine whether processes and quality of care are influenced by understudied areas such as fatigue, staffing levels, physician experience or other factors. We would like to thank R Boone, for his thoughtful comments. We appreciate the assistance of the Calgary Health Region and the Capital Health Authority in supporting data entry by cardiac catheterisation laboratory personnel.

In conclusion, our study findings suggest that primary PCI can be performed outside a clinical trial with acceptable short- and long-term mortalities, during working hours and after-hours. However, our findings taken in the context of other after-hours primary PCI studies, with an almost 25% increase in the risk for short-term mortality, do not provide complete reassurance; nor do they indicate complete equivalency of outcomes to working-hours procedures. This summary finding remains a concern and may be related to previously unexplored areas in after-hours care. Patient satisfaction will also need to be considered. Further research is thus still required to determine whether processes and quality of care are influenced by understudied areas such as fatigue, staffing levels, physician experience or other factors. We would like to thank R Boone, for his thoughtful comments. We appreciate the assistance of the Calgary Health Region and the Capital Health Authority in supporting data entry by cardiac catheterisation laboratory personnel. Funding: The research and creation of this paper were supported by a grant from the Canadian Institutes of Health Research (CIHR). MLK receives partial support from the Libin Trust Fund. WAG is a Senior Health Scholar of the Alberta Heritage Foundation for Medical Research and also supported by a government of Canada Research Chair in Health Services Research and by a Health Scholar Award from the Alberta Heritage Foundation for Medical Research, Edmonton, Alberta. APPROACH was funded in 1995 by the Weston Foundation, with ongoing support from the Canadian Cardiovascular Outcomes Research Team (CCORT), a CIHR Team Grant and the Province-Wide Services Committee of Alberta Health and Wellness. The initiative has also received unrestricted support from Merck Frosst Canada, Monsanto Canada—Searle, Eli Lilly Canada, Guidant Corporation, Boston Scientific, Hoffmann-La Roche and Johnson & Johnson Inc—Cordis.

h Team (CCORT), a CIHR Team Grant and the Province-Wide Services Committee of Alberta Health and Wellness. The initiative has also received unrestricted support from Merck Frosst Canada, Monsanto Canada—Searle, Eli Lilly Canada, Guidant Corporation, Boston Scientific, Hoffmann-La Roche and Johnson & Johnson Inc—Cordis. Competing interests: None. Ethics approval: Ethics approval was provided by the University of Calgary and University of Alberta. Contributors: WAG participated in the study design, analysis and manuscript revision. MLK and MT participated in study design and manuscript revision. DAS participated in the analysis and manuscript revision. The members of the APPROACH steering committee mentioned in the acknowledgements section have also read and approved submission of this paper. Provenance and peer review: Not commissioned; externally peer reviewed.

Contributors: WAG participated in the study design, analysis and manuscript revision. MLK and MT participated in study design and manuscript revision. DAS participated in the analysis and manuscript revision. The members of the APPROACH steering committee mentioned in the acknowledgements section have also read and approved submission of this paper. Provenance and peer review: Not commissioned; externally peer reviewed. Appendix 1 Study Overall N Location Inclusion criteria Exclusion criteria Working hours mortality After-hours mortality RR (95% CI) Dominguez-Rodriguez et al34 90 Spain, 2003 Consecutive primary PCI, single centre None identified 1/51 (1.9%) in hospital 5/39 (12.8%) 6.54 (0.80 to 53.72) Assali et al20 273 Israel, 2001–2004 Consecutive primary PCI, single centre Cardiogenic shock 2/160 (1.25%) in hospital 5/160 (3.1%) at 30 days 7/113 (6.2%) 11/113 (9.7%) 4.96 (1.05 to 23.42) Ortolani et al21 985 Italy, 2003–2005 Consecutive primary PCI, single centre Rescue PCI, in-hospital ST-segment elevation myocardial infarction 29/382 (7.6%) in hospital 49/603 (8.1%) 1.06 (0.69 to 1.63) Saleem et al35 1050 USA, 1998–2002 Consecutive primary PCI, single centre 21/656 (3.2%) in hospital 23/394 (5.8%) 1.82 (1.02 to 3.25) Sadeghi et al18 2036 International CADILLAC randomized controlled trial, all sites 24/7 primary PCI Shock, bleeding, renal insufficiency 17/1047 (1.6%) at 30 days 24/989 (2.4%) 1.49 (0.81 to 2.76) Henriques et al27 1702 Netherlands, 1994–2000 Consecutive primary PCI, within 6 h, single centre Symptom onset >6 h 17/909 (1.0%) at 30 days 33/793 (4.2%) 1.72 (1.18 to 2.51) Magid et al19 33647 USA, 1999–2002 NRMI registry, PCI at 421 centres Transfer patients 728/15419 (4.7%) 859/18228 (4.7%) in hospital 1.0 (0.91 to 1.10) Slonka et al36 1778 Poland, 1998–2003 Consecutive primary PCI, single centre, working hours defined as 0800–1500 33/482 (6.8%) in hospital 80/1296 (6.2%) 0.90 (0.61 to 1.33) Srimachochota37 256 Thailand, 1999–2003 Consecutive primary PCI, single centre 11/107 (10.3%) in hospital 16/149 (10.7%) 1.04 (0.51 to 2.16) Zahn et al26 491 Germany, 1994–1997 MITRA registry, consecutive primary PCI at eight centres during the day and three centres at night (concern for selection bias—23% of patients done after-hours) 33/378 (8.7%) in hospital 6/113 (5.3%) 0.61 (0.26 to 1.41) Garot et al17 288 France Consecutive primary PCI, <6 h after symptom onset, cath lab activated by cath lab staffed after hours by CCU nurses Shock 6/113 (5.3%) 12/175 (6.9%) 1.29 (0.50 to 3.34) Graham 2043 Alberta, 1999–2006 Consecutive primary PCI, three centres Transfer patients 32/89

6/113 (5.3%) 0.61 (0.26 to 1.41) Garot et al17 288 France Consecutive primary PCI, <6 h after symptom onset, cath lab activated by cath lab staffed after hours by CCU nurses Shock 6/113 (5.3%) 12/175 (6.9%) 1.29 (0.50 to 3.34) Graham 2043 Alberta, 1999–2006 Consecutive primary PCI, three centres Transfer patients 32/89 6 (3.6%) 57/1147 (5.0%) 1.39 PCI, percutaneous coronary intervention.

6/113 (5.3%) 0.61 (0.26 to 1.41) Garot et al17 288 France Consecutive primary PCI, <6 h after symptom onset, cath lab activated by cath lab staffed after hours by CCU nurses Shock 6/113 (5.3%) 12/175 (6.9%) 1.29 (0.50 to 3.34) Graham 2043 Alberta, 1999–2006 Consecutive primary PCI, three centres Transfer patients 32/89 6 (3.6%) 57/1147 (5.0%) 1.39 PCI, percutaneous coronary intervention. Appendix 2 Full Cox regression model (30 days and 1 year) 30-day Model fit statistics Criterion Without covariates With covariates –2 log L 1153.810 1050.941 AIC 1153.810 1090.941 SBC 1153.810 1138.075 Testing global null hypothesis: beta=0 Test χ2 df Pr>χ2 Likelihood ratio 102.8685 20 <0.0001 Score 148.7902 20 <0.0001 Wald 120.9245 20 <0.0001 Analysis of maximum likelihood estimates Parameter df Parameter estimate SE χ2 Pr>χ2 HR 95% CI After hour 1 0.22990 0.24166 0.9050 0.3414 1.26 0.78 to 2.02 Age 1 0.41968 0.26344 2.5379 0.1111 1.52 0.91 to 2.55 Sex 1 −0.62751 0.24654 6.4781 0.0109 0.53 0.33 to 0.87 COPD 1 −0.05550 0.39203 0.0200 0.8874 0.95 0.44 to 2.04 CEVD 1 0.51473 0.38828 1.7574 0.1849 1.67 0.78 to 3.58 Creat 1 1.20996 0.38643 9.8038 0.0017 3.35 1.57 to 7.15 Diabetes 1 0.69557 0.26984 6.6446 0.0099 2.01 1.18 to 3.40 Dialysis 1 −0.21161 0.71513 0.0876 0.7673 0.81 0.20 to 3.29 HTN 1 −0.64629 0.25774 6.2878 0.0122 0.52 0.32 to 0.87 Lipid 1 −1.04936 0.29260 12.8618 0.0003 0.35 0.20 to 0.62 Liver/GI 1 0.25309 0.64277 0.1550 0.6938 1.29 0.37 to 4.54 Malignancy 1 −1.02538 1.02202 1.0066 0.3157 0.36 0.05 to 2.66 Old MI 1 −0.15068 0.37330 0.1629 0.6865 0.86 0.41 to 1.79 Lytic 1 −0.12468 0.72190 0.0298 0.8629 0.88 0.21 to 3.63 PVD 1 0.87345 0.36897 5.6040 0.0179 2.40 1.16 to 4.94 Ef 35 1 −0.51292 0.28934 3.1425 0.0763 0.60 0.34 to 1.06 Ef 20 1 0.09989 0.40171 0.0618 0.8036 1.11 0.50 to 2.43 Ef under20 1 1.08157 1.06411 1.0331 0.3094 2.95 0.37 to 23.74 d1 1 0.83690 0.26202 10.2018 0.0014 2.31 1.38 to 3.86 d2 1 1.39078 0.40562 11.7563 0.0006 4.02 1.81 to 8.90 1-year Model fit statistics Criterion Without covariates With covariates –2 log L 1668.884 1528.640 AIC 1668.884 1568.640 SBC 1668.884 1623.187 Testing global null hypothesis: beta=0 Test χ2 df Pr>χ2 Likelihood ratio 140.2445 20 <0.0001 Score 210.1061 20 <0.0001 Wald 166.3952 20 <0.0001 Analysis of maximum likelihood estimates Parameter df Parameter estimate SE χ2 Pr>χ2 HR 95% CI After hour 1 0.07764 0.19810 0.1536 0.6951 1.08 0.73 to 1.59 Age 1 0.72647 0.21061 11.8978 0.0006 2.07 1.37 to 3.12 Sex 1 −0.49431 0.20938 5.5735 0.0182 0.61 0.41 to 0.92 COPD 1 −0.40181 0.34744 1.3375 0.2475 0.67 0.34 to 1.32 CEVD 1

of maximum likelihood estimates Parameter df Parameter estimate SE χ2 Pr>χ2 HR 95% CI After hour 1 0.07764 0.19810 0.1536 0.6951 1.08 0.73 to 1.59 Age 1 0.72647 0.21061 11.8978 0.0006 2.07 1.37 to 3.12 Sex 1 −0.49431 0.20938 5.5735 0.0182 0.61 0.41 to 0.92 COPD 1 −0.40181 0.34744 1.3375 0.2475 0.67 0.34 to 1.32 CEVD 1 0.61525 0.31582 3.7951 0.0514 1.85 1.00 to 3.44 Creat 1 1.32714 0.31433 17.8261 <0.0001 3.77 2.04 to 6.98 Diabetes 1 0.81857 0.21788 14.1147 0.0002 2.27 1.48 to 3.48 Dialysis 1 −0.49067 0.61541 0.6357 0.4253 0.61 0.18 to 2.05 HTN 1 −0.50147 0.21114 5.6409 0.0175 0.61 0.40 to 0.92 Lipid 1 −0.67676 0.22209 9.2852 0.0023 0.51 0.33 to 0.79 Liver/GI 1 0.28563 0.49074 0.3388 0.5605 1.33 0.51 to 3.48 Malignancy 1 0.43663 0.44218 0.9750 0.3234 1.55 0.65 to 3.68 Old MI 1 0.05295 0.28561 0.0344 0.8529 1.05 0.60 to 1.85 Lytic 1 −0.10150 0.59082 0.0295 0.8636 0.90 0.28 to 2.89 PVD 1 0.82531 0.31353 6.9289 0.0085 2.28 1.24 to 4.22 Ef 35 1 −0.58321 0.24534 5.6509 0.0174 0.56 0.35 to 0.90 Ef 20 1 0.08877 0.33432 0.0705 0.7906 1.09 0.57 to 2.10 Ef under20 1 0.37509 1.05195 0.1271 0.7214 1.46 0.19 to 11.44 d1 1 0.78256 0.21761 12.9322 0.0003 2.19 1.43 to 3.35 d2 1 1.44161 0.32632 19.5168 <0.0001 4.23 2.23 to 8.01 COPD – chronic pulmonary disease CEVD – cerebrovascular disease HTN – hypertension MI – myocardial infarction Lytic – thrombolytic therapy PVD – peripheral vascular disease Ef – Ejection Fraction

Introduction Patient safety represents a global public health problem which affects countries at all levels of development. WHO Patient Safety (formerly known as the World Alliance for Patient Safety) was established in 2004 to mobilise global efforts to improve the safety of healthcare for patients in all WHO Member States. WHO estimates that millions of patients worldwide suffer disabling injuries or death every year due to unsafe medical practices and care.1 While nearly one in ten patients is harmed while receiving healthcare in well-funded and technologically advanced hospital settings, there is little evidence about the burden of unsafe care in developing countries, where the risk of patient harm may be even greater due to limitations in infrastructure, technology and human resources, either in hospital or in primary care and community settings.

-funded and technologically advanced hospital settings, there is little evidence about the burden of unsafe care in developing countries, where the risk of patient harm may be even greater due to limitations in infrastructure, technology and human resources, either in hospital or in primary care and community settings. WHO Patient Safety gives special emphasis to research advancement as one of the essential building blocks for achieving safer care. Patient safety research is defined as: An action-oriented field of scientific enquiry that aims to determine: 1) the type and magnitude of harm caused by unsafe care; 2) the contributing factors and causal pathways that are potentially modifiable, including unsafe systems, processes and behaviours; and 3) cost-effective and locally adapted interventions that can successfully prevent, reduce or mitigate unsafe care to reduce harm. More knowledge - and better use of the knowledge available - are essential for understanding the extent and causes of patient harm, and for developing solutions that can be used in different contexts. To address the lack of research capacity in patient safety research worldwide, especially in developing and transitional countries, WHO Patient Safety convened a task force of world experts in patient safety research, curriculum development and research capacity strengthening from a wide range of countries in early 2008. The initial goal was to develop a set of core competencies for patient safety research to guide the development of education and training opportunities for promoting capacity strengthening in this area. Below, we describe the group's approach to competency development, present the competencies, and discuss implications and next steps. The full report is available on the WHO website (http://www.who.int/patientsafety/).2

velopment of education and training opportunities for promoting capacity strengthening in this area. Below, we describe the group's approach to competency development, present the competencies, and discuss implications and next steps. The full report is available on the WHO website (http://www.who.int/patientsafety/).2 Methods The 21-member international task force convened by WHO Patient Safety followed a seven-stage process (figure 1) to develop patient safety research competencies that would be applicable in developing, transitional and developed countries. The first stage involved preparing a background paper and initial framework for patient safety research competencies.3 These documents were discussed at the first meeting of the task force in February 2008 (Stage 2). An expanded literature review of competencies relating to patient safety, research and knowledge translation (Stage 3) was combined with the initial framework from Stage 1 and feedback from Stage 2 to create a preliminary list of patient safety research competencies (version 0.1). An internal consultation with the task force (Stage 4) used an on-line survey to identify key documents or competencies that did not emerge from the Stage 3 literature review, and to suggest changes to the content or format of the preliminary list of competencies. Feedback was incorporated into version 0.2 of the competencies. A large-scale external consultation with potential end users of the competencies (Stage 5) used a snowball technique to send an on-line survey to researchers, practitioners and policy-makers in developing and transitional countries working in the area of patient safety, to determine whether the list of competencies (version 0.2) would be easy to understand, appropriate to local contexts and useful for training future patient safety researchers. Feedback was used to revise the competencies to version 0.3. A second external consultation with international experts in patient safety (Stage 6) involved sending an online questionnaire to a convenience sample of 155 experts chosen from several key meeting and conference lists to assess the face validity of the proposed list of competencies. The sample, which included the external leads of WHO Patient Safety as well as advisory council members for the Research Programme, involved experts from the six WHO world regions, although the predominance was from high-income countries in Europe and North America where patient safety research has been more developed.

The sample, which included the external leads of WHO Patient Safety as well as advisory council members for the Research Programme, involved experts from the six WHO world regions, although the predominance was from high-income countries in Europe and North America where patient safety research has been more developed. Version 0.4 of the patient safety research competencies was presented to the task force for discussion at a 2-day conference (Stage 7) aimed at (a) achieving consensus on a first edition of the Patient Safety Research Competencies (version 1.0), as well as (b) identifying steps for further validation and dissemination of the competencies, and for their incorporation into existing or new training programmes in developing and transitional countries in particular. Figure 1 Seven-stage patient safety research competency development process.

Version 0.4 of the patient safety research competencies was presented to the task force for discussion at a 2-day conference (Stage 7) aimed at (a) achieving consensus on a first edition of the Patient Safety Research Competencies (version 1.0), as well as (b) identifying steps for further validation and dissemination of the competencies, and for their incorporation into existing or new training programmes in developing and transitional countries in particular. Figure 1 Seven-stage patient safety research competency development process. Results In January 2008, when the seven-stage process of competency development began, little had been published on the nature, boundaries and challenges of patient safety research.4–9 Since that time, a great deal more has emerged in the literature.10–18 The initial framework for patient safety research competencies that emerged from Stage 1 suggested that competencies may differ for different target audiences, different regional contexts and different levels of research advancement (table 1).3 The primary discussion points of the international task force (Stage 2) focused on (a) how to develop patient safety research competencies that would be relevant for developing and transitional countries, and (b) who should be the main target audience of patient safety research education and training. Table 1 Initial framework for patient safety research competencies*

Results In January 2008, when the seven-stage process of competency development began, little had been published on the nature, boundaries and challenges of patient safety research.4–9 Since that time, a great deal more has emerged in the literature.10–18 The initial framework for patient safety research competencies that emerged from Stage 1 suggested that competencies may differ for different target audiences, different regional contexts and different levels of research advancement (table 1).3 The primary discussion points of the international task force (Stage 2) focused on (a) how to develop patient safety research competencies that would be relevant for developing and transitional countries, and (b) who should be the main target audience of patient safety research education and training. Table 1 Initial framework for patient safety research competencies* Advanced knowledge and skills needed in the following competency areas: Academic researcher track Clinician researcher track Policy/ manager researcher track 1. Science of patient safety √ + √ 2. At least one patient safety subspecialty √ – – 3. Research design and methodology + √ √ 4. Conducting research + √ √ 5. Statistics and data analysis + √ √ 6. Knowledge translation √ √ + √, essential competency; –, competency could be de-emphasised; +, additional emphasis may be required. * Adapted from Ginsburg and Norton.3

Advanced knowledge and skills needed in the following competency areas: Academic researcher track Clinician researcher track Policy/ manager researcher track 1. Science of patient safety √ + √ 2. At least one patient safety subspecialty √ – – 3. Research design and methodology + √ √ 4. Conducting research + √ √ 5. Statistics and data analysis + √ √ 6. Knowledge translation √ √ + √, essential competency; –, competency could be de-emphasised; +, additional emphasis may be required. * Adapted from Ginsburg and Norton.3 Twenty-nine documents were identified for inclusion in the literature review and synthesis of competencies relating to patient safety research (Stage 3). These included documents on competencies for promoting patient safety,19–22 for conducting health services research23 24 and for knowledge translation.25 26 A preliminary list of patient safety research competencies was created (version 0.1) based on key themes which emerged from the literature (figure 2) relating to patient safety, research, and knowledge translation. Feedback on this preliminary list was obtained from nine out of 21 members (43%) of the task force (Stage 4). Respondents to the internal consultation highlighted the need for further attention to questions about whether and how patient safety research competencies may differ for different researcher profiles (eg, academic researcher, policy-maker, healthcare practitioner interested in research), different regional, social, economic and cultural contexts, and the extent to which the competencies take into account unique contextual issues in developing countries.

arch competencies may differ for different researcher profiles (eg, academic researcher, policy-maker, healthcare practitioner interested in research), different regional, social, economic and cultural contexts, and the extent to which the competencies take into account unique contextual issues in developing countries. Figure 2 Emerging themes relating to patient safety research competencies. In response to these concerns, Stage 5 solicited feedback from potential end users of the competencies in developing and transitional countries. The external consultation yielded usable data from 73 respondents in 35 developing and transitional countries across all six WHO regions. Respondents were largely early to mid-career academics or physicians involved in conducting research relating to patient safety (96%), with one-third focussing on safe medicines and devices, and another third focussing on healthcare associated infections. Respondents indicated that the proposed patient safety research competencies were easy to understand and relevant in developing and transitional countries, and would be useful for training patient safety researchers in these regions. Respondents also indicated that few patient safety research training opportunities presently exist (see table 2). At least half of respondents believed that each of the three competency areas is important for practitioners, policy-makers and academic researchers alike, thus indicating that a single list of competencies for all three profiles of patient safety researchers should be retained in version 0.3.

ntly exist (see table 2). At least half of respondents believed that each of the three competency areas is important for practitioners, policy-makers and academic researchers alike, thus indicating that a single list of competencies for all three profiles of patient safety researchers should be retained in version 0.3. Table 2 External consultation key results Potential end users from developing and transitional countries (Stage 5 respondents, n=73) International experts in patient safety research (Stage 6 respondents, n=46) Percentage reporting the competencies Are easy to understand 63 (86%) 39 (85%) Do not require modification 60 (82%) 32 (70%) Are well adapted to local contexts 64 (88%) 40 (87%) Would be useful for training patient safety researchers 73 (100%) 43 (93%) Percentage reporting the competencies would be useful As a systematic basis for training 60 (82%) 37 (80%) For defining learning objectives 55 (75%) 33 (72%) To emphasise different knowledge and skills needed 53 (73%) 37 (80%) As a basis to be tailored to different trainee profiles 37 (51%) 25 (54%) To evaluate the progress of trainees 57 (78%) 23 (50%) Competency area considered the main priority for training patient safety researchers in their own country: Patient safety theory and practice 31 (42%) NA Designing and conducting research 24 (33%) Translating findings into safer care 18 (25%)

Percentage reporting the competencies would be useful As a systematic basis for training 60 (82%) 37 (80%) For defining learning objectives 55 (75%) 33 (72%) To emphasise different knowledge and skills needed 53 (73%) 37 (80%) As a basis to be tailored to different trainee profiles 37 (51%) 25 (54%) To evaluate the progress of trainees 57 (78%) 23 (50%) Competency area considered the main priority for training patient safety researchers in their own country: Patient safety theory and practice 31 (42%) NA Designing and conducting research 24 (33%) Translating findings into safer care 18 (25%) Percentage aware of training opportunities for patient safety researchers in their country 13 (18%) NA Of the 46 international experts who responded in Stage 6, 85% found the competencies easy to understand, 87% considered them to be appropriate for local contexts, and 93% thought they would be helpful for training future patient safety researchers internationally (see table 2). They also emphasised the need to avoid the use of technical jargon (such as ‘disseminating’ and ‘surveillance’) and the need for subdividing competencies that cover multiple concepts and grouping together those where there is overlap.

helpful for training future patient safety researchers internationally (see table 2). They also emphasised the need to avoid the use of technical jargon (such as ‘disseminating’ and ‘surveillance’) and the need for subdividing competencies that cover multiple concepts and grouping together those where there is overlap. In the final stage of the competency development process (Stage 7), 17 of the 21 members of the task force, as well as three key informants, attended a consensus conference for an in-depth discussion of version 0.4 of the patient safety research competencies. During the conference, modifications were made to better reflect that the main target audience are researchers (as opposed to end users of research such as clinicians or policy-makers) and to ensure that the final document would employ appropriate terminology commonly used in the field of competency development. Consensus was reached that since all competencies contained basic and advanced levels, there was no need to specify competencies as basic or advanced for trainees at different academic levels. The first edition of the Competencies for Patient Safety Research (version 1.0—box 1) was agreed upon at the close of the meeting. Box 1 First edition of Core Competencies for Patient Safety Research Patient safety researchers are able to:Describe the fundamental concepts of the science of patient safety, in their specific social, cultural and economic context. These concepts include the following items, among others: 1.1. Basic definitions and foundational concepts, including human factors and organisational theory

Box 1 First edition of Core Competencies for Patient Safety Research Patient safety researchers are able to:Describe the fundamental concepts of the science of patient safety, in their specific social, cultural and economic context. These concepts include the following items, among others: 1.1. Basic definitions and foundational concepts, including human factors and organisational theory 1.2. The burden of unsafe care 1.3. The importance of a culture of safety 1.4. The importance of effective communication and collaboration in care delivery teams 1.5. The use of evidence-based strategies for improving the quality and safety of care 1.6. The identification and management of hazards and risks 1.7. The importance of creating environments for safe care 1.8. The importance of educating and empowering patients to be partners for safer care Design and conduct patient safety research. These competencies include the ability to perform, but are not necessarily restricted to, the following: 2.1. Search, appraise and synthesise the existing research evidence 2.2. Involve patients and carers in the research process starting with defining the research objectives 2.3. Identify research questions that address important knowledge gaps 2.4. Select an appropriate qualitative or quantitative study design to answer the research question 2.5. Conduct research using a systematic approach, valid methodologies and information technology 2.6. Employ valid and reliable data measurement and data analysis techniques 2.7. Foster interdisciplinary research teams and supportive environments for research 2.8. Write a grant proposal 2.9. Obtain research funding

2.4. Select an appropriate qualitative or quantitative study design to answer the research question 2.5. Conduct research using a systematic approach, valid methodologies and information technology 2.6. Employ valid and reliable data measurement and data analysis techniques 2.7. Foster interdisciplinary research teams and supportive environments for research 2.8. Write a grant proposal 2.9. Obtain research funding 2.10. Manage research projects 2.11. Write-up research findings and disseminate key messages 2.12. Evaluate the impact of interventions as well as feasibility and resource requirements 2.13. Identify and evaluate indicators of patient safety for use in monitoring and surveillance 2.14. Ensure professionalism and ethical conduct in research Be part of the process of translating research evidence to improve the safe care of patients. The skills involved include, but are not restricted to, the ability to contribute to the following: 3.1. Appraise and adapt research evidence to specific social, cultural and economic contexts 3.2. Use research evidence to advocate for patient safety 3.3. Define goals and priorities for making healthcare safer 3.4. Translate research evidence into policies and practices that reduce harm 3.5. Partner with key stakeholders in overcoming barriers to change 3.6. Promote standards and legal frameworks to improve safety 3.7. Institutionalise changes to build supportive systems for safer care 3.8. Apply financial information for knowledge translation 3.9. Promote leadership, teaching and safety skills.

3.4. Translate research evidence into policies and practices that reduce harm 3.5. Partner with key stakeholders in overcoming barriers to change 3.6. Promote standards and legal frameworks to improve safety 3.7. Institutionalise changes to build supportive systems for safer care 3.8. Apply financial information for knowledge translation 3.9. Promote leadership, teaching and safety skills. Discussion Patient safety research competencies represent the fundamental knowledge, ability, skills and expertise needed to carry out research in this area and to use patient safety research evidence to make healthcare safer. The current paper describes the methods and results of a multistage process, including literature reviews and consultations with experts and stakeholders that were used to develop patient safety research competencies. The competencies that emerged are intended to guide patient safety researchers in acquiring the knowledge and skills needed to conduct research that aims to better understand the magnitude and type of patient harm, identify potentially modifiable contributing factors, and design and test cost-effective and locally adapted interventions that can successfully prevent, reduce or mitigate unsafe care to reduce harm.

knowledge and skills needed to conduct research that aims to better understand the magnitude and type of patient harm, identify potentially modifiable contributing factors, and design and test cost-effective and locally adapted interventions that can successfully prevent, reduce or mitigate unsafe care to reduce harm. The first edition of Competencies for Patient Safety Research (version 1.0—box 1) proposed in this paper emphasises (a) understanding the science of patient safety, (b) conducting valid and ethical research and (c) translating research into practice. Our results indicate that both experts and potential end users believe the competencies can be tailored to different audiences, local and national contexts, and levels of educational attainment. Our results further suggest the core competencies have potential to be used by educational and research institutions in high- as well as low- and middle-income country contexts to inform curricula and the development of training programmes. In particular, the core competencies can be used to (a) provide a systematic basis for training, (b) define key learning objectives that should be covered, (c) demonstrate the many different types of knowledge and skills that are needed to carry out patient safety research, (d) tailor training programmes and curricula to different professional profiles, contexts and needs, and (e) help evaluate the progress of patient safety research trainees.

ives that should be covered, (c) demonstrate the many different types of knowledge and skills that are needed to carry out patient safety research, (d) tailor training programmes and curricula to different professional profiles, contexts and needs, and (e) help evaluate the progress of patient safety research trainees. Experience utilising the proposed patient safety research competencies will be essential to achieve a better understanding of the completeness, specificity and acceptability by different user groups in different socio-economic environments, as well as the feasibility of incorporating the competencies into a variety of educational programmes and training modalities. This will require the more general competency areas of ‘designing and conducting patient safety research’ and ‘translating research evidence to improve the safe care of patients’ to be adapted to the field of patient safety research by drawing upon context-specific examples, case studies, data collection tools and study designs that are particularly useful in trying to address research questions relating to patient safety.27

d ‘translating research evidence to improve the safe care of patients’ to be adapted to the field of patient safety research by drawing upon context-specific examples, case studies, data collection tools and study designs that are particularly useful in trying to address research questions relating to patient safety.27 Implications for policy and practice Relatively little is known about the epidemiology of patient safety in developing or transitional countries, and about what strategies will be effective in improving patient safety in these regions. Further research is therefore needed, which will require the creation of a critical mass of newly trained researchers able to fill the many knowledge gaps that exist. For many, the goal of education and training in patient safety research is to create research that leads to actions that improve patient safety. However, the link between training patient safety researchers and these more distal outcomes will likely be difficult to demonstrate in the short-term. Accordingly, process indicators such as increases in the number of courses offered in patient safety research, the number of graduates, the local adoption of evidence-based safe practices or the number of policies aimed at improving safety would also be important measures of success.

t to demonstrate in the short-term. Accordingly, process indicators such as increases in the number of courses offered in patient safety research, the number of graduates, the local adoption of evidence-based safe practices or the number of policies aimed at improving safety would also be important measures of success. Building research capacity is a long-term process that requires sustained effort, in terms of formal training opportunities but, more importantly, in providing a nurturing environment for trainees to continue to develop their knowledge and skills by being involved in conducting research. The Core Competencies for Patient Safety Research provide a framework for the ongoing education and training of patient safety researchers worldwide. Establishing formal training programmes at accredited academic institutions across all WHO regions may take many years. However, the growth of research networks and the availability of targeted funding to support patient safety research can already be used to promote research-capacity strengthening, particularly in developing and transitional countries where such research is urgently needed.

s across all WHO regions may take many years. However, the growth of research networks and the availability of targeted funding to support patient safety research can already be used to promote research-capacity strengthening, particularly in developing and transitional countries where such research is urgently needed. The authors would like to thank the members of the WHO Patient Safety Research Training and Education Expert Working Group for their important contribution to this work: Ahmed Al-Mandhari (Sultan Qaboos University Hospital, Sultanate of Oman); Antoine Geissbuhler (Hôpitaux Universitaires de Genève, Switzerland); John Goshbee (University of Michigan, USA); Festus Ilako (African Medical and Research Foundation,AMREF, Kenya); Mark Joshi (University of Nairobi, Kenya); Rakesh Lodha (All India Institute of Medical Sciences, India); Sergio Muñoz Navarro (Universidad de La Frontera, Chile); Hillegonda Maria Dutilh Novaes (University of São Paolo, Brazil); John Orav (Harvard School of Public Health, USA); Alan Pearson (Johanna Briggs Institute, Australia); Peter Pronovost (Johns Hopkins School of Medicine, USA); Miguel Recio Segoviano (Universidad Carlos III de Madrid, Spain); Jiruth Sriratanaban (Chulalongkorn University, Thailand); Naruo Uehara (Tohoku University, Japan); Steven Wayling (Special Programme for Research and Training in Tropical Diseases, World Health Organization, Switzerland); Maria Woloshynowych (Imperial College, United Kingdom); and Zhao Yue (Peking University People's Hospital, P.R. China). The authors would also like to thank Jason Frank (Royal College of Physicians and Surgeons of Canada, Canada) for sharing his expertise in the area of competency development, as well as the many interns who contributed at various stages to this project: Khalifa Elmusharaf (University of Khartoum, Sudan); Aimee McHale (University of North Carolina, USA); Jean-Louis Keene (McGill University, Canada); Naomi Dove (University of British Columbia, Canada); Ruramayi Rukini (University of Bristol, United Kingdom) and Shannon Gibson (University of Victoria, Canada). The authors are also grateful to all the experts and stakeholders from around the world who provided feedback on earlier drafts of this work.