Browse the corpus

ticipant may have been asked to verify medication names in five scenarios in Condition 1, but only one of the five scenarios contained a planted error in the medication name. Each scenario contained a maximum of one planted error. Further details regarding the scenarios are presented in online supplementary appendix 1. Participants Nurses from the ambulatory chemotherapy unit were recruited via a sign-up sheet located in the unit, and were eligible to participate if they worked in the unit and routinely administered chemotherapy at the time of the study. In accordance with institutional ethics protocols, nurses provided informed consent and were remunerated for their participation with an amount commensurate with their hourly wages. Participant characteristics are summarised in table 2. A χ2 test of homogeneity revealed no significant demographic differences between the two participant cohorts. Table 2 Characteristics of participants in preintervention and postintervention experiments

INTRODUCTION Non-adherence to medication is a ‘worldwide problem of striking magnitude’.1 It has consequences for the health of patients and is a great concern for healthcare providers in terms of patient outcomes and healthcare costs. Theoretical research has mainly focused on intentional non-adherence: for example, when people choose not to take their medication. However, unintentional non-adherence also accounts for a significant proportion of the problem: when people mean to take their medication in the right way but do not. This area is under-researched. In this paper, we bring a new perspective to this problem by exploring what contribution ‘resilience’ could make to it. Resilience engineering focuses on a system's ability to maintain performance, avoid error, compensate for poor circumstances and cope with disturbances. So, rather than focus on how things go wrong, we propose to exploit and enhance how things go right.

blem by exploring what contribution ‘resilience’ could make to it. Resilience engineering focuses on a system's ability to maintain performance, avoid error, compensate for poor circumstances and cope with disturbances. So, rather than focus on how things go wrong, we propose to exploit and enhance how things go right. Non-adherence The National Health Service in England processed 962 million prescriptions in 2011, which had an ingredient cost of £8.8 billion,2 and it is estimated that 30%–50% of prescribed medication are not taken correctly.3 A report prepared for a summit of European health ministers in 2012 estimated that non-adherence contributes approximately 57% of $500 billion total avoidable costs attributed to suboptimal medicine use globally each year.4 Horne et al3 provide a broad and comprehensive review of the research in this area. Their report shows that the problem is complex and multifaceted, and they call for more research to tackle this important issue. They argue that improving the effectiveness of adherence interventions could have a greater impact on public health than improvements in specific medical treatments.3

e research in this area. Their report shows that the problem is complex and multifaceted, and they call for more research to tackle this important issue. They argue that improving the effectiveness of adherence interventions could have a greater impact on public health than improvements in specific medical treatments.3 Haynes et al5 note that current successful interventions for non-adherence are multifaceted, complex, labour intensive and at best have modest effects; they call for innovative approaches to assist patients to follow prescriptions for medications. To help gain traction on this problem, different theoretical frameworks have been applied. Barber et al6 point out that the most commonly used explanations of non-adherence have their roots in psychology: for example, social cognition models such as the theory of reasoned action and the self-regulatory model. However, these give little insight into unintentional non-adherence.

l frameworks have been applied. Barber et al6 point out that the most commonly used explanations of non-adherence have their roots in psychology: for example, social cognition models such as the theory of reasoned action and the self-regulatory model. However, these give little insight into unintentional non-adherence. Barber7 and Barber et al6 have found theoretical leverage for understanding unintentional non-adherence by applying the human error literature to their data. Reason8 distinguishes between three broad categories of error that have different cognitive causes: slips and lapses, mistakes, and violations. Slips and lapses occur when someone has a momentary lapse of attention and forgets to do something or does the wrong thing in a routine procedure despite knowing what to do; mistakes are the misapplication of knowledge so that people make the wrong decision about what to do; and a violation is where people have the correct knowledge but intentionally choose to break rules. Barber et al6 applied this theory to data about instances of non-adherence and found that it provided a way of seeing different patterns, problems and solutions in the situation. First and foremost, it provided a structure for analysing instances of non-adherence in terms of their cognitive causes. This was particularly useful for gaining insight into causes of unintentional non-adherence, which are largely neglected in the literature. Second, it moved blame away from the patient to investigate the contribution of the wider system in cases where someone did not adhere. These include system level factors that are not purely cognitive, such as poor communication from medical staff, the patient's financial situation, their life style, poor labelling on medication and confusing instructions.

atient to investigate the contribution of the wider system in cases where someone did not adhere. These include system level factors that are not purely cognitive, such as poor communication from medical staff, the patient's financial situation, their life style, poor labelling on medication and confusing instructions. A resilience perspective Efforts to improve quality and safety in healthcare tend to focus on when things go wrong, for example, ‘to err is human’.9 However, this can be to the detriment of noticing the informal and interesting ways in which systems go right despite adversity.

atient to investigate the contribution of the wider system in cases where someone did not adhere. These include system level factors that are not purely cognitive, such as poor communication from medical staff, the patient's financial situation, their life style, poor labelling on medication and confusing instructions. A resilience perspective Efforts to improve quality and safety in healthcare tend to focus on when things go wrong, for example, ‘to err is human’.9 However, this can be to the detriment of noticing the informal and interesting ways in which systems go right despite adversity. Resilience engineering is a relatively new research field that focuses on the positive aspects of resilient systems:10 how systems anticipate, monitor, respond and learn so that they are able to avoid and recover from error and maintain successful performance.11 This perspective has been applied to emergency scenarios where systems have faced an extreme situation that they have not had to cope with before, and it has been applied to everyday aspects of system performance that keep a system working well. It has been applied to a wide variety of contexts within healthcare, including emergency departments,12 handover13 and surgery.14 In studies of programming infusion pumps, Furniss et al15 found that nurses organise their work in intelligent ways to improve their performance and to reduce the risk of making errors. Methodologically, it is important not just to hold errors and incidents up to scrutiny, but also to examine ‘normal’ practice to identify the intelligent ways in which things go right despite poor behaviour, poor design, poor systems and poor circumstances.16

e their performance and to reduce the risk of making errors. Methodologically, it is important not just to hold errors and incidents up to scrutiny, but also to examine ‘normal’ practice to identify the intelligent ways in which things go right despite poor behaviour, poor design, poor systems and poor circumstances.16 One area of theoretical and practical interest related to this work is the recognition and development of ‘resilience strategies’ which can help people avoid error and improve performance.16 17 These include creating cues to remember things, separating similar objects and tasks so that there is less likelihood of confusion, and checking you have got the correct resources before committing to an action. These strategies provide a new perspective on problems associated with adherence.

improve performance.16 17 These include creating cues to remember things, separating similar objects and tasks so that there is less likelihood of confusion, and checking you have got the correct resources before committing to an action. These strategies provide a new perspective on problems associated with adherence. A change of perspective for non-adherence Positive behaviours related to adherence have been noted in other studies, but they have not had the strength of theoretical motivation and focus we propose here. For example, Barber et al6 refer to the strategies that patients employ to help them adhere by adjusting routines and creating reminders, and Eliasson18 reported many strategies to aid adherence in patients prescribed imatinib (a life-saving drug for chronic myeloid leukaemia), such as taking medication with food, crushing tablets and dissolving them in water, using a TV programme and family members as a prompt, and using a visual prompt in the bathroom or kitchen. These strategies help the patients fit imatinib into their daily routine, use prompts to help them remember to take their medication and find ways of coping with side effects, which can all influence whether they adhere or not.19 These strategies help people to better cope with the challenges of adherence and reduce the likelihood of non-adherence, particularly if unintentional. Although we have dichotomised forms of non-adherence in this paper, as is common, they interact, which forms part of the story of resilience and adherence. For example, weak intent can lead some not to create strategies to guard against unintentional difficulties to ensure they take the drug reliably; in contrast, constant unintentional difficulties in managing to take a drug can lead to dwindling intent, particularly if there are no overt consequences of non-adherence. We focus in this paper on unintentional non-adherence because it is a significant problem that is under-theorised.

sure they take the drug reliably; in contrast, constant unintentional difficulties in managing to take a drug can lead to dwindling intent, particularly if there are no overt consequences of non-adherence. We focus in this paper on unintentional non-adherence because it is a significant problem that is under-theorised. A resilience perspective has concepts and frameworks that can be used to better explore unintentional non-adherence. For example, Furniss et al17 identify abstract categories of resilience strategies that can be used to recognise and identify resilience more broadly. Table 1 shows these abstract categories of resilience with examples, which illustrate their application to medication taking/adherence. Table 1 Resilience strategies with examples for adherence (adapted from Furniss et al17)

A resilience perspective has concepts and frameworks that can be used to better explore unintentional non-adherence. For example, Furniss et al17 identify abstract categories of resilience strategies that can be used to recognise and identify resilience more broadly. Table 1 shows these abstract categories of resilience with examples, which illustrate their application to medication taking/adherence. Table 1 Resilience strategies with examples for adherence (adapted from Furniss et al17) Resilience strategy Definition Example Cue creation to support prospective memory A cue is created as a reminder about something in the future Setting an alarm to remember to take medication at a particular time Premature-completion awareness Action is taken as a reminder about ‘X’ after the main goal has been achieved, where ‘X’ is normally a secondary task Leaving used and empty medication packaging out, rather than putting it straight into the bin, as a reminder to order more if it is needed Pre-emptive separation and disambiguation Things are separated or differentiated so they are not mixed up Moving similar looking pills into monitored dosing boxes or labelling them in a different and salient way Precommitment check Things are checked before committing to a course of action Making sure all the parts for home nebulisation of drugs are present before starting the procedure Managing resource availability Resources are managed so they are available for action Having medication at work and at home just in case it is forgotten at one location Routine adjustment Routine is adjusted in response to a threat or opportunity Adjusting time of taking medicines when travelling between time zones Safety reinforcement Where some safety barrier, procedure or practice is reinforced Double checking blood glucose levels and insulin dosage before injecting Some of the strategies in table 1 are observed in Eliasson;18 for example, using an alarm on a watch is an example of cue creation; taking medication as part of a set morning routine and avoiding the evening which could be more unpredictable and more prone to forgetting is one example of routine adjustment; and carrying around medication so that it is available wherever the patient may be is an example of managing resource availability. These categories allow one to reflect on strategies at a more abstract level. They provide a focus and a vocabulary so we can talk about these behaviours collectively and in new ways.

ent; and carrying around medication so that it is available wherever the patient may be is an example of managing resource availability. These categories allow one to reflect on strategies at a more abstract level. They provide a focus and a vocabulary so we can talk about these behaviours collectively and in new ways. Further research needs to explore the quality of individual strategies. For example, Furniss et al17 note that the quality of a cue depends on whether the cue is salient, linked to the thing to be recalled and task related. We might speculate that using a TV programme as a cue is vulnerable to failure because it is indirectly linked to taking medication, watching the programme is not task related and it depends on watching the TV. In contrast, using a mobile phone alarm as a cue could specify which medication to take, it can be snoozed until the medication has actually been taken so it integrates with the task more directly and it can be with the person while on the move.

ching the programme is not task related and it depends on watching the TV. In contrast, using a mobile phone alarm as a cue could specify which medication to take, it can be snoozed until the medication has actually been taken so it integrates with the task more directly and it can be with the person while on the move. Further research also needs to develop resilience theory. Eliasson18 reports problems and strategies that do not fit into table 1. For example, some patients reported difficulties in remembering whether they had taken their medication or not, which could lead them to take it again and overdosing (a form of non-adherence), or lead them to missing a dose for fear of taking it twice. This is an issue for retrospective memory that does not appear in table 1. Furniss et al17 note that the concepts they propose still require further research and development. Empirical observations such as these can be used to suggest new resilience strategies and in doing so can contribute to the development of this theory, as well as the theory contributing to improving adherence.

table 1. Furniss et al17 note that the concepts they propose still require further research and development. Empirical observations such as these can be used to suggest new resilience strategies and in doing so can contribute to the development of this theory, as well as the theory contributing to improving adherence. In addition to the categorisation of resilience strategies, the resilience markers framework16 can support understanding of resilience in practice by helping to describe how adherence takes place. Particularly, how performance is maintained in the rich varieties of ‘normal’ and more extreme contexts in which people try to adhere, and to include the effects of their illness, type of treatment, emotional stress, cognitive impairments, and patients’ social and financial situations. By focusing on these and other system level factors, we widen the possible explanations for adherence and non-adherence. The resilience markers framework elaborates how individuals in different contexts can use similar strategies, what dependencies might influence the use of different strategies and what difficulties they might be trying to overcome.

ystem level factors, we widen the possible explanations for adherence and non-adherence. The resilience markers framework elaborates how individuals in different contexts can use similar strategies, what dependencies might influence the use of different strategies and what difficulties they might be trying to overcome. Briefly, at the heart of the framework is a resilience repertoire which is a ‘library’ of strategies that can be adapted and deployed in different contexts; this would be something similar to the examples in table 1. The use of these strategies will depend on whether the person has the resources to use them, for example, you cannot set a phone alarm without a phone. It will also be influenced by the mode of operation where certain classes of behaviour will have an impact on the vulnerabilities of the system and the strategies that can be used. For example, if a patient is illiterate then strategies that involve reading will not help, and if a patient is alone then strategies that involve help from friends and family cannot help. More positively, if patients have a poor short term memory then developing support for this might be particularly helpful, and if they are comfortable with new technology then developing a strategy that involves this could contribute to a successful outcome.

tegies that involve help from friends and family cannot help. More positively, if patients have a poor short term memory then developing support for this might be particularly helpful, and if they are comfortable with new technology then developing a strategy that involves this could contribute to a successful outcome. The reasons for non-adherence are diverse and the strategies for tackling it need to be equally diverse, being tailored to individual patients and their circumstances. Patients, carers and clinical staff could create a library of strategies for broader populations of patients, carers and clinicians to adopt and adapt to support adherence. This resonates with the suggestion of Vermeire et al20 of a menu of compliance-enhancing strategies from which an appropriate strategy could be selected for individual patients and their treatment, but gives further detail and a theoretical basis to move forward from.

clinicians to adopt and adapt to support adherence. This resonates with the suggestion of Vermeire et al20 of a menu of compliance-enhancing strategies from which an appropriate strategy could be selected for individual patients and their treatment, but gives further detail and a theoretical basis to move forward from. Conclusions Resilience is more than just the avoidance of error; it can exploit and enhance successful performance in its own right. For adherence, a resilience perspective is a new approach that focuses on enhancing strategies to cope with adherence and reducing the likelihood of unintentional non-adherence. We have seen that episodes of resilient behaviour can be identified and reflected upon through more abstract categories of resilience strategies. We have also seen that understanding the variability of situations in which they are used can be supported by using the resilience markers framework. We need to understand the resilience strategies that patients develop, the best ways of sharing these strategies and testing whether other patients adopt them, adapt them and find them useful for successfully adhering to their own medication regimes. These efforts could work towards developing a library of resilience strategies that are pertinent to different communities of patients, carers and clinicians for use in different circumstances so that more people are able to manage their medications effectively. We would like to thank the reviewers for their comments, which have strengthened the paper.

Conclusions Resilience is more than just the avoidance of error; it can exploit and enhance successful performance in its own right. For adherence, a resilience perspective is a new approach that focuses on enhancing strategies to cope with adherence and reducing the likelihood of unintentional non-adherence. We have seen that episodes of resilient behaviour can be identified and reflected upon through more abstract categories of resilience strategies. We have also seen that understanding the variability of situations in which they are used can be supported by using the resilience markers framework. We need to understand the resilience strategies that patients develop, the best ways of sharing these strategies and testing whether other patients adopt them, adapt them and find them useful for successfully adhering to their own medication regimes. These efforts could work towards developing a library of resilience strategies that are pertinent to different communities of patients, carers and clinicians for use in different circumstances so that more people are able to manage their medications effectively. We would like to thank the reviewers for their comments, which have strengthened the paper. Contributors: NB had the initial idea for the paper. This idea has been explored, developed and written by DF in collaboration with NB. AB, LE and IL have contributed their knowledge and expertise to sharpen and validate ideas in the paper. Further, we have used examples in the paper that originate from a previous study by LE.

tors: NB had the initial idea for the paper. This idea has been explored, developed and written by DF in collaboration with NB. AB, LE and IL have contributed their knowledge and expertise to sharpen and validate ideas in the paper. Further, we have used examples in the paper that originate from a previous study by LE. Funding: This work was supported by the UK Engineering and Physical Sciences Research Council (EP/G059063/1). Competing interests: None. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Several reports, including the Institute of Medicine's To Err is Human1 and the Agency for Healthcare Research and Quality's The Effect of Health Care Working Conditions on Patient Safety2 have identified interruptions and distractions as factors contributing to medical errors. Distractions were cited as causal factors in nearly half of all medication error reports submitted to the United States national error-reporting database, and were the most frequently reported factor contributing to patient harm.3

have identified interruptions and distractions as factors contributing to medical errors. Distractions were cited as causal factors in nearly half of all medication error reports submitted to the United States national error-reporting database, and were the most frequently reported factor contributing to patient harm.3 Although interruptions may occur at any stage of the medication process, the medication administration stage is of particular interest because it represents the last opportunity for an error to be intercepted before reaching the patient.4 Nurses have cited interruptions and distractions as a top cause of errors during medication administration,5 and such interruptions are significantly associated with a variety of medication administration errors (eg, administering wrong medication, dose, infusion rate).6 Thus, there is a strong need to develop interventions that can reduce interruption-related errors during medication administration. To date, a variety of interventions have been proposed, including: prohibition of non-essential conversation, phone calls and pages7 8; use of ‘Do Not Disturb’ vests and signage9 10; use of a medication administration checklist9 10; and use of a clearly demarcated ‘No Interruption Zone’11 or physical barrier12 in medication preparation areas. Notably, most of the above interventions were designed to reduce the number of interruptions occurring during medication administration, with limited evaluation of the resulting impact on medication administration error rates. Indeed, a recent review suggests that there is only weak evidence regarding the effectiveness of such interventions in reducing interruptions and resulting medication errors.13 Thus, there is a need to develop effective interventions for interruption-related errors, and to assess the impact of these interventions on medication error rates.

ew suggests that there is only weak evidence regarding the effectiveness of such interventions in reducing interruptions and resulting medication errors.13 Thus, there is a need to develop effective interventions for interruption-related errors, and to assess the impact of these interventions on medication error rates. In a previous ethnographic study14 in an ambulatory chemotherapy unit at a large cancer centre in Toronto, we identified two broad categories of safety-critical tasks prone to interruptions (ie, medication verification tasks and medication administration tasks) that could lead to errors. Medication verification tasks consisted of checking the five rights of medication administration (ie, right patient, right medication, right dose, right route, right time), and were found to be primarily susceptible to errors of detection (eg, failing to notice a discrepancy between the medication order and medication label). In contrast, medication administration tasks such as administering medication via infusion pumps or intravenous push were found to be susceptible to errors of commission (eg, setting the wrong infusion rate). In the current study we aimed to (A) investigate the association, if any, between interruptions and medication verification and administration errors, (B) design interventions to reduce such errors in the presence of interruptions, and (C) assess the effectiveness of these interventions in reducing the identified medication verification and administration errors arising from interruptions. We conducted a simulation laboratory experiment to assess the effectiveness of interventions as a prerequisite to live clinical implementation.

interruptions, and (C) assess the effectiveness of these interventions in reducing the identified medication verification and administration errors arising from interruptions. We conducted a simulation laboratory experiment to assess the effectiveness of interventions as a prerequisite to live clinical implementation. Methodology The current work was conducted in three phases over a time period of 6 months. An overview of the three phases is shown in figure 1. Details of each phase are described in the following sections. Figure 1 An overview of the three phases, Phase A: Preintervention Experiment, Phase B: Intervention Design, Phase C: Postintervention Experiment. Phases A and C: preintervention and postintervention experiments Study setting Experiments conducted in phases A and C took place in a high-fidelity simulation laboratory, where nurses were asked to carry out medication verification and administration tasks within a highly realistic but controlled setting. This experimental design was chosen as it allows test administrators to make detailed observations of the impact of interruptions and interventions in a manner that would be impractical and unduly disruptive in a live clinical environment.

ation and administration tasks within a highly realistic but controlled setting. This experimental design was chosen as it allows test administrators to make detailed observations of the impact of interruptions and interventions in a manner that would be impractical and unduly disruptive in a live clinical environment. The simulation laboratory was equipped with theatre-style rooms, one-way glass and cameras (see online supplementary figures A1 and A2 in appendix 1) that allowed realistic simulation of an ambulatory chemotherapy unit, including patient beds, chairs, computerised physician order entry (CPOE) system, intravenous infusion equipment and paperwork. Manikins were used instead of patients. All medication bags, syringes, intravenous tubing sets, paper medication orders, medication labels and computerised medication order screens were identical to those used in the institution's regular practice. Coloured water or saline was used in place of real medications. An audio recording of a busy hospital unit was played throughout the experiment to provide realistic ambient noise. An actor-facilitator playing the role of a charge nurse guided participants through each scenario. To further recreate the busy, interruption-filled environment, actors played the roles of patients, family members and fellow nurses. Four actors participated in this study, playing the roles of a charge nurse, a family member and two patients. A fifth person, whose primary role was to assist the investigator in the observation room, also played the interjectory role of a physician. Additionally, three realistic patient manikins were placed in beds and chairs, thereby bringing the total number of mock patients to five. Thus, the simulated environment mimicked the cognitive load experienced by nurses working in the chemotherapy unit. Further details regarding the simulation setting are provided in online supplementary appendix 1.

tient manikins were placed in beds and chairs, thereby bringing the total number of mock patients to five. Thus, the simulated environment mimicked the cognitive load experienced by nurses working in the chemotherapy unit. Further details regarding the simulation setting are provided in online supplementary appendix 1. Study design An initial preintervention experiment was conducted to understand whether or not interruptions were associated with medication errors. Nurses were asked to perform medication verification and administration tasks under two conditions: uninterrupted (Condition 1) and interrupted (Condition 2). Thus, the experiment was a 2 (interruption condition)×7 (task type) within-subjects (repeated measures) design. The order of interruption and non-interruption tasks was counterbalanced to avoid carryover effects. Results emanating from the preintervention Condition 2 were used as a baseline (control) for the postintervention experiment. In other words, the postintervention experiment compared Condition 2 (where nurses were interrupted, with no interventions) to Condition 3 (interrupted, with interventions) using a between-subjects design. To permit comparability across the three conditions, equivalent scenarios, planted errors, and type/timing of interruptions (where applicable) were used in all conditions, as listed in table 1. The postintervention experiment took place approximately 2 months following the preintervention experiment, as the time in between was used to develop interventions (ie, Phase B).

equivalent scenarios, planted errors, and type/timing of interruptions (where applicable) were used in all conditions, as listed in table 1. The postintervention experiment took place approximately 2 months following the preintervention experiment, as the time in between was used to develop interventions (ie, Phase B). Table 1 Description of tasks, interruptions, planted errors, performance metrics and applicable interventions in simulation experiments

equivalent scenarios, planted errors, and type/timing of interruptions (where applicable) were used in all conditions, as listed in table 1. The postintervention experiment took place approximately 2 months following the preintervention experiment, as the time in between was used to develop interventions (ie, Phase B). Table 1 Description of tasks, interruptions, planted errors, performance metrics and applicable interventions in simulation experiments Task Number and timing of interruptions* Planted error Performance metric† Applicable intervention(s)‡ Medication verification tasks (assessment of error detection) 1. Verifying medication name Participant was required to verify medication name on label against electronic and paper medication orders During medication verification, participant was interrupted by: 2 requests from nursing colleague; 1 question from patient Medication name on label did not match name on medication orders. Sound-alike, look-alike medications were chosen (eg, Carboplatin vs Cisplatin) Task was coded as ‘fail’ if participant did not detect the planted error Verification booth, standardised workflow 2. Verifying medication dosage Participant was required to verify medication dosage on label against electronic and paper medication orders During medication verification, participant was interrupted by: 1 question from patient; 1 work-related call; 1 request from physician; 1 background infusion pump alarm Dosage on the medication label did not match that in the medication order ‘Fail’ if participant did not detect the planted error Verification booth, standardised workflow 3. Verifying medication volume in syringe Participant was required to verify medication volume on syringe label against electronic and paper medication orders During medication verification, participant was interrupted by: 1 request from patient's family The syringe contained an incorrect volume of medication (underfilled by 5 mL—a clinically significant amount) ‘Fail’ if participant did not detect the planted error Verification booth, standardised workflow 4. Verifying medication volume in ambulatory infusion pump (AIP) Participant was required to verify medication volume programmed in AIP against medication order and medication label During medication verification, participant was interrupted by: 1 question from nursing colleague, 1 question from patient's family The medication volume programmed in the AIP did not match that on the medication order ‘Fail’ if participant did not detect the planted error Verification booth, standardised workflow 5.

l During medication verification, participant was interrupted by: 1 question from nursing colleague, 1 question from patient's family The medication volume programmed in the AIP did not match that on the medication order ‘Fail’ if participant did not detect the planted error Verification booth, standardised workflow 5. Verifying patient identification (ID) Participant was required to verify patient name on medication label against the patient's armband During patient armband verification, participant was interrupted by: 1 question from patient, 1 request from nursing colleague The name on the medication label did not match that on the patient's armband. Sound-alike, look-alike names were chosen (eg, Pamela Chan vs Patricia Chan) ‘Fail’ if participant did not detect the planted error Speaking aloud Medication administration tasks (assessment of error commission) 6. Intravenous push Participant was required to administer a chemotherapeutic agent to a patient via manual intravenous push over the pharmacy-prescribed timeline of 6–10 min During the intravenous push, the participant was interrupted by: conversations from patient and family, 1 request from nursing colleague, 1 question from patient, repeated background infusion pump alarms No error was planted in this task ‘Fail’ if participant did not administer medication within pharmacy-prescribed timeline (eg, 6–10 min for vinorelbine) Visual timers 7. Pump programming and infusion initiation§ Participant was required to administer medication by correctly hanging the medication bag, closing the clamp of previously hanging medication tubing set, opening the clamp for medication to be delivered, and programming an infusion pump with the prescribed administration rate and volume of medication During pump programming and infusion initiation, the participant was interrupted by: requests from nursing colleague, requests from patient, conversations with patient's family, and background infusion pump alarms No error was planted in this task ‘Fail’ if participant programmed pump with incorrect rate or volume, forgot to open/close appropriate clamps, hung medication bags at incorrect heights such that the wrong medication was being infused, or forgot to start the infusion entirely No interruption zones with motion-activated indicators, speaking aloud, reminder signage *Applicable to Conditions 2 and 3 only. The number and timing of interruptions was kept consistent between the two conditions to permit comparability.

the wrong medication was being infused, or forgot to start the infusion entirely No interruption zones with motion-activated indicators, speaking aloud, reminder signage *Applicable to Conditions 2 and 3 only. The number and timing of interruptions was kept consistent between the two conditions to permit comparability. †Participants were instructed to report detected errors to the charge nurse (played by an actor-facilitator). ‡Applicable to postintervention condition (ie, Condition 3) only. §As described in online supplementary appendix 1, the pump programming task occurred in four of the five scenarios. Therefore, Pass/Fail performance was determined using collective criterion; that is, participants had to correctly program the pump in all four scenarios to receive a ‘Pass’.

‡Applicable to postintervention condition (ie, Condition 3) only. §As described in online supplementary appendix 1, the pump programming task occurred in four of the five scenarios. Therefore, Pass/Fail performance was determined using collective criterion; that is, participants had to correctly program the pump in all four scenarios to receive a ‘Pass’. Table 1 describes all tasks (some of which contained planted errors), interruptions and performance metrics pertinent to the simulation experiment. The tasks, planted errors and interruptions were designed based on extensive ethnographic observations gathered during a prior study in this care area.14 Specifically, interruptions were selected based on the frequency with which they occurred during each task, as observed during the ethnographic study. To further ensure that the experiment accurately reflected participants’ real-world practice, the tasks were presented to participants in realistic scenarios. Participants encountered each planted error only once per experiment, even if they performed that task in multiple scenarios. For example, a participant may have been asked to verify medication names in five scenarios in Condition 1, but only one of the five scenarios contained a planted error in the medication name. Each scenario contained a maximum of one planted error. Further details regarding the scenarios are presented in online supplementary appendix 1.

Participants Nurses from the ambulatory chemotherapy unit were recruited via a sign-up sheet located in the unit, and were eligible to participate if they worked in the unit and routinely administered chemotherapy at the time of the study. In accordance with institutional ethics protocols, nurses provided informed consent and were remunerated for their participation with an amount commensurate with their hourly wages. Participant characteristics are summarised in table 2. A χ2 test of homogeneity revealed no significant demographic differences between the two participant cohorts. Table 2 Characteristics of participants in preintervention and postintervention experiments Characteristic Participants in Phase A: Preintervention experiment (n=18) Participants in Phase C: Postintervention experiment (n=19) Age 18–29 years 5 3 30–39 years 8 7 40–49 years 3 5 50–65 years 1 3 >65 years 1 1 Sex Male 3 2 Female 15 17 Years of nursing experience <1 year 0 0 1–10 years 11 7 11–20 years 6 8 21–30 years 0 1 >31 years 1 3 Frequency of administering chemotherapy via infusion pumps <Once a week 1 5 1–5 times per week 12 7 2–3 times per day 0 0 >3 times per day 5 7 Experimental procedure At the start of the study, the investigator introduced the participant to the lab environment and briefly described the process of simulation testing. In the preintervention condition, participants were asked to start carrying out the medication verification and administration tasks. In the postintervention condition, the participant received 30 min of training on the interventions prior to carrying out the medication tasks. Specifically, in the training session, the investigator explained each applicable intervention and how to use it. The participant was then asked to practice using each intervention and resolve any doubts before starting the experiment. Actors playing the roles of family members and patients also assisted in the training process by providing interruptions during the participant's practice with interventions. The training process was concluded once the participant had demonstrated his/her ability to correctly use each intervention by successfully completing each practice task and using each intervention when applicable. The actor playing the role of the charge nurse then proceeded to start the experiment by directing the participant towards the first scenario.

ce the participant had demonstrated his/her ability to correctly use each intervention by successfully completing each practice task and using each intervention when applicable. The actor playing the role of the charge nurse then proceeded to start the experiment by directing the participant towards the first scenario. Data collection Two trained observers collected live data from an observation room located behind one-way glass while the experiment was in session. Specifically, observers documented errors (ie, Pass, Fail) on an Excel worksheet containing a list of all tasks. If there was an intervention for which compliance was dependent on the participant (eg, speaking aloud), observers additionally documented whether or not the intervention was used at each instance where an opportunity for use was present. Observers compared notes after each session to ensure consensus. Any discrepancies between observer notes were resolved by consulting video recordings of the session.

g, speaking aloud), observers additionally documented whether or not the intervention was used at each instance where an opportunity for use was present. Observers compared notes after each session to ensure consensus. Any discrepancies between observer notes were resolved by consulting video recordings of the session. Data analysis Data emanating from the experiment were coded according the criteria described in table 1 (‘Performance Metrics’ column). McNemar's χ2 test was used to assess differences in error rates between Conditions 1 and 2 in the preintervention experiment. Fisher's exact test was used to assess differences in error rates between Conditions 2 and 3 following the postintervention experiment. These comparisons were justified because all tasks, interruptions and scenarios were kept equivalent between the two experiments. An α of 0.05 was used for all statistical tests. All data were analysed using SPSS V.18.0 for Mac.

rences in error rates between Conditions 2 and 3 following the postintervention experiment. These comparisons were justified because all tasks, interruptions and scenarios were kept equivalent between the two experiments. An α of 0.05 was used for all statistical tests. All data were analysed using SPSS V.18.0 for Mac. Phase B: intervention development To ensure a participatory design approach (ie, an approach where key stakeholders and end-users are involved in intervention design), nine nurses from the chemotherapy unit who had participated in previous phases of the study were recruited to take part in focus groups, where they brainstormed potential error mitigation strategies and iterated upon the design of interventions. When appropriate, designs for interventions were sketched on paper. Qualitative input regarding nurses’ impressions of the potential effectiveness, uptake and feasibility of implementation of each solution was gathered during each discussion. Focus group data therefore served as a form of requirements gathering (supplemented by prior observational studies) to inform intervention design. The resulting interventions are described below. With the exception of the patient ID verification task, all other tasks employed multiple applicable interventions at a time (ie, interventions were employed as a system, as shown in table 1). Interventions for medication verification tasks (errors of detection)

Phase B: intervention development To ensure a participatory design approach (ie, an approach where key stakeholders and end-users are involved in intervention design), nine nurses from the chemotherapy unit who had participated in previous phases of the study were recruited to take part in focus groups, where they brainstormed potential error mitigation strategies and iterated upon the design of interventions. When appropriate, designs for interventions were sketched on paper. Qualitative input regarding nurses’ impressions of the potential effectiveness, uptake and feasibility of implementation of each solution was gathered during each discussion. Focus group data therefore served as a form of requirements gathering (supplemented by prior observational studies) to inform intervention design. The resulting interventions are described below. With the exception of the patient ID verification task, all other tasks employed multiple applicable interventions at a time (ie, interventions were employed as a system, as shown in table 1). Interventions for medication verification tasks (errors of detection) Verification Booth: Results of previous ethnography revealed that nurses were interrupted 57% of the time while verifying medication label information against the CPOE system.14 With this in mind, a ‘Verification Booth’ (figure 2A) was developed to provide nurses with a physically distinct quiet space to conduct verifications at computer stations. The booth was a transparent enclosure fitting around computer stations that allowed nurses to monitor and access their patients in case of medical emergency.15 Strategic signage was placed on the booth to remind passers-by of the criticality of tasks taking place within.

e to conduct verifications at computer stations. The booth was a transparent enclosure fitting around computer stations that allowed nurses to monitor and access their patients in case of medical emergency.15 Strategic signage was placed on the booth to remind passers-by of the criticality of tasks taking place within. Standardised Workflow: During preceding phases of the study,14 it was observed that nurses rarely followed a standardised workflow for verifying medications prior to reaching the patient. When interrupted, nurses often omitted verification of medications against the CPOE, paper order or patient's armband. The dual paper/electronic order system used in the unit exacerbated the potential for such omissions. To mitigate errors resulting from these omissions, nurses’ workflow was standardised through training, Information Technology (IT) cues, and making use of physical space. Nurses were requested to pick up medications from the pharmacy area, and proceed directly to the Verification Booth rather than approaching the patient first. Nurses would then check each medication label against the electronic order, followed by the paper order, and would document on screen and paper that the medications had been checked. A redesigned prototype of the CPOE software interface was created16 that accommodated a forced verification check process, and displayed visual indicators of the status of verification of each medication. Any discrepancies would therefore be resolved before the medications reached the point of care and had the potential to cause harm.

totype of the CPOE software interface was created16 that accommodated a forced verification check process, and displayed visual indicators of the status of verification of each medication. Any discrepancies would therefore be resolved before the medications reached the point of care and had the potential to cause harm. Speaking Aloud: Nurses were asked to use a ‘Speak Aloud’ protocol when verifying medication labels against the patient's armband.15 This required the nurse to verbalise identifying information (eg, patient's name, date of birth and medical record number) during verification. It was hypothesised that this action of speaking aloud would alert patients and coworkers of the critical task at hand, and help increase nurses’ focus on the numerical matching task. An analogous scenario would be a bank teller counting money out loud before customers; in the medication administration environment, the action of speaking aloud cues patients and coworkers to wait until the critical task is complete before asking questions or otherwise engaging the nurses’ attention. Figure 2 Photographs depicting, (A) Verification Booth, (B) No Interruption Zones with Motion-activated Indicator, (C) Visual Timers, (D) Reminder Signage.

Speaking Aloud: Nurses were asked to use a ‘Speak Aloud’ protocol when verifying medication labels against the patient's armband.15 This required the nurse to verbalise identifying information (eg, patient's name, date of birth and medical record number) during verification. It was hypothesised that this action of speaking aloud would alert patients and coworkers of the critical task at hand, and help increase nurses’ focus on the numerical matching task. An analogous scenario would be a bank teller counting money out loud before customers; in the medication administration environment, the action of speaking aloud cues patients and coworkers to wait until the critical task is complete before asking questions or otherwise engaging the nurses’ attention. Figure 2 Photographs depicting, (A) Verification Booth, (B) No Interruption Zones with Motion-activated Indicator, (C) Visual Timers, (D) Reminder Signage. Interventions for medication administration tasks (errors of commission) The following interventions were proposed for medication administration tasks15: Visual timers for intravenous pushes: Results of a preceding phase revealed that nurses lost track of time when they were interrupted during administration of intravenous push medications. This resulted in medications being administered too quickly or too slowly, both of which can have severe physiological consequences for patients.17 To mitigate such errors, it was proposed that a visual timer (figure 2C) be attached to each intravenous pole with the infusion pump. Rather than a numerical stopwatch-like function, the timer counted down by proportionally reducing the visual coloured indicator, with no audible alarms or distractions. Nurses would start the timer prior to commencing manual intravenous pushes.

r (figure 2C) be attached to each intravenous pole with the infusion pump. Rather than a numerical stopwatch-like function, the timer counted down by proportionally reducing the visual coloured indicator, with no audible alarms or distractions. Nurses would start the timer prior to commencing manual intravenous pushes. No interruption zones with motion-activated indicators: The immediate area surrounding infusion pump poles was visually demarcated as a ‘No Interruption Zone’ (figure 2B). A motion-activated ‘busy’ indicator was mounted on top of the intravenous pole, and would light up when nurses stepped in front of an intravenous pole to hang bags, adjust tubing or program infusion pumps. This served as an automatic indicator to passers-by that the nurse was conducting a critical task and should not be interrupted. Speaking aloud: For the reasons listed previously (see point 3 under Interventions for Medication Verification), nurses were also asked to speak aloud when programming infusion pumps. For instance, a nurse would say, ‘I'm programming a volume of 250 mL at a rate of 500 mL/h.’

No interruption zones with motion-activated indicators: The immediate area surrounding infusion pump poles was visually demarcated as a ‘No Interruption Zone’ (figure 2B). A motion-activated ‘busy’ indicator was mounted on top of the intravenous pole, and would light up when nurses stepped in front of an intravenous pole to hang bags, adjust tubing or program infusion pumps. This served as an automatic indicator to passers-by that the nurse was conducting a critical task and should not be interrupted. Speaking aloud: For the reasons listed previously (see point 3 under Interventions for Medication Verification), nurses were also asked to speak aloud when programming infusion pumps. For instance, a nurse would say, ‘I'm programming a volume of 250 mL at a rate of 500 mL/h.’ Reminder signage: To aid nurses in recovering from interruptions during pump programming, and to assist them in programming infusion parameters correctly even after being interrupted, strategic signage was placed on and near infusion pumps (figure 2D). The signage reminded nurses to check infusion parameters, clamps and tubing connections. The prominent presence of this signage directly on the intravenous pole served as a visual cue, reminding nurses to double-check infusion parameters prior to administration.

ignage was placed on and near infusion pumps (figure 2D). The signage reminded nurses to check infusion parameters, clamps and tubing connections. The prominent presence of this signage directly on the intravenous pole served as a visual cue, reminding nurses to double-check infusion parameters prior to administration. Results Intervention utilisation The use of some interventions (such as the Verification Booth, No Interruption Zone, Standardised Workflow and CPOE enhancements) was forced upon the participant according to the design of the physical environment. For interventions that required active use by participants, the rate of utilisation was as follows: Visual Timers: 100% utilisation; Speaking Aloud during Pump Programming: 53%; Speaking Aloud during Patient Identification Verification: 74%. Error rates in medication verification and administration Error rates for medication verification and administration tasks under all three experimental conditions are shown in table 3. The results show that interruptions were associated with a significant increase in error rates for the following four tasks: verifying volume in a syringe, verifying volume in an ambulatory infusion pump, intravenous push and infusion pump programming. The number of nurses committing errors in these four tasks significantly decreased in the postintervention condition. However, use of interventions did not significantly decrease error rates for other medication verification tasks. Table 3 Error rates in medication verification and administration tasks, under all three conditions

Error rates in medication verification and administration Error rates for medication verification and administration tasks under all three experimental conditions are shown in table 3. The results show that interruptions were associated with a significant increase in error rates for the following four tasks: verifying volume in a syringe, verifying volume in an ambulatory infusion pump, intravenous push and infusion pump programming. The number of nurses committing errors in these four tasks significantly decreased in the postintervention condition. However, use of interventions did not significantly decrease error rates for other medication verification tasks. Table 3 Error rates in medication verification and administration tasks, under all three conditions Task Number of nurses committing error (%) Preintervention experiment Postintervention experiment Condition 1: uninterrupted (n=18) Condition 2: interrupted (n=18) Significance (Condition 1 vs 2)* Condition 3: interrupted (n=19) Significance (Condition 2 vs 3)† Medication verification tasks (assessment of error detection) 1. Verifying medication name 3 (17%) 6 (33%) No (p=0.160) 4 (21%) No (p=0.319) 2. Verifying medication dosage 4 (22%) 4 (22%) No (p=0.595) 1 (5%) No (p=0.153) 3. Verifying medication volume in syringe 9 (50%) 16 (89%) Yes (p=0.003) 11 (58%) Yes (p=0.038) 4. Verifying medication volume in AIP 10 (56%) 17 (94%) Yes (p=0.002) 11 (58%) Yes (p=0.012) 5. Verifying patient ID 7 (39%) 6 (33%) No (p=0.591) 6 (32%) No (p=0.593) Medication administration tasks (assessment of error commission) 6. Intravenous push 8 (44%) 16 (89%) Yes (p=0.02) 6 (32%) Yes (p=0.001) 7. Pump programming and infusion initiation 0 (0%) 7 (39%) Yes (p=0.03) 1 (5%) Yes (p=0.017) *McNemar's χ2 test (within-subjects analysis).

) 6 (33%) No (p=0.591) 6 (32%) No (p=0.593) Medication administration tasks (assessment of error commission) 6. Intravenous push 8 (44%) 16 (89%) Yes (p=0.02) 6 (32%) Yes (p=0.001) 7. Pump programming and infusion initiation 0 (0%) 7 (39%) Yes (p=0.03) 1 (5%) Yes (p=0.017) *McNemar's χ2 test (within-subjects analysis). †Fisher's exact test (between-subjects analysis). Discussion To our knowledge, this is the first study to make use of controlled high-fidelity simulation to explicitly examine the relationship between interruptions, error rates and the effect of interventions on medication error rates. We identified that nurses committed significantly more errors in infusion pump programming and intravenous push delivery, and failed to detect errors in several critical parameters of medication verification when interrupted. These findings provide important insight into understanding the contribution of work interruptions to medication errors. More significantly, we identified characteristics of interventions that were effective at mitigating these error types.

ors in several critical parameters of medication verification when interrupted. These findings provide important insight into understanding the contribution of work interruptions to medication errors. More significantly, we identified characteristics of interventions that were effective at mitigating these error types. Intravenous push delivery errors were significantly reduced through use of a simple visual timer that allowed nurses to temporally monitor the push without requiring them to perform mental calculations of elapsed time or remember numerical starting time values. Nurses commented that the timer display provided an easy visual reference without detracting from their ability to teach, monitor and care for patients throughout the duration of the push. Nurses were extremely eager to use the timers in their own care environments, which is an encouraging finding given the simple implementation and low-cost nature of this intervention.

easy visual reference without detracting from their ability to teach, monitor and care for patients throughout the duration of the push. Nurses were extremely eager to use the timers in their own care environments, which is an encouraging finding given the simple implementation and low-cost nature of this intervention. Similarly, pump programming errors were significantly reduced through a combination of No Interruption Zones, motion-activated indicators, speak-aloud protocols and infusion pump signage. Because our study design tested these interventions as a system rather than individually, it is difficult to conclusively identify the specific mechanisms that led to this result. Speaking aloud may have helped improve nurses’ focus on pump programming parameters by increasing the distinctiveness of the information being verbalised,18 and the presence of the No Interruption Zones and associated signage may have acted as final visual cues for nurses, reminding them to conduct one last check of pump parameters prior to administration. Thus, a combination of environmental modifications and simple speak-aloud interventions may provide a low-cost method of mitigating pump programming and infusion initiation errors caused by interruptions.

final visual cues for nurses, reminding them to conduct one last check of pump parameters prior to administration. Thus, a combination of environmental modifications and simple speak-aloud interventions may provide a low-cost method of mitigating pump programming and infusion initiation errors caused by interruptions. Interestingly, the speak-aloud intervention was not effective when applied to patient identification verification tasks. We suggest that this differential effect may be due to the very different nature of medication verification vs medication administration. In contrast to the unpredictable and constantly evolving nature of medication administration, medication verification is a highly mechanistic and predictable task19 that may be more prone to habituation, confirmation bias and complacency effects. Thus, reliance on a ‘people-dependent’ intervention such as speaking aloud may be less effective at reducing errors because it is ultimately reliant on human memory, vigilance and adherence to rules.20 21 After the experiment, some nurses commented that they may not remember to consistently speak out loud when interrupted in the real environment, suggesting that there is a ‘ceiling effect’ to the effectiveness of this intervention. Studies suggest that technological solutions that automate tasks (eg, bar code medication administration systems), force functions and relieve the memory burden placed on humans may be more effective at reducing adverse events,20 21 and this automation may be particularly well-suited to tasks that involve mechanistic comparison or routine checking of information.19 22 The real value of the speak-aloud intervention might be in deterring people from interrupting nurses. However, we were not able to evaluate this hypothesis because all interruptions were held constant in our experiments.

y well-suited to tasks that involve mechanistic comparison or routine checking of information.19 22 The real value of the speak-aloud intervention might be in deterring people from interrupting nurses. However, we were not able to evaluate this hypothesis because all interruptions were held constant in our experiments. For other tasks involving mechanistic verification of information, interventions such as the Verification Booth and standardised workflow with CPOE enhancements were effective at reducing wrong volume errors in syringes and AIPs. We suggest that our enhancements to the CPOE system (ie, forced checks of all medication parameters and clearly visible verification status) acted as a cueing function that encouraged task resumption by reminding nurses of outstanding verification items after being interrupted. This finding is in line with research suggesting that use of cueing functions on clinical IT systems can encourage task resumption by reminding the user of the task at hand.23–25 Interestingly, the same intervention was not effective at mitigating wrong medication name and wrong dose errors. We attribute this finding to two reasons. First, the preintervention error rate for these two tasks was already relatively low, indicating that there was less room for improvement compared with the other verification tasks. This may be the result of nurses being more vigilant in verifying medication name and dosage compared with other medication information. Second, the limited nature of the CPOE enhancements may have had an effect: while the prototype incorporated layout changes and visual cues, it did not incorporate interventions such as TALLman lettering (eg, CARBOplatin vs CISplatin) that specifically targeted ‘look alike, sound alike’ medications. This further highlights the need for more specificity in automated interventions to reduce nurses’ reliance on vigilance and memory for error detection.

l cues, it did not incorporate interventions such as TALLman lettering (eg, CARBOplatin vs CISplatin) that specifically targeted ‘look alike, sound alike’ medications. This further highlights the need for more specificity in automated interventions to reduce nurses’ reliance on vigilance and memory for error detection. Limitations of the study We acknowledge that there are limitations to this study. First, participants were aware that they were being observed during the high-fidelity simulation experiment. It is possible that their behaviour may have been altered as a consequence (ie, the Hawthorne effect), though post-test debriefs suggested that this was not a significant problem given the high fidelity of the simulation. Second, the number of errors planted in the simulation experiment was artificially high compared with real life, and may have caused participants to become more vigilant for errors as the experiment progressed. However, the order of presentation of task types was counterbalanced to limit this effect. Lastly, we were able to assess the effectiveness of interventions when they were grouped together as a system, but our study design did not allow us to definitively assess the effectiveness of each individual intervention. We also did not assess the longitudinal impact of interventions. Conducting these additional assessments is a goal of future research.

fectiveness of interventions when they were grouped together as a system, but our study design did not allow us to definitively assess the effectiveness of each individual intervention. We also did not assess the longitudinal impact of interventions. Conducting these additional assessments is a goal of future research. Conclusions The present research identifies that interruptions increase the chances of nurses committing safety-critical errors when delivering high-risk medications. Our study adds to the literature by providing examples of low-cost interventions (eg, visual timers) that can enhance patient safety by reducing medication administration errors. We found that our proposed interventions were effective at reducing errors of commission in medication administration tasks, but less effective at reducing errors of detection in medication verification tasks. We suggest that routine, predictable errors of detection cannot be successfully mitigated through ‘people-dependent’ interventions alone, but would likely benefit from interventions that are more automated and less reliant on human memory and vigilance. Identifying and testing the effectiveness of such interventions is a potential avenue of future work. Because interruptions represent a highly complex sociotechnical phenomenon26 with potentially different effects on different task types, no single intervention is sufficient to achieve a reduction in error. Rather, mitigation efforts must be designed with a thorough understanding of task and error types to be effective.

ecause interruptions represent a highly complex sociotechnical phenomenon26 with potentially different effects on different task types, no single intervention is sufficient to achieve a reduction in error. Rather, mitigation efforts must be designed with a thorough understanding of task and error types to be effective. Supplementary Material Web supplement The authors are grateful to the oncology nurses who participated in all phases of this study. We also thank Karin Ayanian, Michelle Dowling, Archana Gopal, Melissa Griffin, Diane Kostka and Ilia Makedonov for their assistance in conducting the simulation experiment. Contributors: All authors contributed to this work. VP executed the design and testing of interventions, analysed data, and prepared the manuscript. CK, PS, KT, JS, TM and JAC participated in intervention development and analysis activities. PT conceived, designed and executed the overall research study. All authors contributed to and approved the final manuscript. Funding: This research was funded by the Canadian Patient Safety Institute. The opinions in the present paper are those of the authors and do not necessarily reflect the sponsor's official position. Competing interests: None. Ethics approval: Research Ethics Board approval for this study was obtained from the University Health Network (Reference: #08-0306-BE) and the University of Toronto (Reference: #24457). Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Meaningful evaluation has an essential role in the work of improving healthcare, especially in enabling learning to be shared.1 Evaluations typically seek to identify the aims of an intervention or programme, find measurable indicators of achievement, collect data on these indicators and assess what was achieved against the original aims.2 Evaluating whether a programme works is not necessarily the only purpose of evaluation, however: how and why may be equally important questions,3 4 especially in enabling apparently successful interventions to be reproduced.5 Despite the potential benefits of such efforts, and the welcome given to evaluation by some who run programmes, the literature on programme evaluation has long acknowledged that evaluation can be a source of tension, friction and confusion of purpose:[Evaluation] involves a balancing act between competing forces. Paramount among these is the inherent conflict between the requirements of systematic inquiry and data collection associated with evaluation research and the organizational imperatives of a social program devoted to delivering services and maintaining essential routine activities.6

lancing act between competing forces. Paramount among these is the inherent conflict between the requirements of systematic inquiry and data collection associated with evaluation research and the organizational imperatives of a social program devoted to delivering services and maintaining essential routine activities.6 Healthcare is no exception to the general problems characteristic of programme evaluation: the concerns and interests of the different parties involved in an improvement project and its associated evaluation may not always converge. These parties may include the designers and implementers of interventions (without whose improvement work there would be nothing to evaluate), the evaluators (who may be a heterogeneous mix of different professional groups - including health professionals and others - or academics from different disciplines) and sometimes funders (who may be funding either the intervention, the evaluation or both). Each may have different goals, perspectives, expectations, priorities and interests, professional languages and norms of practice, and they may have very distinct accountabilities and audiences for their work. As a result, evaluation work may—and in fact, often does—present challenges for all involved, ranging from practicalities such as arranging access to data, through conceptual disagreements about the programme and what it is trying to achieve, to concerns about the impartiality and competence of the evaluation team, widely divergent definitions of success and many others.6 Given that it is not unlikely these challenges will occur, the important question is how they can optimally be anticipated and managed.7 8

t the programme and what it is trying to achieve, to concerns about the impartiality and competence of the evaluation team, widely divergent definitions of success and many others.6 Given that it is not unlikely these challenges will occur, the important question is how they can optimally be anticipated and managed.7 8 This article seeks to make a practical contribution by presenting a possible approach to minimising the tensions. Specifically, we propose the co-production and use of a concordat—a mutually agreed compact between all parties, which articulates a set of principles to guide the conduct of the evaluation. The article proceeds in two parts. First, we identify the kinds of challenge often faced in the design, running and evaluation of an improvement programme in healthcare. Second, we present an example of the development of a concordat used in the evaluation of a major improvement project.

the conduct of the evaluation. The article proceeds in two parts. First, we identify the kinds of challenge often faced in the design, running and evaluation of an improvement programme in healthcare. Second, we present an example of the development of a concordat used in the evaluation of a major improvement project. Challenges in conducting programme evaluations A now extensive literature has identified multiple challenges in programme evaluation, dating back to when the field began to develop formally during the 1960s.9 Challenges can arise at virtually every stage—from the design of the evaluation through its conduct and eventual publication—to the extent that ‘evaluation anxiety’ is a known phenomenon.10 Those being evaluated may be subjected to judgements about behaviour and outcomes against externally agreed targets. The detailed examination of individual, group and organisational practices may be experienced as risky and unpleasant, and strains in the relationships between the different parties may easily arise. These strains may, for example, relate to the goals of the evaluation; data management; the discomforts of being studied and disappointing or otherwise unwelcome results (box 1). Box 1 Areas of possible tension and challenge in programme evaluation identified in the literature Securing full consensus on the specifics of evaluation objectives34 Unpacking contrasting interpretations about what and who the evaluation is for6 A desire on the part of evaluators to fix the goals for improvement programmes early in the evaluation process35

Challenges in conducting programme evaluations A now extensive literature has identified multiple challenges in programme evaluation, dating back to when the field began to develop formally during the 1960s.9 Challenges can arise at virtually every stage—from the design of the evaluation through its conduct and eventual publication—to the extent that ‘evaluation anxiety’ is a known phenomenon.10 Those being evaluated may be subjected to judgements about behaviour and outcomes against externally agreed targets. The detailed examination of individual, group and organisational practices may be experienced as risky and unpleasant, and strains in the relationships between the different parties may easily arise. These strains may, for example, relate to the goals of the evaluation; data management; the discomforts of being studied and disappointing or otherwise unwelcome results (box 1). Box 1 Areas of possible tension and challenge in programme evaluation identified in the literature Securing full consensus on the specifics of evaluation objectives34 Unpacking contrasting interpretations about what and who the evaluation is for6 A desire on the part of evaluators to fix the goals for improvement programmes early in the evaluation process35 Evolution of interventions (intentionally or unintentionally) during implementation36 and ongoing negotiation about evaluation scope in relation to implementation evolution37 Fear of evaluation being used for performance management15 Mismatched interpretations of stakeholders’ own role and other partners’ roles12 14

A desire on the part of evaluators to fix the goals for improvement programmes early in the evaluation process35 Evolution of interventions (intentionally or unintentionally) during implementation36 and ongoing negotiation about evaluation scope in relation to implementation evolution37 Fear of evaluation being used for performance management15 Mismatched interpretations of stakeholders’ own role and other partners’ roles12 14 An interpretation of evaluators as friends or confidants, risking a subsequent sense of betrayal16 A lack of shared language or understanding if some partners lack familiarity with the methodological paradigm or data collection tools being proposed13 Conflicts between the burden of evaluation data collection and the work of the programme2 Previous experiences of the dubious value of evaluation leading to disengagement with current evaluation work17 Tensions between an imperative to feedback findings and to respect principles of anonymity and confidentiality38 Encountering the ‘uncomfortable reality’ that a service or intervention is not performing as planned or envisaged and objectives have not been met18 Negotiations with gatekeepers about access to complete and accurate data in a timely fashion A reluctance to share evaluation findings if they are seen as against the ‘organisational zeitgeist’20 or threaten identity and reputational claims21 Pressure from partners, research sponsors or funders to alter the content or scope of the evaluation,20 or to delay their publication22

Negotiations with gatekeepers about access to complete and accurate data in a timely fashion A reluctance to share evaluation findings if they are seen as against the ‘organisational zeitgeist’20 or threaten identity and reputational claims21 Pressure from partners, research sponsors or funders to alter the content or scope of the evaluation,20 or to delay their publication22 A critical first task for all parties is to therefore clarify what is to be achieved through evaluation. This allows an appropriate evaluation design to be formulated, but is also central to establishing a shared vision to underpin activity. This negotiation of purpose may be more or less formal,11 but should be undertaken. The task is to settle questions about purpose and scope, remembering that agreements about these may unravel over the course of the activity.12 Constant review and revisiting of the goals of the evaluation (as well as the goals of the improvement programme) may therefore be necessary to maintain dialogue and avoid unwarranted drift.

settle questions about purpose and scope, remembering that agreements about these may unravel over the course of the activity.12 Constant review and revisiting of the goals of the evaluation (as well as the goals of the improvement programme) may therefore be necessary to maintain dialogue and avoid unwarranted drift. These early discussions are especially important in ensuring that all parties understand the methods and data collection procedures being used in the evaluation.13 A lack of shared language and understanding may lead to confusion over why particular methods are being used, generating uncertainties or suspicion and undermining willingness to cooperate. Regardless of what form it takes, the burden of data collection can be off-putting for those being evaluated and those performing the evaluation. If the evaluation itself is too demanding, there may be conflicts between its requirements and doing the work of the programme.2 For partner organisations, collecting data for evaluation may not seem as much of a priority as delivery, and the issue of who gets to control and benefit from the data they have worked so hard to collect may be difficult to resolve.

there may be conflicts between its requirements and doing the work of the programme.2 For partner organisations, collecting data for evaluation may not seem as much of a priority as delivery, and the issue of who gets to control and benefit from the data they have worked so hard to collect may be difficult to resolve. Even when agreement on goals and scope is reached early on and remains intact, complex evaluations create a multiplicity of possible lines of communication and accountability, as well as ambiguity about roles. Though the role of each party in a programme evaluation may seem self-evident (eg, one funds, one implements, one evaluates), in practice different parties may have mismatched interpretations both of their own role and of others’. Such blind spots can fatally derail collaborative efforts.12 The role of the evaluator may be an especially complex one, viewed in different ways by different parties.14 Outcomes-focused aspects of evaluation—aimed at assessing degree of success in achieving goals—may cast evaluators as ‘performance managers’.15 But the process-focused aspects of evaluation—particularly where they involve frequent contact between evaluators and evaluated, as is usually the case with ethnographic study—may make evaluators seem like friendly confidants, risking a subsequent sense of betrayal.16 Thus, evaluators may be seen as critical friends, co-investigators, facilitators or problem solvers by some, but also as unwelcome intruders who sit in judgement but do not get their hands dirty in the real work of delivering the programme and who have influence without responsibility.

a subsequent sense of betrayal.16 Thus, evaluators may be seen as critical friends, co-investigators, facilitators or problem solvers by some, but also as unwelcome intruders who sit in judgement but do not get their hands dirty in the real work of delivering the programme and who have influence without responsibility. Uncertainties about what information should be shared with whom, when and under what conditions may provide a further source of ethical dilemma, especially when unspoken assumptions and expectations are breached, damaging trust and undermining cooperative efforts. Evaluators must often abide by both the imperative to feedback findings to other stakeholders (especially, perhaps, the funders and clients of the evaluation) and to respect principles of anonymity and confidentiality in determining the limits of what can be fed back, to whom and in how much detail. For these reasons, role perceptions and understandings about information exchange (content and direction) need to be surfaced early in the programme—and revisited throughout—to avoid threats to an honest, critical and uncompromised evaluation process. This is especially important given the asymmetry that may arise between the various parties, which can lead to tensions about who is in charge and on what authority.

rection) need to be surfaced early in the programme—and revisited throughout—to avoid threats to an honest, critical and uncompromised evaluation process. This is especially important given the asymmetry that may arise between the various parties, which can lead to tensions about who is in charge and on what authority. Sometimes, though perhaps not often, the challenges are such that implementers may feel that obstructing evaluation is more in line with their organisational interests. They may, for example, frustrate attempts to evaluate by providing inaccurate, incomplete or tardy data (quantitative or qualitative) or, where they are able to play the role of ‘gatekeeper’, simply deny access to data or key members of staff. A lack of engagement with the process may be fuelled by previous experiences of evaluation that was felt to be time-consuming or of dubious value.17

urate, incomplete or tardy data (quantitative or qualitative) or, where they are able to play the role of ‘gatekeeper’, simply deny access to data or key members of staff. A lack of engagement with the process may be fuelled by previous experiences of evaluation that was felt to be time-consuming or of dubious value.17 Tensions do not, of course, end when the programme and evaluation are complete, and may indeed intensify when the results are published. Those involved in designing, delivering and funding a programme may set out with great optimism; they may invest huge energy, efforts and resource in a programme; they may be convinced of its benefits and success and they may want to be recognised and congratulated on their hard work and achievement. When evaluation findings are positive, they are likely to be welcomed. Robust evidence of the effectiveness of an intervention can be extremely valuable in providing weight to arguments for its uptake and spread, and positive findings from independent evaluation of large-scale improvement programmes help legitimise claims to success. But not every project succeeds, and an evaluation may result in some participants being confronted with the uncomfortable reality that their service or their intervention has not performed as well as they had hoped.18 Such findings may provoke reactions of disappointment, anger and challenge: ‘for every evaluation finding there is equal and opposite criticism’.19

tion may result in some participants being confronted with the uncomfortable reality that their service or their intervention has not performed as well as they had hoped.18 Such findings may provoke reactions of disappointment, anger and challenge: ‘for every evaluation finding there is equal and opposite criticism’.19 When a programme falls short of realising its goals, analysis of the reasons for failure can produce huge net benefits for the wider community, not least in ensuring that future endeavours do not repeat the same mistakes.2 But recognising this value can be difficult given the immediate disappointment that comes with failure. If the evaluation—and the resulting publications—does not present the organisation(s) involved in the intervention in a positive light, there may be a reluctance to ‘wash dirty linen in public’2 and resistance to the implications of findings,20 especially where they threaten reputation.21 Evaluators themselves may not be immune to pressures to compromise their impartiality. The literature contains cautionary examples of pressure from partners or research sponsors who wish to direct the content of the report or analysis,20 or coercion from funders to limit the scope of evaluation, distort results or critically delay their publication.22

immune to pressures to compromise their impartiality. The literature contains cautionary examples of pressure from partners or research sponsors who wish to direct the content of the report or analysis,20 or coercion from funders to limit the scope of evaluation, distort results or critically delay their publication.22 A possible solution: developing a concordat By now it will be clear that challenges in conducting programme evaluation should be anticipated with a view to managing them (box 2). But how should this be done? Attempts to answer this question commonly include exhortations for stakeholders to commit to open dialogue and respect for other stakeholders, to have clear founding principles, a shared vision and transparent mechanisms for conflict resolution.7 23 24 While these are all important, guidance on how to achieve them in practice is limited. We propose that one promising solution lies in evaluation partners (evaluators, designers, implementers, funders and others) working together at the outset of a project to produce a concordat. It requires them to develop a set of principles to guide the conduct of the evaluation and agreeing to abide by these principles, consistent with the approach advocated by the Harvard Negotiation Project.25 We elaborate the rationale behind this proposal by drawing on our experience of developing a concordat for the evaluation of a large, multi-partner patient safety improvement programme. Box 2 Managing risks of conflict and tension in the evaluation of improvement programmes All parties should agree on the purpose and scope of the evaluation upfront, but recognise that both may mutate over time and need to be revisited

t for the evaluation of a large, multi-partner patient safety improvement programme. Box 2 Managing risks of conflict and tension in the evaluation of improvement programmes All parties should agree on the purpose and scope of the evaluation upfront, but recognise that both may mutate over time and need to be revisited An explicit statement of roles may ensure that understandings of the division of labour within an evaluation—and the responsibilities and relationships that imply—are shared The expectations placed on each party in relation to data collection should be reasonable and feasible, and the methodological approach (in its basic principles) should be understood by all parties Clear terms of reference concerning disclosure, dissemination and the limits of confidentiality are necessary from the start All efforts should be made to avoid implementers experiencing discomfort about being studied: through ensuring all parties are fully briefed about the evaluation; sharing formative findings and ensuring appropriate levels of anonymity in reporting findings Commitment to learning for the greatest collective benefit is the overriding duty of all parties involved—it follows from this that all parties should make an explicit commitment to ensuring sincere, honest and impartial reporting of evaluation findings

All efforts should be made to avoid implementers experiencing discomfort about being studied: through ensuring all parties are fully briefed about the evaluation; sharing formative findings and ensuring appropriate levels of anonymity in reporting findings Commitment to learning for the greatest collective benefit is the overriding duty of all parties involved—it follows from this that all parties should make an explicit commitment to ensuring sincere, honest and impartial reporting of evaluation findings The programme we discuss, known as Safer Clinical Systems Phase 2, was a complex intervention in which eight organisations were trained to apply a new approach (adapted from high-risk industries) to the detection and management of risk in clinical settings.26 The work was highly customised to the particularities of these settings. The programme involved a complicated nexus of actors, including the funder (the Health Foundation, a UK healthcare improvement charitable foundation); the technical support team (based at the University of Warwick Medical School), who designed the approach and provided training and support for the participating sites over a 2-year period; the eight healthcare organisations (‘implementers’) and the evaluation team (itself a three-university partnership led by the University of Leicester).

team (based at the University of Warwick Medical School), who designed the approach and provided training and support for the participating sites over a 2-year period; the eight healthcare organisations (‘implementers’) and the evaluation team (itself a three-university partnership led by the University of Leicester). Developing the concordat and its content The evaluation team drew on the literature and previous experience to anticipate potential points of conflict or frustration and to identify principles and values that could govern the relationships and promote cooperation. These were drawn together into the first draft of a document that we called a ‘concordat’. The evaluation team came up with the initial draft, which was then subject to extensive comment, discussion, refinement and revision by the technical support team and funders. The document went through multiple drafts based on feedback, including several meetings where evaluators, technical team and funders came up with possible areas of conflict and possible scenarios illustrating tensions, and tested these against the concordat. Once the final draft was agreed, it was signed by all three parties and shared with the participating sites.

on feedback, including several meetings where evaluators, technical team and funders came up with possible areas of conflict and possible scenarios illustrating tensions, and tested these against the concordat. Once the final draft was agreed, it was signed by all three parties and shared with the participating sites. The first section of the concordat—‘goals and values’—sets out the core principles concerning the purpose of the activity (box 3). These were the constitutional foundations: they emphasised a shared, overarching goal—safer healthcare for patients—and committed all parties to adherence to this principle in all their interactions. In foregrounding these principles, the intention was to address the misconceptions that can occlude understanding of evaluation and to make explicit shared objectives. Box 3 The concordat in outline Goals and values—outlining the partners and their commitment to the programme goal, shared learning, respect for dignity and integrity and open dialogue Responsibilities of the evaluation team—summarising the purpose of the evaluation, making a commitment to accuracy in representation and reporting and seeking to minimise the burden on partners Responsibilities of the support team—a synopsis of the remit of one partner's role in relation to the evaluation team and their agreed interaction Responsibilities of participating sites—outlining how the sites will facilitate access to data for the evaluation team Data collection—agreeing steps to minimise the burden of data collection on all partners and to share data as appropriate

Responsibilities of the support team—a synopsis of the remit of one partner's role in relation to the evaluation team and their agreed interaction Responsibilities of participating sites—outlining how the sites will facilitate access to data for the evaluation team Data collection—agreeing steps to minimise the burden of data collection on all partners and to share data as appropriate Ethical issues—summarising issues about confidentiality, data security and working within appropriate ethical and governance frameworks Publications—confirming a commitment to timely publication of findings, paying particular attention to the possibility of negative or critical findings Feedback—outlining how formative feedback should be provided, received and actioned by appropriate partners

Ethical issues—summarising issues about confidentiality, data security and working within appropriate ethical and governance frameworks Publications—confirming a commitment to timely publication of findings, paying particular attention to the possibility of negative or critical findings Feedback—outlining how formative feedback should be provided, received and actioned by appropriate partners The concordat then sets out the roles and responsibilities of each party, including, for example, an obligation to be even-handed for the evaluation team, and the commitment to sharing information openly on the part of the technical support team (box 3). The concordat also articulated the relationships between the different parties, emphasising the importance of critical distance and stressing that this was not a relationship of performance management. The concordat further sought to address potential disagreements relating to the measures used in the evaluation. Rather than delineate an exhaustive list of what those methods and data would be, the concordat sets out the process through which measures would be negotiated and determined, and made explicit the principles concerning requests for and provision of data that would underpin this process (eg, the evaluation team should minimise duplicative demands for data by the evaluation team, and the participating sites should provide timely and accurate data).

measures would be negotiated and determined, and made explicit the principles concerning requests for and provision of data that would underpin this process (eg, the evaluation team should minimise duplicative demands for data by the evaluation team, and the participating sites should provide timely and accurate data). The values and ethical imperatives governing action and interactions were also made explicit; for example, arrangements around confidentiality, anonymity and dissemination were addressed, including expectations relating to authorship of published outputs. Principles relating to research governance and feedback sought both to mitigate unease at the prospect of evaluation while also enshrining certain inalienable principles that are required for high-quality evaluation: for example, it committed all parties to sharing outputs ahead of publication, but it also protected the impartiality of the evaluation team by making clear that they had the final say in the interpretation and presentation of evaluation findings (though this did not preclude other partners from publishing their own work). Importantly, the concordat sets out a framework that all parties committed to following if disputes did arise. These principles were invoked on a number of occasions during the Safer Clinical Systems evaluation, for example, when trying to reach agreement on measurement or to resolve ambiguities in the roles of the evaluation and support teams. The concordat was also invaluable in ensuring that boundaries and expectations did not have to be continually re-negotiated in response to organisational turbulence, given that the programme experienced frequent changes of personnel over its course.

resolve ambiguities in the roles of the evaluation and support teams. The concordat was also invaluable in ensuring that boundaries and expectations did not have to be continually re-negotiated in response to organisational turbulence, given that the programme experienced frequent changes of personnel over its course. Challenges in developing and using the concordat Of course, neither the process nor the outcome of the concordat for this evaluation was without wrinkles. Some issues arose that had not been anticipated, and some tensions encountered from the start of the programme continued to cause difficulties. These challenges were in some respects unique to this particular context, but may provide general lessons to inform future evaluation work. For instance, the technical support team was charged with undertaking ‘learning capture’, which was not always easy to distinguish from evaluation, and it proved difficult to maintain clear boundaries about this scope. Future projects would benefit from earlier clarification of scope and roles.

future evaluation work. For instance, the technical support team was charged with undertaking ‘learning capture’, which was not always easy to distinguish from evaluation, and it proved difficult to maintain clear boundaries about this scope. Future projects would benefit from earlier clarification of scope and roles. The concordat took considerable time to develop and agree—around 6 months—in part because the process for developing the concordat was being worked on at the same time as developing the concordat itself. One consequence of this was that the participating sites (the implementers) were only given the opportunity to comment rather than engage as full partners. Future iterations should attempt to involve all parties earlier. We share this concordat and its process of development in part to facilitate the speedier creation of future similar agreements. The concordat as a solution: how does developing a concordat support effective collaborative activity? The development of a concordat makes concrete the principles underpinning evaluation as a collaborative activity, and the concordat itself has value as a symbolic, practical and actionable tool for setting expectations and supporting conflict resolution.

developing a concordat support effective collaborative activity? The development of a concordat makes concrete the principles underpinning evaluation as a collaborative activity, and the concordat itself has value as a symbolic, practical and actionable tool for setting expectations and supporting conflict resolution. The concordat as a document provides mutually agreed foundational principles which can be revisited when difficulties arise. In this sense, the concordat has value as a guide and point of reference.21 It also serves a symbolic function, in that it signals recognition—by all parties—of the centrality and importance of collaboration and a shared commitment to the process of evaluation. Formalising a collaborative agreement between parties, in the form of a non-binding contract, has the potential to promote a cooperative orientation among the parties involved and build trust.27 28 That the concordat is written and literally signed up to by all parties is important, as this institutionalisation of the concordat makes it less susceptible to distortion over time and better able to ensure that mutual understanding is more than superficial. Further, because it is explicitly not a contract, it offers a means of achieving agreement on core principles, goals and values separate from any legal commitments, and it leaves open the possibility of negotiation and renegotiation.

ver time and better able to ensure that mutual understanding is more than superficial. Further, because it is explicitly not a contract, it offers a means of achieving agreement on core principles, goals and values separate from any legal commitments, and it leaves open the possibility of negotiation and renegotiation. Much of the value in developing a concordat, however, lies in the process of co-production by all parties—a case of ‘all plans are useless, but planning is indispensable’. Though we did not directly evaluate its use, we feel that its development had a number of benefits for all stakeholders. First, rather than waiting for contradictions to materialise as disruptive conflicts that impede the evaluation, the process of discussing and (re)drafting a concordat offers an opportunity to anticipate, identify and make explicit differences in interpretations and perspectives on various aspects of the joint activity. Each party must engage in a process of surfacing and reflecting on their own assumptions, interpretations and interests, and sharing these with other parties. This allows difference and alternative interpretations to be openly acknowledged (rather than denied or ignored)—a respectful act of recognition and a prerequisite of open dialogue.29 30 Thus, the production of the concordat acts as a mechanism for establishing the kind of open dialogue and shared understanding so commonly exhorted.

fference and alternative interpretations to be openly acknowledged (rather than denied or ignored)—a respectful act of recognition and a prerequisite of open dialogue.29 30 Thus, the production of the concordat acts as a mechanism for establishing the kind of open dialogue and shared understanding so commonly exhorted. Second, by explicitly reflecting on and articulating the various roles and contributions of each party, the concordat-building process helps to foreground the contribution that each partner makes to the project and its evaluation, showing that all are interdependent and necessary.7 This emphasis on the distributed nature of contributions can help to offset the dominance of asymmetrical, hierarchical positionings (such as evaluator and evaluated, funder and funded, for example).21 It can therefore enable all those involved to see the opportunities as well as the challenges within an evaluation process, and reinforce a shared understanding of the value of a systematic, well-conducted evaluation.

trical, hierarchical positionings (such as evaluator and evaluated, funder and funded, for example).21 It can therefore enable all those involved to see the opportunities as well as the challenges within an evaluation process, and reinforce a shared understanding of the value of a systematic, well-conducted evaluation. Conclusions Programme evaluation is important to advancing the science of improvement. But it is unrealistic to suppose that there will be no conflict within an evaluation situation involving competing needs, priorities and interests: the management of these tensions is key to ensuring that a productive collaboration is maintained. Drawing on empirical and theoretical literature, and our own experience, we have outlined a practical approach—co-production and use of a concordat—designed to optimise and sustain the collaboration on which evaluation activity depends. A concordat is no substitute for sincere, faithful commitment to an ethic of learning on the part of all involved parties,31 and even with goodwill from all parties, it may not succeed in eliminating discord entirely. Nonetheless, in complex, challenging situations, having a clear set of values and principles that all parties have worked through is better than not having one.

ent to an ethic of learning on the part of all involved parties,31 and even with goodwill from all parties, it may not succeed in eliminating discord entirely. Nonetheless, in complex, challenging situations, having a clear set of values and principles that all parties have worked through is better than not having one. A concordat offers a useful component in planning an evaluation that runs smoothly by providing a framework for both anticipating and resolving conflict in collaborative activity. This approach is premised on recognition that evaluation depends on collaboration between diverse parties, and is therefore, by its collective nature, prone to tension about multiple areas of practice.32 Key to the potential of a concordat is its value, first, as an institutionalised agreement to be used as a framework for conflict resolution during evaluation activity, and, second, as a mechanism through which potential conflicts can be anticipated, made explicit and acknowledged before they arise, thereby establishing dialogue and a shared understanding of the purpose, roles, methods and procedures entailed in the evaluation.

ework for conflict resolution during evaluation activity, and, second, as a mechanism through which potential conflicts can be anticipated, made explicit and acknowledged before they arise, thereby establishing dialogue and a shared understanding of the purpose, roles, methods and procedures entailed in the evaluation. The concordat we developed for the Safer Clinical Systems evaluation (see online supplementary appendix 1) is not intended to be used directly as template for others, although, with appropriate acknowledgement, its principles could potentially be adapted and used. Understanding the principles behind the use of a concordat (how and why it works) is critical.33 In accordance with the rationale behind the concordat approach, we do not advocate that other collaborations simply adopt this example of a concordat ‘as is’. To do so would eliminate a crucial component of its value—the process of collective co-production. The process of articulating potential challenges in the planned collaboration, and testing drafts of the concordat against these, is particularly important in helping to uncover the implicit assumptions and expectations held by different parties, and to identify ambiguities about roles and relationships. All parties must be involved, in order to secure local ownership and capitalise on the opportunity to anticipate and surface tensions, establish dialogue and a shared vision and foreground the positive interdependence of all parties.

held by different parties, and to identify ambiguities about roles and relationships. All parties must be involved, in order to secure local ownership and capitalise on the opportunity to anticipate and surface tensions, establish dialogue and a shared vision and foreground the positive interdependence of all parties. Supplementary Material Web appendix We thank all of the many organisations and individuals who participated in this programme for their generosity and support, and our collaborators and the advisory group for the programme. Mary Dixon-Woods thanks the University of Leicester for grant of study leave and the Dartmouth Institute for Health Policy and Clinical Practice for hosting her during the writing of this paper. Collaborators: The Safer Clinical Systems Phase 2 Core Group Collaboration & Writing Committee, Nick Barber, Julian Bion, Matthew Cooke, Steve Cross, Hugh Flanagan, Christine Goeschel, Rose Jarvis, Peter Pronovost and Peter Spurgeon. Contributors: All authors contributed to and approved the final manuscript. Funding: Wellcome Trust, Health Foundation. Competing interests: Mary Dixon-Woods is deputy editor-in-chief of BMJ Quality and Safety. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Voluntary incident reporting systems have been recommended by various bodies as a way to improve patient safety.1–4 Yet, incident reporting systems are complex sociotechnical systems5 6 that have come under intense scrutiny;7–11 they have been criticised12–14 and praised.15–17 One of the main objectives of incident reporting systems is to attain organisational learning.1 18 This learning is restricted if only realised incidents are reported, but it could be greatly enhanced if patient safety hazards (defined as conditions that could lead to patient harm) are reported as well.19 20 While many studies have researched obstacles to and enablers of incident reporting by front-line healthcare workers, this study focuses on a specific but significant type of information—problems that the practitioners themselves can typically resolve. Few studies have looked at the goal conflict associated with the decision made at the front end of fixing an encountered patient safety problem on the spot and forgetting about it, or fixing the problem and reporting it into a reporting system. For example, a practitioner is about to administer medication to a patient, when s/he realises that the dosage by far exceeds what is recommended. The practitioner has two options: (1) seek clarification and change the dosage, administer the proper medication, move on to other tasks, and forego incident reporting, or (2) seek clarification and change the dosage, administer the proper medication, and fill out an incident report.

by far exceeds what is recommended. The practitioner has two options: (1) seek clarification and change the dosage, administer the proper medication, move on to other tasks, and forego incident reporting, or (2) seek clarification and change the dosage, administer the proper medication, and fill out an incident report. Several conceptualisations of what constitutes an incident have been suggested.21–23 The Canadian Patient Safety Institute (CPSI) presents three types of incidents: a harmful incident (reached the patient, and harm resulted—typically well accepted in healthcare under various terms such as ‘adverse events’, ‘sentinel events’ and ‘critical incidents’) (ref. 24, p.9), a no-harm incident (reached the patient, but no discernible harm resulted) and a near miss (did not reach the patient). Given that all of these deserve analysis, it can be argued that they should be reported into a reporting system. The WHO's Patient Safety Curriculum Guide expects multidisciplinary learners to acquire competencies in adverse events and near misses.25 Franklin et al note that “If incident reporting systems include and encourage reports of no-harm incidents in addition to actual patient harm, they can facilitate monitoring the resilience of healthcare processes” (ref. 19, p.765). The Health Foundation's The Measurement and Monitoring of Safety states “The focus is moving from counting harms after the event towards looking at hazards that might give rise to error, or safety failure before harm has occurred” (ref. 26, p.iii). Overall, while historically adverse events were the main focus of hospital reporting systems, near misses and hazards may also be expected in patient safety reporting, thus requiring practitioners to attend to this dimension of their work.

ive rise to error, or safety failure before harm has occurred” (ref. 26, p.iii). Overall, while historically adverse events were the main focus of hospital reporting systems, near misses and hazards may also be expected in patient safety reporting, thus requiring practitioners to attend to this dimension of their work. At a cognitive level, Tucker and Edmondson27–29 studied healthcare practitioners’ first order problem solving (fixing the problem at hand) and second order problem solving (understanding why the problem exists, aiming to correct the drivers, and thereby enhancing organisational learning). This study extends Tucker and Edmondson's work by viewing their concepts as applied to a reporting system, specifically first order problem solving represented as fixing a safety problem in the moment and forgetting about it, and second order problem solving represented as fixing the problem in the moment and reporting it so that wider learning can occur. The choice between these two options may be done more or less purposefully, as we demonstrate in this paper. Regardless, the two options are faced on a regular basis, and are underpinned by competing organisational priorities that require providing effective, expedient care to as many patients who need that care, and investing effort to create awareness of hazards and incidents. Given that front-line workers tend to be in an excellent position to identify safety problems and that they can play an important role in enhancing organisational learning,28 we ask: How do front-line healthcare practitioners choose between (A) fixing a patient safety problem and forgetting about it, and (B) fixing the problem and reporting it into an incident reporting system?

ition to identify safety problems and that they can play an important role in enhancing organisational learning,28 we ask: How do front-line healthcare practitioners choose between (A) fixing a patient safety problem and forgetting about it, and (B) fixing the problem and reporting it into an incident reporting system? Methods This study is part of a larger research project on voluntary incident reporting and safety in a teaching hospital in Ontario, Canada. The reporting system at the hospital is available to employees through any networked device. The reporter enters information using the patient's medical record number and provides a narrative describing the patient safety incident (see the CPSI definition above) using facts. The reports are reviewed by Clinical Managers who investigate them locally, by physician Clinical Reviewers who assess if harm was caused to the patient, and by Core Reviewers who assess larger hospital issues.

and provides a narrative describing the patient safety incident (see the CPSI definition above) using facts. The reports are reviewed by Clinical Managers who investigate them locally, by physician Clinical Reviewers who assess if harm was caused to the patient, and by Core Reviewers who assess larger hospital issues. Our study focused on General Internal Medicine—one of the largest departments in the hospital. The study began in spring 2012, with a quality review meeting whereby the researchers were introduced to key personnel who would later become interviewees. Over 5 months, two researchers (independently and together) confidentially interviewed hospital personnel. The Chief Physician and Clinical Director were key informants, each of whom recommended other personnel to interview based on our request to sample individuals with a variety of views and practices related to incident reporting. This process yielded 26 interviews purposefully sampled. Additional interviewees were recruited through email requests for practitioners to participate in the study. Overall, 40 participants were recruited as shown in table 1. Table 1 Number of interviewees by job title

Our study focused on General Internal Medicine—one of the largest departments in the hospital. The study began in spring 2012, with a quality review meeting whereby the researchers were introduced to key personnel who would later become interviewees. Over 5 months, two researchers (independently and together) confidentially interviewed hospital personnel. The Chief Physician and Clinical Director were key informants, each of whom recommended other personnel to interview based on our request to sample individuals with a variety of views and practices related to incident reporting. This process yielded 26 interviews purposefully sampled. Additional interviewees were recruited through email requests for practitioners to participate in the study. Overall, 40 participants were recruited as shown in table 1. Table 1 Number of interviewees by job title Job title Number of participants Clinical Director 1 Physicians (Chief Physician, Clinical Observer, clinical reviewers, physicians) 8 Residents 3 Clinical managers, nurse educators 4 Registered nurses, registered practical nurses 15 Clerks, orderlies 3 Physiotherapists 3 Pharmacy (pharmacist, drug distribution supervisor, pharmacy technician) 3 Total 40 The interview included questions about reporting, non-reporting and safety practices. Interviews averaged 45 min, and were digitally recorded and transcribed. Data analysis was undertaken by two researchers who met to discuss the themes in the interviews and the derivation of codes based on the data gathered. Atlas ti software (GmbH, Berlin, Germany) was used to code the interviews and retrieve quotations. The analysis involved a deductive and inductive approach.30 Through a reading of the literature, we were informed about concepts and approaches related to incident reporting systems and patient safety (deductive approach). Our analysis of the data revealed local practices related to the use of the incident reporting system (inductive approach). Through an iterative process of moving between the literature and the data, we identified three themes pertaining to how patient safety problems that healthcare practitioners can solve are handled with respect to reporting.

vealed local practices related to the use of the incident reporting system (inductive approach). Through an iterative process of moving between the literature and the data, we identified three themes pertaining to how patient safety problems that healthcare practitioners can solve are handled with respect to reporting. Results The Canadian tertiary care hospital where this voluntary incident reporting system was in place had instructions on the types of events expected to be entered: a ‘patient safety event’ is ‘Any circumstance where a patient experiences harm, or potential harm, due to medical care’ (Hospital literature). This is a broad statement; leaders refined and defined expectations regarding incident reporting, but these expectations were not always met by practitioners. A nursing leader described two different ways that events might be dealt with:We had dry alcohol swabs…So nurses would go ‘oh dry, dry, dry’, they'd go through 5–6 dry ones, get a wet one, and move on with their day. Finally somebody comes to me and says ‘why are all of our alcohol swabs coming up dry?’… So it takes that initiative. Some nurses have that initiative … they think about systems. Others think about moments, ‘this is an issue right now, this is something I can deal with…’ not ‘oh I think the hospital needs to know that this isn't working’. (Educator 1)

says ‘why are all of our alcohol swabs coming up dry?’… So it takes that initiative. Some nurses have that initiative … they think about systems. Others think about moments, ‘this is an issue right now, this is something I can deal with…’ not ‘oh I think the hospital needs to know that this isn't working’. (Educator 1) In the situation where the nurse opens alcohol swabs that are dry, and continues to do so until finding a wet one, the nurse is ‘fixing and forgetting’. The goal of getting a wet alcohol swab was reached—the nurse fixed the problem—so work can continue. However, the nurse who questions why there were dry swabs is ‘fixing and reporting’. The nurse fixed the problem—getting the wet alcohol swab as well—but ensured that the problem was reported before attending to the next task. The report allowed an investigation and went beyond only tending to the immediate problem at hand. This illustrates a situation that healthcare practitioners can address, followed by their choice to report or not to report, which is the subject of our study. We examine the context surrounding practitioners’ decision to ‘fix and forget’ or ‘fix and report’. Through this analysis, we revealed three themes: handling near misses, fixing individual patients’ safety problems and adapting to imperfections.

heir choice to report or not to report, which is the subject of our study. We examine the context surrounding practitioners’ decision to ‘fix and forget’ or ‘fix and report’. Through this analysis, we revealed three themes: handling near misses, fixing individual patients’ safety problems and adapting to imperfections. Handling near misses Near misses have various definitions,23 24 but in all these definitions, the incident has not been realised—some intervention prevented the near miss from progressing to a harmful incident. Some near misses, if reported, would overwhelm the practitioners:You have a crazy shift… Somebody has turned around and quickly they grab an IV fluid, they go to hang it, they realize it's in their hand and say ‘oh it's the wrong one’ and they go put it back. I basically have been told that that's a near miss, you should be doing an incident report. But we would drown in paperwork if we did that, right? (Registered Nurse 9)

d around and quickly they grab an IV fluid, they go to hang it, they realize it's in their hand and say ‘oh it's the wrong one’ and they go put it back. I basically have been told that that's a near miss, you should be doing an incident report. But we would drown in paperwork if we did that, right? (Registered Nurse 9) The main issue here was that reporting this type of near miss would create overburdening paperwork, and the subtext of the patient not being harmed helped justify the decision to ‘fix and forget’. In this case, putting the incorrect intravenous fluid back and getting the right intravenous fluid for the patient's need, and continuing on with various tasks was prioritised, without reporting the near miss. However, near misses were not reported for other reasons as well.If the physician wrote in the wrong chart they [nurses] will just call the physician up and say ‘hey you wrote the order in the wrong chart come and write it in the right chart’…They wouldn't fill out an incident report unless something had happened because of an order being written wrong… If they had a chance to fix it, it's not considered a near miss. Like I said, most near misses are not reported… there's no agreement on what constitutes a near miss. (Educator 2)

rite it in the right chart’…They wouldn't fill out an incident report unless something had happened because of an order being written wrong… If they had a chance to fix it, it's not considered a near miss. Like I said, most near misses are not reported… there's no agreement on what constitutes a near miss. (Educator 2) What constituted a near miss was poorly understood, and there was general agreement that they were under-reported. Generally, the view was that if a problem had not progressed to the patient (CPSI near miss), there would be no need to report. The potential for harm to the patient notwithstanding, the patient was not harmed, so ‘fixing and forgetting’ as opposed to ‘fixing and reporting’ was how near misses were generally handled. However, some did recognise that near misses were worth reporting:(Interviewer) Would you fill out [an incident report] for a dose that's a hundred times too high on the prescription and you got it clarified down to what it should have been? (Registered Practical Nurse 2) I should, yeah, that would be a near miss type thing, because it never got given but it had the potential to be harmful. This nurse considered that the potential for harm was a trigger to writing an incident report, and in contrast to his/her colleagues, engaged in ‘fixing and reporting’.

However, some did recognise that near misses were worth reporting:(Interviewer) Would you fill out [an incident report] for a dose that's a hundred times too high on the prescription and you got it clarified down to what it should have been? (Registered Practical Nurse 2) I should, yeah, that would be a near miss type thing, because it never got given but it had the potential to be harmful. This nurse considered that the potential for harm was a trigger to writing an incident report, and in contrast to his/her colleagues, engaged in ‘fixing and reporting’. Fixing individual patients’ safety problems Healthcare providers prioritise caring for individual patients, and if a problem occurs, providers tend to treat the situation as ‘a one-off’, or ‘a one-time’ event. Hazards are conditions or situations that could cause harm, but if the patient was not harmed, the situation was seen as not worth reporting. A prescription or order was ambiguous, but instead of viewing this as a problem worthy of reporting, the nurse got clarification so that the proper drugs were administered to his/her patient, and did not subsequently fill an incident report.Anything that adversely affects my patient is an incident report … If there's a potential, it becomes more of a judgement thing … ‘If we can fix it, don't report it’ type idea… A doctor writes an order, I don't understand what they've [written], I call the doctor and say ‘what was the dosage…’ They tell me. It hasn't affected my patient …. If there wasn't any effect on the patient—I fixed the problem—so I don't do an incident report. (Registered Nurse 9)

an fix it, don't report it’ type idea… A doctor writes an order, I don't understand what they've [written], I call the doctor and say ‘what was the dosage…’ They tell me. It hasn't affected my patient …. If there wasn't any effect on the patient—I fixed the problem—so I don't do an incident report. (Registered Nurse 9) This quote demonstrates two concepts—severity determining reporting, as well as responsibility towards an assigned patient. This nurse was certain that an incident that affects the patient is worthy of reporting (a CPSI harmful incident), but if a situation is fixed, the severity would decrease, and there would be no need to report. Additionally, the quote illustrates the responsibility felt towards assigned patients (‘my’ patient) and the desire to fix problems for individuals under a provider's direct care. An incident report would have little effect on an individual patient, so ‘fixing and forgetting’ was seen as far more aligned with what the healthcare provider believed to be his or her role. In a different scenario, where a patient was affected (a CPSI no harm incident), an incident report was not thought to be important.[The nurses] called me [and] we fixed the situation—we checked [the patient's] blood sugars and everything was fine … Because the mistake was recognized early and because the appropriate recourses were taken, it wasn't something that needed to be reported. (Resident 2)

an incident report was not thought to be important.[The nurses] called me [and] we fixed the situation—we checked [the patient's] blood sugars and everything was fine … Because the mistake was recognized early and because the appropriate recourses were taken, it wasn't something that needed to be reported. (Resident 2) In this case, short acting insulin had been injected instead of the intended long acting insulin, but the healthcare team realised it early enough to prevent any serious harm to the patient. The reactive response of the healthcare providers was considered appropriate, and ‘everything was fine’ for this patient; an incident report was seen not to be necessary. This is in contrast to one interviewee, who noted:[We should report] having two patients side by side with the exact same last name—which I have seen multiple times—you're asking for error to happen. (Physician 2) This physician talked of a hazard that can affect more patients than just an individual patient assigned to a practitioner as being worthy of reporting since the possibility of causing an incident is foreseeable. Reporting a hazard aligns with the hospital's expectation of reporting, ‘A circumstance where a patient experiences potential harm due to medical care’ (hospital literature). This is a case that is closer to ‘fixing and reporting’. It should be noted, however, that most interviewees espoused the ‘fix and forget’ option and only a few spoke about reporting hazards.

expectation of reporting, ‘A circumstance where a patient experiences potential harm due to medical care’ (hospital literature). This is a case that is closer to ‘fixing and reporting’. It should be noted, however, that most interviewees espoused the ‘fix and forget’ option and only a few spoke about reporting hazards. Adapting to imperfections Fixes, or adapting to unfixed problems, can become routinised normal work, and may not be noticed any longer. The extract below followed a discussion that only major events would be reported.(Interviewer) So [a patient whose paperwork wasn't complete], staying for the extra 2 days of the weekend isn't major? (Physician 3) Happens all the time. There's delays. My day is chock a block full of dealing with these sort of things as well as trying to take care of patients and do other duties… They're just things that occur day-to-day, that have always sort of occurred day-to-day in various different ways. That is part of the practice of medicine in a big large teaching hospital. The sense of inevitability of ‘these sorts of things’ is evident in stating the daily occurrence of seemingly minor problems that this practitioner needed to deal with to attend to patients and other duties. Reporting these problems, although identified as potentially harmful to patients, was not undertaken—was not considered. Rather, practitioners adapted to these seemingly minor issues and considered these as routine occurrences.(Resident 1) [Requisitions] get lost all the time and that's never reported. (Interviewer) And it's not reported?

The sense of inevitability of ‘these sorts of things’ is evident in stating the daily occurrence of seemingly minor problems that this practitioner needed to deal with to attend to patients and other duties. Reporting these problems, although identified as potentially harmful to patients, was not undertaken—was not considered. Rather, practitioners adapted to these seemingly minor issues and considered these as routine occurrences.(Resident 1) [Requisitions] get lost all the time and that's never reported. (Interviewer) And it's not reported? (Resident 1) Oh no. Everybody agrees that there's probably some black box …where all these radiology reqs that have been lost—that happens all the time… They say ‘oh well we never got the fax’… And that never gets reported, and it delayed treatment or delayed assessment. Here, although the potential for harm was acknowledged by the provider, the problem was not reported—rather the resident (and other practitioners) adapted to the frequent loss of requisitions and delayed assessments. Engaging in workarounds, such as hand delivering a requisition, became a routinised practice that escaped attention as a problem to be reported.

acknowledged by the provider, the problem was not reported—rather the resident (and other practitioners) adapted to the frequent loss of requisitions and delayed assessments. Engaging in workarounds, such as hand delivering a requisition, became a routinised practice that escaped attention as a problem to be reported. In short, ‘fixing and forgetting’ was the main choice that most practitioners made in situations where they faced problems that they themselves could resolve. These situations included (A) handling near misses, which were seen as unworthy of reporting since they did not result in actual harm to the patient, (B) prioritising solving individual patients’ safety problems, which were viewed as unique or one-time events, and (C) encountering re-occurring safety problems, which were framed as inevitable, routine events.

ear misses, which were seen as unworthy of reporting since they did not result in actual harm to the patient, (B) prioritising solving individual patients’ safety problems, which were viewed as unique or one-time events, and (C) encountering re-occurring safety problems, which were framed as inevitable, routine events. Discussion This study looked at specific encounters with hazards and problems where a healthcare practitioner might either fix the problem and move on, or fix the problem and report it into the reporting system. We identified three themes: handling near misses, fixing individual patients’ safety problems and adapting to imperfections. In these scenarios, the practitioners nearly always chose to fix and forget—or to engage in first order problem solving. It is worth reflecting here on Tucker and Edmondson's study, where they found that “on average, 33 min were lost per 8 h shift due to coping with system failures that could have been addressed and removed” (ref. 27, p.60). These different types of problems that practitioners fix and forget are lost to organisational learning, and may be costing them time as well in their workarounds. Choosing to be efficient in the moment may ironically cause the front-line providers to be far less efficient over time than intended.

ved” (ref. 27, p.60). These different types of problems that practitioners fix and forget are lost to organisational learning, and may be costing them time as well in their workarounds. Choosing to be efficient in the moment may ironically cause the front-line providers to be far less efficient over time than intended. One of the problems was near misses. Near misses in our study were poorly understood—a common occurrence in other healthcare contexts. Kessels-Habraken et al23 argued that healthcare has focused more on incidents that did reach the patient but did not cause harm (CPSI no harm incident—the insulin example in this study) than incidents that did not reach the patient (CPSI near miss—ambiguous prescription example in this study). Kessels-Habraken et al indicate that due to this focus, “valuable safety-related information about successful error recovery mechanisms remains unavailable or gets lost” (ref. 23, p.1302). Other healthcare studies have shown near misses not being tapped for their potential. Mattioli et al31 noted a preference for reporting incidents causing harm over near miss reporting in a paediatric surgical department. Jeffs et al32 also found that near misses generate three typical responses: doing a quick fix and nothing else (‘fixing and forgetting’ in our study), which was the most frequent response, and two types of ‘fixing and reporting’, whereby the report falls into a black hole, or is used as a catalyst for organisational change. In transportation, accident investigation crews examine crash sites extensively to find out why the crash occurred and how to prevent it in the future. They thus generate lessons for the industry, but only after lives have been lost in the crash. Alternately, near misses are seen as ‘free lessons’ where learning can occur, but without any deaths.33 Burnett, Carthey and Vincent stated that high-risk industries have shifted from focusing attention on incidents and realisation of harm to hazards and conditions that create safety.26 In healthcare, Schildmeijer et al recommend that random chart reviews be undertaken to find no-harm incidents,34 although this view is not unanimous.35 Franklin et al (ref. 19, p.770) state that “The challenge, across all areas of harm, is now to create and use data on… low-harm occurrences to test the resilience of safety practices and systems”.

et al recommend that random chart reviews be undertaken to find no-harm incidents,34 although this view is not unanimous.35 Franklin et al (ref. 19, p.770) state that “The challenge, across all areas of harm, is now to create and use data on… low-harm occurrences to test the resilience of safety practices and systems”. Fixing an individual patient's safety problem is common. As mentioned, healthcare providers often personalise the assignment of patients to them, and the care given (including fixing problems) to ‘my’ patient is then a source of professional pride and responsibility. Jeffs et al noted a physician stating “part of what we pride ourselves in is to be able to get ourselves out of tricky situations” (ref. 32, p.289). This professional pride in fixing problems is common to many front-line workers, including in rail36 and nursing.28 Practitioners view their ability to solve problems ‘… as a strong sign of their expertise and competence’ (ref. 37, p.101). Fixing problems that would otherwise harm that patient (to the exclusion of reporting the hazard) is common. However, as Jeffs et al (ref. 32, p.289) point out “As a result of such unreported quick-fix scenarios, learning remained local and confined to the individual level. Notably, learning is limited to the individual who initiated the fix.” Haradan adds (ref. 38, p.2) “We fix it for that particular person or family and the immediate surgical team involved learns, which is very important, but what about preventing such errors from happening to anyone else?”. Tucker and Edmondson27 note that nurses are encouraged to use vigilance to solve problems to the detriment of organisational improvement. This aligns directly with first order problem solving (fixing and forgetting), to the exclusion of second order problem solving (fixing and reporting).27–29 In fixing and reporting, the lessons are disseminated beyond the local circumstance of the individual patient, and can benefit far more patients as a result.

ment. This aligns directly with first order problem solving (fixing and forgetting), to the exclusion of second order problem solving (fixing and reporting).27–29 In fixing and reporting, the lessons are disseminated beyond the local circumstance of the individual patient, and can benefit far more patients as a result. It is worth noting that opting to report or not to report is not always a purposeful decision. For example, if fixing and forgetting becomes the norm, adapting to imperfections will follow, and the need to report hazards or problems will escape attention. Waring writes “the inevitability of error leads to more than acceptance [of errors], but also to their ‘normalisation’. This is where some common mistakes are regarded as routine and normal within the context of medical work, and in consequence these events are not perceived as problematic or worth reporting” (ref. 6, pp.1931–1932). Dekker explains Vaughan's influential term coined through her study of the Challenger launch decision: “The ‘normalisation of deviance’ describes a process whereby a group's construction of risk can persist even in the face of continued (and worsening) signals of potential danger… Small departures from an earlier established norm are often not worth remarking or reporting on” (ref. 39, p.538). Thinking of imperfections as routine occurrences could result in hazards or safety problems never being reported.

risk can persist even in the face of continued (and worsening) signals of potential danger… Small departures from an earlier established norm are often not worth remarking or reporting on” (ref. 39, p.538). Thinking of imperfections as routine occurrences could result in hazards or safety problems never being reported. However, not all healthcare practitioners are blind to the imperfections. Tucker and Edmondson describe a desirable employee as a “disruptive questioner who will not leave well enough alone. The person is constantly questioning, rather than accepting and committed to, current practices” (ref. 27, p.68). The success of the Central Line Blood Stream Infection initiative in the USA40—an excellent example of a patient safety intervention—began with a physician challenging the status quo, or the inevitability of error. Reporting problems can save front-line healthcare workers from encountering the same incident repeatedly, or engaging in workarounds that reinforce working with an imperfect system.27 Even in our study, a nurse saw the potential for patient harm through an order clarification, and a physician found two patients with the same last name in close proximity troubling, and both deemed the situation worthy of fixing and reporting.

at an organisational level. In contrast measures of patient experience and staff culture were not significantly different across Trusts. However, caution should be taken in interpretation given that safety outcomes are strongly influenced by case mix (eg, pressure sores and falls are more common on elderly care wards). The potential value of the PMOS tool as a proactive diagnostic tool should not be underestimated. While measures such as the friends and family test are parsimonious and may, with sufficiently high response rates, offer a means of comparing healthcare organisations or identifying problems, they provide information that is of little value to healthcare providers in delivering improvement. The collection of data without any real purpose is rife in the NHS. Policy makers should think carefully about what a measure offers before requiring providers to spend time and resources collecting data and providers should embrace measures which provide them with valuable information about what they can do to improve. We believe PMOS does just that.

engaging in workarounds that reinforce working with an imperfect system.27 Even in our study, a nurse saw the potential for patient harm through an order clarification, and a physician found two patients with the same last name in close proximity troubling, and both deemed the situation worthy of fixing and reporting. At this point, one might ask whether it is worth reporting all realised incidents and all potential incidents. Might this not result in costly and far too many reports, for the reporters,41 and those who analyse the reports?20 42 And more importantly, would such reporting yield safety benefits? For example, it has been pointed out that most events that are entered into reporting systems, such as patient falls, ‘provide little incremental value about the insight of safety systems… questioning the benefit of having the user report them in the first place’ (ref. 20, p.155). In contrast, other studies43 have looked at patient falls with an aim to get more of them reported. Importantly, most studies suggest that better criteria should be set to guide practitioners about what and how to report. Additionally, ‘operational ‘know how’ and conceptual ‘know why’’ are important for front-line engagement in quality improvement projects (ref. 20, p.125). Certainly the most serious harm to patients is dealt with expediently using hospital processes of risk management and senior personnel,42–44 often followed by extensive investigations. Equally important, however, is giving consideration to reporting near misses and hazards, and using them as the focus of investigation and intervention. This would enable investigations to have a preventive (and not only a reactive) approach to patient safety.44

rsonnel,42–44 often followed by extensive investigations. Equally important, however, is giving consideration to reporting near misses and hazards, and using them as the focus of investigation and intervention. This would enable investigations to have a preventive (and not only a reactive) approach to patient safety.44 Limitations This study has limitations. The focus on the fixable types of problems is to the exclusion of other types of reports that healthcare professionals write. Further, this study was undertaken in one hospital department, and may not be generalisable to other departments. However, the results concerning the under-reporting of near miss incidents have been found in other studies of hospital departments, notably paediatric surgery,31 paediatric ICU,45 radiation oncology,46 47 emergency48 and surgery.49 50 In addition, it should be noted that the strength of qualitative studies lies not in generalisability, but in the ability to provide an in-depth view of micro dynamics present in sites.51 In fact, findings from qualitative studies can be transferred to other contexts that are similar to the one studied.30 The quotes and interpretation should serve to illuminate the context sufficiently to allow the reader to assess if the findings of this study are transferable to other contexts.

ics present in sites.51 In fact, findings from qualitative studies can be transferred to other contexts that are similar to the one studied.30 The quotes and interpretation should serve to illuminate the context sufficiently to allow the reader to assess if the findings of this study are transferable to other contexts. Conclusion This study investigated a particular type of possible report—a problem that the practitioners themselves can typically resolve—and looked at their choice between ‘fixing and forgetting’ and ‘fixing and reporting’. In considering the themes of handling near misses, fixing individual patients’ safety problems, and adapting to imperfections, we found that generally healthcare providers do not prioritise reporting if a situation is fixed. This suggests a number of practice recommendations. ‘Fixing and forgetting’ could engender normalisation as exceptions become the norm, but recognition of hazards and problems may help healthcare guard against normalising deviance, and may help improve patient safety by focusing on a more preventive approach. Communication and training on what should be reported, why and how should be considered more carefully. Further, recognising the ‘disruptive questioner’27 as a desirable employee would be one step towards encouraging this new approach. Finding individuals who ‘fix and report’, enabling them to share their reporting approach with their colleagues, and recognising their efforts would underlie informal training that could result in more healthcare workers fixing and reporting hazards and problems, and help organisational learning that improves patient safety.

g individuals who ‘fix and report’, enabling them to share their reporting approach with their colleagues, and recognising their efforts would underlie informal training that could result in more healthcare workers fixing and reporting hazards and problems, and help organisational learning that improves patient safety. The authors thank Dr Alan Forster, Principal Investigator of the research programme, who enabled this study, and Saskia Vanderloo and Chantal Backman who provided project management for the study. The authors thank the participants of this study for generously giving their time and sharing their frank opinions. The authors also thank several individuals at the hospital who helped facilitate this study. The associate editor and anonymous reviewer comments received were very helpful in improving the manuscript. Twitter: Follow Tanya Hewitt at @HewittTanya Collaborators: Alan Forster; Saskia Vanderloo; Chantal Backman. Contributors: TAH: contributed to the conceptualisation of the study and the development of the study design, collected data, carried out data analysis and drafted the manuscript. SC: contributed to the development of the study design, collected data, reviewed the data analysis and the drafted manuscript. Both authors worked on and approved the final manuscript. Funding: Ontario Research Fund (#RE-05-070); University of Ottawa (Excellence Scholarship); Technical Standards and Safety Authority (Safety Education Graduate Research Scholarship). Competing interests: None.

Contributors: TAH: contributed to the conceptualisation of the study and the development of the study design, collected data, carried out data analysis and drafted the manuscript. SC: contributed to the development of the study design, collected data, reviewed the data analysis and the drafted manuscript. Both authors worked on and approved the final manuscript. Funding: Ontario Research Fund (#RE-05-070); University of Ottawa (Excellence Scholarship); Technical Standards and Safety Authority (Safety Education Graduate Research Scholarship). Competing interests: None. Ethics approval: Ottawa Health Science Network Research Ethics Board; University of Ottawa Office of Research Ethics and Integrity. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction A recent report on the measurement of safety1 concluded that there is no ‘single’ measure of patient safety, but that monitoring and measurement needs to be backward-looking, present-looking and forward-looking. That is, we need to know about and learn from past harm, understand what is going on in real time and anticipate future harm. The authors concluded that while healthcare organisations have grappled with the first of these (through incident reporting systems, root cause analysis and more recently the patient safety thermometer) and have made some progress with the second (eg, via early warning scores), the third remains somewhat elusive.

harm. The authors concluded that while healthcare organisations have grappled with the first of these (through incident reporting systems, root cause analysis and more recently the patient safety thermometer) and have made some progress with the second (eg, via early warning scores), the third remains somewhat elusive. Moreover, safety measurement has relied almost exclusively on eliciting information from staff (directly, eg, through incident reports and questionnaire measures, or indirectly, eg, through case note review) or requiring them to record information (eg, on the number of falls or pressure sores). More recently, however, following reports in the UK such as Francis;2 Keogh3 and Berwick4, commentators have emphasised the importance of patients as the ‘smoke detectors’ for safety.5 These ideas have a growing evidence base with increasing numbers of international studies reporting that patients are able to make a valuable contribution to our understanding of safety.6–8 Indeed, patients do have a voice, at least in some countries, and there are a growing number of measures that ask patients to report on their experience of care in the UK (eg, the Picker survey9 and the friends and family test10) and elsewhere, (eg, Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS)11 in the USA and the Australian Patient Experience Questionnaire (2014;12 see Castle et al,13 for a review). Rarely, however, are patients asked about the safety of their care or their care environment and to date most studies have asked patients to report on outcomes (patient safety incidents, eg, The National Danish Survey of Patient Experiences14) rather than on the factors that might represent failures in structures and processes (ie, systems of care). An exception to this is the Patient Measure of Safety (PMOS15 16) developed by the Yorkshire Quality and Safety Research Group. PMOS provides a proactive assessment of the organisational and local system factors known to contribute to patient safety incidents in hospitals.

tructures and processes (ie, systems of care). An exception to this is the Patient Measure of Safety (PMOS15 16) developed by the Yorkshire Quality and Safety Research Group. PMOS provides a proactive assessment of the organisational and local system factors known to contribute to patient safety incidents in hospitals. In other words, it is a forward-looking measurement instrument that provides information, based on patient perceptions, about how the ward/hospital is performing in a number of safety critical domains including communication and team work, access to resources, organisation and care planning, roles and responsibilities, ward type and layout, information flow, staff training, delays and equipment; those factors known to contribute to patient safety incidents.17 The PMOS tool has been extensively piloted and found to be reliable, valid and well received by patients.15 16 What is yet untested is the extent to which these tools provide the same or a different perspective on safety from more traditional patient safety measurement tools that are based on staff perceptions of safety or safety outcomes (as reported by staff).

nd found to be reliable, valid and well received by patients.15 16 What is yet untested is the extent to which these tools provide the same or a different perspective on safety from more traditional patient safety measurement tools that are based on staff perceptions of safety or safety outcomes (as reported by staff). The aims of this study were to investigate (A) the extent to which the staff and patient measures of safety correlate together and with the percentage of patients receiving harm-free care (% harm-free care) (patient safety thermometer data) (b) to explore the extent to which the staff and patient perspective on safety provide information that can explain variation in % harm-free care and (c) describe variation in these measures in three acute hospital Trusts. Methods Participants Participants were patients in one of 33 hospital wards, across five acute hospitals in three acute hospital Trusts in the Yorkshire and Humber region of the UK during May–July 2013. Patients who were deemed to have the capacity to respond to the measures (following discussion with the nurse in charge of the ward) and who had been admitted to hospital a minimum of 4 h previously, were approached by one of a team of six research nurses and invited to take part in the study. The aim was to collect data from a minimum of 15 patients per ward, recognising that in some wards (eg, elderly medicine) patient throughput would be much slower than in others (medical admissions unit).

um of 4 h previously, were approached by one of a team of six research nurses and invited to take part in the study. The aim was to collect data from a minimum of 15 patients per ward, recognising that in some wards (eg, elderly medicine) patient throughput would be much slower than in others (medical admissions unit). Staff on each of the participating wards were also asked to complete the four outcome measures of the Agency for Healthcare Research and Quality (AHRQ) Hospital Survey of Patient Safety Culture18 (hereafter referred to as the HSOPSC) patient safety culture tool.

um of 4 h previously, were approached by one of a team of six research nurses and invited to take part in the study. The aim was to collect data from a minimum of 15 patients per ward, recognising that in some wards (eg, elderly medicine) patient throughput would be much slower than in others (medical admissions unit). Staff on each of the participating wards were also asked to complete the four outcome measures of the Agency for Healthcare Research and Quality (AHRQ) Hospital Survey of Patient Safety Culture18 (hereafter referred to as the HSOPSC) patient safety culture tool. Measures Two measures of patient experience and safety and one measure of staff safety culture were used Patient perceptions of safety. The PMOS includes 44 items measuring 9 constructs: communication and teamwork (9 items); organisation and care planning (5 items); ward type and layout (11 items); equipment (2 items); roles and responsibility (4 items); access to resources (4 items); information flow (3 items); staff training (2 items) and delays (2 items). A score was calculated for each construct by taking a mean of the responses to all those items making up the domain. Where scores were not available for at least two items within a construct, the domain score was coded as missing data. The questionnaire is supplied as an online supplementary file here because the wording of some items differs from the measure previously published, due to changes following further piloting. Items are presented as statements with which patients are asked the extent to which they agree or disagree on a five-point Likert scale, with a higher score representing stronger agreement. Negatively worded items are recoded. The PMOS scale showed strong internal reliability (α=0.93). A mean of the construct scores was calculated to give a PMOS mean score between 1 and 5. Where scores were not available for one or more of the nine constructs, the PMOS mean score was coded as missing.

stronger agreement. Negatively worded items are recoded. The PMOS scale showed strong internal reliability (α=0.93). A mean of the construct scores was calculated to give a PMOS mean score between 1 and 5. Where scores were not available for one or more of the nine constructs, the PMOS mean score was coded as missing. Patient experience. The friends and family test was also completed by each patient. This asks “How likely are you to recommend this ward to your friends and family if they needed similar care or treatment?” Responses are provided on a six-point scale from ‘extremely unlikely’ to ‘extremely likely’. As part of routine data collection this question is asked of patients at discharge or no more than 48 h afterwards. In this case, we asked patients to complete the measure at the same time as the PMOS which was always prior to discharge. Staff safety culture. Nurses on participating wards were asked to complete the four outcome measures from the HSOPSC18. These are (1) perceptions of patient safety (four items, 1=‘strongly disagree’ to 5=‘strongly agree’); (2) patient safety grade (one item, 1=‘failing’ through to 5=‘excellent’); (3) frequency of events reported (three items, 1=‘never’ through to 5=‘always’); and (4) number of events reported (one item, categorical response options ranging from 1=‘no event reports’ through to 6=‘21 event reports or more’). The brief questionnaire was devised to promote a high response rate by minimising the burden on staff time.

vents reported (three items, 1=‘never’ through to 5=‘always’); and (4) number of events reported (one item, categorical response options ranging from 1=‘no event reports’ through to 6=‘21 event reports or more’). The brief questionnaire was devised to promote a high response rate by minimising the burden on staff time. Patient safety outcomes Patient safety outcome data is publicly available for all NHS Trusts in England. The NHS Safety Thermometer (http://www.hscic.gov.uk/thermometer)19 asks staff on hospital wards to record the presence or absence of four harms on a single day each month: pressure ulcers, falls, urinary tract infections in patients with a catheter and new venous thromboembolisms. Based on these data a score is computed for the percentage of patient assessments which showed none of the harms and therefore represents ‘harm-free’ care. We downloaded the data for all wards in the three participating acute hospital Trusts. Data were downloaded for each of the 4 months in which PMOS and staff data were collected at our participating hospitals to generate a ‘harm-free’ score and dividing by 4 (months) we created a mean ‘harm-free’ care score. This score is therefore a proportion (%) of patients on each ward who are recorded as not having been exposed to any of the four harms, based on the mean of 4 months data.

a were collected at our participating hospitals to generate a ‘harm-free’ score and dividing by 4 (months) we created a mean ‘harm-free’ care score. This score is therefore a proportion (%) of patients on each ward who are recorded as not having been exposed to any of the four harms, based on the mean of 4 months data. Procedure for data collection Patients were approached by a research nurse and provided with a brief synopsis of the purpose of the study. Interested patients were asked to read the information sheet and give their informed consent to participate. Patients were asked if they wanted to complete the questions directly or whether they wanted the research nurse to ask them the questions and then record their responses on the tablet computer (19% chose to complete the questionnaire themselves). The friends and family test was also completed by all patients as part of the collection of other data. Where problems or incidents were reported that were identified by the research staff as safety critical and requiring an immediate response, the patient was informed that the information would be passed on to the person in charge. We aimed to recruit 25 patients per ward (with a minimum of 15 patients) over a 4 week period. This number was a balance between achieving sufficient patients per ward to reasonably estimate parameters while capturing data within a short enough period so that substantial ward changes in that time frame were unlikely.

. We aimed to recruit 25 patients per ward (with a minimum of 15 patients) over a 4 week period. This number was a balance between achieving sufficient patients per ward to reasonably estimate parameters while capturing data within a short enough period so that substantial ward changes in that time frame were unlikely. Staff culture questionnaires were prepared for members of staff employed on each ward using staff lists collected from the ward prior to the recruitment period. When research nurses began recruitment of patients they met with the ward sister/manager and asked him/her to distribute the questionnaires to staff and encourage completion. Staff returned questionnaires using drop boxes which were collected from the ward no longer than a month after patient recruitment finished. This was to ensure that feedback from staff and patients was collected during the same time period. To allow for the collection of data from staff who were not employed by a particular ward/unit, for example, doctors, five blank questionnaires were also made available for each ward and the sister was asked to ensure these were all distributed. All questionnaires were labelled with a code that allowed identification of the trust and ward, but not the individual. The aim was to recruit a minimum of 50% of the staff on each ward. Where this target was not reached at the end of the patient data collection period, research nurses visited the wards on up to two further occasions to encourage questionnaire completion (blank copies were made available if necessary).

Staff culture questionnaires were prepared for members of staff employed on each ward using staff lists collected from the ward prior to the recruitment period. When research nurses began recruitment of patients they met with the ward sister/manager and asked him/her to distribute the questionnaires to staff and encourage completion. Staff returned questionnaires using drop boxes which were collected from the ward no longer than a month after patient recruitment finished. This was to ensure that feedback from staff and patients was collected during the same time period. To allow for the collection of data from staff who were not employed by a particular ward/unit, for example, doctors, five blank questionnaires were also made available for each ward and the sister was asked to ensure these were all distributed. All questionnaires were labelled with a code that allowed identification of the trust and ward, but not the individual. The aim was to recruit a minimum of 50% of the staff on each ward. Where this target was not reached at the end of the patient data collection period, research nurses visited the wards on up to two further occasions to encourage questionnaire completion (blank copies were made available if necessary). Analysis Frequencies and descriptive summaries were run to explore the numbers and types of patients and staff on each ward and to check for any anomalous values and explore patterns of responding across wards.

The aim was to recruit a minimum of 50% of the staff on each ward. Where this target was not reached at the end of the patient data collection period, research nurses visited the wards on up to two further occasions to encourage questionnaire completion (blank copies were made available if necessary). Analysis Frequencies and descriptive summaries were run to explore the numbers and types of patients and staff on each ward and to check for any anomalous values and explore patterns of responding across wards. The unit of analysis here was ward. All measurements from staff or patients were averaged at the ward level before entering into the data file. The harm-free care score was only available at ward level. Pearson correlations were run to assess the association between harm-free care score (averaged score across the 4 months of data collection), PMOS (total score for each ward) and the four patient safety outcomes from the HSOPSC Hospital Survey of Patient Safety. Correlation coefficients of 0.1 were considered small; 0.3 moderate and 0.5 and above large.20 All variables are coded such that a positive score equals good levels of safety, so that we anticipated positive correlations among all variables. There was one exception to this. Positive correlations were predicted for all variables with the exception of the number of events reported. Higher number of events reported can indicate either good or bad levels of safety (ie, it could be that staff are doing a good job of reporting all errors (good) or it could be that there are a lot of errors to report (bad)). Scatterplots (see online resources) were run to assess whether relationships between variables were linear and whether particular wards represented outliers with respect to patterns of correlations. Spearman's rank correlations were also computed to ensure that the pattern of findings were the same if assumptions of linearity were not met.

ine resources) were run to assess whether relationships between variables were linear and whether particular wards represented outliers with respect to patterns of correlations. Spearman's rank correlations were also computed to ensure that the pattern of findings were the same if assumptions of linearity were not met. Linear regressions were computed to assess the predictive value of the different measures of safety. Only those variables that demonstrated univariate correlations with the % harm-free care score (dependent variable) were entered into the regression. In each regression ward size (number of patients treated on the ward), type (medical or surgical) and average age of patients treated on the ward were first entered into the model as these represent the minimisation factors in our sampling strategy. The first regression model tests the amount of variance accounted for when only staff measures of safety (independent variable 1) are regressed onto % harm-free care. The second model investigates the variance accounted for when only the patient measures (PMOS; independent variable 2) is included in the model. The final model investigates the additional variance that is accounted for (r2 change) by the PMOS score having already accounted for the staff measures of safety in the model. Although ward was the unit of analysis there is further clustering at the Trust level but with only three Trusts we could not run a multilevel model accounting for this. A MANOVA was run to compare Trusts across the four measures and to identify any differences in safety scores across Trusts.

Linear regressions were computed to assess the predictive value of the different measures of safety. Only those variables that demonstrated univariate correlations with the % harm-free care score (dependent variable) were entered into the regression. In each regression ward size (number of patients treated on the ward), type (medical or surgical) and average age of patients treated on the ward were first entered into the model as these represent the minimisation factors in our sampling strategy. The first regression model tests the amount of variance accounted for when only staff measures of safety (independent variable 1) are regressed onto % harm-free care. The second model investigates the variance accounted for when only the patient measures (PMOS; independent variable 2) is included in the model. The final model investigates the additional variance that is accounted for (r2 change) by the PMOS score having already accounted for the staff measures of safety in the model. Although ward was the unit of analysis there is further clustering at the Trust level but with only three Trusts we could not run a multilevel model accounting for this. A MANOVA was run to compare Trusts across the four measures and to identify any differences in safety scores across Trusts. Results Data were collected from 822 patients and 648 members of staff. The number and type of patients recruited varied across wards (see table 1). Recruitment was at the planned level (with a mean of 25 patients) for all but three of the participating wards. Response rates were high, above 80% on all but three wards and only falling below 70% on one ward. Those wards (1, 11 and 25) where we did not reach the minimum target of 15 respondents were elderly wards with a very slow turnover of patients (wards 1 and 25) or had a large number of patients who were unable to consent due to a lack of capacity, or severity of illness (ward 11). A response rate of 50% or above was achieved for the HSOPSC outcomes in Trust A and Trust B, but was poorer in Trust C (the largest of our three participating Trusts). All wards in all Trusts were nevertheless included in the analyses.

were unable to consent due to a lack of capacity, or severity of illness (ward 11). A response rate of 50% or above was achieved for the HSOPSC outcomes in Trust A and Trust B, but was poorer in Trust C (the largest of our three participating Trusts). All wards in all Trusts were nevertheless included in the analyses. Table 1 Response rate for patient and staff surveys and demographic characteristics for patients recruited to the study

were unable to consent due to a lack of capacity, or severity of illness (ward 11). A response rate of 50% or above was achieved for the HSOPSC outcomes in Trust A and Trust B, but was poorer in Trust C (the largest of our three participating Trusts). All wards in all Trusts were nevertheless included in the analyses. Table 1 Response rate for patient and staff surveys and demographic characteristics for patients recruited to the study Patient response rate n (%) Mean age % Female % White British Staff response rate n (%) % nursing Trust A Ward 1 14 (82) 73 14 100 20 (69) 45 Ward 2 35 (92) 55 0 97 25 (71) 60 Ward 3 35 (80) 50 71 100 19 (53) 47 Ward 4 26 (93) 72 54 92 21 (66) 67 Ward 5 30 (97) 60 37 100 30 (73) 67 Ward 6 31 (89) 68 100 100 25 (68) 36 Ward 7 18 (78) 56 61 100 17 (74) 71 Ward 8 16 (80) 66 67 100 26 (58) 46 Trust B Ward 9 26 (84) 58 35 85 20 (51) 55 Ward 10 34 (89) 66 59 91 24 (92) 58 Ward 11 14 (54) 69 43 100 18 (55) 67 Ward 12 32 (100) 59 31 94 15 (63) 47 Ward 13 29 (100) 60 69 90 15 (50) 40 Ward 14 22 (92) 60 48 86 17 (61) 53 Ward 15 18 (78) 61 83 94 19 (49) 53 Ward 16 26 (90) 62 31 88 26 (49) 62 Ward 17 27 (96) 52 100 88 27 (96) 70 Ward 18 25 (100) 63 52 96 25 (54) 44 Trust C Ward 19 18 (82) 64 29 100 15 (56) 60 Ward 20 32 (84) 66 0 90 17 (55) 59 Ward 21 30 (97) 47 30 87 28 (54) 57 Ward 22 29 (94) 51 45 86 13 (46) 38 Ward 23 29 (91) 51 3 97 8 (36) 63 Ward 24 29 (97) 62 72 96 10 (38) 30 Ward 25 8 (89) 86 100 100 20 (65) 40 Ward 26 22 (76) 67 38 91 11 (31) 45 Ward 27 18 (90) 60 0 88 16 (67) 38 Ward 28 20 (100) 57 100 85 15 (52) 53 Ward 29 16 (94) 76 50 88 19 (46) 58 Ward 30 27 (96) 62 93 96 22 (50) 55 Ward 31 29 (91) 53 3 86 24 (59) 67 Ward 32 32 (84) 56 47 97 22 (61) 45 Ward 33 25 (89) 61 48 92 19 (53) 42 Correlation of different measures of the quality of care The distributions for each of the seven variables to be correlated appeared normal and skew and kurtosis values for all variables were below 1, with the exception of frequency of events reported where values were slightly outside this range (−1.17 and 1.22, respectively). Thus the data were treated as normal and Pearson's correlations were run.i The correlations between the four types of data (PMOS, harm-free care score, the four staff safety culture (HSOPSC) outcomes and friends and family test) are shown below in table 2. Scatterplots showing the distribution of variables and the patterns of associations between variables are shown in online supplementary figure S1.

elations between the four types of data (PMOS, harm-free care score, the four staff safety culture (HSOPSC) outcomes and friends and family test) are shown below in table 2. Scatterplots showing the distribution of variables and the patterns of associations between variables are shown in online supplementary figure S1. Table 2 Mean (SD) for safety measures and correlations between measures (N=33 wards) Mean (SD) % harm-free care Friends and family test PMOS mean score HSOPSC—perceptions of safety HSOPSC—patient safety grade HSOPSC—frequency of events HSOPSC number of events reported % harm-free care (0–100) 91.61 (5.97) Friends and family test (1–6) 4.38 (0.25) 0.29 PMOS mean score (0–5) 3.84 (0.17) 0.58*** 0.69*** HSOPSC—perceptions of safety 3.54 (0.39) 0.48** -0.05 0.25 HSOPSC—patient safety grade 3.80 (0.44) 0.54*** 0.11 0.30 0.91*** HSOPSC—frequency of events 4.04 (0.33 0.04 0.12 0.03 0.48** 0.56** HSOPSC—number of events reported 2.39 (0.52) −0.41* −0.39* −0.43* −0.47** −0.44* −0.36* *p<0.05; ** p<0.01; *** p<0.001. % harm-free care, percentage of patients receiving harm-free care; HSOPSC, Hospital Survey of Patient Safety Culture; PMOS, Patient Measure of Safety.

Mean (SD) % harm-free care Friends and family test PMOS mean score HSOPSC—perceptions of safety HSOPSC—patient safety grade HSOPSC—frequency of events HSOPSC number of events reported % harm-free care (0–100) 91.61 (5.97) Friends and family test (1–6) 4.38 (0.25) 0.29 PMOS mean score (0–5) 3.84 (0.17) 0.58*** 0.69*** HSOPSC—perceptions of safety 3.54 (0.39) 0.48** -0.05 0.25 HSOPSC—patient safety grade 3.80 (0.44) 0.54*** 0.11 0.30 0.91*** HSOPSC—frequency of events 4.04 (0.33 0.04 0.12 0.03 0.48** 0.56** HSOPSC—number of events reported 2.39 (0.52) −0.41* −0.39* −0.43* −0.47** −0.44* −0.36* *p<0.05; ** p<0.01; *** p<0.001. % harm-free care, percentage of patients receiving harm-free care; HSOPSC, Hospital Survey of Patient Safety Culture; PMOS, Patient Measure of Safety. The correlations in table 2 show that while the friends and family test score is correlated with the PMOS, it is not significantly correlated with either the staff safety culture (HSOPSC) or the % harm-free care reported in participating wards. The friends and family test was therefore excluded from the regression analysis. The strongest correlations were between the PMOS and the % harm-free care (r=0.58) score and between the HSOPSC patient safety grade and the % harm-free care score (r=0.54). Perceptions of patient safety (0.48) and number of patient safety events reported (−0.41) were also significantly correlated with % harm-free care. The latter negative correlation indicates that as the number of patient safety events reported by staff increased, the % harm-free care decreased. The frequency of reporting of events was not associated with harm-free care and therefore will not be included in the subsequent analysis. Interesting too was the lack of a significant correlation between the perceptions of patient safety, the patient safety grade and the PMOS score. However, the number of events reported by staff did show a significant negative correlation (r=−0.43) such that the more safety events reported by staff, the lower the PMOS score. The very high correlations between the HSOPSC patient safety grade and perceptions of safety (r=0.91) indicated that these two scales were measuring the same thing and therefore to avoid multicollinearity, only the patient safety grade score was entered in the regression analysis.

ed by staff, the lower the PMOS score. The very high correlations between the HSOPSC patient safety grade and perceptions of safety (r=0.91) indicated that these two scales were measuring the same thing and therefore to avoid multicollinearity, only the patient safety grade score was entered in the regression analysis. Factors associated with safety outcome (harm-free care score) Three separate regressions were computed. In each case, ward characteristics were entered on the first step to control for any variation in patient safety outcomes due to factors that might be anticipated to co-vary with safety outcomes (ward size, average age of patients and whether the ward was surgical or medical). Together the ward characteristics accounted for 15% (r2=0.15) of the variance and this model was not statistically significant (p=0.06). Regression model 1 Staff measures of safety (patient safety grade and number of events reported in the last 12 months), were entered on the second step and the regression model was significant (F (5,26)=5.04, p<0.01) and the adjusted r2 was 0.39. The standardised regression coefficient was significantly different from zero for patient safety grade (β=0.43, p<0.05) but not for number of events reported (β=−0.31, p=0.08). Regression model 2 PMOS was entered on step 2 and the regression model was significant (F (4, 27)=4.39, p<0.01) and the adjusted r2 was 0.30. The standardised regression coefficient for PMOS was significant in the model (β=0.50, p<0.05).

Regression model 1 Staff measures of safety (patient safety grade and number of events reported in the last 12 months), were entered on the second step and the regression model was significant (F (5,26)=5.04, p<0.01) and the adjusted r2 was 0.39. The standardised regression coefficient was significantly different from zero for patient safety grade (β=0.43, p<0.05) but not for number of events reported (β=−0.31, p=0.08). Regression model 2 PMOS was entered on step 2 and the regression model was significant (F (4, 27)=4.39, p<0.01) and the adjusted r2 was 0.30. The standardised regression coefficient for PMOS was significant in the model (β=0.50, p<0.05). Regression model 3 In this model, the staff measures of safety were entered on step 2, followed by the patient measures of safety on step 3. See model 1 above for the results of step 2. On step 3, the adjusted r2 was 0.49 and the model was statistically significant (F(6,25)=6.02, p<0.001). Patient safety grade (β=0.42, p<0.01), PMOS (β=0.40, p<0.05) and average age of patients on the ward (β=−0.38, p<0.05) were significant predictors of % harm-free care, such that more positive staff grades of safety, more positive patient perceptions of safety and wards with younger patients were all associated with less harm to patients.

e (β=0.42, p<0.01), PMOS (β=0.40, p<0.05) and average age of patients on the ward (β=−0.38, p<0.05) were significant predictors of % harm-free care, such that more positive staff grades of safety, more positive patient perceptions of safety and wards with younger patients were all associated with less harm to patients. Variation in scores between hospitals Table 3 shows the means and SDs for each of the participating NHS Trusts on the four types of measure. The friends and family test is consistently high and positive, across Trusts. The scores on the PMOS and the HSOPC were lower and less positive overall. The % harm-free care scores indicate that in Trust A, 7.00% of patients are harmed, in Trust B this is lower at 5.66% and highest in Trust C at 11.13%. The lowest performing Trust across all four of the measures was Trust C. Table 3 Means (SD) for each Trust on each of the patient safety measures Trust A Trust B Trust C F (sig) p Value Friends and family test 4.48 (0.23) 4.43 (0.25) 4.30 (0.24) 1.53 0.23 PMOS score 3.91 (0.16) 3.93 (0.08) 3.74 (0.19) 5.55 0.009 HSOPSC total 3.93 (0.33) 3.82 (0.27) 3.75 (0.35) 0.81 0.46 Harm-free care score 93.00 (6.01) 94.34 (2.95) 88.87 (6.68) 3.10 0.06 HSOPSC, Hospital Survey of Patient Safety Culture; PMOS, Patient Measure of Safety.

est 4.48 (0.23) 4.43 (0.25) 4.30 (0.24) 1.53 0.23 PMOS score 3.91 (0.16) 3.93 (0.08) 3.74 (0.19) 5.55 0.009 HSOPSC total 3.93 (0.33) 3.82 (0.27) 3.75 (0.35) 0.81 0.46 Harm-free care score 93.00 (6.01) 94.34 (2.95) 88.87 (6.68) 3.10 0.06 HSOPSC, Hospital Survey of Patient Safety Culture; PMOS, Patient Measure of Safety. A multivariate analysis of variance was used to investigate the differences in scores across the four measures for the three Trusts from which we collected data. There was a main effect of Trust (F(5,25)=2.14, p<0.05). Inspection of the univariate differences revealed that while the friends and family test and the HSOPSC outcomes did not differ significantly, the PMOS scores did demonstrate a significant difference between Trusts. The harm-free care score was close to significance. Bonferroni pairwise comparisons revealed that only Trust B and Trust C differed significantly (p=0.009) on the PMOS measure, with Trust A and Trust C showing differences, but which did not reach statistical significance (p=0.06). Discussion Patient safety is a multifaceted and complex concept which may be assessed differently by staff and patients.16 Indeed there is some empirical evidence which suggests that patients might offer a different perspective on safety from staff.21 22

A multivariate analysis of variance was used to investigate the differences in scores across the four measures for the three Trusts from which we collected data. There was a main effect of Trust (F(5,25)=2.14, p<0.05). Inspection of the univariate differences revealed that while the friends and family test and the HSOPSC outcomes did not differ significantly, the PMOS scores did demonstrate a significant difference between Trusts. The harm-free care score was close to significance. Bonferroni pairwise comparisons revealed that only Trust B and Trust C differed significantly (p=0.009) on the PMOS measure, with Trust A and Trust C showing differences, but which did not reach statistical significance (p=0.06). Discussion Patient safety is a multifaceted and complex concept which may be assessed differently by staff and patients.16 Indeed there is some empirical evidence which suggests that patients might offer a different perspective on safety from staff.21 22 We found that two measures showed strong correlations with patient safety outcomes as measured by %harm-free care. These were the staff measure of safety culture (HSOPSC outcomes) and the PMOS. The regression models showed that staff perceptions (adjusted r2=0.39) and patient perceptions (adjusted r2=0.30) of safety independently predicted safety outcomes. When entered together both measures accounted for 49% of the variance in safety outcomes (adjusted r2=0.49), suggesting that there is some overlap but also a unique variance explained by these two measures. In other words, what patients tell us about the safety of their care may partially overlap with staff perceptions, but patients tell us something that helps us to understand more about the safety outcomes on hospital wards. The friends and family test, which is purported to measure patient experience of care was significantly correlated with the PMOS, which measures patient perceptions of the safety of care. However the friends and family test did not show any relationship with patient safety outcomes or staff measures of safety. This suggests that while overall patient experience and patient experience of safety may overlap, only the latter, more specific, measure provides a potentially useful measure of the safety of care.

wever the friends and family test did not show any relationship with patient safety outcomes or staff measures of safety. This suggests that while overall patient experience and patient experience of safety may overlap, only the latter, more specific, measure provides a potentially useful measure of the safety of care. It could be argued that in completing questionnaires about their perceptions of safety, staff will be influenced by recent harm-free care scores, raising the possibility that the relationship between these two variables is bidirectional. The same is unlikely to be true for patient scores of safety. Some, but not all wards, make their harm-free care scores publicly available through wall displays in wards. However, with nothing in the way of comparator data, patients are unlikely to know whether the scores are positive or not and therefore the impact on their ratings of safety are unlikely to be strong. The PMOS measure includes a large number of items and therefore is somewhat onerous to complete, taking approximately 10–15 min per patient. However, it has a number of advantages that make it an important addition to the battery of measurement tools for patient safety. First, asking patients, rather than staff to complete surveys, reduces the burden on staff. Second, the 44-item PMOS questionnaire provides feedback on the general perceptions of patients about safety, and offers unique information about the performance of a ward or unit in nine domains of safety.15 16 This information can be used by wards to inform and plan local improvements.

s, reduces the burden on staff. Second, the 44-item PMOS questionnaire provides feedback on the general perceptions of patients about safety, and offers unique information about the performance of a ward or unit in nine domains of safety.15 16 This information can be used by wards to inform and plan local improvements. A key attribute for any measure of quality and safety is that it should be able to describe opportunities for improvement. Many patient measures (eg, patient satisfaction and the friends and family test) suffer from a ceiling effect where patients are consistently positive in their responses. In this study we explored the extent to which measures might discriminate across Trusts. The findings here revealed that the PMOS scores do vary across Trusts and that it is possible to identify significant differences in patient perceptions of safety that might provide an important and proactive measure of safety at an organisational level. In contrast measures of patient experience and staff culture were not significantly different across Trusts. However, caution should be taken in interpretation given that safety outcomes are strongly influenced by case mix (eg, pressure sores and falls are more common on elderly care wards).

y real purpose is rife in the NHS. Policy makers should think carefully about what a measure offers before requiring providers to spend time and resources collecting data and providers should embrace measures which provide them with valuable information about what they can do to improve. We believe PMOS does just that. Limitations The 50% target response rate to the staff culture survey was not achieved on some wards, particularly in Trust C. Although research staff administered and distributed the questionnaires, the nature of data collection meant there was a reliance on ward managers/sisters to encourage their staff to complete the survey. Encouragement was often related to how engaged the senior ward staff were with the study and also their style of leadership. Some ward managers were more proactive than others and personally handed them out, some did nothing at all to encourage completion. After two further visits by research nurses to wards with low response rates, the research team agreed that data collection should cease to ensure that the patient and staff feedback was elicited over the same period of time. Moreover, we achieved good response rates for the majority of wards and data were analysed at ward level so variation in response rates is very unlikely to explain the relationships between variables.

t data collection should cease to ensure that the patient and staff feedback was elicited over the same period of time. Moreover, we achieved good response rates for the majority of wards and data were analysed at ward level so variation in response rates is very unlikely to explain the relationships between variables. We did not recruit patients who either lacked capacity or were too ill to participate, to complete PMOS. While this is ethically justifiable, we may have also inadvertently excluded those patients who, through experiencing a broader range of treatment related interventions and sometimes for a longer time period, might have provided an important perspective on organisational safety. Our research nurses were aware of this and were careful to return to patients who they were not able to recruit on day 1 to check whether they were able to consent on a later day (eg, when they were 2 days postop or if they were having a more lucid episode). It may also be appropriate, in future, to elicit the views of these groups by asking friends or relatives to complete PMOS on the patient's behalf. However, further research will need to be undertaken to assess the extent to which the views of friends and relatives reflect those of the patients, being on the wards, as they generally are, for a limited time and at particular times of the day.

by asking friends or relatives to complete PMOS on the patient's behalf. However, further research will need to be undertaken to assess the extent to which the views of friends and relatives reflect those of the patients, being on the wards, as they generally are, for a limited time and at particular times of the day. In this study, data were collected by dedicated research nurses who have been working on this project since the empirical phase began. They have developed a specific repertoire of skills that allow them to engage with patients about their safety and have received training in the theory of patient safety. Such preparation would have some resource implications for an organisation who might wish to adopt the interventions. It will be necessary to explore alternative and potentially more efficient approaches to data collection, for example, via hospital volunteers or undergraduate medical and nursing students. Further research It is important to note that the harm-free care score assesses a limited number of harms. For example, it does not assess medication errors, other infections, misdiagnosis, etc. It is not possible to conclude that the same pattern of relationships will exist for other safety outcome measures. Further exploration of these relationships for other safety outcome measures is necessary.

limited number of harms. For example, it does not assess medication errors, other infections, misdiagnosis, etc. It is not possible to conclude that the same pattern of relationships will exist for other safety outcome measures. Further exploration of these relationships for other safety outcome measures is necessary. PMOS has been designed for the acute care sector. However, the design has a number of principles that have the potential for transferability to other sectors. It is entirely feasible therefore that equivalent measures could be developed for primary, community and mental healthcare organisations albeit with some modification to the domains and the questions used to assess them within the questionnaire. The PMOS questionnaire, as well as providing a quantitative measure of the safety of the care environment through the eyes of patients, was developed primarily as a diagnostic tool. Wards/departments can use the scores on the nine domains to proactively identify local areas of strength and weakness and to plan for safety improvements using this information. A cluster randomised study is currently underway to evaluate the impact on safety outcomes of using the PMOS tool in this way. Further research will be needed to understand whether approaches such as this represent cost-effective solutions to improving safety in hospitals and the extent to which this varies as a function of different data collection methods.

underway to evaluate the impact on safety outcomes of using the PMOS tool in this way. Further research will be needed to understand whether approaches such as this represent cost-effective solutions to improving safety in hospitals and the extent to which this varies as a function of different data collection methods. Conclusion This study showed that a patient measure of ward safety and a staff culture measure were associated with the level of harm-free care. PMOS provides a distinctive patient-centred measure of patient safety that has the potential to inform safety improvement actions appropriate to individual wards. Supplementary Material Web supplement Web figure Twitter: Follow Jane O'Hara at @janekohara Contributors: RL, GA and JW conceived the original research proposal. JKO, LS, GA and RL developed the protocol and managed the project. RL and KC conducted and interpreted analyses, with support from CR. RL drafted the manuscript. All authors commented on and approved the final draft. Funding: This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Improving safety through the involvement of patients: grant reference number RP-PG-0108-10049). Competing interests: None. Patient consent: Obtained. Ethics approval: South Yorkshire Research Ethics Committee (13/YH/0077). Provenance and peer review: Not commissioned; externally peer reviewed.

Funding: This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Improving safety through the involvement of patients: grant reference number RP-PG-0108-10049). Competing interests: None. Patient consent: Obtained. Ethics approval: South Yorkshire Research Ethics Committee (13/YH/0077). Provenance and peer review: Not commissioned; externally peer reviewed. i Spearman's rank correlations were also computed, but the patterns and significance of the relationships did not change when using this method, although the sizes of the correlations were consistently more conservative.

Introduction Patient experience of care in general practice is routinely monitored in England using the national General Practice Patient Survey (GPPS).1 Recent analyses of GPPS data identified differential overall satisfaction with routine primary care across patient sociodemographic factors, such as age and ethnicity.2 Further analyses of GPPS data have indicated that interpersonal aspects of care were the strongest drivers of patient satisfaction, whereas ease of access to care was a weaker driver of patient satisfaction.3 However, younger adult patients (aged 18–25 years) valued access relatively more highly (compared with patients aged 55–64 years); patients from Asian and Chinese ethnic backgrounds valued telephone access relatively more highly (compared with white patients). At present, little is known about how the management of appointment requests, for example through the use of triage systems, may have an impact on patient experience of care. While providing important data on patients’ experiences in primary care, both routine and out of hours, GPPS does not specifically address patient experience following requests for a prompt consultation within routine practice hours, for example a telephone request to see a general practitioner (GP) that same day.

are. While providing important data on patients’ experiences in primary care, both routine and out of hours, GPPS does not specifically address patient experience following requests for a prompt consultation within routine practice hours, for example a telephone request to see a general practitioner (GP) that same day. A systematic review of patient satisfaction with telephone triage models4 identified two randomised controlled trials (RCTs).5 6 The more recent of these was conducted in two practices in Scotland and randomised patients who contacted their practice to request a same-day appointment to either a face-to-face appointment on that day or to telephone triage by a GP.6 That trial found no significant difference in patient satisfaction; however, the control group received a same-day face-to-face GP consultation, which may not routinely be available to all patients requesting such a consultation in all practices. Thus, the comparison of GP telephone triage versus a same-day face-to-face GP consultation may not be generalisable to many practices.

ction; however, the control group received a same-day face-to-face GP consultation, which may not routinely be available to all patients requesting such a consultation in all practices. Thus, the comparison of GP telephone triage versus a same-day face-to-face GP consultation may not be generalisable to many practices. The ESTEEM three-arm cluster RCT investigated the effects of two forms of triage, one led by GPs and the other led by nurses using computer-supported decision-making software, compared with usual care (defined as the care that the patient would normally receive from his/her practice when requesting a same-day GP consultation). The design and primary findings of ESTEEM have been described elsewhere,7–9 as well as a detailed account of the process evaluation10 and a discussion of differences in communication comparing GPs and nurses performing telephone triage.11 Conducted in four regions of England, this large cluster RCT (randomisation was conducted at the level of the practice) collected data from 20 990 patients within 42 GP practices. The aim of the trial was to compare the effects of the two triage systems with usual care, with regard to healthcare resource use, safety, health-related outcomes and patient satisfaction outcomes, among patients who had requested a same-day face-to-face consultation with a GP. Initial analyses of the trial data indicated that patients’ reports of their experience of care were generally positive, although there was some evidence that nurse triage was less positively regarded by patients compared with both usual care and GP triage.8 9

d requested a same-day face-to-face consultation with a GP. Initial analyses of the trial data indicated that patients’ reports of their experience of care were generally positive, although there was some evidence that nurse triage was less positively regarded by patients compared with both usual care and GP triage.8 9 This paper presents more detailed analyses of the ESTEEM trial data regarding patients’ experience of, or satisfaction with, care. The secondary analyses reported here sought to (i) identify any patient characteristics associated with patient experience and (ii) determine whether there were any differences in reporting of experience across trial arms among different groups as defined by patient characteristics. We address these issues with regard to patient sociodemographic characteristics (gender, age, ethnic group and deprivation status) and health/lifestyle characteristics (presence of a long-standing health condition and self-reported ability to take time away from work during the patient's typical working hours if relevant). Methods The data collection and analysis methods of the ESTEEM trial have been reported elsewhere.7–9

This paper presents more detailed analyses of the ESTEEM trial data regarding patients’ experience of, or satisfaction with, care. The secondary analyses reported here sought to (i) identify any patient characteristics associated with patient experience and (ii) determine whether there were any differences in reporting of experience across trial arms among different groups as defined by patient characteristics. We address these issues with regard to patient sociodemographic characteristics (gender, age, ethnic group and deprivation status) and health/lifestyle characteristics (presence of a long-standing health condition and self-reported ability to take time away from work during the patient's typical working hours if relevant). Methods The data collection and analysis methods of the ESTEEM trial have been reported elsewhere.7–9 Patient questionnaire All eligible patients within ESTEEM were sent a postal questionnaire (including a participant information sheet and reply-paid envelope) 4 weeks after the initial same-day consultation request. Further details on the development of the ESTEEM questionnaire have been reported elsewhere.9 To maximise response rates, non-respondents to initial mailings were sent up to two reminders (after 2 and 4 weeks had elapsed respectively). Implied consent to participate was evidenced by return of the questionnaire. The questionnaire included six evaluative items inviting the patient (or a parent/carer) to rate aspects of the care received by the patient on the day of the consultation request. We investigated three aspects of patient care: (i) overall satisfaction with care (scored on a Likert scale of 1–5: 1—very satisfied; 2—fairly satisfied; 3—neither satisfied not dissatisfied; 4—fairly dissatisfied; 5—very dissatisfied); (ii) ease of getting medical help or advice for the problem (scored on a Likert scale of 1–5: 1—very easy; 2—fairly easy; 3—neither easy nor difficult; 4—fairly difficult; 5—very difficult); and (iii) convenience of care (scored on a Likert scale of 1–4: 1—very convenient; 2—fairly convenient; 3—not very convenient; 4—not at all convenient).

dical help or advice for the problem (scored on a Likert scale of 1–5: 1—very easy; 2—fairly easy; 3—neither easy nor difficult; 4—fairly difficult; 5—very difficult); and (iii) convenience of care (scored on a Likert scale of 1–4: 1—very convenient; 2—fairly convenient; 3—not very convenient; 4—not at all convenient). Patient characteristics Data on age and gender were available for all patients within the trial (supplied from practice records), with data on deprivation being available for patients whose postcode (supplied from practice records) could be mapped to an Indices of Multiple Deprivation (IMD) 2010 score.12 Data on ethnicity, presence of a long-standing health condition and ease of taking time away from work were only available if the patient returned a questionnaire with the relevant item completed. Age was categorised into six ranges: 0–4 years; 5–11 years; 16–24 years; 25–59 years (reference category); 60–74 years; 75 years and older (patients aged 12–15 years were excluded from the trial due to reasons of confidentiality, as some patients in this age group may wish their parents/guardians to be unaware of their clinician consultation; should the parents/guardians open a questionnaire addressed to the patient, they would be alerted to this fact).

r (patients aged 12–15 years were excluded from the trial due to reasons of confidentiality, as some patients in this age group may wish their parents/guardians to be unaware of their clinician consultation; should the parents/guardians open a questionnaire addressed to the patient, they would be alerted to this fact). The ESTEEM questionnaire invited patients to report their ethnic group using five categories. Ethnic group was then dichotomised for the purposes of these analyses as ‘white’ and ‘other ethnic group’ (comprising Mixed/multiple ethnic groups, Asian/Asian British, Black/African/Caribbean/Black British, and Other ethnic group) due to the small number of patients from ethnic minority backgrounds. Deprivation status was based on IMD 2010 scores mapped to the patient's residential postcode. Deprivation was divided into five quintiles based on rank (using national quintiles as cut-off points), using the least deprived quintile as the reference category. Patients reported how easy it was for them to attend a GP consultation at their practice during working hours (categorised as easily, with difficulty or unable to attend during working hours), with the option to record if the question was not relevant (eg, if the patient was a child or non-working adult); the ‘not relevant’ category was used as the reference group within analyses. Patients also reported the presence/absence of a long-standing health condition.

culty or unable to attend during working hours), with the option to record if the question was not relevant (eg, if the patient was a child or non-working adult); the ‘not relevant’ category was used as the reference group within analyses. Patients also reported the presence/absence of a long-standing health condition. Statistical methods The satisfaction/experience outcome variables were linearised on a scale of 0–100 (lower values indicating lower levels of negative response) to facilitate ease of interpretation.2 On this scale, a difference of <3 points is considered to be of small magnitude in practical terms.13 All analyses took the form of multilevel linear regression models (using the Stata command ‘xtmixed’) with a random effect on practice (cluster).14 The multilevel structure provides a ‘within practice’ approach, evaluating differences across sociodemographic characteristics within each practice and allowing each practice to have its own baseline score within the model. This approach allows for ‘clustering’ of participants with specific sociodemographic characteristics within a practice.2 However, our aim was not to evaluate the degree of variation in an outcome that was attributable to practice level variation,14 as this was not an issue of interest in our analyses; rather, we aimed to evaluate the effects of sociodemographic covariates after adjusting for practice level variables and accounting for the multilevel nature of the data. All minimisation variables used in the cluster randomisation procedure of the ESTEEM trial (practice list size (small, medium, large), practice deprivation (deprived/non-deprived) and location (Bristol, Devon, Norwich, Warwick)7–9) were included as fixed effect variables in all analyses as was trial arm (using usual care as the reference group).

used in the cluster randomisation procedure of the ESTEEM trial (practice list size (small, medium, large), practice deprivation (deprived/non-deprived) and location (Bristol, Devon, Norwich, Warwick)7–9) were included as fixed effect variables in all analyses as was trial arm (using usual care as the reference group). A series of multivariable models were fitted to investigate potential associations between the sociodemographic/lifestyle variables and each outcome variable individually. Potential interactions of sociodemographic variables with trial arm were also investigated, by inclusion of one potential interaction term within an individual model. Although the mode of management actually received by individual patients varied within each arm as well as across arms,9 we have taken the pragmatic approach of analysing patient experience outcomes by allocated trial arm rather than by the mode of management received. Sociodemographic/lifestyle variables that were found to be not significantly associated with an individual outcome variable (within a multivariable model) were excluded from the final model for that outcome variable. The p value for statistical significance was set at <0.05 for main effects and <0.1 for an interaction term.

demographic/lifestyle variables that were found to be not significantly associated with an individual outcome variable (within a multivariable model) were excluded from the final model for that outcome variable. The p value for statistical significance was set at <0.05 for main effects and <0.1 for an interaction term. Marginal means were reported for statistically significant interaction effects between trial arm and the specified sociodemographic/lifestyle factor. Marginal mean scores indicated the expected score for specified patient characteristic groups (eg, by gender) across the three trial arms individually (eg, male patients receiving usual care, male patients receiving GP triage, and so on to encompass the six possible combinations of three trial arms and two genders). Such scores are based on the assumption that all patients in the model take the specified values for trial arm and gender, while all other variables within the model retain their observed values. All analyses were performed on an intention-to-treat basis (not all patients received the management method allocated to their practice9) using complete case data. All analyses were conducted using Stata v.12.

Marginal means were reported for statistically significant interaction effects between trial arm and the specified sociodemographic/lifestyle factor. Marginal mean scores indicated the expected score for specified patient characteristic groups (eg, by gender) across the three trial arms individually (eg, male patients receiving usual care, male patients receiving GP triage, and so on to encompass the six possible combinations of three trial arms and two genders). Such scores are based on the assumption that all patients in the model take the specified values for trial arm and gender, while all other variables within the model retain their observed values. All analyses were performed on an intention-to-treat basis (not all patients received the management method allocated to their practice9) using complete case data. All analyses were conducted using Stata v.12. Results Study population characteristics Of the 20 990 patients within ESTEEM, 12 132 (58%) returned a questionnaire that included at least one completed question. A smaller proportion of patients who were sent a questionnaire responded to the question on overall satisfaction in nurse triage (3704/7012; 53%) compared with 4093/7283 (56%) in usual care and 4034/6695 (60%) in GP triage. Of 12 132 questionnaire respondents, 11 831 (98%) completed the question regarding overall satisfaction; their demographic characteristics are presented in table 1; demographic data for questionnaire respondents and a detailed analysis of the factors associated with questionnaire response are reported elsewhere.9 The predominantly female sample reflected the overall study sample. However, patients in older age groups (60 and over) were over-represented within the sample of patients who provided satisfaction data compared with the overall study sample, whereas young adults (aged 16–24 years) were under-represented. Patients in the most deprived quintile were under-represented both in the overall study sample and the sample providing experience data.9 Of 12 132 questionnaire respondents, 11 119 (92%) provided data on the ease of getting medical help or advice; 11 783 (97%) provided data on convenience of care. The frequencies for each response category are set out in the ESTEEM report.9

nted both in the overall study sample and the sample providing experience data.9 Of 12 132 questionnaire respondents, 11 119 (92%) provided data on the ease of getting medical help or advice; 11 783 (97%) provided data on convenience of care. The frequencies for each response category are set out in the ESTEEM report.9 Table 1 Baseline patient sociodemographic characteristics for patients who responded to overall satisfaction question

nted both in the overall study sample and the sample providing experience data.9 Of 12 132 questionnaire respondents, 11 119 (92%) provided data on the ease of getting medical help or advice; 11 783 (97%) provided data on convenience of care. The frequencies for each response category are set out in the ESTEEM report.9 Table 1 Baseline patient sociodemographic characteristics for patients who responded to overall satisfaction question Usual care (UC; N=4093) GP triage (GPT; N=4034) Nurse triage (NT; N=3704) Individual patient characteristics derived from practice data*† Gender; n (%) Male 1579 (38.6) 1601 (39.7) 1417 (38.3) Female 2514 (61.4) 2433 (60.3) 2287 (61.7) Age (years); mean (SD) 46.8 (24.0) 49.6 (24.8) 47.3 (25.4) By category; n (%) Under 5 347 (8.5) 325 (8.1) 380 (10.2) 5–11 222 (5.4) 210 (5.2) 214 (5.8) 16–24 297 (7.3) 227 (5.6) 220 (5.9) 25–59 1757 (42.9) 1591 (39.4) 1487 (40.2) 60–74 1047 (25.6) 1052 (26.1) 899 (24.3) 75 and over 423 (10.3) 629 (15.6) 504 (13.6) Deprivation (IMD 2010† score); mean (SD), n 16.6 (9.6), 4069 16.2 (10.6), 4025 16.2 (10.5), 3673 Deprivation (IMD 2010† quintile based on rank); n (%) Quintile 1 (least deprived) 817 (20.1) 779 (19.4) 870 (23.7) Quintile 2 1136 (27.9) 1220 (30.3) 1038 (28.3) Quintile 3 1089 (26.8) 1198 (29.8) 864 (23.5) Quintile 4 824 (20.3) 590 (14.7) 665 (18.1) Quintile 5 (most deprived) 203 (5.0) 238 (5.9) 236 (6.4) Individual patient characteristics derived from questionnaire Ethnicity—by ethnic group; n (%) White 3927 (96.5) 3851 (96.1) 3498 (95.2) Other ethnic group 143 (3.5) 158 (3.9) 175 (4.8) Total N 4070 4009 3673 Able to attend surgery during work hours; n (%) Not relevant 1956 (48.6) 2049 (51.8) 1811 (49.8) Yes, easily 792 (19.7) 787 (19.9) 721 (19.8) Yes, with difficulty 877 (21.8) 829 (20.9) 768 (21.1) No 400 (9.9) 294 (7.4) 335 (9.2) Total N 4025 3959 3635 Long-standing health conditions; n (%) Yes 1923 (48.0) 1973 (50.1) 1669 (46.0) No 2087 (52.0) 1967 (49.9) 1957 (54.0) Total N 4010 3940 3626 *Age and gender derived directly from practice records.

9) 721 (19.8) Yes, with difficulty 877 (21.8) 829 (20.9) 768 (21.1) No 400 (9.9) 294 (7.4) 335 (9.2) Total N 4025 3959 3635 Long-standing health conditions; n (%) Yes 1923 (48.0) 1973 (50.1) 1669 (46.0) No 2087 (52.0) 1967 (49.9) 1957 (54.0) Total N 4010 3940 3626 *Age and gender derived directly from practice records. †IMD 2010 score and rank derived from residential postcode data (provided by the patient's practice) mapped to lower super output area; https://www.gov.uk/government/publications/english-indices-of-deprivation-2010. IMD, Indices of Multiple Deprivation.

9) 721 (19.8) Yes, with difficulty 877 (21.8) 829 (20.9) 768 (21.1) No 400 (9.9) 294 (7.4) 335 (9.2) Total N 4025 3959 3635 Long-standing health conditions; n (%) Yes 1923 (48.0) 1973 (50.1) 1669 (46.0) No 2087 (52.0) 1967 (49.9) 1957 (54.0) Total N 4010 3940 3626 *Age and gender derived directly from practice records. †IMD 2010 score and rank derived from residential postcode data (provided by the patient's practice) mapped to lower super output area; https://www.gov.uk/government/publications/english-indices-of-deprivation-2010. IMD, Indices of Multiple Deprivation. Overall satisfaction with care Overall, patients were satisfied with their care, with approximately 90% of responding patients in each trial arm reporting that they were ‘very’ or ‘fairly’ satisfied with their care, although only 59% of responding patients in the nurse triage arm were ‘very’ satisfied, compared with around 65% in the usual care and GP triage arms.9 Gender and patient deprivation were not significantly associated with overall satisfaction and were therefore excluded from subsequent analyses. Older patients were more satisfied compared with adults aged 25–59 years; for example, patients aged 75 years and over had a mean difference in score of −3.83, with a 95% CI of −5.25 to −2.40 (table 2, Model A), whereas young adults (aged 16–24 years) were less satisfied (mean difference 4.35, 95% CI 2.74 to 5.97; table 2, Model A). Patients from ethnic minorities reported reduced satisfaction compared with white patients (mean difference 5.00, 95% CI 2.96 to 7.04; table 2, Model A). Also, patients who reported that they were unable to attend the practice during working hours, or could only do so with difficulty, were less satisfied than those patients for whom this issue was not relevant (mean difference (95% CI) 5.41 (3.89 to 6.94) and 2.46 (1.29 to 3.62) respectively; table 2, Model A); there was little evidence to indicate that this reduced satisfaction varied by trial arm (see online supplementary appendix table A1). Patients who were easily able to attend the practice during working hours did not report significantly lower satisfaction than those patients for whom attendance during working hours was not relevant. The presence of a long-standing health condition was also significantly associated with reduced satisfaction, but to a small extent. Patients from ethnic minority backgrounds and patients who were unable to take time away from work to attend their surgery during working hours both reported poorer satisfaction of approximately 5 points compared with the relevant reference group.

gnificantly associated with reduced satisfaction, but to a small extent. Patients from ethnic minority backgrounds and patients who were unable to take time away from work to attend their surgery during working hours both reported poorer satisfaction of approximately 5 points compared with the relevant reference group. Table 2 Overall satisfaction with care: sociodemographic associations and interactions with trial arm

gnificantly associated with reduced satisfaction, but to a small extent. Patients from ethnic minority backgrounds and patients who were unable to take time away from work to attend their surgery during working hours both reported poorer satisfaction of approximately 5 points compared with the relevant reference group. Table 2 Overall satisfaction with care: sociodemographic associations and interactions with trial arm Mean difference in overall satisfaction with care* (95% CI) Global p value Model A†: N=11 343 Trial arm Reference; usual care GP triage 1.18 (−0.69 to 3.06) <0.001 Nurse triage 3.78 (1.88 to 5.69) Patient characteristic Age (reference: 25–59 years) Under 5 years 0.90 (−0.62 to 2.42) <0.001 5–11 years 0.60 (−1.21 to 2.41) 16–24 years 4.35 (2.74 to 5.97) 60–74 years −2.70 (−3.81 to −1.60) 75 years and over −3.83 (−5.25 to −2.40) Ethnic group (reference; white) Other ethnic group 5.00 (2.96 to 7.04) <0.001 Ease of taking time away from work to attend surgery (reference: not relevant‡) Can take time away from work easily −0.63 (−1.75 to 0.49) <0.001 Can take time away from work with difficulty 2.46 (1.29 to 3.62) Cannot take time away from work 5.41 (3.89 to 6.94) Presence of long-standing health condition (reference: none) Long-standing health condition present 1.83 (0.98 to 2.67) <0.001 Mean difference in overall satisfaction with care* (95% CI) Global p value Marginal mean value§ (95% CI) Model B¶: N=11 343 GP triage interaction with ethnicity Other ethnic group −5.21 (−10.17 to −0.25) 0.101 Nurse triage interaction with ethnicity Other ethnic group −1.44 (−6.41 to 3.52) Marginal mean values Usual care White 11.0 (9.7 to 12.3) Other ethnic group 18.3 (14.6 to 22.0) GP triage White 12.4 (11.1 to 13.7) Other ethnic group 14.5 (11.0 to 18.1) Nurse triage White 14.8 (13.5 to 16.2) Other ethnic group 20.7 (17.2 to 24.2) *Positive mean difference indicates reduced overall satisfaction with care in comparator group; scale 0–100.

11.0 (9.7 to 12.3) Other ethnic group 18.3 (14.6 to 22.0) GP triage White 12.4 (11.1 to 13.7) Other ethnic group 14.5 (11.0 to 18.1) Nurse triage White 14.8 (13.5 to 16.2) Other ethnic group 20.7 (17.2 to 24.2) *Positive mean difference indicates reduced overall satisfaction with care in comparator group; scale 0–100. †Adjusted for practice site, size and practice-level deprivation. ‡Patient does not work, for example, a child or non-working adult. §Higher value indicates reduced satisfaction; scale 0–100. ¶Adjusted as for Model A, with inclusion of interaction between trial arm and ethnic group. GP, general practitioner.

11.0 (9.7 to 12.3) Other ethnic group 18.3 (14.6 to 22.0) GP triage White 12.4 (11.1 to 13.7) Other ethnic group 14.5 (11.0 to 18.1) Nurse triage White 14.8 (13.5 to 16.2) Other ethnic group 20.7 (17.2 to 24.2) *Positive mean difference indicates reduced overall satisfaction with care in comparator group; scale 0–100. †Adjusted for practice site, size and practice-level deprivation. ‡Patient does not work, for example, a child or non-working adult. §Higher value indicates reduced satisfaction; scale 0–100. ¶Adjusted as for Model A, with inclusion of interaction between trial arm and ethnic group. GP, general practitioner. While there was little evidence for a significant interaction between GP triage and nurse triage compared with usual care and individual sociodemographic characteristics, there was some evidence to indicate an interaction between ethnic group and GP triage only compared with usual care (p value 0.040). Patients from ethnic minorities reported lower satisfaction compared with white patients for all three trial arms, whereas for GP triage, the marginal mean score was closer to that of white patients (table 2, Model B; figure 1A). This interaction between ethnicity and trial arm indicates a different pattern in satisfaction across ethnic groups, whereby white patients reported greatest satisfaction with usual care, followed by GP triage and then nurse triage. In contrast, the pattern seen among patients from ethnic minorities indicated that GP triage was associated with greatest satisfaction, followed by usual care and then nurse triage. The full results of Model A are presented in online supplementary appendix table A1. No other statistically significant interactions were observed between trial arm and sociodemographic or lifestyle factors included in Model A (see online supplementary appendix table A1, Models H–J) or between non-significant sociodemographic characteristics (gender and patient deprivation) when individually added to Model A with the appropriate interaction term (data not presented).

een trial arm and sociodemographic or lifestyle factors included in Model A (see online supplementary appendix table A1, Models H–J) or between non-significant sociodemographic characteristics (gender and patient deprivation) when individually added to Model A with the appropriate interaction term (data not presented). Figure 1 Marginal mean values (with 95% CI) across ethnic groups for (A) overall satisfaction with care and (B) ease of getting medical help or advice. *Lower values indicate higher overall satisfaction with care/greater ease of getting medical help or advice.

een trial arm and sociodemographic or lifestyle factors included in Model A (see online supplementary appendix table A1, Models H–J) or between non-significant sociodemographic characteristics (gender and patient deprivation) when individually added to Model A with the appropriate interaction term (data not presented). Figure 1 Marginal mean values (with 95% CI) across ethnic groups for (A) overall satisfaction with care and (B) ease of getting medical help or advice. *Lower values indicate higher overall satisfaction with care/greater ease of getting medical help or advice. Ease of getting medical help/advice Gender and patient deprivation were not significantly associated with this aspect of patient care, and were therefore excluded from subsequent analyses. Patients in the nurse triage arm reported increased difficulty in obtaining medical help/advice compared with usual care (mean difference 4.81, 95% CI 2.49 to 7.12; table 3, Model C). There was some evidence for an interaction between trial arm and ethnicity (p value 0.079). Patients from ethnic minorities reported relatively greater ease of getting medical help/advice in the GP triage arm (versus usual care) compared with white patients (p value 0.025), although again, patients from ethnic minorities appeared to report poorer absolute scores than white patients across all three trial arms (eg, in the nurse triage arm, the marginal mean score for white patients was 19.9, 95% CI 18.3 to 21.5, compared with 26.0, 95% CI 22.1 to 30.0 for patients from ethnic minorities; table 3, Model D; figure 1B). The full results of Model C are presented in online supplementary appendix table A2. No significant interactions between other patient characteristics included in Model C were observed (see online supplementary appendix table A2, Models K–M); nor were there any significant interactions between trial arm and gender or deprivation when individually added to Model C with the appropriate interaction term (data not presented).

actions between other patient characteristics included in Model C were observed (see online supplementary appendix table A2, Models K–M); nor were there any significant interactions between trial arm and gender or deprivation when individually added to Model C with the appropriate interaction term (data not presented). Table 3 Ease of getting medical help or advice: sociodemographic associations and interactions with trial arm

actions between other patient characteristics included in Model C were observed (see online supplementary appendix table A2, Models K–M); nor were there any significant interactions between trial arm and gender or deprivation when individually added to Model C with the appropriate interaction term (data not presented). Table 3 Ease of getting medical help or advice: sociodemographic associations and interactions with trial arm Mean difference in ease of getting medical help or advice* (95% CI) Global p value Model C†: N=10 683 Trial arm Reference; usual care GP triage −0.30 (−2.59 to 1.98) <0.001 Nurse triage 4.81 (2.49 to 7.12) Patient characteristic Age (reference: 25–59 years) Under 5 years 1.19 (−0.47 to 2.86) <0.001 5–11 years 1.54 (−0.44 to 3.51) 16–24 years 2.59 (0.83 to 4.35) 60–74 years −3.22 (−4.44 to −1.99) 75 years and over −3.36 (−4.96 to −1.76) Ethnic group (reference; white) Other ethnic group 5.86 (3.60 to 8.12) <0.001 Ease of taking time away from work to attend surgery (reference: not relevant‡) Can take time away from work easily −0.87 (−2.11 to 0.38) Can take time away from work with difficulty 3.74 (2.46 to 5.03) <0.001 Cannot take time away from work 7.65 (5.97 to 9.32) Presence of long-standing health condition (reference: no long-standing health condition) Long-standing health condition present 2.01 (1.07 to 2.94) <0.001 Mean difference in overall satisfaction with care* (95% CI) Global p value Marginal mean value§ (95% CI) Model D¶: N=10 683 GP triage interaction with ethnicity Other ethnic group −6.29 (−11.78 to −0.79) Nurse triage interaction with ethnicity 0.079 Other ethnic group −2.78 (−8.28 to 2.73) Marginal mean values Usual care White 14.9 (13.3 to 16.5) Other ethnic group 23.9 (19.7 to 28.0) GP triage White 14.9 (13.3 to 16.5) Other ethnic group 17.6 (13.6 to 21.6) Nurse triage White 19.9 (18.3 to 21.5) Other ethnic group 26.0 (22.1 to 30.0) *Positive mean difference indicates increased difficulty in getting medical help or advice in comparator group; scale 0–100.

.5) Other ethnic group 23.9 (19.7 to 28.0) GP triage White 14.9 (13.3 to 16.5) Other ethnic group 17.6 (13.6 to 21.6) Nurse triage White 19.9 (18.3 to 21.5) Other ethnic group 26.0 (22.1 to 30.0) *Positive mean difference indicates increased difficulty in getting medical help or advice in comparator group; scale 0–100. †Adjusted for practice site, size and practice-level deprivation. ‡Patient does not work, for example, a child or non-working adult. §Higher value indicates increased difficulty in getting medical help or advice; scale 0–100. ¶Adjusted as for Model C, with inclusion of interaction between trial arm and ethnic group. GP, general practitioner.

†Adjusted for practice site, size and practice-level deprivation. ‡Patient does not work, for example, a child or non-working adult. §Higher value indicates increased difficulty in getting medical help or advice; scale 0–100. ¶Adjusted as for Model C, with inclusion of interaction between trial arm and ethnic group. GP, general practitioner. Convenience of care Convenience of care demonstrated slightly different patterns of association with patient characteristics. Although little evidence was found to support an association between gender and convenience of care, strong evidence was found to support an association between patient deprivation and convenience of care (p value 0.001). This association appeared to be driven by increased convenience reported by patients in the most deprived quintile compared with patients in the least deprived quintile (mean difference −4.05, 95% CI −6.35 to −1.75; table 4, Model E). There was some evidence for an interaction effect between deprivation and trial arm (p value 0.041). This interaction appeared to be driven by lower reported convenience for nurse triage by patients in the less deprived groups (based on marginal mean values; eg, in the usual care arm the marginal mean for patients in the least deprived quintile was 12.8 (95% CI 10.4 to 15.1), in the GP triage arm 14.0 (95% CI 11.5 to 16.6), and in the nurse triage arm 21.3 (95% CI 19.0 to 23.7); supplementary online appendix table A3, Model G), although the global p value (0.130) for the interaction between nurse triage compared with usual care, and deprivation group, was not significant. Some evidence for an interaction between age and trial arm was observed (p value 0.072); the interaction between age and nurse triage only versus usual care had a p value of 0.019. This interaction appeared to be driven by relatively lower convenience reported by patients (or more accurately their parents/guardians) in the 5–11 years age bracket compared with patients aged 25–59 years in the nurse triage arm (marginal mean for patients aged 5–11 years 26.7, 95% CI 23.3 to 30.2, for patients aged 25–59 years 18.8, 95% CI 16.9 to 20.8; online supplementary appendix table A3, Model F). The results of Model E are presented in full in online supplementary appendix table A4.

with patients aged 25–59 years in the nurse triage arm (marginal mean for patients aged 5–11 years 26.7, 95% CI 23.3 to 30.2, for patients aged 25–59 years 18.8, 95% CI 16.9 to 20.8; online supplementary appendix table A3, Model F). The results of Model E are presented in full in online supplementary appendix table A4. No further significant interactions were found between trial arm and sociodemographic and lifestyle factors included in Model E (online supplementary appendix table A4, Models O–Q) or between trial arm and gender when added to Model E with the appropriate interaction term (data not presented). Table 4 Convenience of care: sociodemographic associations and interactions with trial arm

No further significant interactions were found between trial arm and sociodemographic and lifestyle factors included in Model E (online supplementary appendix table A4, Models O–Q) or between trial arm and gender when added to Model E with the appropriate interaction term (data not presented). Table 4 Convenience of care: sociodemographic associations and interactions with trial arm Mean difference in convenience of care* (95% CI) Global p value Model E†: N=11 243 Trial arm Reference: usual care GP triage 1.99 (−0.49 to 4.47) <0.001 Nurse triage 5.69 (3.19 to 8.20) Patient characteristic Age (reference: 25–59 years) Under 5 years 3.11 (1.48 to 4.74) 5–11 years 4.28 (2.35 to 6.21) 16–24 years 4.24 (2.51 to 5.97) <0.001 60–74 years −3.35 (−4.54 to −2.17) 75 years and over −2.68 (−4.21 to −1.15) Patient deprivation‡ (reference: Quintile 1; least deprived) Quintile 2 −1.27 (−2.54 to <−0.01) Quintile 3 −1.05 (−2.39 to −0.28) 0.001 Quintile 4 −0.01 (−1.59 to 1.57) Quintile 5 −4.05 (−6.35 to −1.75) Ethnic group (reference: white) Other ethnic group 6.36 (4.16 to 8.56) <0.001 Ease of taking time away from work to attend surgery (reference: not relevant§) Can take time away from work easily −1.56 (−2.76 to −0.36) Can take time away from work with difficulty 4.64 (3.39 to 5.89) <0.001 Cannot take time away from work 9.02 (7.39 to 10.65) Presence of long-standing health condition (reference: no long-standing health condition) Long-standing health condition present 1.90 (1.00 to 2.81) <0.001 *Positive mean difference indicates lower convenience of care in comparator group; scale 0–100.

to 5.89) <0.001 Cannot take time away from work 9.02 (7.39 to 10.65) Presence of long-standing health condition (reference: no long-standing health condition) Long-standing health condition present 1.90 (1.00 to 2.81) <0.001 *Positive mean difference indicates lower convenience of care in comparator group; scale 0–100. †Adjusted for practice site, size and practice-level deprivation. ‡IMD 2010 score and rank derived from residential postcode data mapped to lower super output area; https://www.gov.uk/government/publications/english-indices-of-deprivation-2010. §Patient does not work, for example, a child or non-working adult. GP, general practitioner; IMD, Indices of Multiple Deprivation. Discussion Summary Based on secondary analyses of a cluster RCT (ESTEEM) that investigated the clinical effectiveness of two telephone triage management systems for patients requesting a same-day consultation with a GP,8 9 we sought to explore whether patients’ reported experiences of care were associated with their sociodemographic, health, and lifestyle characteristics.

uster RCT (ESTEEM) that investigated the clinical effectiveness of two telephone triage management systems for patients requesting a same-day consultation with a GP,8 9 we sought to explore whether patients’ reported experiences of care were associated with their sociodemographic, health, and lifestyle characteristics. We found little evidence for a significant interaction with regard to overall satisfaction between all three trial arms and sociodemographic groups. However, there was a significant interaction between GP triage only versus usual care and ethnic group with regard to overall patient satisfaction, and between trial arms and ethnic group regarding ease/difficulty of getting medical help or advice. Patients from ethnic minorities, although reporting less positively than white patients on both of these outcomes, reported scores closer to those of white patients when receiving GP triage than when receiving either usual care or nurse triage. A possible explanation for this could be that patients from ethnic minority backgrounds value rapid access to a GP more highly as a driver of satisfaction compared with white patients, and that telephone access is acceptable. However, a similar effect was not found with nurse triage, possibly indicating that rapid telephone access to a nurse is less acceptable than equivalent access to a GP.

y backgrounds value rapid access to a GP more highly as a driver of satisfaction compared with white patients, and that telephone access is acceptable. However, a similar effect was not found with nurse triage, possibly indicating that rapid telephone access to a nurse is less acceptable than equivalent access to a GP. We observed reduced convenience reported by parents/guardians of children aged 5–11 years in the nurse triage arm, which may be due to the inconvenience of waiting for a call back from the nurse at a time when the child is attending school. Patients in the nurse triage arm were also more likely (compared with patients receiving GP triage) to receive triage and then to be called in to the practice for a face-to-face consultation with a GP or nurse.9 This may be a further source of inconvenience when the child is attending school and the parents/guardians may be at work. There is some evidence to indicate that delays in receiving care and/or uncertainty of what to do while waiting for definitive care may also be a driver of dissatisfaction,15 and these issues may be heightened for parents of children in this age range, and are less well addressed by nurse triage than other management approaches.

ome evidence to indicate that delays in receiving care and/or uncertainty of what to do while waiting for definitive care may also be a driver of dissatisfaction,15 and these issues may be heightened for parents of children in this age range, and are less well addressed by nurse triage than other management approaches. These analyses indicate that triage systems do not alleviate the difficulties working patients may have in obtaining care in a flexible manner. It is possible that the telephone call back approach employed in both triage arms may have been inconvenient for working patients, due to difficulties in receiving calls while travelling to or being at work, for example due to being unable to stop their work to take the call or due to privacy issues in the workplace. The telephone triage took place during regular surgery hours, so this may explain why these outcomes (satisfaction, convenience, and ease of obtaining medical help or advice) did not appear to be more positively reported for patients with difficulty attending their surgery during regular hours in the triage arms.

lace. The telephone triage took place during regular surgery hours, so this may explain why these outcomes (satisfaction, convenience, and ease of obtaining medical help or advice) did not appear to be more positively reported for patients with difficulty attending their surgery during regular hours in the triage arms. Strengths and limitations ESTEEM was a large multicentre trial, which examined a range of aspects of patient experience. The large size of the trial and availability of data for many patients on a range of health and lifestyle factors as well as sociodemographic factors enabled us to undertake these analyses. Our response rate of 58% is relatively high for this type of patient survey within a general practice setting, for example, by comparison with the annual iterations of GPPS.1 A survey in the Netherlands suggested that non-response bias was small with regard to overall satisfaction with out-of-hours primary care.16 Therefore, we felt that our sample was sufficiently large to militate against the use of data imputation methods. However, the ESTEEM trial was not powered to detect interactions with regard to patient satisfaction, and therefore the results of these exploratory analyses should be viewed with caution.

s primary care.16 Therefore, we felt that our sample was sufficiently large to militate against the use of data imputation methods. However, the ESTEEM trial was not powered to detect interactions with regard to patient satisfaction, and therefore the results of these exploratory analyses should be viewed with caution. Practices were given a 3-week ‘run-in’ period to establish the new routine of the triage system,9 but we acknowledge that this is a short time for staff and patients to become accustomed to the new approach. It is possible that reduced satisfaction in the triage arms may be due to this period of transition and that patient experience scores may improve as the triage system becomes more familiar to both staff and patients.

acknowledge that this is a short time for staff and patients to become accustomed to the new approach. It is possible that reduced satisfaction in the triage arms may be due to this period of transition and that patient experience scores may improve as the triage system becomes more familiar to both staff and patients. For safety reasons, patients were excluded from the trial if they were unable to communicate effectively by telephone, for example due to hearing/speech difficulties or difficulties with spoken English. In view of this, the reduced satisfaction reported by patients from ethnic minority backgrounds may be less likely (compared with other reports of reduced satisfaction among patients from ethnic minorities2 17 18) to have arisen due to difficulties with communication in spoken English (although we acknowledge that verbal communication in a second language, even if of a high standard, may be more difficult than communicating in the native language, especially when using the telephone, as visual cues are not available). However, a limitation of these analyses was that due to the small number of patients from minority ethnic groups, it was necessary to dichotomise ethnic group (into ‘white’ and ‘other ethnic group’ categories); hence, we were unable to investigate differences in experience among patients from different ethnic minority groups. The exclusion of patients who were unable to communicate effectively in English using the telephone may have contributed to the small number of patients from ethnic minority backgrounds. Furthermore, we did not elicit information on English language ability, which may have been associated with patient experience.

oups. The exclusion of patients who were unable to communicate effectively in English using the telephone may have contributed to the small number of patients from ethnic minority backgrounds. Furthermore, we did not elicit information on English language ability, which may have been associated with patient experience. A further limitation was that we did not collect data regarding the patient's presenting complaint—the reason for the patient's initial request for a same-day GP consultation. Hence, we were unable to link satisfaction to the nature and/or severity of the medical problem resulting in the initial same-day consultation request. Such data may have been of particular relevance with regard to patients with a long-standing health condition (although we did not collect any information regarding the nature and/or severity of long-standing health conditions), older patients or children, in terms of trying to understand reasons for reporting their experience of care as they did. Nor did we request patients who were unable to take time away from work (or were able to do so only with difficulty) to report their occupation or typical working hours; such information may have provided insights into the reduced satisfaction among these patients. We did not aim to link patient experience with the timeliness of care or the quality of clinical care received; therefore, we are unable to comment on the potential association between patients’ reported experience of care and objective measures of the quality of care.

nto the reduced satisfaction among these patients. We did not aim to link patient experience with the timeliness of care or the quality of clinical care received; therefore, we are unable to comment on the potential association between patients’ reported experience of care and objective measures of the quality of care. Participants were requested to report their experience of care received approximately 4 weeks previously. Although this is a relatively short time period, there is a possibility for recall bias, with regard to identifying the specific date in question, and with regard to recalling experience of care, especially if the patient is a frequent attender at his/her practice. For example, older patients may not accurately report health service resource use over a relatively short time frame (less than 12 weeks).19 Such difficulties with recall may also extend to patients’ ratings of experience of care. We acknowledge that these analyses are probably underpowered with regard to detecting interaction effects, despite the use of a more liberal p value of 0.1 (as opposed to 0.05 for the main effects) in assessing statistical significance. By presenting multiple analyses across the three outcomes, there is also a risk of detecting a statistically significant effect by chance. However, given that the p values for statistically significant main effects of patient characteristics were low (<0.001), we believe our interpretation of effects is largely unaffected.

resenting multiple analyses across the three outcomes, there is also a risk of detecting a statistically significant effect by chance. However, given that the p values for statistically significant main effects of patient characteristics were low (<0.001), we believe our interpretation of effects is largely unaffected. Comparison with existing literature Previous research has identified that certain sociodemographic groups consistently report lower levels of patient satisfaction with primary care services, for example, patients who are young adults or are from ethnic minorities.2 17 As reported elsewhere, patients in this trial reported significantly reduced overall satisfaction within the nurse triage arm compared with usual care.8 9 We found significantly increased satisfaction among older patients (aged 60–74 years, and 75 years and over), with reduced satisfaction among young adults (16–24 years) and among ethnic minority patients, which is consistent with other research into patient satisfaction, using patient–doctor communication as a proxy for overall satisfaction.2 The interactions between trial arm and sociodemographic characteristics such as ethnicity, age and deprivation may be due to differences in factors associated with patient evaluation of care across sociodemographic groups.3

o patient satisfaction, using patient–doctor communication as a proxy for overall satisfaction.2 The interactions between trial arm and sociodemographic characteristics such as ethnicity, age and deprivation may be due to differences in factors associated with patient evaluation of care across sociodemographic groups.3 Our results are consistent with those of an earlier patient survey within general practice, the GP Access Survey, conducted in 2007–2008.20 That survey found that patient employment status was associated with poorer experience of aspects of access to GP care. Among patients in employment, those patients able to take time away from work to visit their GP reported more positive experience, compared with patients unable to do so. More recently, analyses of data from the English GPPS regarding service users' experience of out of hours GP care have indicated poorer experience among service users who were unable to take time away from work to see a GP during their regular working hours, compared with non-working service users.21

s unable to do so. More recently, analyses of data from the English GPPS regarding service users' experience of out of hours GP care have indicated poorer experience among service users who were unable to take time away from work to see a GP during their regular working hours, compared with non-working service users.21 Implications for practice and research Provision of timely care within a primary care setting has many challenges, for example due to patients having difficulty in attending consultations during regular general practice hours. Further research is warranted into how best to serve patients whose employment commitments make attendance at their practice during regular general practice hours difficult or impossible; these analyses provided no indication that a telephone triage system improves the experience of such patients. Lower general satisfaction of ethnic minority patients may be best addressed using qualitative methods to better understand the drivers of satisfaction; our results indicate that rapid access to a GP is of importance and this could be pursued further. Any further research in this field would be enhanced by the inclusion of data regarding the specific nature of the request for urgent care, which would provide the opportunity to explore case type/severity with regard to patient satisfaction.

that rapid access to a GP is of importance and this could be pursued further. Any further research in this field would be enhanced by the inclusion of data regarding the specific nature of the request for urgent care, which would provide the opportunity to explore case type/severity with regard to patient satisfaction. These additional analyses of patient experience data derived from the ESTEEM trial have enhanced our understanding of patients’ experiences of provision of prompt primary care access, including telephone triage approaches. We have identified patient groups whose reported experience of care following a same-day GP consultation request was less positive than comparator groups, specifically patients from ethnic minorities, school-age children and their parents/guardians, and patients who find it difficult to attend a practice consultation during routine hours. These findings may indicate areas for further research into how best to tailor prompt management in primary care to meet the needs of these patients. Supplementary Material Web appendix We would like to thank all patients and practices who participated in the ESTEEM trial, and all of our colleagues at the four ESTEEM locations for their support in completing ESTEEM.

These additional analyses of patient experience data derived from the ESTEEM trial have enhanced our understanding of patients’ experiences of provision of prompt primary care access, including telephone triage approaches. We have identified patient groups whose reported experience of care following a same-day GP consultation request was less positive than comparator groups, specifically patients from ethnic minorities, school-age children and their parents/guardians, and patients who find it difficult to attend a practice consultation during routine hours. These findings may indicate areas for further research into how best to tailor prompt management in primary care to meet the needs of these patients. Supplementary Material Web appendix We would like to thank all patients and practices who participated in the ESTEEM trial, and all of our colleagues at the four ESTEEM locations for their support in completing ESTEEM. Contributors: FCW performed the analyses and drafted the first version of the manuscript. RC and EF co-ordinated the trial, and with AV, collected and managed the data. All authors contributed to the development and operationalisation of the ESTEEM trial protocol and to the preparation and drafting of this manuscript. All authors approved the final draft. FCW and JLC act as guarantors of the paper. Funding: This article presents independent research funded by the National Institute for Health Research (NIHR) Health Technology Assessment Programme (project 08/53/15).

Contributors: FCW performed the analyses and drafted the first version of the manuscript. RC and EF co-ordinated the trial, and with AV, collected and managed the data. All authors contributed to the development and operationalisation of the ESTEEM trial protocol and to the preparation and drafting of this manuscript. All authors approved the final draft. FCW and JLC act as guarantors of the paper. Funding: This article presents independent research funded by the National Institute for Health Research (NIHR) Health Technology Assessment Programme (project 08/53/15). Disclaimer: The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The UK NIHR had no direct involvement in the design and conduct of the study, the collection, management, analysis and interpretation of the data, or in the preparation, review or approval of the manuscript. Competing interests: None declared. Ethics approval: South West No. 2 Research Ethics Committee. Provenance and peer review: Not commissioned; externally peer reviewed.

Reducing the number of avoidable deaths in hospital is the focus of many quality improvement initiatives worldwide.1 Comparing indicators of avoidable mortality between different hospitals could help to target improvement efforts, but optimally defining and measuring hospital deaths that could be deemed preventable remains a challenge.2 Unlike performance comparisons based on hospital standardised mortality ratio (HSMR), a new policy initiative announced by the UK Government will rank hospitals for avoidable mortality based on case reviews of 2000 deaths in English hospitals each year. Although this initiative aims to overcome limitations of current policies, two statistical properties of the proposed approach mean that it is unsuitable for classifying hospital performance.

e UK Government will rank hospitals for avoidable mortality based on case reviews of 2000 deaths in English hospitals each year. Although this initiative aims to overcome limitations of current policies, two statistical properties of the proposed approach mean that it is unsuitable for classifying hospital performance. The first issue relates to the ability to identify whether any one death really was avoidable on a case-by-case basis. It would appear3 that the planned process is based on work by Hogan et al4 using retrospective case record review (RCRR). In line with previous studies using RCRR, these investigators asked experienced clinicians to rate whether a death was preventable on a 6-point Likert scale.4 5 Their study recognised that the use of a semicontinuous scale better reflects ‘the probabilistic nature of reviewers’ decision making more closely than requiring a simple “yes” or “no” response’.4 5 However, in operationally defining an avoidable death, the probabilistic component of the instrument is lost because a fixed cut-off is used such that deaths where it is judged that there is more than a 50% chance that the death was preventable are classified as avoidable, and those below 50% are not. (It should be noted that the somewhat arbitrary choice of a 50% cut-off value is not the real issue here, but rather the dichotomisation itself is. However, hereafter, we assume a 50% cut-off value is used as proposed).4

50% chance that the death was preventable are classified as avoidable, and those below 50% are not. (It should be noted that the somewhat arbitrary choice of a 50% cut-off value is not the real issue here, but rather the dichotomisation itself is. However, hereafter, we assume a 50% cut-off value is used as proposed).4 By dichotomising cases into being avoidable or not, the information about the distribution is lost. One might naively argue that the probabilities above 50% will average out with those below 50% to give the right answer. In fact, this is only true when the mean chance of a death being avoided (where the chance is greater than zero) is 50%. This is a strong assumption that will nearly always be untrue. To illustrate this further, we can consider two scenarios. First, a scenario where there were 100 deaths, each with a 60% chance of preventability, implying that 60 deaths would have been avoided if there were no problems in care (assuming independence between cases); and another scenario where there were 100 deaths, each with a 20% chance of preventability, implying that 20 deaths would have been avoided. By only focusing on deaths where the judged preventability is greater than 50% (ie, the proposed operational definition of an avoidable death), we would have estimated 100 ‘avoidable deaths’ in the first scenario and zero ‘avoidable deaths’ in the latter—both conclusions being evidently untrue. These errors arise as a result of ignoring that there will be a range of risks that deaths are preventable. In reality, the distribution of risk that deaths are preventable is likely to be highly skewed, with many deaths where there is a small chance that it was preventable and a few deaths where the chance that the death was preventable is large. By ignoring this continuum of risk, only those deaths with a high risk will be counted, and the collective impact of those deaths with a small risk of being preventable will be missed.

deaths where there is a small chance that it was preventable and a few deaths where the chance that the death was preventable is large. By ignoring this continuum of risk, only those deaths with a high risk will be counted, and the collective impact of those deaths with a small risk of being preventable will be missed. The second issue with the plan to rank hospitals on the basis of case review is one of small numbers. While challenges in quality measurement associated with small-number phenomena are not new,6 7 it is worth considering how small numbers pose challenges in this specific proposed initiative: the emerging policy is to review 2000 hospital deaths every year. Given that previous work with this method found that around 5% of deaths were preventable,4 we might expect that about 100 deaths will be identified as preventable every year. One would hope that an equal number of cases were reviewed per hospital to make comparisons fair, and because there are around 160 acute hospitals in England, this equates to less than one death per hospital on average. Statistical theory is not required to realise that this sample will be grossly inadequate in terms of precision and reliability.

number of cases were reviewed per hospital to make comparisons fair, and because there are around 160 acute hospitals in England, this equates to less than one death per hospital on average. Statistical theory is not required to realise that this sample will be grossly inadequate in terms of precision and reliability. An alternative approach might be to use an algorithm designed to identify a set of deaths with a much higher chance of being avoidable, based on routine data (eg, information included in patients’ electronic health records). Even if a highly efficient algorithm could be developed, which selected deaths of which 50% were found to be preventable, assuming 2000 hospital deaths sampled from 160 hospitals as proposed, such an algorithm would still result in hospital ranks based on approximately six potentially preventable deaths per hospital on average. While selecting deaths with a 50% chance of being preventable based on routine data alone is highly unrealistic, even in this scenario, it is very unlikely that the number of observations will be adequate. To better understand the implications, we can consider these data in the context provided by the binomial distribution. If there were 13 cases reviewed per hospital (2080 in total), and on average, 50% of those deaths were flagged as preventable, we would expect a range of observed numbers of preventable deaths across all hospitals just by chance alone. A useful analogy here is one of flipping a coin. One person flipping a coin 13 times may get 10 heads and three tails whereas another person may get only six heads. If each person continued flipping the coin many times, we would expect both to get a head close to half of the time, but when considering only a small number of flips, large variation can be seen.

a coin. One person flipping a coin 13 times may get 10 heads and three tails whereas another person may get only six heads. If each person continued flipping the coin many times, we would expect both to get a head close to half of the time, but when considering only a small number of flips, large variation can be seen. Figure 1 shows the expected distribution for any 1 year in the scenario described above (160 hospitals with 13 deaths reviewed, of which 50% are deemed preventable). This distribution has very large variability. The, apparently, worst hospitals have 5.5 times as many preventable deaths than the, apparently, best hospitals. This, however, is due to chance alone. Additionally, we expect eight hospitals on average to have 10 or more of these 13 cases flagged as preventable even if there is no true variation between hospitals. This illustration is for a typical year, and once every 4 years, we would expect one hospital to have all but one of the cases reviewed flagged as preventable, purely due to chance variability. It should be remembered that this is almost certainly an unrealistic scenario, and performance of any algorithm would almost always be worse than that outlined here. Furthermore, the chance variation illustrated will be further exaggerated when extrapolating back to the entire hospital cohort using an appropriate weighting scheme. Previous simulation studies have also provided concordant evidence.8

d performance of any algorithm would almost always be worse than that outlined here. Furthermore, the chance variation illustrated will be further exaggerated when extrapolating back to the entire hospital cohort using an appropriate weighting scheme. Previous simulation studies have also provided concordant evidence.8 Figure 1 Distribution of the expected number of avoidable deaths for 160 hospitals, each having 13 deaths reviewed, where deaths selected for review have been chosen such that 50% are expected to be avoidable. Even with this unrealistically high probability of preventable deaths, chance variation alone produces substantial variation, with some hospitals apparently having five times the number of preventable deaths as others.

viewed, where deaths selected for review have been chosen such that 50% are expected to be avoidable. Even with this unrealistically high probability of preventable deaths, chance variation alone produces substantial variation, with some hospitals apparently having five times the number of preventable deaths as others. The two issues outlined here are not without potential cure. The use of a dichotomising cut-off when defining preventable deaths can be overcome by recognising the full range of probabilities and factoring them into calculations. For example, this can be done by preserving the case reviewers’ estimate of preventability. By doing so, the number of deaths that could be avoided can be estimated by multiplying the mean chance of a death being preventable by the total number of hospital deaths. In turn, the issue of small numbers may be addressed simply by reviewing more cases. In order to determine how many cases is enough, one should consider the statistical reliability (also known as rankability) of the resulting metrics.9–12 Reliability, in this context, is defined as the proportion of observed variance in hospital scores/metrics that is explained by the true (underlying) variance in hospital performance or, put another way, the proportion of the overall observed variation not due to noise. When sample sizes for individual hospitals are small, the uncertainty on those individual hospital scores will artificially inflate the apparent variance between hospitals. When the difference between true and observed variance is small, metrics may be considered reliable; however, there are times when the true variability accounts for only a minority of the observed variability,12 and metrics are highly unreliable. It should be noted that reliability depends on sample size and the degree of true variability between hospitals as it is easier to distinguish hospitals when there is larger variation between them. Of course, it may not be feasible or practical to review enough cases to produce reliable ranks.

cs are highly unreliable. It should be noted that reliability depends on sample size and the degree of true variability between hospitals as it is easier to distinguish hospitals when there is larger variation between them. Of course, it may not be feasible or practical to review enough cases to produce reliable ranks. A related concept worth consideration is the identification of outliers. There are many ways in which this is done, but often hospitals for which there is statistical evidence that their performance differs from the national average are flagged as outliers. Some hospitals are likely to be flagged as outliers even when reliability is low. In such situations (and to a lesser degree when reliability is high), we can observe the apparently paradoxical situation when hospitals flagged as better or as worse than average are not statistically significantly different from some hospitals flagged as average.13 This apparent paradox is easily explained when one considers that the hypothesis considered when testing if a hospital is different from the national average is not the same as when testing if a hospital is different to another individual hospital.14 It is easier to distinguish a hospital from a national average than it is to distinguish from another hospital. When ranking hospitals, implicit comparisons between individual hospitals are being made, and thus, it is the latter distinction that matters rather than the former. It should also be borne in mind that when reliability is very high, nearly all hospitals will be statistically significantly different from the national average. For such reasons, it is often preferable to account for the real variation between hospitals, and the resulting overdispersion, when identifying outliers.15

. It should also be borne in mind that when reliability is very high, nearly all hospitals will be statistically significantly different from the national average. For such reasons, it is often preferable to account for the real variation between hospitals, and the resulting overdispersion, when identifying outliers.15 In conclusion, classifying the performance of English hospitals for avoidable hospital mortality based on a review of 2000 hospital deaths per year will result in both invalid and unreliable rankings. In spite of the limitations of HSMR and similar metrics, they do not suffer from the issues described here. For example, when estimating the HSMR value for a given hospital, no attempt is made to classify individual deaths as preventable or not; rather, the total number of deaths is simply compared with the expected number given the case-mix of patients. Further, given all deaths are counted, small number issues are minimised. Given known methodological limitations of HSMR use, this leaves the improvement community and policy-makers in an uncertain, in fact uncomfortable, place. It should be acknowledged that the very notion of case reviews may be beneficial in itself. For example, it can help to engage hospital leaders on reflecting about their own performance, and incentivise and motivate local improvements. It may also be useful for determining a useful national benchmark.4 Therefore, this new initiative may result in quality improvements; it will, however, remain a grossly inadequate measure for judging comparative hospital performance in respect of avoidable mortality.

e, and incentivise and motivate local improvements. It may also be useful for determining a useful national benchmark.4 Therefore, this new initiative may result in quality improvements; it will, however, remain a grossly inadequate measure for judging comparative hospital performance in respect of avoidable mortality. Contributors: GA conceived the original argument and presented and drafted the manuscript. GL helped develop the argument and contributed to the writing and editing of the manuscript. Funding: GA received no specific funding for this piece of work. GL is supported by a Cancer Research UK Clinician Scientist Fellowship award (A18180). Competing interests: None declared. Provenance and peer review: Not commissioned; internally peer reviewed.

luded a before-and-after study of patient and staff experience, a quasi-experimental before-and-after study of safety outcomes using two control hospitals, and a cost analysis. Data were collected in the new hospital between September 2012 and June 2013. A fuller report of the study and methods can be found elsewhere.6 Patient and staff experience Trust-wide data comprised 20 pre-interviews and 21 post-interviews with senior managers (eg, director of nursing, of infection control and of therapies), clinicians (doctors, allied health professionals, ward managers), ward clerk and staff managing the move or architects/builders (see online supplementary appendix). Other patient and staff data were collected from four case study adult inpatient wards, purposively selected to encompass a range of different clinical areas and patient groups: medical assessment unit (MAU), surgical, medical (older people) and maternity. These data comprised 119 hours of structured ward observations pre and 250 hours post; 24 pre and post semi-structured ward staff interviews. The observation of practice was to understand how and where staff spent their time and determine whether the proportion of time they spent on each activity changed following the move to single-room accommodation. The researcher shadowed staff and recorded their activities on a personal digital assistant using HanDBase software in a structured time and motion data collection tool developed by the research team. This drew on a similar tool used in healthcare research designed and developed by Westbrook and Ampt32 (work observation method by activity timing) (see online supplementary appendix for observation categories). The following data were also collected in each of the four case studies: 32 pre and post semi-structured patient interviews; 55 pre and post staff surveys and pedometer data from 53 staff pre move and 56 post move. Further details can be found in the online supplementary appendix.

Introduction The reasons for ordering blood tests in primary care are varied, and yet, the rapid and accurate communication of results remains central to ensuring patients receive timely and appropriate care.1 As the numbers of tests ordered in primary care continue to increase, there is a need for increasingly flexible, yet robust, systems for managing testing and result communication. The total testing process (TTP) is complex, encompassing test ordering, phlebotomy, dispatch of sample, dissemination of results and the initiation of appropriate follow-up.2 The success of the TTP is potentially hindered by the absence of satisfactory guidelines, and relies on a range of practice staff (including those without clinical expertise), external groups in laboratory and hospital settings, and patients. Errors in the process can lead to serious harm for patients,1 3–6 and medicolegal concerns for healthcare providers.7–9 General practices in the UK recently identified the handling of test results as one of the top 10 risks for patient safety,10 with some reporting that up to one-third of patients are not notified of abnormal results.4

cess can lead to serious harm for patients,1 3–6 and medicolegal concerns for healthcare providers.7–9 General practices in the UK recently identified the handling of test results as one of the top 10 risks for patient safety,10 with some reporting that up to one-third of patients are not notified of abnormal results.4 In analysing the TTP in the USA, errors have been attributed to a number of social and organisational factors at practice level, these include documentation errors, delays in responding to clinical information, difficulties in contacting patients, time constraints of practice staff and forgetfulness.4 There is evidence that these errors are compounded by the limited training of practice staff and a lack of awareness of the scale of the problem.11 Recent studies report that only half of family practices have written protocols for result management.5 In particular, there was no agreed set of operating procedures describing what was to be done if the result was positive, or for allowing the patient some discretion over the method by which results would be communicated.6 In response, there are increasing calls for a more unambiguous process, linking the ordering of a test, the mechanism by which the results may be communicated and, if appropriate, further action being initiated.3 6 12–14 It is apparent that any system redesign must reference the perspectives and preferences of all involved in the process, and that using methodology such as ‘experience-based co-design’ means that patients’ preferences can be reconciled with available resources.13 14

appropriate, further action being initiated.3 6 12–14 It is apparent that any system redesign must reference the perspectives and preferences of all involved in the process, and that using methodology such as ‘experience-based co-design’ means that patients’ preferences can be reconciled with available resources.13 14 To date, only one other study based in the UK has explored the TTP from either staff or patient perspectives.15 Due to this, little is known of what works well in NHS general practices or which aspects of the process can be usefully improved. Here, we collate data from a series of focus group discussions with patients and staff in order to assess strengths and weaknesses of current systems, and identify areas and develop strategies for improvement that account for patient preference, staff capabilities and logistical feasibility. Methods Four general practices were selected from 10 previously collaborating in Birmingham and Lambeth Liver Evaluation Testing Strategies (BALLETS), a prospective study of abnormal liver function tests in England.16 During BALLETS, we discovered that methods of test result communication varied between practices. Judgement sampling, based on our knowledge of the practices, was used to purposively select four practices to take part in focus groups. These practices reflected a range of size, socioeconomic environment and communication pathways encompassing a range of overlapping methods and systems (see table 1).17 Table 1 Breakdown of general practices where participating patients are registered

Methods Four general practices were selected from 10 previously collaborating in Birmingham and Lambeth Liver Evaluation Testing Strategies (BALLETS), a prospective study of abnormal liver function tests in England.16 During BALLETS, we discovered that methods of test result communication varied between practices. Judgement sampling, based on our knowledge of the practices, was used to purposively select four practices to take part in focus groups. These practices reflected a range of size, socioeconomic environment and communication pathways encompassing a range of overlapping methods and systems (see table 1).17 Table 1 Breakdown of general practices where participating patients are registered General practice study ID Number of patients registered Number of full-time equivalent GPs IMD code* Practice 1 23 727 7.3 15 066 Practice 2 5914 3.0 13 866 Practice 3 7059 6.3 871 Practice 4 27 430 12.3 8447 *Index of multiple deprivation (IMD) ranking out of 32 482 lower super output levels in England. The IMD codes, produced by the UK Government and first released in 2004 and updated in 2010, provide indicators of deprivation in local authority areas to inform health and social policy.18

3 871 Practice 4 27 430 12.3 8447 *Index of multiple deprivation (IMD) ranking out of 32 482 lower super output levels in England. The IMD codes, produced by the UK Government and first released in 2004 and updated in 2010, provide indicators of deprivation in local authority areas to inform health and social policy.18 Focus groups were conducted in two phases. The first phase of groups was practice specific, and we met with staff and patients separately (see online supplementary appendix table S1).19 Participants in staff focus groups were selected from all staff currently involved in the communication of test results; general practitioners, practice nurses, healthcare assistants, receptionists and practice managers. Participants in patient focus groups were drawn from those with experience of receiving test results, while the groups maintained the maximum variability of patient characteristics, such as age, gender and ethnicity to include a range of opinions and experiences. The topic guides explored what staff and patients perceived to be the strengths and weaknesses of their existing system, the role of patients and staff, how both groups felt the service could be improved, alternative methods for communicating test results in primary care, including patient preferences and how they might best be accommodated (see figure 1 for themes explored). Figure 1 Themes for discussion for Phase I and Phase II focus groups.

Focus groups were conducted in two phases. The first phase of groups was practice specific, and we met with staff and patients separately (see online supplementary appendix table S1).19 Participants in staff focus groups were selected from all staff currently involved in the communication of test results; general practitioners, practice nurses, healthcare assistants, receptionists and practice managers. Participants in patient focus groups were drawn from those with experience of receiving test results, while the groups maintained the maximum variability of patient characteristics, such as age, gender and ethnicity to include a range of opinions and experiences. The topic guides explored what staff and patients perceived to be the strengths and weaknesses of their existing system, the role of patients and staff, how both groups felt the service could be improved, alternative methods for communicating test results in primary care, including patient preferences and how they might best be accommodated (see figure 1 for themes explored). Figure 1 Themes for discussion for Phase I and Phase II focus groups. Each focus group was attended by a moderator and a researcher who acted as an observer and took down field notes to be used in conjunction with the digitally recorded discussions, which were then transcribed verbatim. Each transcript was examined closely, and the findings analysed thematically by IL, LB and SG who met and agreed on emerging themes to decide on a coding framework. Transcripts were analysed alongside the field notes, using constant comparative analysis.20

lly recorded discussions, which were then transcribed verbatim. Each transcript was examined closely, and the findings analysed thematically by IL, LB and SG who met and agreed on emerging themes to decide on a coding framework. Transcripts were analysed alongside the field notes, using constant comparative analysis.20 The group discussions from Phase I were used to create a process map first. These maps display a set of activities (and their respective values) as a series of steps that are involved in creating a product or providing a service, and are fundamental to process-improvement methodologies such as ‘lean’. Each map provides the opportunity to understand interaction between steps, the roles of various individuals and the flow of materials and information required to provide a service.21–23 The detailed process map allowed us to create a service blueprint, a tool originally used in the service industry to diagnose problems with operational inefficiency and now increasingly used in the healthcare environment to drive service innovation.24 25 Our blueprint was to focus on the process from the different perspectives of general practitioner (GP) and patient, identifying areas of delay and failure in the process.

industry to diagnose problems with operational inefficiency and now increasingly used in the healthcare environment to drive service innovation.24 25 Our blueprint was to focus on the process from the different perspectives of general practitioner (GP) and patient, identifying areas of delay and failure in the process. The second phase consisted of focus groups, comprised of both staff and patients combined from all four participating practices, some of whom had participated in Phase I group discussions (see online supplementary appendix table S2). The topic guide for these discussions was informed by the findings of Phase I, including the service blueprint, and included suggestions for improving the current system (see figure 1). Groups were again attended by a moderator and a researcher who acted as observers, and the same individuals and method of analysis as used in Phase I were employed.

ussions was informed by the findings of Phase I, including the service blueprint, and included suggestions for improving the current system (see figure 1). Groups were again attended by a moderator and a researcher who acted as observers, and the same individuals and method of analysis as used in Phase I were employed. Results From our discussions with staff and patients, it became apparent that those urgent tests with potentially serious implications were followed closely by the GP and patient. However, for the vast majority of diagnostic blood tests, the following default pathway for communicating results emerged. Following the decision to order a test, the patient meets with a phlebotomist to provide a blood sample, which is then collected for testing by a laboratory located within a large local hospital. Following testing, the result is issued to practices electronically using the Pathology Messaging Implementation Programme (PMIP) standard.26 Typically, patients are asked to telephone practices 7 days after providing a sample to learn their result and to arrange any appropriate follow-up. Service blueprint From patient and staff accounts, we were able to create a service blueprint of the components of the TTP using the example of a diagnostic blood test (figure 2). The blueprint shows the twin perspectives of GP and patient, and locates both areas of delay (waiting points) and where the process can fail (failure points). Figure 2 Service blueprint for (diagnostic) blood test communication in primary care.

Service blueprint From patient and staff accounts, we were able to create a service blueprint of the components of the TTP using the example of a diagnostic blood test (figure 2). The blueprint shows the twin perspectives of GP and patient, and locates both areas of delay (waiting points) and where the process can fail (failure points). Figure 2 Service blueprint for (diagnostic) blood test communication in primary care. The GP has fewer steps to take in the process than patients who may be required to make repeat visits to the practice, and deal with a variety of staff when they do so. Four key sources of delay were identified, and are primarily visible to patients. The first the wait for phlebotomy (W1) and second the time taken to analyse the sample (W2). The third and fourth waiting points concern retrieving results, either via busy phone lines (W3) or in person via the GP (W4).

f when they do so. Four key sources of delay were identified, and are primarily visible to patients. The first the wait for phlebotomy (W1) and second the time taken to analyse the sample (W2). The third and fourth waiting points concern retrieving results, either via busy phone lines (W3) or in person via the GP (W4). There were six areas identified where the process may fail, only one of which was visible to GPs—the failure to receive the result from the laboratory. The other five locations of potential failure were associated with patients failing to take appropriate action at the correct time. This ranged from booking an appointment with the phlebotomist (F1) to failing to attend an appointment with the GP to receive their result (F6). A total of six areas within the TTP emerged where improvements could be usefully and practically implemented. These consisted of reducing delay prior to blood sampling, a fail-safe to detect missing and delayed results, improvement in managing calls from patients seeking results, addressing the role of non-clinical staff in result communication, routine communication of non-critical results by the practice and defining and disseminating among patients and staff a protocol for the TTP. Taken together, these improvements will facilitate a more efficient and expeditious route for patients to access results.

the role of non-clinical staff in result communication, routine communication of non-critical results by the practice and defining and disseminating among patients and staff a protocol for the TTP. Taken together, these improvements will facilitate a more efficient and expeditious route for patients to access results. Analysis of focus groups Delay in blood sample being taken Following a GP's decision to order a blood test, they may take a blood sample during the consultation. More typically however, patients are required to visit reception and book an appointment with a phlebotomist or nurse. This usually requires a return visit to the surgery up to 14 days later dependent upon the availability of appointments and the patients’ own schedule.Well if the doctor says “make an appointment for a blood test” I make it at the reception desk you see, while I'm there, but it's usually a couple of weeks before I can have it. (Phase I Focus Groups: Patient 9, Practice 3) Patients also expressed concern that making another visit to the practice could have a detrimental impact on their responsibilities elsewhere.Well the other thing is the cost to the patient. I mean, I'm retired, I can find time for appointments, but if I'm doing a job and you know, tomorrow I've got to go to Germany, and the day after I've got to be somewhere else, taking time out is a cost to the project that I'm working on. (Phase I: Patient 1, Practice 4)

Well the other thing is the cost to the patient. I mean, I'm retired, I can find time for appointments, but if I'm doing a job and you know, tomorrow I've got to go to Germany, and the day after I've got to be somewhere else, taking time out is a cost to the project that I'm working on. (Phase I: Patient 1, Practice 4) The delay in the blood sample being taken can also provide anxious patients with the opportunity to avoid seeing the phlebotomist.Patients are scared so they don't turn up [to appointments] and they'll do it continually until we finally do get either hold of them and do it or…that's it, they don't book another. (Phase II: Phlebotomist, Practice 3) Fail-safe In the current system, laboratories return blood test results to practices electronically using the PMIP format. The specifics of how these results are dealt with varies from practice to practice, some are assigned to the ordering GP while other practices have a central ‘in-box’ where results are collated before being processed by the duty GP. What was common across all practices was that none had a means of detecting when individual results had been returned to the practice from the laboratory, and if so, whether said result had reached the patient. Practices acknowledged that frequently they would be aware of missing results only if prompted by a patient enquiry to look for the result.You've got no way of knowing that one of those [results] hasn't come back unless the patient rings in to say: “is my blood result back?” (Phase II: Practice Secretary, Practice 2)

. Practices acknowledged that frequently they would be aware of missing results only if prompted by a patient enquiry to look for the result.You've got no way of knowing that one of those [results] hasn't come back unless the patient rings in to say: “is my blood result back?” (Phase II: Practice Secretary, Practice 2) It's hard, if the patient hasn't called for the result we may never know that they didn't get the result, especially with an abnormal result. (Phase I: GP, Practice 3) Improve the management of patients telephoning the practice for results The typical instruction for patients seeking results was to call the practice reception. Practices spoke about the large volume of calls this generates and the time and resources it takes to respond to these calls. Patients also expressed frustration at time spent waiting for their call to be answered on busy lines.I must say [a queuing system] would help ‘cause I found it just a little annoying that I didn't know where I was in the queue waiting for a response on the phone. (Phase I: Patient 1, Practice 4) Role of non-clinical staff in communicating results Receptionists were frequently responsible for taking calls from patients seeking results, and used a script written by the GP to provide further information. Patients expressed concerns over the suitability of receptionists to communicate results, particularly those which may have serious consequences.I would never accept results off the receptionist unless its cholesterol. (Phase I: Patient 23, Practice 1)

and used a script written by the GP to provide further information. Patients expressed concerns over the suitability of receptionists to communicate results, particularly those which may have serious consequences.I would never accept results off the receptionist unless its cholesterol. (Phase I: Patient 23, Practice 1) Do they have any guidelines about what results they should give? Because there must be some that are not very good for them to [give]. (Phase I: Patient 15, Practice 1) Patients also felt that receptionists should have an awareness of the potential impact a result may have on patients.It's not so much about being given the results over the phone as the person giving the results understanding the impact of what they are saying to the person on the end of the phone. (Phase I: Patient 12, Practice 2) Receptionists are also unable to answer further questions about results or provide clinical information that is not included in the doctor's script. This may remove any reassurance that patients might usually gain from normal results.In those cases you actually need a straightforward bland, “There is no problem what so ever” then maybe that's OK to get that from the receptionist, but that might not be particularly helpful to you if you still have the ache, the pain, the twinge, and you're not feeling so well … “Did they do the right test?” (Phase I: Patient 13, Practice 2)

ly need a straightforward bland, “There is no problem what so ever” then maybe that's OK to get that from the receptionist, but that might not be particularly helpful to you if you still have the ache, the pain, the twinge, and you're not feeling so well … “Did they do the right test?” (Phase I: Patient 13, Practice 2) The content of the script provided for receptionists by GPs to accompany the result varies depending on the GP. Inconsistencies in the amount of information provided from one test to the next can create anxiety and confusion in patients.There are some clinicians that like to add lots of comments…some that don't add any comments…one time the result comes back to a doctor who put some very helpful comments…and the next time the result comes back to one of the clinicians who simply used the default comments and doesn't put anything else there. (Phase I: Practice Manager, Practice 4)

t like to add lots of comments…some that don't add any comments…one time the result comes back to a doctor who put some very helpful comments…and the next time the result comes back to one of the clinicians who simply used the default comments and doesn't put anything else there. (Phase I: Practice Manager, Practice 4) Routine communication of normal results Current systems required patients to call the practice for the majority of results. Due to the large number of tests ordered in general practice, relaying normal results in this fashion places a considerable burden on practice resources. Senior staff felt that time spent communicating normal results was a waste of practice resources.The majority of these [results] are normal, they're fine. They're not results that the doctor has felt the patient needs; that's taking valuable clinical time out. Or even if the senior receptionists are doing it, taking valuable time [rather than getting on with other, perhaps more pertinent tasks] talking about results which are normal. (Phase I: Practice Manager, Practice 4) Despite practice staff questioning the value of relaying normal results, patients repeatedly expressed a preference for receiving all results.If you had a test, and there's no fault found as it were, it would be nice to have that confirmed. (Phase I: Patient 1, Practice 4)

Routine communication of normal results Current systems required patients to call the practice for the majority of results. Due to the large number of tests ordered in general practice, relaying normal results in this fashion places a considerable burden on practice resources. Senior staff felt that time spent communicating normal results was a waste of practice resources.The majority of these [results] are normal, they're fine. They're not results that the doctor has felt the patient needs; that's taking valuable clinical time out. Or even if the senior receptionists are doing it, taking valuable time [rather than getting on with other, perhaps more pertinent tasks] talking about results which are normal. (Phase I: Practice Manager, Practice 4) Despite practice staff questioning the value of relaying normal results, patients repeatedly expressed a preference for receiving all results.If you had a test, and there's no fault found as it were, it would be nice to have that confirmed. (Phase I: Patient 1, Practice 4) We discussed with staff and patients the feasibility of and preferences for alternative methods for routinely communicating normal results that would impact less on practice time and resource yet still meet patient preferences. Patients in our group appeared more comfortable with SMS than staff who felt it was inappropriate or foresaw problems in maintaining accurate records of mobile telephone numbers.I think text messaging is a good idea for a routine test. (Phase I: Patient 20, Practice 2)

e and resource yet still meet patient preferences. Patients in our group appeared more comfortable with SMS than staff who felt it was inappropriate or foresaw problems in maintaining accurate records of mobile telephone numbers.I think text messaging is a good idea for a routine test. (Phase I: Patient 20, Practice 2) There's an SMS text thing on the pick-up menu and I presume whatever you write in gets texted out. I haven't used it because I don't think it's particularly appropriate. (Phase I: GP 3, Practice 1) It relies on the mobile numbers being up to date…or the partner's mobile being up to date and things like that, so I'm not a fan of it. (Phase I: GP 2, Practice 1) Protocol of testing and result communication Staff confirmed that no formal protocols were currently in place for communicating results, and though a default pathway for delivering results existed, it was not consistently adhered to.Well, here there is a set procedure and the degree to which the clinicians use the procedure is, I think, variable to a degree. The first thing to say is that it is at the clinician's discretion. (Phase I: Practice Manager; Practice 4) Patients were aware of the general instruction to phone the practice for results, but appeared unaware of what happens in the eventuality of an abnormal test.… a bit of communication about a system a bit of reassurance—explanation about what will happen if it shouldn't be normal just so you know what to expect. (Phase I: Patient 17, Practice 1)

general instruction to phone the practice for results, but appeared unaware of what happens in the eventuality of an abnormal test.… a bit of communication about a system a bit of reassurance—explanation about what will happen if it shouldn't be normal just so you know what to expect. (Phase I: Patient 17, Practice 1) One of the problems identified by the authors, although not explicitly by focus group members, was that practices only ever gave out instructions for the default method for retrieving results. Therefore, patients were often confused when contacted seemingly ‘out of the blue’ by the practice and informed of their results.They usually say “we will have your results in a couple of weeks” but I have also been called by the surgery to give me results over the phone, randomly and I thought “why are they calling me?” (Phase II: Patient 12, Practice 2)

ed when contacted seemingly ‘out of the blue’ by the practice and informed of their results.They usually say “we will have your results in a couple of weeks” but I have also been called by the surgery to give me results over the phone, randomly and I thought “why are they calling me?” (Phase II: Patient 12, Practice 2) Discussion Summary The accurate, consistent and timely reporting of test results to GPs and then to the patient reduces the likelihood of medical error, and promotes timely and appropriate follow-up. Many services develop over a long period of time in an ad hoc fashion. Such processes are accepted without question, and may even assume a kind of validation through use. Unlike evolution, however, there is no invisible narrative process to ensure that they undergo continuous adaptation and improvement. This is where process and operation management techniques have a role to play as they provide an opportunity to redesign processes that have become increasingly unfit for purpose.27–29 In applying aspects of lean methodologies to the process of testing and result management, we located key areas where delay or failure may occur.21 24

rocess and operation management techniques have a role to play as they provide an opportunity to redesign processes that have become increasingly unfit for purpose.27–29 In applying aspects of lean methodologies to the process of testing and result management, we located key areas where delay or failure may occur.21 24 Strengths and limitations Patient participants were drawn from those with recent experience of the testing and result communication process. Where possible, we created groups, which were mixed by age, gender and ethnicity, though younger patients, more likely to be working full time with potentially different needs and expectations, failed to attend focus groups, and as the time available for recruitment was finite, this led to a preponderance of participants aged over 60 years. This is, however, reflective of primary care, where the majority of patients are older adults. While we cannot claim that the perspectives of patients at the study practices are internationally representative, previous studies in the USA have also found that the process for communicating test results is haphazard and that dissatisfaction with current practice is pervasive.30 31 Staff focus groups contained a range of primary healthcare professionals, and there was a range of staff types, gender and ethnicity. That these focus groups consisted of mixed staff grades may have inhibited the openness of some participants. However, creating groups of mixed staff reflected the reality of the practice environment where a range of practice staff interact with each other throughout the process.32

of staff types, gender and ethnicity. That these focus groups consisted of mixed staff grades may have inhibited the openness of some participants. However, creating groups of mixed staff reflected the reality of the practice environment where a range of practice staff interact with each other throughout the process.32 We did not explore in depth the use of alternative technologies, for example, the direct communication of results from laboratories to patients or the potential impact on result communication of an increase in point of care testing as these are not yet freely available in the NHS. Despite the number of focus groups being relatively limited, this is within the range reported in existing literature,33 and similar experiences were repeatedly described by both patients and staff across our groups suggesting that we were approaching theoretical saturation.34

We did not explore in depth the use of alternative technologies, for example, the direct communication of results from laboratories to patients or the potential impact on result communication of an increase in point of care testing as these are not yet freely available in the NHS. Despite the number of focus groups being relatively limited, this is within the range reported in existing literature,33 and similar experiences were repeatedly described by both patients and staff across our groups suggesting that we were approaching theoretical saturation.34 Main findings Both patients and staff acknowledged the unnecessary delay that can be encountered across current communication processes. Timeliness is frequently acknowledged as a key area for improvement,35 and reducing delay in healthcare delivery is more typically a matter of improving administration rather than medical science, yet it has the potential to have a large and positive impact on the quality of healthcare. As delay reduction necessitates improving administrative capabilities, lessons may be learned from the experience of time-based management accrued in other industries. For example, by reducing unproductive time (delay), companies in the production and manufacturing industries have been able to reduce costs, improve quality and provide improved services to customers.36 In this study, one area where patients repeatedly expressed frustration was the delay experienced between the decision to test and the phlebotomy appointment. Patients reported feeling anxious at the anxiety felt by the subsequent delay in diagnosis and the impact on their time and resources of an additional visit to the practice. In the UK, the importance of an accessible phlebotomy service was previously identified by the Carter Report, which called for samples to be taken at times and in places convenient to patients.37 Pilot studies in secondary care have used lean methodologies to improve quality, productivity and patient experience within phlebotomy with notable success.38 The same could be true in primary care settings. By reconfiguring provision of appointments to better meet the demand of patient access to phlebotomy can be improved while minimising impact on practice resources.

odologies to improve quality, productivity and patient experience within phlebotomy with notable success.38 The same could be true in primary care settings. By reconfiguring provision of appointments to better meet the demand of patient access to phlebotomy can be improved while minimising impact on practice resources. Missing results have serious implications for patient safety; particularly where patients who hold to the adage that ‘no news is good news’ are unaware of their responsibility for collecting results, or wrongfully assume that practices initiate contact in all instances of abnormal results.6 30 39 Despite this, none of the practices we spoke to had a fail-safe in place to ensure that results are returned to staff or patients. The three clinical management systems (Egton Medical Information Systems (EMIS), SystmOne and Invision) currently employed by practices in the UK and two of which by our practices, all possess the capability to collate information on tests ordered, and record the return of results from laboratories. However, staff in our study appeared unclear on the advantages that could be gained, and appeared reluctant to use these features. This hesitancy to engage with health information technology has been observed previously, and has been attributed to a combination of technical factors, which concern the design, usability and functionality of the system,40 41 and social factors centred around characteristics of staff, the organisation and the broader socioeconomic environment.42–47 Evidence suggests that developers could help overcome some of these barriers, and increase the confidence of customers by sharing a broader knowledge base of the implications of using these systems.42 47 If practices are to make the most of the functionality offered by modern software systems then the software needs to possess a demonstrable advantage and be compatible with varying existing methods of work and used by staff comfortable with the technology.41 This being the case, practices may be more willing to commit the resources necessary for the relevant staff to be provided with appropriate training. The timely response to phone calls is considered a key feature of patient-centred primary care,48 yet the number of tests ordered and the common instruction to call practices for results mean that large volumes of calls are generated, and both staff and patients in our study recognised the difficulty of retrieving results from practice staff via telephone.

considered a key feature of patient-centred primary care,48 yet the number of tests ordered and the common instruction to call practices for results mean that large volumes of calls are generated, and both staff and patients in our study recognised the difficulty of retrieving results from practice staff via telephone. Previous studies in the UK reported that one-third of practices has too few people answering the phone at peak hours and that ensuing GP ‘phone line jams’ can lead to increased hospital admissions.49 50 Greater detail in the time and date given to patients to phone may prevent time wasted on calls for tests not yet completed. In addition, improving the management of practice telephone lines by implementing straightforward measures such as call waiting may help alleviate pressure on practice telephone systems, and reduce the frustration and anxiety of patients waiting on busy lines. There are minimal requirements for the training of practice receptionists,51 yet their role is central to a successful testing and result communication process. It became apparent that patients were apprehensive at such a pivotal role being performed by apparently underqualified staff, and during the course of our discussions, three areas of concern emerged.

raining of practice receptionists,51 yet their role is central to a successful testing and result communication process. It became apparent that patients were apprehensive at such a pivotal role being performed by apparently underqualified staff, and during the course of our discussions, three areas of concern emerged. First, result information relayed by receptionists varied depending on the GP who had written the related script, and often lacked objective data. Receptionists could not provide either supporting information themselves or suggest to patients where supporting information may be located. The surrounding uncertainty about the meaning or accuracy of ‘normal’ results can lead to additional costly and unnecessary medical visits and diagnostic procedures.52–55 Second, there is currently no training requirement for receptionists handling confidential or sensitive patient data, and patients questioned the advisability of receptionists relaying results over the phone in busy public areas of the practice.51

First, result information relayed by receptionists varied depending on the GP who had written the related script, and often lacked objective data. Receptionists could not provide either supporting information themselves or suggest to patients where supporting information may be located. The surrounding uncertainty about the meaning or accuracy of ‘normal’ results can lead to additional costly and unnecessary medical visits and diagnostic procedures.52–55 Second, there is currently no training requirement for receptionists handling confidential or sensitive patient data, and patients questioned the advisability of receptionists relaying results over the phone in busy public areas of the practice.51 Third, patients felt reception staff lacked sensitivity when communicating results, and this perception can be related to the style of discourse employed by receptionists. Previous research exploring this issue has identified that receptionists frequently use a task-centred discourse style, which patients can find overly direct, and impact on both a patient's satisfaction and even appointment attendance.56 57 Conversely, where reception staff are perceived as friendly (perhaps using a conventionally polite or rapport-building discourse style), patients report greater satisfaction with their staff encounter and the practice as a whole.58–60 Previously, the use of well-developed scripts in telephone conversations have been found to increase consistency, save time and facilitate courtesy,61 62 and it may be that this dialogue between patients and staff can be improved by providing receptionists with a more structured framework to guide their response.

58–60 Previously, the use of well-developed scripts in telephone conversations have been found to increase consistency, save time and facilitate courtesy,61 62 and it may be that this dialogue between patients and staff can be improved by providing receptionists with a more structured framework to guide their response. We found, as have others, that patients preferred to be informed of normal results, and where this preference is met, it can help reduce patient anxiety.63–66 Though we acknowledge that electronic methods of communication are not suitable for all patients, our discussions identified SMS as an acceptable method to patients and the most timely and cost-effective way for practices to communicate normal results in advance of the introduction of patients access to online medical records. Despite the potential to reduce the workload of practice staff and improve timeliness in communicating results,67 none of our practices yet use SMS to deliver normal results. However, as the model of healthcare delivery in the UK continues to change in line with initiatives designed to place patients at the centre of their care, the use of modern information and communication technologies to communicate normal results appears inevitable.63 Both patients and staff recognised that the TTP needed clarification. Many of the downstream problems with test result communication that were discussed can be ameliorated upstream by agreeing beforehand with the patient how the result will be transmitted, and how this may depend on portent of result. For example, if the patient wishes to receive the result by SMS, the mobile phone number can be confirmed at the point of ordering; subsequent actions, contingent on the result, can also be agreed at this point, and patients can be provided information on the implications of the result, providing reassurance and reducing requests for explanation from unqualified staff downstream.53 There are a number of ways in which awareness of the TTP can be increased. One such solution discussed in our groups was the provision of patients with a hard copy of salient information during the initial consultation, facilitated by existing clinical systems and informing patients of the tests being undertaken, preventing confusion if the return of results from multiple tests are staggered and clarifying where responsibility lies in the agreed method of communicating results.

y of salient information during the initial consultation, facilitated by existing clinical systems and informing patients of the tests being undertaken, preventing confusion if the return of results from multiple tests are staggered and clarifying where responsibility lies in the agreed method of communicating results. Conclusion The complex TTP offers practices a number of opportunities for reducing delay and improving efficiency while increasing patient safety and satisfaction. Undoubtedly, the increase in affordability and accuracy of point of care testing means that when it becomes more widespread, it will have a positive impact on many of the issues we highlight here, as will online access to personal health records for patients.68 In considering existing systems, delay is a significant cause of anxiety in patients, and can be dramatically reduced by better management of phlebotomy services and practice telephone systems. Introducing a fail-safe to ensure that results have been returned to practices and ultimately to patients will further reduce the risk of delayed or even missed diagnoses. The use of automated systems for returning routine results can release resources to be more productively spent elsewhere. Finally, by providing patients with confirmation of the tests ordered, information on the implications of those tests and the precise route for retrieving results, we can improve engagement, and help ensure the patients remain as complicit as possible in their care. Further work is planned to implement and evaluate practical interventions to improve the TTP in some of these key areas.

ed, information on the implications of those tests and the precise route for retrieving results, we can improve engagement, and help ensure the patients remain as complicit as possible in their care. Further work is planned to implement and evaluate practical interventions to improve the TTP in some of these key areas. Supplementary Material Web supplement The authors thank the NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRC) for Birmingham and Black Country.

ed, information on the implications of those tests and the precise route for retrieving results, we can improve engagement, and help ensure the patients remain as complicit as possible in their care. Further work is planned to implement and evaluate practical interventions to improve the TTP in some of these key areas. Supplementary Material Web supplement The authors thank the NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRC) for Birmingham and Black Country. Contributors: IL made substantial contributions to the conception or design of the work, the acquisition, analysis and interpretation of data for the work; was involved in drafting the work and revising it critically for important intellectual content; had final approval of the version to be published; agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. LB and AH made substantial contributions to the acquisition, analysis and interpretation of data for the work; were involved in drafting the work; had final approval of the version to be published and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. RJM made substantial contributions to the conception or design of the work; was involved in revising the work critically for important intellectual content; had final approval of the version to be published; agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. SG made substantial contributions to the conception or design of the work, the analysis and interpretation of data for the work; was involved in drafting the work and revising it critically for important intellectual content; had final approval of the version to be published and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. RL made substantial contributions to the conception or design of the work, the analysis and interpretation of data for the work; was involved in revising the work critically for important intellectual content; had final approval of the version to be published and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

was involved in revising the work critically for important intellectual content; had final approval of the version to be published and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding: National Institute for Health Research (NIHR), Research for Patient Benefit programme in England (PB-PG-1208-18219); United Kingdom Comprehensive Research Network (UKCRN 10055). Competing interests: None declared. Ethics approval: NHS Black Country Research Ethics Committee. Provenance and peer review: Not commissioned; externally peer reviewed.

Background Historically hospital design was based on Florence Nightingale's nineteenth-century observations about the advantages of natural light, ventilation and cleanliness.1 Shared patient accommodation of 30 beds became the standard inpatient accommodation in hospitals globally.2 The suitability of such wards in modern hospitals is now questioned in terms of quality, safety and experience.3 Subdivisions of wards into smaller rooms and bays has become the norm, and internationally, the case is being made for more single-room accommodation in new hospital designs. Arguments for the abolition of all shared accommodation4 are based largely on the belief that patients prefer single rooms and benefit from improved patient outcomes compared with open hospital wards.5 Most current evidence derives from the USA and Scandinavia,6 and while some evidence may be transferable, variation in financial, cultural and organisational systems means that generalisation should not be assumed. The available evidence is both weak and equivocal, suggesting a range of potential benefits for patients and staff but also potential disadvantages.

and Scandinavia,6 and while some evidence may be transferable, variation in financial, cultural and organisational systems means that generalisation should not be assumed. The available evidence is both weak and equivocal, suggesting a range of potential benefits for patients and staff but also potential disadvantages. Potential advantages of single-room accommodation for patients include increased patient privacy, dignity, comfort and less disruption from other patients7–9 and also enhanced patient satisfaction with communication with staff.7–10 Patients rated privacy and personal space as important but they also said that when ill they wanted nurses to be closer.11–13 Potential advantages for staff of working in single rooms include more personalised contact with patients and fewer interruptions for care delivery.14 Improved safety outcomes could include fewer medication errors,15 faster patient recovery rates7–9 and reduced infection rates.7 16–18 Perceived economic advantages include the potential for cost savings associated with reduced length of stay and infection rates and fewer medication errors.19 20 Potential disadvantages of single-room accommodation for patients include increased rates of slips, trips and falls and other adverse events through less patient surveillance,7–9 21 and reduced social interaction22 and patient isolation7–9 23 with consequent negative impacts on patients’ mental well-being and behaviour.24

Potential disadvantages of single-room accommodation for patients include increased rates of slips, trips and falls and other adverse events through less patient surveillance,7–9 21 and reduced social interaction22 and patient isolation7–9 23 with consequent negative impacts on patients’ mental well-being and behaviour.24 Suggested disadvantages of single rooms for staff/organisations include an increase in staff walking distances (reducing time for direct care)25 and workload3; potential need for an increase in staffing levels and/or adjustments to staff skill-mix8 9 24; increase in staff stress26 27 and potential risks to staff through working in isolation.26–30 The international healthcare literature on the advantages and disadvantages of single-room accommodation for patients and staff is of variable quality, and some aspects have been studied more closely than others. Research on the impact of single-room accommodation on staff experiences of work and care provision is limited, the evidence does not clearly point to a preference for single rooms among patients and little is known about patient preferences across different age and cultural groups.31 There is very little evidence from well-designed research studies, and most claims are unsupported by evidence of changes in clinical outcomes; research designs are weak, typically relying on staff opinion or simple before-and-after comparisons.

nown about patient preferences across different age and cultural groups.31 There is very little evidence from well-designed research studies, and most claims are unsupported by evidence of changes in clinical outcomes; research designs are weak, typically relying on staff opinion or simple before-and-after comparisons. Finally, it is clear that decisions around ward design are complex and trade-offs are likely to be necessary between design considerations, patient and staff preferences, changes in clinical needs/demand and economic considerations. Objectives We aimed to identify the impact of the move to a newly built acute National Health Service (NHS) hospital in England with 100% single rooms on staff and patient experience, patient safety and costs. Design and methods Study setting In 2011, Tunbridge Wells Hospital opened, replacing old facilities (some 100 years old) at Pembury and Kent and Sussex hospitals. This was the first NHS hospital in England to have 100% single inpatient rooms in all wards. Staff and patients moved from predominantly large bays and open Nightingale wards in two phases in January and September 2011. Design The study was a mixed methods evaluation of the impact of the move and included a before-and-after study of patient and staff experience, a quasi-experimental before-and-after study of safety outcomes using two control hospitals, and a cost analysis. Data were collected in the new hospital between September 2012 and June 2013. A fuller report of the study and methods can be found elsewhere.6

appendix for observation categories). The following data were also collected in each of the four case studies: 32 pre and post semi-structured patient interviews; 55 pre and post staff surveys and pedometer data from 53 staff pre move and 56 post move. Further details can be found in the online supplementary appendix. The structured observation, pedometer and staff survey data were entered into Excel and analysed in SPSS using descriptive and parametric statistics.6 Staff perceptions of the ward environment pre and post move were compared statistically using an unpaired t test. Travel distance data were analysed using a repeated-measures general linear mixed model (SPSS MIXED) with steps per hour as the dependent variable and pre-post new build, ward (maternity, surgical, older people, MAU), observation session (repeated measure), staff group (midwife, registered nurse (RN), healthcare assistants (HCAs)) and day of the week as independent variables. Midwives were only employed on the maternity ward, and therefore, this introduced an element of confounding. For this reason, the maternity ward was excluded from the modelling. Its exclusion resulted in an improved model fit. A sensitivity analysis to assess the effect of repeated measures on the small number of people who contributed more than one data collection session found that there was no effect and so the data were treated as a between-subjects design. The moderating effects of ward and staff group on the pre-post move were also tested by fitting these two interactions to the main effects model. F tests and adjusted means with 95% CIs have been presented.

data collection session found that there was no effect and so the data were treated as a between-subjects design. The moderating effects of ward and staff group on the pre-post move were also tested by fitting these two interactions to the main effects model. F tests and adjusted means with 95% CIs have been presented. Interviews were recorded, transcribed verbatim and analysed using a framework approach, a method that involves a systematic analysis of verbatim interview data within a thematic matrix.33 Data synthesis occurred as part of the analytical process, and connections were made between qualitative and quantitative data sources in order to identify core themes and connections.34 35

ed using a framework approach, a method that involves a systematic analysis of verbatim interview data within a thematic matrix.33 Data synthesis occurred as part of the analytical process, and connections were made between qualitative and quantitative data sources in order to identify core themes and connections.34 35 Safety outcomes We recruited two control hospitals; one in which no move occurred (steady-state control) and one that moved to a new build with an increase in single rooms (but fewer than 100% single rooms) during the study period (new-build control). Patient outcomes (safety events—falls, pressure ulcers, medication errors—and hospital-acquired infections—methicillin resistant Staphylococcus aureus (MRSA) and Clostridium difficile) were collated for a 3-year period from January 2010 to December 2012 for three of the four wards (MAU, older people and general surgery) of the same type as the study site. Maternity was excluded as pressure ulcers and falls were not applicable and infection rates low. Data sources were discharge data from the patient administration system (including patient age and diagnosis) and incident reporting system for safety surveillance and infection control purposes (see online supplementary appendix). Time series were analysed by means of statistical process control charts (SPC, u-charts) to look for evidence of special cause variation associated with the move to single rooms.36 Where special cause variation was found, we further explored data to assess whether it could be attributed to the move to 100% single rooms. To attribute a change to single rooms, we explored change in case mix as an alternative explanation and looked for a similar but attenuated pattern in the new-build control where the proportion of single rooms increased. For space reasons, we report safety or infection events (eg, per falls or medications errors) per 1000 patient-days in four phases: ≥10 months before moving, 1–9 months before moving, 1–9 months after moving and ≥10 months after moving.

ed pattern in the new-build control where the proportion of single rooms increased. For space reasons, we report safety or infection events (eg, per falls or medications errors) per 1000 patient-days in four phases: ≥10 months before moving, 1–9 months before moving, 1–9 months after moving and ≥10 months after moving. Costs associated with single rooms We conducted a comparative analysis of costs associated with single rooms. To do this, we collected data on bed occupancy, administrative data on cleaning costs, nurse staffing, payroll, length of stay and build costs. Structured observation data were also analysed for proportion of time spent on direct care with patients. We modelled the impact of these costs over the lifetime of the hospital in discounted cash flow/net present value terms. In addition, 12 experts from the architecture, construction, hospital and facilities management and operations were interviewed to inform analysis on the relative impact of different hospital designs on costs and resource use (see online supplementary appendix).

hospital in discounted cash flow/net present value terms. In addition, 12 experts from the architecture, construction, hospital and facilities management and operations were interviewed to inform analysis on the relative impact of different hospital designs on costs and resource use (see online supplementary appendix). Results Patient experiences At interview, patients in the shared accommodation wards reported enjoying the security of being very visible to staff and to each other, which resulted in good patient camaraderie for some. However, the disadvantages of shared patient accommodation included exposure to confused and disruptive patients, lack of privacy and lack of physical comfort. Four dimensions reflect patients’ overall experiences of shared patient accommodation: security, community (advantages and disadvantages), lack of privacy and lack of physical comfort. Four dimensions reflected patients’ overall experience of single-room wards: comfort, control, connection and isolation (table 1). Table 1 Patient advantages and disadvantages of shared patient accommodation and single-room accommodation Shared patient accommodation in an open ward Advantages Patient interview data 1. Security—visibility and staff proximity Shared patient accommodation meant nurses’ station visible from patient beds: (i) Enabled patients to see staff and evaluate staff competence, leading to a sense of safety (ii) Staff were close by at all times, and this proximity also increased patients’ perceptions of security

Shared patient accommodation in an open ward Advantages Patient interview data 1. Security—visibility and staff proximity Shared patient accommodation meant nurses’ station visible from patient beds: (i) Enabled patients to see staff and evaluate staff competence, leading to a sense of safety (ii) Staff were close by at all times, and this proximity also increased patients’ perceptions of security (iii) Shared patient accommodation allowed patients to witness the care of others which helped patients feel they were in a safe environment “You were so close to the station anyway so if something did go wrong you could call somebody. They didn't draw the curtains so they could see you all night…they did keep a good watch on you” Patient, MAU (female—age 55) “You could always see that there was a nurse there….You were never ever left alone…and to me that's important because if you press the buzzer, they look up, they're with you within seconds….” Patient, surgical ward (male—age 73) 2. Community—patient camaraderie (i) Proximity of other patients created a ‘ready-made’ community which many patients appreciated (ii) Watching or observing other patients passed the time or distracted patients from their own discomfort (iii) Actively interacting with other patients to provide and obtain emotional support was noted by many as important.

“You were so close to the station anyway so if something did go wrong you could call somebody. They didn't draw the curtains so they could see you all night…they did keep a good watch on you” Patient, MAU (female—age 55) “You could always see that there was a nurse there….You were never ever left alone…and to me that's important because if you press the buzzer, they look up, they're with you within seconds….” Patient, surgical ward (male—age 73) 2. Community—patient camaraderie (i) Proximity of other patients created a ‘ready-made’ community which many patients appreciated (ii) Watching or observing other patients passed the time or distracted patients from their own discomfort (iii) Actively interacting with other patients to provide and obtain emotional support was noted by many as important. “In the bed I was in, you had a nice little community of us, all within talking distance…It's good for morale…[it] created a wonderful atmosphere between…we got on well with one another. And we all knew what each other had got wrong with them.” Patient, surgical ward (male—age 73) Disadvantages Patient interview data 3. Community—confused /disruptive patients 'Community’ dynamic could change quickly; positive aspects negated: (i) When another patient became confused, or when a disruptive patient was admitted to the ward (ii) Patients described feeling vulnerable and concerned, for the patient's welfare and sometimes their own (iii) Staff were ‘tied up’ caring for or dealing with an individual patient—leading to a reduced sense of security/emotional comfort

3. Community—confused /disruptive patients 'Community’ dynamic could change quickly; positive aspects negated: (i) When another patient became confused, or when a disruptive patient was admitted to the ward (ii) Patients described feeling vulnerable and concerned, for the patient's welfare and sometimes their own (iii) Staff were ‘tied up’ caring for or dealing with an individual patient—leading to a reduced sense of security/emotional comfort “The man who was opposite…he'd got dementia and he was in the shouting stage…shouting out ‘Help’ all the time…they tried everything to help him; they were terribly, terribly patient… most of us, well I think all of us, understood, but it was a nuisance, because in the end it took over, you couldn't do anything else but listen, and see how they were getting on with him.” Patient, surgical ward (male—age 71) 4. Lack of privacy—confidentiality and embarrassment (i) Some patients accepted a lack of privacy as an unavoidable aspect of the environment (ii) Others were left feeling vulnerable, particularly on the maternity ward

“The man who was opposite…he'd got dementia and he was in the shouting stage…shouting out ‘Help’ all the time…they tried everything to help him; they were terribly, terribly patient… most of us, well I think all of us, understood, but it was a nuisance, because in the end it took over, you couldn't do anything else but listen, and see how they were getting on with him.” Patient, surgical ward (male—age 71) 4. Lack of privacy—confidentiality and embarrassment (i) Some patients accepted a lack of privacy as an unavoidable aspect of the environment (ii) Others were left feeling vulnerable, particularly on the maternity ward “It's difficult to keep everybody private…..they'd pull the curtains but you obviously can hear what they're saying… if you're relatively well I think it might be a factor but once you're so ill you really don't care about anything.” Patient, MAU (female—age 55) “I had patients with families peeking through the curtains that didn't fit properly. It was awful. I was so embarrassed…There's no privacy” Patient maternity ward (primiparous) “I was trying to keep him [baby] quiet to try not to disturb everybody else….I was getting really anxious because I was thinking, ‘Oh, don't wake everybody else up.’” Patient, maternity ward (primiparous) 5. Lack of physical comfort—noisy and no control (i) Aspects of physical comfort of importance to patients included location and size of shared toilet facilities, lack of space around the bed, and inability to control lighting, temperature and noise  “I had two drains to carry with me and your drip thing to get to the bathroom, and it pulls all your nightie up at the side ….You're constantly looking somewhere else, trying not to make eye contact because I get very embarrassed.” Patient, surgical ward (female—age 57) “I got no sleep because it is just so noisy…..the other babies screaming…. So it was very difficult to sleep in a maternity ward like that.” Patient, maternity ward (primiparous) 100% single rooms Advantages Patient interview data 1. Comfort—en-suite convenience, rest and sleep-‘hotel’ Patients experienced high levels of comfort in the single rooms, in relation to: (i) En-suite toilet facilities, lighting, ventilation, having a view from a window and noise levels

t, maternity ward (primiparous) 100% single rooms Advantages Patient interview data 1. Comfort—en-suite convenience, rest and sleep-‘hotel’ Patients experienced high levels of comfort in the single rooms, in relation to: (i) En-suite toilet facilities, lighting, ventilation, having a view from a window and noise levels (ii) The single room was frequently compared favourably with a hotel or home environment (iii) The design of the room helped reduce patients’ anxiety, promote rest and sleep, and support physical comfort "You had a shower, toilet, and washing facilities. Got your own TV […] I would rate it really as probably on a par with a three star hotel […] That's how good it was.” Patient, surgical ward (male—age 70). “I only just made it into my bathroom in time […] I would have been so embarrassed, if I'd had to go out of my [room] and across the corridor and feel sick, and wet myself, I would just die of embarrassment […] but there it was absolutely fine.” Patient, surgical ward (female—age 45) “The single room is, I think, the most perfect solution they came up with because […] sometimes babies are crying constantly, […] at least if baby was asleep, I could also go to sleep because there wasn't anyone next to me with a baby crying or anything.” (Postnatal patient, age 26) 2. Control—enhanced privacy and freedom- ‘at home’ (i) Control rarely featured in patients’ accounts of staying on shared patient accommodation

crying constantly, […] at least if baby was asleep, I could also go to sleep because there wasn't anyone next to me with a baby crying or anything.” (Postnatal patient, age 26) 2. Control—enhanced privacy and freedom- ‘at home’ (i) Control rarely featured in patients’ accounts of staying on shared patient accommodation (ii) Control was closely associated with privacy and freedom; not being observed by others; retaining independence and agency- patients valued not having to worry about what other patients might be thinking about them (iii) Patients compared the single room with being at home, in their own space, with similar control and privacy (iv) Privacy and flexibility of the single-room environment for receiving visitors

(ii) Control was closely associated with privacy and freedom; not being observed by others; retaining independence and agency- patients valued not having to worry about what other patients might be thinking about them (iii) Patients compared the single room with being at home, in their own space, with similar control and privacy (iv) Privacy and flexibility of the single-room environment for receiving visitors "Being in a room on my own where I didn't have to talk to people if I didn't want to, I could watch the telly […] And the fact that I'd got privacy, and things like being able to go to sleep when I wanted, and wake up when I wanted, was really beneficial […].” Patient, MAU (female—age 44) “It just gives you time to properly relax, and you haven't got other people shouting across the ward… It's just nice because you can relax in your own time. You're given that privacy, and that's what I like.” Patient, surgical ward (female—age 72) “You can ask them anything […] I had a C-section and I would ask them about the pains, the bleeding, the catheters… Because it's private, I felt comfortable and they could even show me and […] actually demonstrate it for me”. (Postnatal patient, age 26) “I do think visiting in here is very much easier, in single rooms. [There's] more space. And there's more confidentiality, you feel not everyone's hearing your business […] it is nice really [visitors] can come whenever they like and that's a very big bonus.” Patient, old people ward (female—age 80) Disadvantages Patient interview data

"Being in a room on my own where I didn't have to talk to people if I didn't want to, I could watch the telly […] And the fact that I'd got privacy, and things like being able to go to sleep when I wanted, and wake up when I wanted, was really beneficial […].” Patient, MAU (female—age 44) “It just gives you time to properly relax, and you haven't got other people shouting across the ward… It's just nice because you can relax in your own time. You're given that privacy, and that's what I like.” Patient, surgical ward (female—age 72) “You can ask them anything […] I had a C-section and I would ask them about the pains, the bleeding, the catheters… Because it's private, I felt comfortable and they could even show me and […] actually demonstrate it for me”. (Postnatal patient, age 26) “I do think visiting in here is very much easier, in single rooms. [There's] more space. And there's more confidentiality, you feel not everyone's hearing your business […] it is nice really [visitors] can come whenever they like and that's a very big bonus.” Patient, old people ward (female—age 80) Disadvantages Patient interview data 3. Connection—seeing staff and being seen (i) Anticipating the move to single rooms patients in shared patient accommodation expressed anxiety about staff not being in close proximity and able to monitor them sufficiently (ii) Proximity and monitoring not directly mentioned by patients in single-room accommodation who spoke more generally about feeling or not feeling alone

3. Connection—seeing staff and being seen (i) Anticipating the move to single rooms patients in shared patient accommodation expressed anxiety about staff not being in close proximity and able to monitor them sufficiently (ii) Proximity and monitoring not directly mentioned by patients in single-room accommodation who spoke more generally about feeling or not feeling alone (iii) The quality of interaction with staff experienced by patients in single rooms affected how connected they felt to staff (iv) Some patients thought that the single room enabled staff to give them undivided attention and supported one-to-one care, others that staff were ‘very distant’, ‘busy’ and ‘did not have time to chat’ (v) Not being able to see staff going about their work beyond the single room meant patients often had little idea of other patients needs or how busy or not busy staff were

(iv) Some patients thought that the single room enabled staff to give them undivided attention and supported one-to-one care, others that staff were ‘very distant’, ‘busy’ and ‘did not have time to chat’ (v) Not being able to see staff going about their work beyond the single room meant patients often had little idea of other patients needs or how busy or not busy staff were Anticipating the move a patient in shared patient accommodation said: “The problem is if you have these low blood pressure episodes they happen so quickly that you might not have a chance to ring the bell. And that would have worried me. Say you'd got up to go to the bathroom and then had one of these episodes and fallen over…then you just do wonder how often they would come along and check on you….I would have thought that's not a good idea.” Patient, MAU (female—age 55) “They didn't really have time to chat […] They just came in and did what they had to do, the temperatures and blood pressures, and things […] You get more time [in a multi-bed bay], not to chat to the staff, but to ask them questions, because they're in a bay more than they are in a single room. They would only come to your room just to see if you're okay, every so often or if you called them.” Patient, surgical ward (male—age 71) “If you're in hospital, you're ill, you're not there for a holiday, and if you're ill you need a lot of treatment, you have your blood pressure taken three times a day, your temperature …So you are being constantly attended by nurses because of your condition….. so actually, a [single room] is a very busy place, you're not on your own, hardly ever.” Patient, surgical ward (male—age 84) 4. Isolation (i)  Anticipating the move to single rooms patients in shared patient accommodation expressed anxiety about potential isolation in single rooms

y nurses because of your condition….. so actually, a [single room] is a very busy place, you're not on your own, hardly ever.” Patient, surgical ward (male—age 84) 4. Isolation (i)  Anticipating the move to single rooms patients in shared patient accommodation expressed anxiety about potential isolation in single rooms (ii) For some patients in single rooms, lack of interaction with other patients led to strong feelings of isolation (iii) Patients who felt little connection with staff were likely to report feeling isolated in single rooms (iv) Some patients wanted to have the opportunity to socialise with other patients to counter this isolation (v) Patients who missed ‘company’ felt confined in their rooms and unaware of dayroom opportunities or whether it was permissible to walk the corridors or ‘visit’ other patients

(iii) Patients who felt little connection with staff were likely to report feeling isolated in single rooms (iv) Some patients wanted to have the opportunity to socialise with other patients to counter this isolation (v) Patients who missed ‘company’ felt confined in their rooms and unaware of dayroom opportunities or whether it was permissible to walk the corridors or ‘visit’ other patients Anticipating the move a patient in shared patient accommodation said: “I think…single bed [rooms] are a bit isolating. You wouldn't see life going by. Yes, it would be quieter, but perhaps a bit too quiet? [other patients] they're not going to go into an individual room really to say ‘Hello’ and see how you're getting on, rather than if they're just passing by the bed, will they?” Patient, surgical ward (female—age 56) “It would have been nice to have been able to see other people, or maybe just to chat with other mothers […] If you're in an open ward, you just see people coming and going […] you can chat to people across the ward […] I just wanted to get home […] because there isn't anyone to chat to and I just felt a little bit isolated being in the room on my own.” (Postnatal patient, age 36) “It did get a little bit lonely […] you don't really get to talk to many people, other patients. […] I wanted to go for a walk just to get out of the room and maybe have a chat with someone. You do start to feel a little bit stir crazy stuck in a room […] I kind of felt like I had to stay in my room.” Patient, MAU (male—age 44) Patients experienced high levels of comfort in the single rooms and compared the hospital room favourably to a hotel or home environment allowing them to experience the comfort and control of home or a hotel, in hospital. Patients also experienced a high degree of personal control in the single room; feeling not as ‘on show’ as in shared patient accommodation. Patients in single rooms described being able to do as they pleased at any time without worrying about other patients. This was in contrast to patients’ experiences in shared accommodation.

lso experienced a high degree of personal control in the single room; feeling not as ‘on show’ as in shared patient accommodation. Patients in single rooms described being able to do as they pleased at any time without worrying about other patients. This was in contrast to patients’ experiences in shared accommodation. Patients appreciated the confidentiality afforded by the single room, and the privacy and flexibility it gave for visitors. Patients rarely used dayrooms (communal social spaces) (available on two wards); they were only orientated to their room. They did not know where dayrooms were and were uncertain if they could leave their rooms. Interaction with other patients was largely absent in the new hospital, leading to a sense of isolation for some patients. Patients reported regular visits to their single room by various staff, and all patients described nursing staff conducting frequent intentional ward rounds. These nursing rounds were introduced as a method of ensuring all patients were observed and had their needs met regularly and to counter isolation in single rooms. Some patients experienced a good quality of communication and felt connected with staff; others experienced care as largely task-driven and functional, which may have enhanced the disconnection experienced by some patients.

tients were observed and had their needs met regularly and to counter isolation in single rooms. Some patients experienced a good quality of communication and felt connected with staff; others experienced care as largely task-driven and functional, which may have enhanced the disconnection experienced by some patients. Overall, the majority of patients (two-thirds of those interviewed) expressed a clear preference for single rooms. Privacy and en-suite bathrooms were important aspects of this. All the women in the postnatal unit liked single rooms. One-fifth of patients, including almost half the men interviewed, said they preferred multi-bedded accommodation. Staff associated loneliness with older patients, but it was experienced by patients in all age groups in this study. Single rooms were a clear preference for many patients, but a significant minority, including patients of varying ages and patient groups, wanted greater choice, either in terms of having the option of shared accommodation or being able to interact with other patients in communal spaces of the hospital. Approximately one in three patients interviewed (11 of 32) reported lack of interaction with other patients as a main disadvantage of the single room.

ps, wanted greater choice, either in terms of having the option of shared accommodation or being able to interact with other patients in communal spaces of the hospital. Approximately one in three patients interviewed (11 of 32) reported lack of interaction with other patients as a main disadvantage of the single room. Staff experiences At interview, before the move to 100% single-room hospital, staff identified the advantages of shared patient accommodation as good visibility of all patients, which facilitated surveillance and monitoring; supporting good teamwork, which was enhanced by being able to see and find staff easily that enabled staff to support each other. Staff also identified that shared patient accommodation allowed social contact between patients, which helped staff and enhanced recovery (see table 2). Disadvantages of the shared patient accommodation included an inadequate physical environment for patients and for care delivery and poor privacy, sleep and rest for patients. Table 2 Staff advantages and disadvantages of shared patient accommodation and single-room accommodation Shared patient accommodation in an open ward Advantages Staff interview data 1. Visibility—enhanced surveillance and monitoring Visual and aural proximity of staff to patients resulted in three key perceived benefits for staff: (i) Enhanced surveillance and monitoring of patients (ii) Staff ability to monitor and communicate with patients enhanced by the proximity of patients to each other (iii) Staff benefitted from patients acting as an extra pair of eyes on the ward

Shared patient accommodation in an open ward Advantages Staff interview data 1. Visibility—enhanced surveillance and monitoring Visual and aural proximity of staff to patients resulted in three key perceived benefits for staff: (i) Enhanced surveillance and monitoring of patients (ii) Staff ability to monitor and communicate with patients enhanced by the proximity of patients to each other (iii) Staff benefitted from patients acting as an extra pair of eyes on the ward “Even when I'm sitting here by the desk [staff base], I can see…By just looking up, you can automatically see that, ‘Oh, I think he's not breathing well,’ or something is going wrong, and you can act immediately.” E6 Nurse, medical (older people) ward “I was on [the postnatal section] the other night in the early hours of the morning and I was seeing somebody and I could hear this baby throwing up, so I dived behind the curtain so I could get it on its side, pat on the back. So, yeah, you can see the patients as you walk down very easily. You hear things.” M3 Midwife, maternity ward “Where I am today it's been absolutely hectic because we've got confused patients. They're all constantly shouting out. And where we are now, we can shout over, ‘I'll be with you in a minute.’ [It helps with] knowing where to go, how to prioritise yourself really” A1 Healthcare assistant, MAU “We've got patients who look out for [other patients]. If a patient next to them sees that they're having trouble doing something they'll ring their bell for them and say, ‘Oh, nurse, she's doing this, she's doing that, I think she's trying to stand up…’ which is really good.” E3 Healthcare assistant, medical (older people) ward 2. Teamwork and communication

her patients]. If a patient next to them sees that they're having trouble doing something they'll ring their bell for them and say, ‘Oh, nurse, she's doing this, she's doing that, I think she's trying to stand up…’ which is really good.” E3 Healthcare assistant, medical (older people) ward 2. Teamwork and communication (i)  Staff valued being easily able to request or provide assistance where needed (ii)  In staff survey, surgical and medical (older people) ward staff had the highest mean scores for the items ‘Obtaining advice from colleagues’, ‘Finding a staff member’ and ‘Knowing when other staff might need a helping hand’ (mean scores >4) (iii) Staff learning from each other and supporting each other highlighted as important. During observation sessions ward managers were frequently observed on the ward ‘floor’, assisting and encouraging or advising their team “It's one of the lovely things about the Nightingale wards, at least we [nursing staff] can all see each other… you can go, ‘Do you want some help there?’ E2 Healthcare assistant, medical (older people) ward “Sometimes you are faced with a situation that is difficult to manage on your own and so there's nearly always somebody very close by that you can call. Or if maybe you were struggling a bit in any way …somebody else would probably hear and come to your rescue. Those are big advantages.” S3 Nurse, surgical ward 3. Facilitation of social contact between patients (i) Staff were positive about patients being able to see, hear and interact with each other in shared open plan areas

“It's one of the lovely things about the Nightingale wards, at least we [nursing staff] can all see each other… you can go, ‘Do you want some help there?’ E2 Healthcare assistant, medical (older people) ward “Sometimes you are faced with a situation that is difficult to manage on your own and so there's nearly always somebody very close by that you can call. Or if maybe you were struggling a bit in any way …somebody else would probably hear and come to your rescue. Those are big advantages.” S3 Nurse, surgical ward 3. Facilitation of social contact between patients (i) Staff were positive about patients being able to see, hear and interact with each other in shared open plan areas (ii) Social contact between patients was seen as an aid to staff on busy wards, who might not have time to interact for longer periods with patients (iii) Social contact was perceived to support recovery and relieve boredom on the wards

(i) Staff were positive about patients being able to see, hear and interact with each other in shared open plan areas (ii) Social contact between patients was seen as an aid to staff on busy wards, who might not have time to interact for longer periods with patients (iii) Social contact was perceived to support recovery and relieve boredom on the wards “If you have one end of the ward where everybody's quite jolly, it really lifts the spirits of everybody. And it's very distracting. If you're sick in a room on your own and it's quiet and all you've got to think about is your pain, whereas if you've got distractions of people walking around, talking to you, it's a really good therapy.” S6 Nurse, surgical ward “Other patients will give somebody a boost, and they will talk to them [another patient] and they'll say, ‘Oh come on, try and eat a bit more’…that helps people a great deal, if somebody's there talking to them…If we've got two people who are reasonably well and they like a chat…we do try and juggle the beds around so that they're together, so that they can have a talk.” E2 Healthcare assistant, medical (older people) ward Shared patient accommodation in an open ward Disadvantages Staff interview data 4. Inadequate physical space for patients/care (i) Lack of space for staff to provide nursing care to patients emerged as a key constraint (ii) Lack of space in toilets and showers made it difficult for staff to assist patients and created obstacles for patients attempting to mobilise independently

“If you have one end of the ward where everybody's quite jolly, it really lifts the spirits of everybody. And it's very distracting. If you're sick in a room on your own and it's quiet and all you've got to think about is your pain, whereas if you've got distractions of people walking around, talking to you, it's a really good therapy.” S6 Nurse, surgical ward “Other patients will give somebody a boost, and they will talk to them [another patient] and they'll say, ‘Oh come on, try and eat a bit more’…that helps people a great deal, if somebody's there talking to them…If we've got two people who are reasonably well and they like a chat…we do try and juggle the beds around so that they're together, so that they can have a talk.” E2 Healthcare assistant, medical (older people) ward Shared patient accommodation in an open ward Disadvantages Staff interview data 4. Inadequate physical space for patients/care (i) Lack of space for staff to provide nursing care to patients emerged as a key constraint (ii) Lack of space in toilets and showers made it difficult for staff to assist patients and created obstacles for patients attempting to mobilise independently (iii) Problems created by lack of space included risk of musculoskeletal injuries, trip hazards and a perceived increase in time required to undertake tasks

(i) Lack of space for staff to provide nursing care to patients emerged as a key constraint (ii) Lack of space in toilets and showers made it difficult for staff to assist patients and created obstacles for patients attempting to mobilise independently (iii) Problems created by lack of space included risk of musculoskeletal injuries, trip hazards and a perceived increase in time required to undertake tasks “You're banging into the next patient's chair or locker and stuff like that. It's really tight around the bed space, and .. when trying to get a commode in there you're fighting with everything really.” S4 Nurse, surgical ward When you want to hoist someone…it takes so much time, you've got to move the bed that side, you've got to move the [other furniture], you know. It's so, so much energy doing that, rather than if there was enough space, you just wheel it in, hoist your patient, take them away, and that's it.” E6 Nurse, medical (older people) ward 5. Poor physical spaces/facilities for staff (i) Space at staff bases generally considered inadequate—limited space and poor access to the limited number of PCs and telephones (ii)  Staff break rooms had limited facilities and no kitchen. Often dark, cold and shabby, and multipurpose, serving as meeting rooms as well as staff break rooms (iii) Lack of staff toilets was problematic (particularly maternity and surgical wards), meaning staff often had to go away and return later to use facilities

(i) Space at staff bases generally considered inadequate—limited space and poor access to the limited number of PCs and telephones (ii)  Staff break rooms had limited facilities and no kitchen. Often dark, cold and shabby, and multipurpose, serving as meeting rooms as well as staff break rooms (iii) Lack of staff toilets was problematic (particularly maternity and surgical wards), meaning staff often had to go away and return later to use facilities “You often have nurses sitting there, then you'll get physios and OT coming along, they might want to sit down and there's only one screen [PC] in the ward and that's really not enough, we need more screens to be effective because you're waiting to use the screen and so it would be useful to have more computer stations.” S1 Nurse, surgical ward “There are no sofas in it [staff room], as you can see [ ] people's bags and coats, and it's usually like a stock room…There's normally stuff piled…It's normally quite cold in here as well, it's got the old window.” M5 Student midwife, maternity ward “Toilet facilities are horrendous, one toilet for all of us, doctors, nurses, domestics—horrendous. So many times you walk up and there's somebody in there. ..[The] changing facilities…[are] totally inadequate, totally.… there's nowhere to change.” S6 Nurse, surgical ward 6. Poor privacy, sleep and rest for patients (i) Staff were looking forward to improved privacy, sleep and rest for patients (ii) Patients in the old accommodation were disturbed, for example, by general ward activity, or by other patients, which could create anxiety

“You often have nurses sitting there, then you'll get physios and OT coming along, they might want to sit down and there's only one screen [PC] in the ward and that's really not enough, we need more screens to be effective because you're waiting to use the screen and so it would be useful to have more computer stations.” S1 Nurse, surgical ward “There are no sofas in it [staff room], as you can see [ ] people's bags and coats, and it's usually like a stock room…There's normally stuff piled…It's normally quite cold in here as well, it's got the old window.” M5 Student midwife, maternity ward “Toilet facilities are horrendous, one toilet for all of us, doctors, nurses, domestics—horrendous. So many times you walk up and there's somebody in there. ..[The] changing facilities…[are] totally inadequate, totally.… there's nowhere to change.” S6 Nurse, surgical ward 6. Poor privacy, sleep and rest for patients (i) Staff were looking forward to improved privacy, sleep and rest for patients (ii) Patients in the old accommodation were disturbed, for example, by general ward activity, or by other patients, which could create anxiety “On [the antenatal area], because you're doing sort of CTGs the whole time [ ] you can always hear like other peoples’ babies’ heartbeats going on… it's so noisy on there, whatever happens you can hear. Like the other day…someone delivered on the ward [because there wasn't enough time to get her to the Delivery Suite] and…The other women on the ward they were just sort of looking as if to say, ‘Is this really what it's going to be like?’ and so in that sense it's not good.” M5 Student midwife, maternity ward 100% single rooms Advantages Staff interview data 1. Privacy, dignity and confidentiality: more personalised patient care Staff compared single rooms favourably with open-plan wards in relation to: (i) Patients having their ‘own space’, including an en-suite bathroom and toilet, where personal care took place away from other patients

ms Advantages Staff interview data 1. Privacy, dignity and confidentiality: more personalised patient care Staff compared single rooms favourably with open-plan wards in relation to: (i) Patients having their ‘own space’, including an en-suite bathroom and toilet, where personal care took place away from other patients (ii) Allowing patients to sleep, rest and recuperate without disturbance (iii) They could also better accommodate visitors. (iv) Privacy was particularly valued for patients who were seriously or terminally ill and for their families (v) Staff also reported less likelihood of interruption and distraction

ms Advantages Staff interview data 1. Privacy, dignity and confidentiality: more personalised patient care Staff compared single rooms favourably with open-plan wards in relation to: (i) Patients having their ‘own space’, including an en-suite bathroom and toilet, where personal care took place away from other patients (ii) Allowing patients to sleep, rest and recuperate without disturbance (iii) They could also better accommodate visitors. (iv) Privacy was particularly valued for patients who were seriously or terminally ill and for their families (v) Staff also reported less likelihood of interruption and distraction “[I would] hate to go back to nursing patients behind curtains” because “dealing with gastric surgery, it can be embarrassing when patients don't make it to the toilet in time […] now they have the dignity and privacy of being in their own room.” S16 Nurse, surgical ward “I think from a privacy and dignity point of view, when patients are dying, or when relatives are in there, or in extremis, when you don't really want the next patient to be making a noise, then [single rooms are] fantastic. […]” KSI28—consultant “ … [at the old hospital] you'd have to find a room outside the ward to bring the family to, to have a discussion with them, whereas here, you might be able to speak a bit more openly and privately with the family and with the patient present. Almost have a mini MDT with the family and the patient in their room.” AHP40—dietician “You're able to give your patient the attention and care because you're with them; you haven't got someone else shouting from the other side of the room for you.” M01-Healthcare assistant, MAU 2. Improved room design: improved care delivery

have a mini MDT with the family and the patient in their room.” AHP40—dietician “You're able to give your patient the attention and care because you're with them; you haven't got someone else shouting from the other side of the room for you.” M01-Healthcare assistant, MAU 2. Improved room design: improved care delivery (i)  Improved working environment for delivering patient care and rooms described as modern, clean, spacious and safe (ii) The en-suite facilities one of the most positive aspects, allowing nursing staff to more easily assist patients with personal care "[The single rooms] are really accessible and for patients that want to hold on to things while they're walking, they've got bars around the side, from the bed to the bathroom, they've got the bars alongside of the toilet and the bars on the wall to hold onto, bars by the shower, so I think that's really good. The floors are non-slip as well, so I stand patients up quite confidently in there, without worrying that they're going to slip.” S14—Healthcare assistant, surgical ward 3. Improved ward layout and design (i)  Centralised ward support facilities (eg, clean and dirty utility, drug preparation room, kitchen) liked by staff (ii) Secure ward entry system to allow patients and visitors to enter and exit (see also disadvantages) (iii) Dedicated staff break rooms (on surgical ward and the older people ward)

"[The single rooms] are really accessible and for patients that want to hold on to things while they're walking, they've got bars around the side, from the bed to the bathroom, they've got the bars alongside of the toilet and the bars on the wall to hold onto, bars by the shower, so I think that's really good. The floors are non-slip as well, so I stand patients up quite confidently in there, without worrying that they're going to slip.” S14—Healthcare assistant, surgical ward 3. Improved ward layout and design (i)  Centralised ward support facilities (eg, clean and dirty utility, drug preparation room, kitchen) liked by staff (ii) Secure ward entry system to allow patients and visitors to enter and exit (see also disadvantages) (iii) Dedicated staff break rooms (on surgical ward and the older people ward) "The clinical room is a good size, much bigger than at the old hospital […] we've got lots of surface area, whereas before it used to get a little bit cramped, and we'd be getting confused and our drug charts muddled up. Now I think there's much more space to do things quickly and more effectively. S16—Nurse, surgical ward “We've got up to 30 patients, possibly confused, wandering; unless a member of the staff has opened the door, those patients are safe […] that's priceless really.” OP25—junior sister, older people ward “It's much cleaner, much tidier, somewhere to sit down and eat […] you've got access to fresh air which we didn't really have at the old place, so I think the staff areas are much nicer.” S14—Healthcare assistant, surgical ward 4. Perceived reduced risk of infection

celess really.” OP25—junior sister, older people ward “It's much cleaner, much tidier, somewhere to sit down and eat […] you've got access to fresh air which we didn't really have at the old place, so I think the staff areas are much nicer.” S14—Healthcare assistant, surgical ward 4. Perceived reduced risk of infection (i) Staff anticipated that single rooms would reduce infection risk (ii) New wards were perceived by staff to support good hygiene practices, with no-touch clinical hand washbasins in every patient room (iii) Personal protective equipment and alcohol hand gel and antibacterial wipes were available from wall-mounted dispensers outside every room. This provided a visual reminder for staff entering a patient room to comply with infection control procedures

(ii) New wards were perceived by staff to support good hygiene practices, with no-touch clinical hand washbasins in every patient room (iii) Personal protective equipment and alcohol hand gel and antibacterial wipes were available from wall-mounted dispensers outside every room. This provided a visual reminder for staff entering a patient room to comply with infection control procedures "I've taken a picture of the gloves on the wall, just outside the general patients’ room, the gloves, alcohol dispenser, aprons, and the wipes […] they're very clean, clear, easy to refill and they really benefit from being where they are, which is perfect, the fact that they're outside every single patient's room, there's no question about it.” OP25—junior sister, older people ward “Because you've got individual basins, you're not sharing a sink. You're able to wash your hands very quickly after having patient contact, rather than having to walk around looking for a basin to use. […] So you just feel that you're able to carry out infection control procedures more effectively.” S16—Nurse, surgical ward 100% single rooms Disadvantages Staff interview data 5. Reduced visibility: difficulties monitoring and safeguarding patients (i) Staff described visibility of patients from the ward corridor as limited to the patient room they were directly outside, and then only if the door or vision panel in it was open

"I've taken a picture of the gloves on the wall, just outside the general patients’ room, the gloves, alcohol dispenser, aprons, and the wipes […] they're very clean, clear, easy to refill and they really benefit from being where they are, which is perfect, the fact that they're outside every single patient's room, there's no question about it.” OP25—junior sister, older people ward “Because you've got individual basins, you're not sharing a sink. You're able to wash your hands very quickly after having patient contact, rather than having to walk around looking for a basin to use. […] So you just feel that you're able to carry out infection control procedures more effectively.” S16—Nurse, surgical ward 100% single rooms Disadvantages Staff interview data 5. Reduced visibility: difficulties monitoring and safeguarding patients (i) Staff described visibility of patients from the ward corridor as limited to the patient room they were directly outside, and then only if the door or vision panel in it was open (ii) The line of sight into the patient room was interrupted by the wall of the adjacent room's en-suite bathroom (‘in-board’ single-room design) also obscuring the view of patient room doors for staff looking down the corridor (iii) All staff interviewed perceived that lack of visibility of patients in the single-room wards had contributed to an increase in falls in the new hospital

(ii) The line of sight into the patient room was interrupted by the wall of the adjacent room's en-suite bathroom (‘in-board’ single-room design) also obscuring the view of patient room doors for staff looking down the corridor (iii) All staff interviewed perceived that lack of visibility of patients in the single-room wards had contributed to an increase in falls in the new hospital (iv) Experienced nursing staff thought that it had been easier to prevent falls in multi-bed accommodation because they could ‘keep an eye’ on patients and were more aware of warning signals, for example, patients attempting to get up from their chair or bed (v) Staff adapted work patterns to monitor and safeguard patients, but use of cordless telephones to locate each other was variable

(iv) Experienced nursing staff thought that it had been easier to prevent falls in multi-bed accommodation because they could ‘keep an eye’ on patients and were more aware of warning signals, for example, patients attempting to get up from their chair or bed (v) Staff adapted work patterns to monitor and safeguard patients, but use of cordless telephones to locate each other was variable “When we were on an open ward I could walk on the ward and I could view everybody. And when you knew your patient you could see [if they didn't look well]. Whereas now I can walk up and down the rooms, but as soon as I'm in a room I'm away from everybody. […] That time has gone where you could just stand and have a quick chat with a patient while you were still keeping an eye on everybody else.” S11—Nurse, surgical ward “Last week, we had about three people […] climbing out of beds and falling […] And I know on an open ward [patients] can still climb out of bed but at least as you're walking up and down the ward you could physically see them […] now the only time we know somebody has fallen out is when we hear the clump and they're on the floor.” S15—Nurse, surgical ward “You have to adjust your nursing practice just to make sure that everybody is seen and you keep an eye on them the whole time. With the bays you'd go in to see one patient but then subconsciously you're eyeballing everybody else, making sure everybody else is okay. Whereas here you have to physically go into each and everybody's rooms, or stop and have a look at them.” M03—Nurse, MAU “I walk around with my head permanently fixed to the side that all the rooms are on, just checking […] I've just adapted. I now look into every single room every time I walk past and I make the effort to go into the rooms.” M05—junior sister, MAU 6. Social isolation of patients

and have a look at them.” M03—Nurse, MAU “I walk around with my head permanently fixed to the side that all the rooms are on, just checking […] I've just adapted. I now look into every single room every time I walk past and I make the effort to go into the rooms.” M05—junior sister, MAU 6. Social isolation of patients (i) Staff felt social isolation was a disadvantage of single-room wards compounded by ward design with limited day rooms (ii) Social isolation influenced, patients’ satisfaction with their hospital stay and their emotional well-being and recovery (iii) Staff felt that provision of social or communal space for patients should have been given much higher priority (iv) Staff felt older patients likely to be more lonely (though our patient data does not support this from our sample) although staff also recognised that patients of all ages could be disadvantaged by not mixing with others and hearing their experiences, which was thought to help individuals assess their own progress and could be reassuring, motivating and encouraging (v) Socially isolated patients were seen as likely to make more demands on staff, for example pressing call bells frequently or talking a lot to ‘keep’ nurses in the room with them which created tensions for staff

(iv) Staff felt older patients likely to be more lonely (though our patient data does not support this from our sample) although staff also recognised that patients of all ages could be disadvantaged by not mixing with others and hearing their experiences, which was thought to help individuals assess their own progress and could be reassuring, motivating and encouraging (v) Socially isolated patients were seen as likely to make more demands on staff, for example pressing call bells frequently or talking a lot to ‘keep’ nurses in the room with them which created tensions for staff “I just do sometimes feel sorry for the older patients that are in for weeks [and] don't necessarily have a lot of contact with other people. And I guess mood has a massive impact on everything in hospital, on your recovery, on your eating, on how likely you are to get up and work with the physio that day. And sometimes if you've other patients motivating you, or even just speaking to you, it just picks up your mood, it can help.” AHP40—dietician “For postnatal ladies it's important that they can see what's going on around them and it's not all about them and their baby […] they don't understand that every baby feeds all the time and cries and everything. They can't see what is normal and I think they think, ‘My baby is going to feed and go to sleep for six hours.’ But if they see the other women struggling as well it sort of normalises it for them.” PN36—midwife, postnatal ward “It tends to be older patients who have no company at home […] especially if they're being barrier nursed and the door needs to be closed, they find it very isolating, and I've had quite a few older people get quite upset. And then the impact it has on us because they're lonely, they'll be pressing their bell all the time for nothing other than just wanting someone to be there with them, [but] you just don't have that time […] Yeah, and if they don't see you for a little while, patients often think you must not be doing anything.” M01—Healthcare assistant, MAU 7. Maintaining teamwork and communication

be pressing their bell all the time for nothing other than just wanting someone to be there with them, [but] you just don't have that time […] Yeah, and if they don't see you for a little while, patients often think you must not be doing anything.” M01—Healthcare assistant, MAU 7. Maintaining teamwork and communication (i) After the move, staff felt that the quality of teamwork had been difficult to recreate (ii) Nurses described seeing less of their colleagues, being unaware of what was happening in other parts of the ward, and sometimes feeling isolated (iii) One of the main difficulties described by staff was finding colleagues to obtain assistance and information. This was a cause of much frustration, especially for HCAs, use of technology to support this was variable (cordless telephones) (iv) Nursing staff worked in small, decentralised teams, caring for patients in a cluster of eight to ten rooms, rather than one large team. Some staff liked this, but others felt that the mutual support they had experienced on multi-bed wards could not always be relied on in the new hospital (v) Single-room wards impacted on the ability to support, train and develop staff as more difficult to supervise junior staff (vi) There were also fewer opportunities for informal learning than on multi-bed wards

(iv) Nursing staff worked in small, decentralised teams, caring for patients in a cluster of eight to ten rooms, rather than one large team. Some staff liked this, but others felt that the mutual support they had experienced on multi-bed wards could not always be relied on in the new hospital (v) Single-room wards impacted on the ability to support, train and develop staff as more difficult to supervise junior staff (vi) There were also fewer opportunities for informal learning than on multi-bed wards "It can be a bit difficult sometimes, if you're really stuck and, you know, I've been hanging out of rooms calling for a nurse sometimes, but if somebody's in another room, you can't see them. […] [If] you really need a nurse, or you really need somebody to come and help you, then you have to go through all the rooms to try and find them […] So if you do need assistance […] we often just press the call buzzers ourselves.” S14—Healthcare assistant, surgical ward “Well, I suppose sometimes, on a really busy day, you can feel a bit isolated […] I know you shouldn't, but at times it does make you put your own health at risk. I've done that with my own back, you just think, ‘Oh I can't find anyone, [the patient] desperately needs the toilet, I'm going to help them.’” OP23—Healthcare assistant, older people ward ”There's a feeling that it's almost like three separate wards, in a way. You're very much self-contained within your own team. Whereas before, I think there was far more interaction between nurses.” S16—staff nurse, surgical ward “That is a direct result of the environment really, because you can't eyeball nurses enough to know that they're drowning and they need help.” M06—Nurse, MAU “You overhear someone working behind a curtain and you pick up and you think, ‘That was a really nice thing they did for that patient. Maybe I'll try that.’ I think that's definitely missing, picking up on things from each other that way, because […] it's not as easy to hear how they interact with people.” M05—junior sister, MAU MDT, multidisciplinary team.

curtain and you pick up and you think, ‘That was a really nice thing they did for that patient. Maybe I'll try that.’ I think that's definitely missing, picking up on things from each other that way, because […] it's not as easy to hear how they interact with people.” M05—junior sister, MAU MDT, multidisciplinary team. Following the move, staff reported that single rooms improved privacy, dignity and confidentiality for patients and were better for visitors. Single rooms were perceived to facilitate communication with patients, with reduced interruptions and allowed more personalised care. The single rooms provided an improved working environment for care delivery, being modern, clean and spacious, and the en-suite bathrooms made personal care easy to deliver. Staff also noted the improved ward layout and felt the single-room environment reduced the risk of infection (table 2).

ed more personalised care. The single rooms provided an improved working environment for care delivery, being modern, clean and spacious, and the en-suite bathrooms made personal care easy to deliver. Staff also noted the improved ward layout and felt the single-room environment reduced the risk of infection (table 2). Analysis of pre-move and post-move survey data found staff perceived statistically significant improvements in the efficiency of the physical environment, the patient amenity, the effect of the environment on infection control, patient privacy, and family and visitors. The largest increases were found for perceptions of infection control and patient privacy (see table 3). The new wards were perceived by staff to support good hygiene practices; however, there was a perception by infection control staff that single rooms induced a degree of complacency in relation to basic precautions and there was a perceived need to reinforce infection control procedures and protocols. Basic infection control principles were revisited to counter the belief that single rooms protected from hospital-acquired infections. Perceptions of the effect of the single-room environment on teamwork and staff's ability to deliver high-quality care were significantly more negative after the move than before (see table 3). Table 3 Results of t tests comparing staff perceptions of the ward environment pre and post move

Analysis of pre-move and post-move survey data found staff perceived statistically significant improvements in the efficiency of the physical environment, the patient amenity, the effect of the environment on infection control, patient privacy, and family and visitors. The largest increases were found for perceptions of infection control and patient privacy (see table 3). The new wards were perceived by staff to support good hygiene practices; however, there was a perception by infection control staff that single rooms induced a degree of complacency in relation to basic precautions and there was a perceived need to reinforce infection control procedures and protocols. Basic infection control principles were revisited to counter the belief that single rooms protected from hospital-acquired infections. Perceptions of the effect of the single-room environment on teamwork and staff's ability to deliver high-quality care were significantly more negative after the move than before (see table 3). Table 3 Results of t tests comparing staff perceptions of the ward environment pre and post move Scale Phase Mean† SD t p-Value Efficiency of physical environment Pre 2.81 0.67 −3.346 0.001* Post 3.24 0.70 Ability to deliver high-quality care Pre 3.20 0.66 2.59 0.011* Post 2.88 0.67 Staff amenity Pre 2.95 0.67 −0.373 0.710 Post 2.99 0.73 Patient amenity Pre 2.82 0.64 −5.52 <0.001* Post 3.43 0.50 Layout supports infection control Pre 2.75 1.11 −8.39 <0.001* Post 4.25 0.73 Patient privacy and confidentiality Pre 2.54 1.15 −10.14 <0.001* Post 4.31 0.59 Teamwork and training Pre 3.55 0.64 4.34 <0.001* Post 2.96 0.77 Patient safety Pre 3.23 0.69 −7.32 0.466 Post 3.32 0.67 Staff safety Pre 3.19 0.68 0.67 0.502 Post 3.10 0.75 Interaction with family/visitors Pre 2.74 0.77 −8.60 <0.001* Post 3.90 0.63 *p<0.05. †Higher scores on the scales indicate more positive responses than lower scores.

ing Pre 3.55 0.64 4.34 <0.001* Post 2.96 0.77 Patient safety Pre 3.23 0.69 −7.32 0.466 Post 3.32 0.67 Staff safety Pre 3.19 0.68 0.67 0.502 Post 3.10 0.75 Interaction with family/visitors Pre 2.74 0.77 −8.60 <0.001* Post 3.90 0.63 *p<0.05. †Higher scores on the scales indicate more positive responses than lower scores. Staff rated the post-move single-room accommodation as significantly lower for their ability to deliver high-quality care (table 3). A notable disadvantage that staff highlighted at interview was the reduced visibility offered by the single-room design, making patient surveillance difficult. Staff identified various obstacles to safe and efficient working, one unique to this particular hospital design (in-board bathrooms that protrude into corridors obscuring views into more than one room at a time). Staff saw their greatest challenge as monitoring and keeping patients safe, especially those at risk of falls. Although staff found the initial increased rate of falls distressing, their rating of patient safety in the survey was not significantly different from pre move (table 3). All staff interviewed perceived a lack of visibility of patients in the single-room wards had contributed to an initial increase in falls in the new hospital. Experienced nursing staff felt it easier to prevent falls in shared patient accommodation because they could ‘keep an eye’ on patients and were more aware of warning signals, for example, patients attempting to get up from their chair or bed. Nurses had to work differently and adapt their working practices to ensure all patients were seen regularly, requiring teamwork with colleagues and the initiation of regular hourly intentional rounds. Other successful examples of new ways of working included supervisory roles for situation awareness; falls interventions and starting a lunch club in a shared social space for patients on the older people's ward to combat isolation and improve food intake at lunchtimes. Nurses found time management and prioritisation of workloads challenging in the new environment and struggled to divide their attention between all the patients they were caring for (they had found this easier in shared patient accommodation).

der people's ward to combat isolation and improve food intake at lunchtimes. Nurses found time management and prioritisation of workloads challenging in the new environment and struggled to divide their attention between all the patients they were caring for (they had found this easier in shared patient accommodation). Staff recognised that they required different strategies to enable them to divide their time between patients and feel satisfied that they were giving all patients sufficient personalised care (again easier on multi-bedded wards), but this challenge had largely been left to individuals to resolve, with reportedly limited success and associated dissatisfaction (table 2). Trial and error was a feature of innovations, with staff teams trying, for example, different ways of preventing falls and different configurations of decentralised teams.

ards), but this challenge had largely been left to individuals to resolve, with reportedly limited success and associated dissatisfaction (table 2). Trial and error was a feature of innovations, with staff teams trying, for example, different ways of preventing falls and different configurations of decentralised teams. Social isolation was perceived by staff to be a real disadvantage for both staff and patients. Staff also felt that the quality of teamwork they had enjoyed on shared accommodation wards was difficult to recreate. Nurses described seeing less of their colleagues, being unaware of what was happening in other parts of the ward and sometimes feeling isolated. Patients were nursed in three groups of 10 patients by sub-teams of nursing staff. Staff adapted to these decentralised nursing teams, but information exchange within wider nursing teams was perceived to be worse. Locating colleagues to obtain assistance was one of the main difficulties described by staff and was largely unresolved by new ways of working and by use of communication technology (table 2). Pedometer data showed an increase in the mean number of steps per hour for all wards and staff groups (table 4). Table 4 Mean steps per hour by wards, pre/post move and staff group

Social isolation was perceived by staff to be a real disadvantage for both staff and patients. Staff also felt that the quality of teamwork they had enjoyed on shared accommodation wards was difficult to recreate. Nurses described seeing less of their colleagues, being unaware of what was happening in other parts of the ward and sometimes feeling isolated. Patients were nursed in three groups of 10 patients by sub-teams of nursing staff. Staff adapted to these decentralised nursing teams, but information exchange within wider nursing teams was perceived to be worse. Locating colleagues to obtain assistance was one of the main difficulties described by staff and was largely unresolved by new ways of working and by use of communication technology (table 2). Pedometer data showed an increase in the mean number of steps per hour for all wards and staff groups (table 4). Table 4 Mean steps per hour by wards, pre/post move and staff group Pre-new build Post-new build No. of observation sessions Mean Range SD No. of observation sessions Mean Range SD Ward Maternity 14 630 380–1007 194 8 687 463–1008 211 Surgical 17 653 354–996 152 22 793 419–1274 247 Older people 23 664 361–965 158 21 845 553–1229 193 MAU 19 773 479–1007 181 16 880 469–1311 254 Staff group Midwife 7 475 380–640 94 6 583 463–683 99 RN 32 639 354–1007 178 29 827 419–1311 240 HCA 34 768 470–1007 131 32 853 469–1274 220 Total 73 683 354–1007 175 67 817 419–1311 231 HCA, healthcare assistant; RN, registered nurse.

e 23 664 361–965 158 21 845 553–1229 193 MAU 19 773 479–1007 181 16 880 469–1311 254 Staff group Midwife 7 475 380–640 94 6 583 463–683 99 RN 32 639 354–1007 178 29 827 419–1311 240 HCA 34 768 470–1007 131 32 853 469–1274 220 Total 73 683 354–1007 175 67 817 419–1311 231 HCA, healthcare assistant; RN, registered nurse. The modelling, which adjusted for ward, staff group and day of week, showed a significant increase from a mean of 716 (95% CI 660 to 771) before the move to a mean of 839 (95% CI 787 to 892) after the move (F(1, 83)=10.36, p=0.002). HCAs (mean 836, 95% CI 776 to 895) walked significantly further than RNs (mean 719, 95% CI 667 to 772, F(1,83)=8.01, p=0.006), but there was no significant difference between wards in the distance walked (F(2, 89)=2.38, p=0.099). The effect of the move was not moderated by either ward (F(2,82)=0.64, p=0.53) or staff group (F(1,76)=3.48, p=0.066). Analysis of the time and motion observation data showed an increase in the proportion of direct care, indirect care, professional communication and medication tasks and a decrease in ward-related activities such as cleaning and bed-making, but none of these were statistically significant. There were, however, fewer interruptions during tasks and work was less fragmented. There were no significant differences in staff well-being and stress post move.

munication and medication tasks and a decrease in ward-related activities such as cleaning and bed-making, but none of these were statistically significant. There were, however, fewer interruptions during tasks and work was less fragmented. There were no significant differences in staff well-being and stress post move. Overall, when asked their preference, most staff said they would prefer wards with a mix of single rooms and multi-bedded accommodation both before and after the move. After the move, only 18% (n=10) of staff indicated a preference for 100% single rooms (figure 1). The most common preference, before (38% n=21) and after (40% n=22) the move, was a combination of 50% of beds in single rooms and 50% in bays. More staff in the pre-move survey reported a preference for more beds in bays (36%, n=20) than in the post-move phase (22%, n=12). Figure 1 Nurse preferences for single-room or multi-bed accommodation.

Overall, when asked their preference, most staff said they would prefer wards with a mix of single rooms and multi-bedded accommodation both before and after the move. After the move, only 18% (n=10) of staff indicated a preference for 100% single rooms (figure 1). The most common preference, before (38% n=21) and after (40% n=22) the move, was a combination of 50% of beds in single rooms and 50% in bays. More staff in the pre-move survey reported a preference for more beds in bays (36%, n=20) than in the post-move phase (22%, n=12). Figure 1 Nurse preferences for single-room or multi-bed accommodation. Impact on patient safety There were few changes in safety outcomes that might be attributed to the move to single rooms. Only 5 of 15 SPC charts across the 5 outcomes from 3 wards at the single-room hospital showed special cause variation following the move (bolded in table 5). Increases of falls and medication errors in the MAU immediately following the move were found but rates decreased to previous levels 9 months after the move. No equivalent change was observed in the ‘new-build’ control hospital. Increases in C. difficile infection, falls and pressure ulcers post move were observed in the older people's ward. Again, no equivalent change in outcomes was observed in the new-build control hospital. However, a reduction of about 50% in length of stay (LOS) indicates a substantial change in case mix on the older people’s ward and so these outcomes could not clearly be attributed to the move or the environment. Table 5 Safety and infection control outcomes before and after the move

Impact on patient safety There were few changes in safety outcomes that might be attributed to the move to single rooms. Only 5 of 15 SPC charts across the 5 outcomes from 3 wards at the single-room hospital showed special cause variation following the move (bolded in table 5). Increases of falls and medication errors in the MAU immediately following the move were found but rates decreased to previous levels 9 months after the move. No equivalent change was observed in the ‘new-build’ control hospital. Increases in C. difficile infection, falls and pressure ulcers post move were observed in the older people's ward. Again, no equivalent change in outcomes was observed in the new-build control hospital. However, a reduction of about 50% in length of stay (LOS) indicates a substantial change in case mix on the older people’s ward and so these outcomes could not clearly be attributed to the move or the environment. Table 5 Safety and infection control outcomes before and after the move Before After 1–11 months 12–20 months 21–29 months 30–36 months 100% single room MAU LOS Per 1.000 patient-days 1.21 1.03 1.49 1.3 Falls 3.9 (2.5 to 6.1) 2.3 (1.2 to 4.4) 13.9 (11.3 to 17) 0.7 (0.2 to 1.9) Med. errors 1.3 (0.6 to 2.8) 0.5 (0.1 to 1.9) 8.2 (6.2 to 10.6) 0.7 (0.2 to 1.9) Older people's ward LOS 36.17 40.14 20.77 19.01 Falls 2.8 (2.1 to 3.9) 2.4 (1.6 to 3.6) 9.5 (7.8 to 11.7) 13.3 (11 to 16.2) Pressure ulcers 0.4 (0.2 to 0.9) 0.2 (0 to 0.7) 3.3 (2.3 to 4.7) 3.5 (2.4 to 5.2) Clostridium difficile 0.1 (0 to 0.4) 0.1 (0 to 0.4) 0.2 (0 to 0.8) 0.5 (0.2 to 1.4) Control new-build hospital MAU LOS 0.78 0.91 0.95 0.98 Falls 4.1 (1.3 to 11.1) 2.1 (0.5 to 6.6) 7.0 (3.9 to 12.2) 7.5 (2.8 to 18.4) Med. errors 4.6 (2.4 to 8.4) 11.8 (7.1 to 19.2) 7.0 (3.9 to 12.2) 0.1 (0 to 7.1) Older people's ward LOS 8.16 8.36 7.53 7.86 Falls 6.9 (4.6 to 10.4) 9.7 (7.3 to 12.7) 11.2 (9.3 to 13.5) 12.6 (9.1 to 17.4) Pressure ulcers 4.0 (2.7 to 5.9) 3.6 (2.2 to 5.6) 1.4 (0.8 to 2.4) 2.9 (1.4 to 5.7) Clostridium difficile 1.6 (0.9 to 2.8) 1.1 (0.4 to 2.5) 0.3 (0.1 to 0.9) 0.3 (0 to 2.1) Steady-state control hospital MAU LOS 6.88 6.37 5.96 5.42 Falls 23.8 (21.2 to 26.7) 14.6 (12.4 to 17.2) 17.8 (15.2 to 20.8) 20.2 (15.5 to 26.3) Med. errors N/A N/A N/A N/A Older people's ward LOS 21.98 36.59 35.88 45.64 Falls 29.7 (25.9 to 34) 17.4 (14 to 21.5) 12.1 (9.7 to 15.1) 5.9 (3.4 to 10.1) Pressure ulcers 1.4 (0.7 to 2.7) 1.8 (0.9 to 3.5) 0.5 (0.1 to 1.4) 0.4 (0 to 2.7) Clostridium difficile 0.9 (0.3 to 2) 0.2 (0 to 1.3) 0.2 (0 to 1) 0.8 (0.1 to 3.4) LOS, length of stay; MAU, medical assessment unit.

OS 21.98 36.59 35.88 45.64 Falls 29.7 (25.9 to 34) 17.4 (14 to 21.5) 12.1 (9.7 to 15.1) 5.9 (3.4 to 10.1) Pressure ulcers 1.4 (0.7 to 2.7) 1.8 (0.9 to 3.5) 0.5 (0.1 to 1.4) 0.4 (0 to 2.7) Clostridium difficile 0.9 (0.3 to 2) 0.2 (0 to 1.3) 0.2 (0 to 1) 0.8 (0.1 to 3.4) LOS, length of stay; MAU, medical assessment unit. Although nurses perceived increased falls as a ‘single-room issue’ because of diminished visibility, the temporary nature of the increase in Tunbridge Wells Hospital and lack of a similar pattern in the new-build/mixed accommodation control site suggest that the observed adverse outcomes are not directly associated with single rooms. One explanation might be the disruption from the move to a new environment and the need to adjust work patterns, before patterns of falls settled down again. Our staff interview data suggest some of the necessary adaptations that were made in the 100% single rooms (eg, use of technology, non-slip socks and improved assessment to prevent falls). To minimise falls, the rooms had non-slip flooring and in-board bathrooms with bathroom doors on the bed head wall with hand rails (to prevent falls risk by patients walking across an open floor to get to the bathroom). However, as outlined above, the in-board bathrooms meant that staff sight lines were limited to one patient.

ise falls, the rooms had non-slip flooring and in-board bathrooms with bathroom doors on the bed head wall with hand rails (to prevent falls risk by patients walking across an open floor to get to the bathroom). However, as outlined above, the in-board bathrooms meant that staff sight lines were limited to one patient. Impact on costs An all-single-room hospital requires more floor space for wards, but space requirements for other areas are the same. Interviews with architects and construction experts suggested that the cost of building an all-single-hospital in the UK is probably no more than 5% higher than a 50% single-room hospital. There was no evidence of difference in maintenance costs per square metre, nor of an increase in the cost of preparing and serving meals related to single rooms. We modelled the cleaning costs for a 500-bed hospital with a 100% and 50% single-room design (table 6). Total annual costs for cleaning ward areas were 53% higher in the all-single-room design and added approximately £1050 per bed per year for cleaning. For a Trust with approximately 500 beds, this cost would be approximately 0.14% of annual budget. We analysed the impact of a number of other costs, such as catering, falls, length of stay and infections.37 However, it was impossible to attribute any observed differences to a ‘single-room effect’, and these are not therefore included here.

oximately 500 beds, this cost would be approximately 0.14% of annual budget. We analysed the impact of a number of other costs, such as catering, falls, length of stay and infections.37 However, it was impossible to attribute any observed differences to a ‘single-room effect’, and these are not therefore included here. Comparing nurse staffing data in the 19 months before and after the move up to March 2013, there was an overall increase in whole time equivalent (WTE) for nurses and also a change in the skill mix of staff (a slight increase in proportion of RNs). Overall, under the planned investment at Tunbridge Wells Hospital, there was a 0.9% decrease in beds (from 791 to 784 beds), a 3% increase in WTE staff (from 1793 to 1847 WTE) and a 2.7% increase in total staffing costs (from £60 878 000 to £62 548 000). Attributing how much of the planned investment was the result of a ‘single-room effect’ is, however, not possible. In some areas, including maternity, the changes were part of an overall review of nursing levels and the general restructuring of patient pathways. While there may have been some impact on operational costs arising from increased numbers of staff, this is partially due to the change in number of beds, the cost of staff time and change in the skill mix of staff. There is an opportunity cost associated with time spent walking; this time is not available for other activities such as spending time with patients or administrative activities. Table 6 Impact on costs

Comparing nurse staffing data in the 19 months before and after the move up to March 2013, there was an overall increase in whole time equivalent (WTE) for nurses and also a change in the skill mix of staff (a slight increase in proportion of RNs). Overall, under the planned investment at Tunbridge Wells Hospital, there was a 0.9% decrease in beds (from 791 to 784 beds), a 3% increase in WTE staff (from 1793 to 1847 WTE) and a 2.7% increase in total staffing costs (from £60 878 000 to £62 548 000). Attributing how much of the planned investment was the result of a ‘single-room effect’ is, however, not possible. In some areas, including maternity, the changes were part of an overall review of nursing levels and the general restructuring of patient pathways. While there may have been some impact on operational costs arising from increased numbers of staff, this is partially due to the change in number of beds, the cost of staff time and change in the skill mix of staff. There is an opportunity cost associated with time spent walking; this time is not available for other activities such as spending time with patients or administrative activities. Table 6 Impact on costs Cost item Single-bedroom hospital Space requirement More space required for wards but not for other areas Building costs per bed 5% higher in 100% single-bedroom hospital than in a 50% single-room hospital Cleaning costs per bed 53% higher in single bedrooms (extra £1050 per bed per year) Number of beds Decreased from 791 to 784 (−0.9%) Nursing staff WTE Increased from 1793 to 1847 WTE (3%) Nursing staff costs Increased from £60.88 million to 62.55 million (2.7%) WTE, whole time equivalent.

in a 50% single-room hospital Cleaning costs per bed 53% higher in single bedrooms (extra £1050 per bed per year) Number of beds Decreased from 791 to 784 (−0.9%) Nursing staff WTE Increased from 1793 to 1847 WTE (3%) Nursing staff costs Increased from £60.88 million to 62.55 million (2.7%) WTE, whole time equivalent. Discussion A disadvantage of single rooms for approximately a third of patients was a sense of isolation and loneliness confirming a recent survey in Scotland.38 Patients missed opportunities for interaction with other patients, confirming previous findings.7 9 23 Patients and staff noted the potential for loneliness, boredom, loss of shared experience and absence of distraction and social interaction offered by multi-bedded wards. Dayrooms were rarely used, and patients were only orientated to their rooms. However, on balance, most patients in our study preferred single rooms. Comfort, control, privacy and en-suite bathrooms were important aspects that influenced this preference.

distraction and social interaction offered by multi-bedded wards. Dayrooms were rarely used, and patients were only orientated to their rooms. However, on balance, most patients in our study preferred single rooms. Comfort, control, privacy and en-suite bathrooms were important aspects that influenced this preference. Nurses needed to adapt their working practices significantly and felt ill prepared for working on single-room wards, resulting in trial and error of new approaches to care. Staff walking distances increased, and a number of aspects of care, including providing adequate surveillance, were challenging. Staff preference remained for a mix of single rooms and bays, and our findings suggest that a move to all single rooms may have significant implications for the nature of teamwork in the longer term, confirming evidence that suggests 100% single-room facility design made team communication and patient monitoring difficult, and that it limited social interaction among staff.39 Our findings suggest single rooms have implications for teamwork and informal learning (ie, observing how colleagues handle situations/role modelling), something not previously identified in the literature. Future research is needed to explore different room designs and ward layouts and consider the long-term impact of single-room working on staff, including the nature of teamwork and informal learning.

(ie, observing how colleagues handle situations/role modelling), something not previously identified in the literature. Future research is needed to explore different room designs and ward layouts and consider the long-term impact of single-room working on staff, including the nature of teamwork and informal learning. Staff interviews highlighted concern about loss of panoptic surveillance of patients, which was seen as a major disadvantage of the new wards, and linked to the perception that falls had increased because of single rooms. While our results suggested that any increases in falls could be explained by changes in case mix, previous evidence suggests single rooms could mean less surveillance by staff and increased rates of slips, trips and falls40 41 and falls that are unnoticed by staff or other patients.8 9 21 Restricted line of sight into the room and the linear ward layout may have increased walking by nursing staff and reduced efficiency, which confirms other research.42 With a different design (eg, observation points and large glazed windows and doors), such adverse effects on staff-to-patient observation might be mitigated.9 Design features of wards with single rooms and technology need to be maximised to ensure that privacy and comfort for patients does not compromise staff's ability to monitor them.

esign (eg, observation points and large glazed windows and doors), such adverse effects on staff-to-patient observation might be mitigated.9 Design features of wards with single rooms and technology need to be maximised to ensure that privacy and comfort for patients does not compromise staff's ability to monitor them. The temporary nature of falls increases, and lack of a similar pattern in the control wards, which also experienced an increase in single rooms, suggests that it is not an inevitable result of single rooms. This conclusion must also be interpreted in the light of an overall increase in falls at the hospital level associated with changing patient-level risk factors (an increase of comorbidities associated with service reconfigurations). While we did see a sustained increase in falls in the older people's ward, a strong correlation between changes in the fall rate and in patient-level risk factors associated with service reconfigurations makes it difficult to conclude that single rooms are the cause. However, the possibility that some groups, such as older people, are adversely affected cannot be discounted.

er people's ward, a strong correlation between changes in the fall rate and in patient-level risk factors associated with service reconfigurations makes it difficult to conclude that single rooms are the cause. However, the possibility that some groups, such as older people, are adversely affected cannot be discounted. The discrepancies between patient and staff views in terms of preference for single rooms are notable; on balance, two-thirds of patients we interviewed preferred single rooms while only 18% of staff indicated a preference for 100% single rooms. Most staff said they would prefer a mix of single rooms and multi-bedded accommodation, to allow them more panoptic surveillance of very sick and frail elderly patients in particular. A mix of accommodation that allows flexibility or accommodation that can be changed to adapt to patient needs, with sliding glass walls for example, may be a solution. Key for patients was access to an en-suite bathroom, which for many tipped the balance in favour of single rooms, despite some patients experiencing loneliness and isolation. If future innovation in single-room design can enhance visibility for staff, while maintaining access to en-suite bathrooms and facilitate more communal spaces for patients to interact, then staff and patient preferences may become more closely aligned.

ooms, despite some patients experiencing loneliness and isolation. If future innovation in single-room design can enhance visibility for staff, while maintaining access to en-suite bathrooms and facilitate more communal spaces for patients to interact, then staff and patient preferences may become more closely aligned. Strengths and limitations Studying one specific hospital in depth through four case study wards provides rich in-depth data generating important learning for future new hospital builds. Sampling four different wards provides a range of ages; conditions, hospital length of stay and a range of contexts in which to understand staff work experiences and patient experiences of care. However, we were not able to interview patients who were very sick or who had dementia or delirium and we did not interview carers/relatives.

fferent wards provides a range of ages; conditions, hospital length of stay and a range of contexts in which to understand staff work experiences and patient experiences of care. However, we were not able to interview patients who were very sick or who had dementia or delirium and we did not interview carers/relatives. The 152-item survey was designed specifically for this study. Apart from the adapted Teamwork and Safety Climate Survey and some staff well-being questions from the NHS staff survey, the survey was, however, not standardised. Items were generated by reviewing the literature and a pool of potential items were reviewed and revised by a group of health services researchers and pilot tested with a group of 20 nurses. Our controlled before-and-after design provides weak causal inference, but our cautious approach—use of SPC charts within units followed by close scrutiny of other sources of data, including case mix, to assess whether cause was plausible and likely—means that we are unlikely to make incorrect attributions. This is in stark contrast to most previous studies where simple before-and-after comparisons have been used. We used aggregated routine data without individual risk adjustment for patient safety because of the available data (monthly reports of rates), and so a regression-based analysis with individual risk adjustment was not possible. While such routine data are problematic because of coding and reporting bias, our approach to analysis, a time-series-based approach at unit level, using SPC charts, was most appropriate for the data we were able to collect43 and protected against many potential biases.

with individual risk adjustment was not possible. While such routine data are problematic because of coding and reporting bias, our approach to analysis, a time-series-based approach at unit level, using SPC charts, was most appropriate for the data we were able to collect43 and protected against many potential biases. We could not make a causal link between single-bed rooms and changes in the rate of falls, so the cost of falls was not included in the economic analysis, although we recognise that these costs are potentially high. As noted above, differences in healthcare resourcing, cultural and workforce expectations and organisational systems mean it should not be assumed that our findings are necessarily generalisable to other countries.

s was not included in the economic analysis, although we recognise that these costs are potentially high. As noted above, differences in healthcare resourcing, cultural and workforce expectations and organisational systems mean it should not be assumed that our findings are necessarily generalisable to other countries. Conclusion This study is one of very few to have examined in depth the experiences of patients and staff in single-room hospital accommodation and provides valuable evidence to guide policy, planning and hospital design. Based on our data, there appear to be no strong economic or safety reasons for choosing 100% single rooms in the UK. Our findings suggest it would be beneficial for managers planning a similar move to encourage staff to prepare and rehearse for working in single rooms, to plan for a possible increase in nurse staffing, to review and reinforce infection control and falls policies for single-room working and to actively manage and monitor changes in work practices and inpatient experiences before and after any changes. Nurses in our study preferred a mix of single rooms and shared patient accommodation. The majority of patients in our study preferred single rooms, but consideration needs to be given to the minority of patients who experienced isolation in single rooms as well as those very sick patients and frail older people, including those with dementia and delirium whom we did not interview. The possibility that some groups, such as frail older people, are adversely affected cannot be discounted. It is therefore important to give careful consideration to the likely patient population when thinking about the mix of accommodation types in other new hospital builds.

tia and delirium whom we did not interview. The possibility that some groups, such as frail older people, are adversely affected cannot be discounted. It is therefore important to give careful consideration to the likely patient population when thinking about the mix of accommodation types in other new hospital builds. Supplementary Material Web supplement Correction notice: This article has been corrected since it was published Online First. Table 5 title has been corrected.

tia and delirium whom we did not interview. The possibility that some groups, such as frail older people, are adversely affected cannot be discounted. It is therefore important to give careful consideration to the likely patient population when thinking about the mix of accommodation types in other new hospital builds. Supplementary Material Web supplement Correction notice: This article has been corrected since it was published Online First. Table 5 title has been corrected. Contributors: JM led the overall study design, led and undertook data collection, analysis, final report writing and preparation of this manuscript . PG contributed to overall study design; led the comparator study design and analysis, contributed to report writing and gave final approval of manuscript. CP contributed to overall study design, led and conducted data collection and analysis of case study qualitative data, contributed to report writing, and gave final approval of manuscript. MS conducted data collection and analysis of comparator study, drafted that section of this paper and gave final approval of manuscript. JEA contributed to overall study design, led the case study quantitative analysis and report write up and gave final approval of manuscript. GR contributed to overall study design and report write up and gave final approval of manuscript. EP conducted data collection, analysed and wrote up the costing study with JB, and gave final approval of manuscript. JH undertook additional data analysis of case study qualitative data, contributed to report write up and gave final approval of manuscript. TM contributed to overall study design, quantitative analysis and report write up and gave final approval of manuscript. JB contributed to overall study design; worked with EP on the cost study design, analysis and write up, contributed to report writing and gave final approval of manuscript.

al approval of manuscript. TM contributed to overall study design, quantitative analysis and report write up and gave final approval of manuscript. JB contributed to overall study design; worked with EP on the cost study design, analysis and write up, contributed to report writing and gave final approval of manuscript. Funding: Department of Health–National Institute for Health Research–Health Services and Delivery Research (HS&DR) Programme 10/1013/42. Competing interests: None declared. Disclaimer: The views and opinions expressed are those of the authors and do not necessarily reflect those of the [Health Services and Delivery Research Programme], NIHR, NHS or the Department of Health. Ethics approval: King’s College London Psychiatry, Nursing and Midwifery Research Ethics Committee (PNM/09/10-30). Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Interventions to combat health service delivery problems (such as hospital-acquired infections) are often developed in response to a heightened public awareness and mounting pressure to tackle them. Under these circumstances, a groundswell of public and professional opinion may be the stimulus for both a spontaneous change across a health system and formal evaluations of particular interventions within that system. Service delivery interventions are often complex in the sense that they are made up of a number of components, many of which may not be novel and which, unlike pharmaceuticals, are not restricted by licensing requirements. The result is that interventions of various types diffuse into widespread practice in an uncontrolled way while evaluation studies are under way. For example, concerns over hospital-acquired infections may lead hospitals across the system to adopt methods to improve hand hygiene and these same concerns may also stimulate formal research studies to evaluate specific interventions with the same aim. Insofar as these various interventions are effective, they produce a positive secular trend. We shall use the metaphor of a ‘rising tide’ as a short hand for such a secular trend that is contemporaneous with the evaluation of an intervention. Such a rising tide may obscure the measured effect of an intervention in a study with contemporaneous controls. Appreciating the possibility of a rising tide offers additional insight for interpreting null results where both control and intervention sites have improved.

poraneous with the evaluation of an intervention. Such a rising tide may obscure the measured effect of an intervention in a study with contemporaneous controls. Appreciating the possibility of a rising tide offers additional insight for interpreting null results where both control and intervention sites have improved. This paper aims to illustrate the rising tide phenomenon in which this might explain a null result where both intervention and contemporaneous control sites have improved. We discuss evidence that may help distinguish between a rising tide and alternative explanations for the null result, and illustrate this approach with examples. Temporal trends versus other explanations for improvement across intervention and control sites The possibility of a ‘rising tide’ explanation arises when a controlled study with baseline measurement(s) yields a null result in which there has been improvement across both intervention and control sites. Various criteria can be put forward to help distinguish a rising tide from other explanations for such simultaneous improvement—we offer these in the spirit of Bradford Hill's famous criteria for cause–effect explanations in clinical research.1 Leaving aside the play of chance (which will have been calibrated statistically), the probability of the rising tide explanation increases in proportion to evidence for the existence of a rising tide and declines in proportion to evidence supporting rival explanations.

r cause–effect explanations in clinical research.1 Leaving aside the play of chance (which will have been calibrated statistically), the probability of the rising tide explanation increases in proportion to evidence for the existence of a rising tide and declines in proportion to evidence supporting rival explanations. Evidence for a rising tide, from strongest to weakest, consists of the following: Data showing that improvement similar to that in study sites occurred across the healthcare system as a whole. Such external data may be derived from regular population surveys, national registries, or routine administrative databases and provide direct evidence of a positive secular trend. Data showing that the intervention and control sites within a study had started to improve before the intervention came on stream, so that it was a continuation of a trend in both intervention and control sites. Qualitative evidence, say in the form of interviews with staff, showing strong motivation to improve practices in both intervention and control sites. Circumstantial evidence in the form of press articles, government reports, and/or documents from national societies showing that the topic was one of pervading concern.

Data showing that the intervention and control sites within a study had started to improve before the intervention came on stream, so that it was a continuation of a trend in both intervention and control sites. Qualitative evidence, say in the form of interviews with staff, showing strong motivation to improve practices in both intervention and control sites. Circumstantial evidence in the form of press articles, government reports, and/or documents from national societies showing that the topic was one of pervading concern. Contamination is the most immediate rival explanation for simultaneous improvement in both intervention and control groups. Contamination is used here in the standard epidemiological sense that control sites become aware of the intervention and replicate it to some degree,2 3 thereby diluting the estimated effect; the direction of effect is from intervention sites to control sites within a study, biasing results towards the null. The intervention ‘leaks’ from intervention to control sites and must follow allocation to intervention and control conditions. A rising tide, by contrast, impacts on all sites in a system, irrespective of whether they are or are not included in the study and it may precede allocation of intervention and control groups. Contamination should be suspected when it can be demonstrated that participants in the control group were exposed to elements of the intervention that had ‘spilled over’ from the intervention group within the study (rather than from outside).

n the study and it may precede allocation of intervention and control groups. Contamination should be suspected when it can be demonstrated that participants in the control group were exposed to elements of the intervention that had ‘spilled over’ from the intervention group within the study (rather than from outside). It is possible for other sources of bias (see table 1) to create or exaggerate the appearance of an improvement in the control group or even to create the illusion of improvement in the intervention group, when in fact it was mainly or only the control group that had improved. Bias could arise, for example, if there was higher dropout from control than intervention sites or if controls were subject to selection bias. Table 1 Issues to be considered for assessing a rising tide phenomenon and results of assessment for the four case studies*

It is possible for other sources of bias (see table 1) to create or exaggerate the appearance of an improvement in the control group or even to create the illusion of improvement in the intervention group, when in fact it was mainly or only the control group that had improved. Bias could arise, for example, if there was higher dropout from control than intervention sites or if controls were subject to selection bias. Table 1 Issues to be considered for assessing a rising tide phenomenon and results of assessment for the four case studies* SPI2 Critical pathways EQHIV MERIT Positive evidence Direct evidence Improvement in process and/or outcome measures observed in external sites: Yes Yes Yes Yes Timing: before or during evaluation study Before and during Before and during Before During System-wide or specific external site(s) System-wide Specific external sites System-wide System-wide (but 30% participation) Qualitative evidence showing behaviour changes driven by external factors in both study groups Yes Yes Yes Yes Suggestive evidence Baseline measures better than expected, or already showing high standards or improving trend Yes Yes Yes Yes Circumstantial evidence Heightened awareness of the problems Yes Yes Yes Yes Negative evidence† Contamination within study No No Unlikely Unlikely Other potential sources of biases‡ Not apparent Not apparent Attrition bias cannot be ruled out Not apparent *Improvement in process and/or outcome measures were observed in both intervention and control groups in these studies during the evaluation period.

e evidence† Contamination within study No No Unlikely Unlikely Other potential sources of biases‡ Not apparent Not apparent Attrition bias cannot be ruled out Not apparent *Improvement in process and/or outcome measures were observed in both intervention and control groups in these studies during the evaluation period. †Factors of which the impact on study findings could resemble a rising tide phenomenon. ‡Including selection bias (eg, control group being a selective sample of highly motivated units or having more headroom for improvement), bias in outcome assessment (eg, changes in methods of data collection or coding over time) and attrition bias (eg, poor-performing units dropping out and being excluded from analysis). Examples of a putative rising tide phenomenon In this section we provide four examples from published literature in which a rising tide phenomenon may be suspected. We briefly describe the key features of these studies and illustrate how the criteria mentioned above and listed in table 1 can be applied to help inform a judgement on the likelihood of a rising tide explanation versus alternative explanations.

m published literature in which a rising tide phenomenon may be suspected. We briefly describe the key features of these studies and illustrate how the criteria mentioned above and listed in table 1 can be applied to help inform a judgement on the likelihood of a rising tide explanation versus alternative explanations. Our first example, the Safer Patients Initiative phase 2 (SPI2) study, was a controlled before-and-after evaluation of a multicomponent hospital clinical safety programme.4 5 Many dimensions of quality measured in the study improved over the intervention period (spanning from March 2007 to September 2009), but did so equally in both intervention and control groups (figure 1). One of the targets of the intervention was to improve recognition of deteriorating patients in general wards, and the quality of nursing observations (as judged from masked review of the notes) improved markedly and statistically significantly over the study period, but no difference was observed in the rate of improvement across intervention and control sites. Likewise, use of hand washing materials improved over time but at a similar pace across sites. There was evidence of improving standards of monitoring in control and intervention sites (which started before the intervention was implemented).5 There were widespread national initiatives to improve the standard of monitoring on the wards,6 and external evidence showed increased use of hand wash materials and reduced infection rates across the whole of England over the study period.7 8 Contamination, in the sense described above, is very unlikely—controls were recruited retrospectively and data were obtained retrospectively from case notes and routine data. For these reasons, the controls would not have been aware that they were controls at the time of intervention. This is an example of an arguably unusual situation where there is specific strength in retrospective selection of control sites.

tively and data were obtained retrospectively from case notes and routine data. For these reasons, the controls would not have been aware that they were controls at the time of intervention. This is an example of an arguably unusual situation where there is specific strength in retrospective selection of control sites. Figure 1 Key characteristics and findings of the Safer Patients Initiatives phase 2 (SPI2). This controlled before-and-after study evaluated a multi-component intervention using an organisation-wide approach to improve patient safety. The key components included interventions to facilitate generic improvement in the hospital system to reduce adverse events (such as building a good leadership to support a culture of safety as well as interventions targeting specific clinical processes that carry a relatively high risk of adverse events (such as procedures aiming to enhance infection control). Various outcomes were measured, including staff morale, culture and opinion, the quality of acute medical care and perioperative care, use of consumables for hand hygiene, adverse events and hospital mortality in older patients with acute respiratory disease, intensive care unit mortality, infection rates associated with healthcare and patients satisfaction. SPI2 was preceded by a pilot phase (SPI1) that provides data on the pre-implementation phase for certain end-points, including the two outcomes illustrated here.

rtality in older patients with acute respiratory disease, intensive care unit mortality, infection rates associated with healthcare and patients satisfaction. SPI2 was preceded by a pilot phase (SPI1) that provides data on the pre-implementation phase for certain end-points, including the two outcomes illustrated here. The Critical Pathway Program was an initiative started in 1993 in the Brigham and Women's Hospital (Boston, USA) to improve efficiency in service delivery for high-cost, high-volume surgical procedures.9 A controlled before-and-after evaluation for its application in colectomy, total knee replacement, and coronary artery bypass graft surgery showed substantial and statistically significant reductions in the average length of hospital stay for all three procedures in both intervention and control sites. Data from the 2 years before intervention suggested that length of stay had started to decline in both intervention and control hospitals before the intervention was initiated in the former (figure 2), and external nation-wide US data showed a continuous decrease in average length of hospital admission spanning the period of the Critical Pathways Intervention, from 9.1 days in 1990 to 7.8 days in 1995 and 7.0 days in 1999.10 Staff interviews at control hospitals provided evidence that competitive pressure, rather than contamination, had triggered efforts to reduce length of stay and improve efficiency.

ital admission spanning the period of the Critical Pathways Intervention, from 9.1 days in 1990 to 7.8 days in 1995 and 7.0 days in 1999.10 Staff interviews at control hospitals provided evidence that competitive pressure, rather than contamination, had triggered efforts to reduce length of stay and improve efficiency. Figure 2 Key characteristics and findings of the Critical Pathways Intervention. This controlled before-and-after study compared the effects of a quality improvement initiative utilising critical pathway framework on post-operative length of stay in a large teaching hospital with concurrent data from 2-3 similar neighbouring hospitals without the intervention. Multidisciplinary teams identified critical steps in the care process and specified required actions and desirable outcomes for each step. Patients entered into critical pathways were monitored and various methods of benchmarking and feedback were used in pathway management. The primary outcome was post-operative length of stay. Hospital charges and other process and clinical outcomes were also examined.

uired actions and desirable outcomes for each step. Patients entered into critical pathways were monitored and various methods of benchmarking and feedback were used in pathway management. The primary outcome was post-operative length of stay. Hospital charges and other process and clinical outcomes were also examined. EQHIV was a controlled before-and-after study evaluating the effectiveness of a suite of interventions to improve the quality of care in clinics treating HIV-infected patients.11 Among the outcome measures, the proportion of patients whose viral load was adequately suppressed increased significantly within each group—by a greater extent in the intervention group (11%; from 41% to 52%) than the control group (6%; from 44% to 50%). However, the between-group difference was not statistically significant (p=0.18). Compliance with a prescription guideline was already high at baseline and did not increase further in either group after the intervention (figure 3). National data from the HIV Cost and Services Utilization Study showed that EQHIV was preceded by significant improvement in care of HIV-infected adults.12 Interview of clinical directors in study sites suggested minimal contamination, as those in control sites reported many fewer quality improvement initiatives compared with intervention sites. However, attrition bias cannot be ruled out, as only 63% (25/40) of selected control sites provided sufficient data to be included in analysis.

inical directors in study sites suggested minimal contamination, as those in control sites reported many fewer quality improvement initiatives compared with intervention sites. However, attrition bias cannot be ruled out, as only 63% (25/40) of selected control sites provided sufficient data to be included in analysis. Figure 3 Key characteristics and findings of the EQHIV study. This controlled before-and-after study evaluated the effectiveness of the ‘Breakthrough Series’, a multi-institutional quality improvement collaborative led by Institute for Healthcare Improvement, on improving the quality of case for clinics treating HIV infected patients. The 16-month intervention involved a series of meetings (learning sessions) covering the theory and practice of quality improvement in the intervention clinics and sharing ideas and progress between them. The primary outcome measures were rates of optimal antiretroviral therapy use and control of HIV viral load.

ients. The 16-month intervention involved a series of meetings (learning sessions) covering the theory and practice of quality improvement in the intervention clinics and sharing ideas and progress between them. The primary outcome measures were rates of optimal antiretroviral therapy use and control of HIV viral load. MERIT was a cluster randomised controlled trial of the effectiveness of emergency teams for deteriorating non-terminal hospital patients in reducing the combined outcome of cardiac arrests without a pre-existing not-for-resuscitation order, unplanned intensive care unit admissions, and unexpected deaths.13 Before the intervention began, the incidence of the outcome appeared to have already improved from 26 per 1000 admissions estimated in a previous study,14 to 6.6 and 7.1 per 1000 admissions observed at baseline for intervention and control hospitals, respectively.13 15 Further improvement was observed in both intervention and control groups after the intervention, with no significant difference between groups (reduction of 0.39 vs 1.41 per 1000 admission for intervention vs control, p=0.30). Similar findings were observed for secondary outcomes (figure 4). External evidence of a secular trend and widespread adoption of medical emergency teams comes from a national registry in which about 30% of all intensive care units provided relevant data.16 The risk of contamination was minimised by agreement of control hospitals not to publicise the intervention internally and not to change the operation of their cardiac arrest team during the study period.

gency teams comes from a national registry in which about 30% of all intensive care units provided relevant data.16 The risk of contamination was minimised by agreement of control hospitals not to publicise the intervention internally and not to change the operation of their cardiac arrest team during the study period. Figure 4 Key characteristics and findings of the MERIT study. This cluster randomised controlled trial (RCT) investigated the effectiveness of introducing a medical emergency team (MET) that could be summoned when non-terminal hospital patients showed signs of physiological instability and deterioration. Staff in the intervention hospitals were trained over 4-months to identify and respond to patients requiring the attention of the team. The primary outcome was the combined incidence of cardiac arrests without a pre-existing not-for-resuscitation order. unplanned intensive care unit admissions and unexpected deaths measured before and after implementation. In summary, there is evidence for a secular trend in all four cases. The case for a rising tide is strongest for SPI2 and MERIT where data from both within and outside the study pointing towards a system-wide secular trend and evidence for alternative explanations can largely be ruled out. For the remaining cases, there is some uncertainty, mainly arising from lack of evidence to eliminate alternative explanations. On the whole, the evidence (summarised in table 1) indicates that a secular trend is likely to have contributed to the null results observed in all four studies.

ions can largely be ruled out. For the remaining cases, there is some uncertainty, mainly arising from lack of evidence to eliminate alternative explanations. On the whole, the evidence (summarised in table 1) indicates that a secular trend is likely to have contributed to the null results observed in all four studies. Discussion What causes a rising tide? Widespread concern about an issue such as hospital-acquired infection or medication error may motivate multiple changes throughout a system that includes, but is not limited to, sites involved in a research study. Exactly how these changes are propagated is a large subject for study, save to say that human behaviour is strongly influenced by prevailing social attitudes and practice.17 Two points can be made about the phenomenon of the spread of behaviour in a community of practitioners: It is not necessary to postulate that the way in which organisations respond to social ‘forces’ is the same everywhere. Services may be improved in a number of separate ways,18 and improvement across the system might arise from intervention and non-intervention sites adopting the same practices, separate practices of similar efficiency, or a mixture of similar and different practices. An analogy of the multifarious ways that social forces may cause a rising tide is shown in box 1. The intervention group is also subject to the rising tide. The measured intervention effect in a study inclines towards the null if the effect of the intervention attenuates with increasing ‘dose’ and/or if the headroom for further improvement is consumed.

s that social forces may cause a rising tide is shown in box 1. The intervention group is also subject to the rising tide. The measured intervention effect in a study inclines towards the null if the effect of the intervention attenuates with increasing ‘dose’ and/or if the headroom for further improvement is consumed. A rising tide can only produce a null result if there is at least some temporal overlap between widespread promulgation of the interventions and the evaluation of a particular intervention. However, it is not necessary for system-wide change and research to start simultaneously and such timing is unlikely given the lag in establishing research projects. Indeed improvement originated before the study got underway in three of the four above examples (table 1).

nd the evaluation of a particular intervention. However, it is not necessary for system-wide change and research to start simultaneously and such timing is unlikely given the lag in establishing research projects. Indeed improvement originated before the study got underway in three of the four above examples (table 1). Box 1 Evolutionary analogy of the rising tide phenomenon A naturalist studying desert fauna may notice that they have certain features in common. It might first be observed that mice are sand-coloured. Then that the snakes, lizards and small birds are all of similar hue. The naturalist may observe that this would increase their efficiency as both prey and/or predator (both, in the case of the snake). Under the same external influence (changes in physical environment), organisms evolve similar outcomes (a sandy colour) by different means (distinct biochemical pathways). So it is that, under the same primary driver (changes in the social environment), organisations evolve similar outcomes (fewer infections) by different means (such as promoting hygienic practices and screening all patients for resistant bacteria on admission).

colour) by different means (distinct biochemical pathways). So it is that, under the same primary driver (changes in the social environment), organisations evolve similar outcomes (fewer infections) by different means (such as promoting hygienic practices and screening all patients for resistant bacteria on admission). Detecting a rising tide explanation A rising tide phenomenon is, in essence, a pronounced secular trend created by social responses to a particular issue which has gained widespread attention. While it is impossible to find incontrovertible proof of a rising tide explanation, we have assembled a set of criteria that should be taken into account in the interpretation of controlled evaluations that have generated a null result associated with similar improvement in both intervention and control groups (table 1). Indeed, possible influence of a secular trend was mentioned or alluded to by the authors of all the case examples we presented here. It must be emphasised that a rising tide does not preclude a positive result. The intervention may augment widespread contemporaneous change because the intervention is different (at least in part), and/or administered with different intensity. For instance, in a controlled evaluation of a team training programme for operating room personnel, a statistically significant reduction in risk-adjusted surgical mortality rate was observed, despite a 7% decrease in annual mortality rate in the control group.19 A tide may also recede, in which case a successful intervention may be one that arrests decline, but this would be manifested as a positive result, not a null result.

cally significant reduction in risk-adjusted surgical mortality rate was observed, despite a 7% decrease in annual mortality rate in the control group.19 A tide may also recede, in which case a successful intervention may be one that arrests decline, but this would be manifested as a positive result, not a null result. Rising tide in applied health research Situations analogous to the rising tide phenomenon can occur in a variety of applied health research—for example, in a trial of screening for prostate cancer, where a substantial proportion of the population (and hence the control group) underwent screening.20 Likewise, an educational package for general practitioners to apply more intensive antidepressant treatment was evaluated at a time when the idea was already getting national publicity.21 Many other examples can be found in the realm of service delivery interventions.22 23 Most recently, the rising tide phenomenon is likely to have contributed to the null findings from two independent analyses of the effect of participation in the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP), where mortality and certain other outcomes improved in both intervention and control groups,24 25 and in the English Matching Michigan study where the rate of decline in central venous catheter bloodstream infection following the introduction of the intervention in intensive care units was not significantly different from a concurrent temporal trend.23

outcomes improved in both intervention and control groups,24 25 and in the English Matching Michigan study where the rate of decline in central venous catheter bloodstream infection following the introduction of the intervention in intensive care units was not significantly different from a concurrent temporal trend.23 Does it matter? We have described a set of criteria to help decide whether a null result in the face of improving outcomes can be attributed to a rising tide (table 1). One subset of criteria concerns a convincing alternative explanation, particularly contamination. It could be argued that a null result needs no further explanation once one is satisfied that it has been measured with sufficient precision and decided that an alternative explanation, such as contamination, can be excluded. Contrary arguments are now given based on two rather distinct philosophical traditions. We draw attention to a distinction made by Schwartz and Lellouch26 between pragmatic and explanatory motivations for a study. The former consists of generating information to inform a particular prespecified decision, and the second consists of generating an understanding of causal mechanisms. A null result in the face of a rising tide fulfils the first, but not the second, requirement. It fulfils the first (pragmatic) requirement because, if a study designed (and powered) around the decision makers’ requirements is assumed, an incremental effect size sufficient to justify the marginal costs of the intervention is excluded. However, the second (explanatory) requirement is unsatisfied, since it does not indicate what the effect of the study intervention would be in a system that was not experiencing a positive temporal trend. In such a system, the intervention would not be ‘competing’ with other positive changes in the system.

is excluded. However, the second (explanatory) requirement is unsatisfied, since it does not indicate what the effect of the study intervention would be in a system that was not experiencing a positive temporal trend. In such a system, the intervention would not be ‘competing’ with other positive changes in the system. The second philosophical argument turns on the idea that it is wrong to make decisions based solely on a statistical convention,27 as pointed out in Sir Bradford Hill's famous lecture.1 To put this another way, data should contribute to an understanding of causal mechanisms (theory), and the rising tide may help explain why an intervention that was expected to prove effective yielded a null result. Recommendations for future practice Having discussed the idea of a secular trend phenomenon, we propose here some options that can be considered alongside established guidelines28–31 during the design of evaluation studies for service and policy interventions in order to facilitate correct interpretation of study findings. In many cases, at least some of the study end points will be available from routine administrative databases or independent surveys regularly carried out nationally. This will allow verification of whether a change observed in the evaluation study is associated with the study participation itself or is similarly observed elsewhere outside the study, thereby providing strong evidence of a secular trend, at least as far as shared end points are concerned. This was the case in the SPI2 study.

l allow verification of whether a change observed in the evaluation study is associated with the study participation itself or is similarly observed elsewhere outside the study, thereby providing strong evidence of a secular trend, at least as far as shared end points are concerned. This was the case in the SPI2 study. Qualitative data may provide evidence to explain study results;5 32 in the case of SPI2, behaviour change was driven by factors in the external environment in both intervention and control sites. Obtaining multiple measurements spanning the pre- and post-intervention period —that is, a controlled interrupted time series.29 Multiple observations before the intervention phase may provide evidence of long-term secular trends in both control and intervention groups.33 34 Prior to the start of data collection, the sample size can be adjusted to take account of secular trends when these are expected. Such analysis can be used to assess the feasibility and value of an evaluation study before it is commissioned, or to inform a decision on whether to extend an ongoing study by increasing its size or to terminate it on grounds of ‘futility’.35 36

an be adjusted to take account of secular trends when these are expected. Such analysis can be used to assess the feasibility and value of an evaluation study before it is commissioned, or to inform a decision on whether to extend an ongoing study by increasing its size or to terminate it on grounds of ‘futility’.35 36 Considering designs that allow temporal effects to be modelled. One example is a step wedge design,37 which uses randomisation as a method to determine the order in which centres on a waiting list receive the intervention. It has many logistical, political and even ethical advantages over a parallel design,28 38 39 and (given a sufficiently large sample) also allows the intervention effect, general temporal effects, and any effect on the intervention at the time it was introduced to be modelled. Conclusion Social pressure that triggers the development and evaluation of a service delivery intervention may at the same time drive spontaneous, widespread changes in a health system leading to improvement across the board, which we describe here as a rising tide. Controlled evaluation studies undertaken amidst a rising tide may yield a null result because incremental effects are similar between intervention and non-intervention sites. Recognition of a rising tide is important because, while the null result demonstrates pragmatically that the intervention does not produce sufficient incremental benefit in this particular scenario, it leaves open the possibility that the intervention could work in a different scenario where a rising tide is absent.

s. Recognition of a rising tide is important because, while the null result demonstrates pragmatically that the intervention does not produce sufficient incremental benefit in this particular scenario, it leaves open the possibility that the intervention could work in a different scenario where a rising tide is absent. In this paper we offer four case studies of evaluations of complex interventions to illustrate a rising tide phenomenon, and suggest a framework to assess evidence either supporting or refuting its presence. Our aim is to raise awareness of the phenomenon and of its potential implications in the design and interpretation of evaluation studies. Further work to gather empirical evidence on the occurrence of such a phenomenon and to develop methods to delineate its impact from other bias is the next step. This in turn will provide guidance for health services researchers and decision makers on the optimal actions to take in the face of a rising tide. Contributors: RJL conceived the idea for the paper, wrote the first draft, led the writing of the paper and is the guarantor. Y-FC compiled the case examples and helped to draft the paper. KH and AJS read and commented on drafts and provided critical insight into the development of the paper.

In this paper we offer four case studies of evaluations of complex interventions to illustrate a rising tide phenomenon, and suggest a framework to assess evidence either supporting or refuting its presence. Our aim is to raise awareness of the phenomenon and of its potential implications in the design and interpretation of evaluation studies. Further work to gather empirical evidence on the occurrence of such a phenomenon and to develop methods to delineate its impact from other bias is the next step. This in turn will provide guidance for health services researchers and decision makers on the optimal actions to take in the face of a rising tide. Contributors: RJL conceived the idea for the paper, wrote the first draft, led the writing of the paper and is the guarantor. Y-FC compiled the case examples and helped to draft the paper. KH and AJS read and commented on drafts and provided critical insight into the development of the paper. Funding: RJL and Y-FC have received funding from the UK National Institute for Health Research (NIHR) through the Collaboration for Leadership in Applied Health Research and Care West Midlands (CLAHRC WM) programme. RJL and KH had financial support from the Medical Research Council (MRC) Midland Hub for Trials Methodology Research (grant No G0800808). Disclaimer: The views expressed here are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: RJL was the principal investigator for the evaluation study of SPI2 described in this paper.

Funding: RJL and Y-FC have received funding from the UK National Institute for Health Research (NIHR) through the Collaboration for Leadership in Applied Health Research and Care West Midlands (CLAHRC WM) programme. RJL and KH had financial support from the Medical Research Council (MRC) Midland Hub for Trials Methodology Research (grant No G0800808). Disclaimer: The views expressed here are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: RJL was the principal investigator for the evaluation study of SPI2 described in this paper. Provenance and peer review: Not commissioned; externally peer reviewed.

The drive for innovation Across health systems there is a drive to roll out innovative models of care that will deliver better value for money and improve the quality of care. Innovation in service delivery has been defined as ‘a novel set of behaviours, routines, and ways of working that are discontinuous from previous practice, directed at improving health outcomes, administrative efficiency, cost effectiveness or users’ experience and that are implemented by planned and coordinated actions’.1 Undertaking this type of innovation at scale is increasingly viewed as crucial to the long-term sustainability of health systems. In the USA, the Affordable Care Act has provided a legislative framework that promotes innovation in service delivery. Accountable care organisations (ACOs) have emerged from this as innovative payment and delivery models that aim to improve the coordination and quality of care, enhance population health while containing the growth in healthcare costs.2 In England,3 4 the Health and Social Care Act has made innovation in the provision of health services a statutory duty and a major initiative to drive innovation at scale is underway.5 Fifty ‘Vanguard’ sites are now starting to act as test beds for multicomponent innovations in service delivery (see table 1), supported by a £200 million transformation fund from National Health Service England. Table 1 New models of care: the NHS Vanguards5

uded meaningful progress with this QI intervention. Interestingly, resolution of the persistent issues raised in the HEADS-UP briefings appeared to depend less on each one's inherent safety threat, than on agreement between clinical staff and managers of the need for change. (Incentives and actions for middle managers) The organisation focused heavily on clinical quality, investing in external consultants to help develop new clinical services and improve the efficiency of existing ones. Board members also dedicated substantial time to clinical quality issues. In fact, clinical quality was the focus of numerous committees and subcommittees, with a complex, devolved governance structure. However, the regulatory environment did not allow for a self-determined quality strategy, with quality priorities established largely by the local healthcare commissioning body and a national quality inspectorate. Board-level meetings were awash with clinical quality metrics, among which the ‘softer data’ emerging from the HEADS-UP briefings had a less certain place.96 As the organisation moved to merge with another institution, fewer resources were available for continuous QI in the interim. (Incentives and actions for senior managers)

In the USA, the Affordable Care Act has provided a legislative framework that promotes innovation in service delivery. Accountable care organisations (ACOs) have emerged from this as innovative payment and delivery models that aim to improve the coordination and quality of care, enhance population health while containing the growth in healthcare costs.2 In England,3 4 the Health and Social Care Act has made innovation in the provision of health services a statutory duty and a major initiative to drive innovation at scale is underway.5 Fifty ‘Vanguard’ sites are now starting to act as test beds for multicomponent innovations in service delivery (see table 1), supported by a £200 million transformation fund from National Health Service England. Table 1 New models of care: the NHS Vanguards5 Integrated primary and acute care systems Eight test bed sites joining up general practice, hospital, community and mental health services Enhanced health in care homes Six test bed sites offering older people better, joined up health, care and rehabilitation services Multispecialty community providers Fourteen test bed sites focused on moving specialist outpatient and ambulatory care out of hospitals into the community Urgent and emergency care Eight test bed sites developing new approaches to simplify and improve the coordination of services and reduce pressure on emergency departments Acute care collaborations Test bed sites (yet to be announced) will link together local hospitals to improve their clinical and financial viability NHS, National Health Service.

ght test bed sites developing new approaches to simplify and improve the coordination of services and reduce pressure on emergency departments Acute care collaborations Test bed sites (yet to be announced) will link together local hospitals to improve their clinical and financial viability NHS, National Health Service. The danger of innovation without evaluation The sheer scale of this initiative serves as a timely reminder that innovation without adequate evaluation can lead to misattribution of effects and worse, the wider adoption of technologies, practices and ways of working without proven benefits over existing alternatives. Health systems, and the NHS in particular, can ill afford compromised decision-making in relation to continuation or wider spread on such a scale. Previous large-scale policy initiatives such as the drive to rapidly implement telehealth technologies despite known uncertainties relating to complexity, costs and benefits6–8 have led to what might be considered inappropriate allocation of finite resources. While there is some recognition that previous roll out and testing of service innovation has been suboptimal4 and that there is a need to establish evidence on ‘what works’,9 this on its own will not be sufficient.

costs and benefits6–8 have led to what might be considered inappropriate allocation of finite resources. While there is some recognition that previous roll out and testing of service innovation has been suboptimal4 and that there is a need to establish evidence on ‘what works’,9 this on its own will not be sufficient. Service innovation at scale is inherently complex and context dependent; what may appear to be successful in one setting may not work the same way in a different context.1 10 The inter-relationship between the mix of intervention, process, workforce and system changes and the contextual setting within which they are introduced is likely to generate both intended and unintended consequences. Understanding this complexity and its impact on apparent outcomes necessitates more nuanced approaches to evaluation than are currently used. To date, this level of understanding has been lacking in early evaluations of the spending and performance of ACOs,11 12 which suggest modest reductions in total Medicare expenditures and differential improvements in the quality of care. Yet the underlying mechanisms within each ACO are complex and evolving and whether or how these will continue to impact on quality and costs in a sustained and positive way or whether they can be reproduced elsewhere is less clear.13

s in total Medicare expenditures and differential improvements in the quality of care. Yet the underlying mechanisms within each ACO are complex and evolving and whether or how these will continue to impact on quality and costs in a sustained and positive way or whether they can be reproduced elsewhere is less clear.13 What type of evaluation is needed? Approaches that are context-sensitive and that address the what, why, how, where and for whom of innovation implementation are necessary if we are to be able to recognise and reproduce ‘success’ at the service level. There is always a tension between rigorous evaluation and ‘good enough’ evidence,14 15 and there will always be a trade-off between what evidence is desirable and what is possible. Nevertheless, those implementing complex service innovations should be considering what sort of outcome and process evaluation is necessary from the outset; to do otherwise, would be to reduce value and increase waste.16

14 15 and there will always be a trade-off between what evidence is desirable and what is possible. Nevertheless, those implementing complex service innovations should be considering what sort of outcome and process evaluation is necessary from the outset; to do otherwise, would be to reduce value and increase waste.16 Prospective pathways for undertaking rigorous outcome and process evaluation of complex and large-scale interventions have been well defined by the UK Medical Research Council (MRC).17–19 The aim of these frameworks is to help researchers and funders recognise and adopt appropriate methods to evaluate the systematic and planned introduction of complex interventions. But there are recognised weaknesses within the existing MRC framework, notably the time and resources needed to operationalise it fully.20 Others have criticised the focus on the intervention rather than on the dynamic influence of the context into which an intervention is introduced or from which it may emerge.21 This lack of acknowledgement that innovation can be unintended as well as intended raises the possibility that other alternative frameworks may be better suited to guiding more nuanced approaches to innovation development and evaluation.

he context into which an intervention is introduced or from which it may emerge.21 This lack of acknowledgement that innovation can be unintended as well as intended raises the possibility that other alternative frameworks may be better suited to guiding more nuanced approaches to innovation development and evaluation. The IDEAL framework The IDEAL framework was developed to provide a systematic and incremental pathway to aid the transfer of surgical innovations into practice.22–25 The framework presents a five-phase approach to innovation implementation encompassing: idea, development, exploration, assessment and long-term study (see table 2). Although broadly following the MRC framework in its latter phases, IDEAL better recognises the planned and unplanned nature of innovation and the need to adapt methods to the innovation process rather than doing the opposite. Table 2 Phases of the IDEAL framework22

The IDEAL framework The IDEAL framework was developed to provide a systematic and incremental pathway to aid the transfer of surgical innovations into practice.22–25 The framework presents a five-phase approach to innovation implementation encompassing: idea, development, exploration, assessment and long-term study (see table 2). Although broadly following the MRC framework in its latter phases, IDEAL better recognises the planned and unplanned nature of innovation and the need to adapt methods to the innovation process rather than doing the opposite. Table 2 Phases of the IDEAL framework22 Idea Initial report Innovation may be planned, accidental or forced Focus on explanation and description Development Rapid iterative modification of innovation Small experience from one centre Focus on technical details and feasibility Exploration Innovation now more stable Replication by others Focus on assessing potential benefits and adverse effects Assessment Innovation gaining wide acceptance Considered as possible replacement for current treatment, ways of working Focus on comparative evaluation against current standards or best practice Long-term monitoring Long-term surveillance of the innovation Focus on monitoring late and rare problems, changes in use IDEAL was developed to confront problems with the evaluation of surgical procedures but as the uncontrolled introduction of innovations is not specific to surgery, there is increasing interest in its potential value beyond this field. The Food and Drug Administration has been working with IDEAL to explore how best to advance the infrastructure and methodology for evaluating medical devices.26 In England, IDEAL is also being promoted as a tool to help commissioners (payers) to consider requests for funding for novel services and make other resource allocation decisions (about interventions not routinely funded) in a more objective and systematic way.27 28 IDEAL therefore has the potential to be adapted and applied to a wide range of situations in which complex innovations are in need of rigorous development and evaluation.

r novel services and make other resource allocation decisions (about interventions not routinely funded) in a more objective and systematic way.27 28 IDEAL therefore has the potential to be adapted and applied to a wide range of situations in which complex innovations are in need of rigorous development and evaluation. With new models of care some interventions, processes and ways of working are likely to be planned from the outset but others may emerge by accident or occur through the unintended consequences of planned innovation. A focus on explaining early intentions and describing what actually happens over time is therefore key. The idea and development phases of the IDEAL framework recognises this and advocates the iterative development and evaluation of innovations to define and test whether it is likely to succeed in a particular setting and allow for adaptation, refinement and system integration before more widespread and rigorous testing is pursued.23

development phases of the IDEAL framework recognises this and advocates the iterative development and evaluation of innovations to define and test whether it is likely to succeed in a particular setting and allow for adaptation, refinement and system integration before more widespread and rigorous testing is pursued.23 With this initial focus on iterative development, IDEAL also recognises that rapid cycle evaluative approaches that traditionally feature in quality improvement initiatives may be most appropriate in these early phases.29 Common to these approaches is the generation of real-time data driven learning and decision-making. This makes them ideally suited to addressing questions such as ‘Should we do this again?’ and ‘Should we pursue this further?’ Approaches such as plan-do-study-act and statistical process control also have the benefit of being widely used as part of efforts to improve the quality and safety of care across health systems. But if iterative evaluation of this type is to be pursued, well documented shortcomings in planning, execution, analysis and reporting of these methods have to be avoided or their effects minimised.30–32

the benefit of being widely used as part of efforts to improve the quality and safety of care across health systems. But if iterative evaluation of this type is to be pursued, well documented shortcomings in planning, execution, analysis and reporting of these methods have to be avoided or their effects minimised.30–32 And this is where one of the key strengths of IDEAL can best be harnessed; that of transparency through registration and reporting. IDEAL recommends that all surgical innovations are prospectively registered, and that it is unethical not to do so.22 In England, infrastructure already exists to facilitate the prospective registration of innovations although the extent to which it is used is unclear, and it may not be intended for service innovations.33 For example, at present it is unclear whether a comprehensive register exists of areas of England where ‘integrated care’ in various forms and at different scales is being attempted. Prospective registration will aid transparency by providing a permanent record of how an innovation was intended to be introduced, whether it was adapted and how it was ultimately implemented in a given context.

l working roles to maintaining and enhancing the performance of the organisation, which itself recognises this commitment in supporting and encouraging high quality care’.28 This working definition makes it clear that real engagement is a very different entity to staff acquiescence, for which it is often confused.29 30 In QI, however, clinical engagement has been summarised simply as staff's ‘active involvement’,31 with no recognition of the possible dialogue between clinicians and those seeking to improve their performance. That no organisational contribution is expected may go some way to explaining why clinical engagement has been problematic. Although there is specific literature pertaining to physician engagement, in this discussion (unless specified otherwise) we group clinical healthcare professionals together. QI interventions are typically interdisciplinary, and securing greater engagement of a single staff group is not an end in itself, only a step towards an ‘organisational culture where all staff feel valued and involved’.30

sts of areas of England where ‘integrated care’ in various forms and at different scales is being attempted. Prospective registration will aid transparency by providing a permanent record of how an innovation was intended to be introduced, whether it was adapted and how it was ultimately implemented in a given context. IDEAL also recommends that the early development and evaluation of any innovation should be adequately reported (regardless of ‘success’ or ‘failure’). Insights and contextual information which could inform wider spread and learning should be systematically captured in a standardised format. There are now a vast number of reporting frameworks available but given the focus on reporting initial innovation development, refinement and implementation, the SQUIRE2 guidelines may be the most appropriate.34 Although designed for quality improvement, SQUIRE2 recognises that innovation can be complex, multidimensional and is inherently context dependent. With a focus on explaining issues such as ‘Why did you start?’, ‘What did you do?’, ‘What did you find?’ and ‘What does it mean?’, SQUIRE2 encourages more nuanced reporting of what happened over time and reflection on the meaning of the outcomes and events that were observed.

sional and is inherently context dependent. With a focus on explaining issues such as ‘Why did you start?’, ‘What did you do?’, ‘What did you find?’ and ‘What does it mean?’, SQUIRE2 encourages more nuanced reporting of what happened over time and reflection on the meaning of the outcomes and events that were observed. Beyond the idea and development phases, IDEAL recommends that plans for wider spread should be accompanied by robust and independent comparative evaluation. The small-scale iterative evaluations of the early phases can only ever give tentative indications of benefit and can be prone to misinterpretation and bias; as such they should only ever represent a starting point. Further corroboration, explanation and evaluation should be sought along the IDEAL pathway and prospective comparative quasi-experimental designs that will generate economic, outcome and process data should be the preferred method of evaluation. Here IDEAL recognises and adopts the methods advocated by the MRC for evaluating complex interventions.17–19

and evaluation should be sought along the IDEAL pathway and prospective comparative quasi-experimental designs that will generate economic, outcome and process data should be the preferred method of evaluation. Here IDEAL recognises and adopts the methods advocated by the MRC for evaluating complex interventions.17–19 Adapting IDEAL As IDEAL was originally devised for surgical innovations it does have shortcomings when considered in relation to other complex and emergent innovations. One such shortcoming is the failure to advocate the use of clearly articulated theory to explain how and why an innovation would be expected to deliver benefits over existing practices. This may be due in part to the fact that in surgery the theory is usually implicit in the innovation itself. But with increasing complexity (surgical or otherwise) theory becomes an essential lens through which we can predict, identify and describe the key features that will influence the implementation of any innovation.35 36 Particularly important is ensuring that there is clarity about what the innovation is together with clarity over the wider system, process and contextual features that will be need to be addressed if its causal mechanism is to function as planned. Encouragingly, use of logic models is being promoted and constructed for each Vanguard site. But this clarity over purpose should also inform the selection of evaluation methods that reflect an innovation's phase of development.37 Applying theory in the idea and development phases should help to focus attention on those evaluation strategies that are best equipped to capture innovation adaptation and refinement over time. Doing so will also offer a more efficient and meaningful method to generalise and predict the potential for successful replication in other settings.

dea and development phases should help to focus attention on those evaluation strategies that are best equipped to capture innovation adaptation and refinement over time. Doing so will also offer a more efficient and meaningful method to generalise and predict the potential for successful replication in other settings. Another major shortcoming with IDEAL is consideration of cost. Although IDEAL incorporates cost effectiveness in its latter phases, it is absent from the idea and development phases. Yet innovation (large or small) is not without cost or consequences. There will be opportunity costs as innovation competes with other health activity and services for finite resources. Determining whether the expected gains in health benefits from innovation implementation will exceed those lost as other activities are displaced is therefore crucial.38 This tension has been perfectly illustrated by a recent analysis of the impact of making routine hospital services available 6 days a week in England; the costs of the proposed solution reportedly outweigh the health benefits that are likely to be realised.39 Early assessment therefore may help to identify and eliminate cost-ineffective innovations. At the very least, innovators should be encouraged to systematically capture and report resource use and costs associated with early stage implementation from a clearly stated perspective—especially important if judgements are to be made on wider spread.

may help to identify and eliminate cost-ineffective innovations. At the very least, innovators should be encouraged to systematically capture and report resource use and costs associated with early stage implementation from a clearly stated perspective—especially important if judgements are to be made on wider spread. Conclusions An adapted IDEAL framework offers a pathway for prospective longitudinal evaluation of the Vanguard test bed sites and similar complex and emergent innovations. Adopting a common analytical framework will encourage iterative development, reporting and evaluation of innovations from the outset and before more widespread and rigorous testing is pursued. Contributors: This paper is based on a discussion paper produced by PMW, RB and GH for the MRC/NIHR Research Methodology Panel. PMW wrote the first draft of this paper and RB and GH commented on this and subsequent drafts. All authors approved the final version and PMW is the guarantor. Funding: PMW, RB and GH all receive funding from the he NIHR Collaboration for Leadership in Applied Health Research and Care Greater Manchester. The views expressed in this article are those of the authors and not necessarily those of the NHS, NIHR or the Department of Health. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Repeated calls have been made to engage and involve patients and the public and to place them at the centre of healthcare. Serious clinical and service failings in the UK1 2 and internationally3–5 increase the urgency and importance of addressing this problem.6–8 Developing stronger patient and public involvement (PPI) in the organisation and delivery of healthcare is now central to health reform across Western economies.9–11 This recognition reflects evidence that patients and the wider public can be involved and make a difference at most stages of healthcare and in service planning and delivery.12 This, however, does not mean that all patients choose to be involved or indeed should have to be responsible for monitoring care, or indeed may not even be the most reliable way to do this, given their vulnerable condition.13 Despite this supportive policy context, progress to achieve greater involvement is patchy and slow and often concentrated at the lowest levels of involvement. By this we mean that consultation is more often the norm, than collaboration.10 14–16 Some healthcare professionals and organisations have not embraced the idea of partnership with patients and even feel threatened by the notion of active involvement.10 17 Though individuals, teams and organisations may be interested and deeply committed to involving patients and family members, they may lack clarity about what the issues are, who to involve and the goals of involvement.18 19

of partnership with patients and even feel threatened by the notion of active involvement.10 17 Though individuals, teams and organisations may be interested and deeply committed to involving patients and family members, they may lack clarity about what the issues are, who to involve and the goals of involvement.18 19 Methods This article drew upon a selective narrative review20 of various sources of information and evidence connected to PPI. This was not meant to be a systematic review. We searched for literature up to March 2016 and omitted any literature published before 1969. The search was guided by the authors’ broad expertise and experience covering a range of disciplines from social work, health and social care, policy and research, clinical care and quality improvement. We used recent published systematic literature reviews to identify relevant authors and publications. Google and hand searches of journal articles and reference lists and reports augmented identification of recent evidence. Expert advice was sought from some cited authors. We selected literature that provided an overview of a range of arguments and methods about the benefits and difficulties with involvement and discussed conclusions. Personal experience in writing peer-reviewed publications in this field informed the analysis and synthesis of the overview.

ought from some cited authors. We selected literature that provided an overview of a range of arguments and methods about the benefits and difficulties with involvement and discussed conclusions. Personal experience in writing peer-reviewed publications in this field informed the analysis and synthesis of the overview. Results A growing body of evidence suggests that patients can be involved and contribute to healthcare in various ways: from helping to reach an accurate diagnosis, choosing an appropriate treatment, management strategy or safe provider, ensuring treatment is properly adhered to and monitored and identifying adverse events and side effects and acting upon them.12 21 Involving patients, their families and the public can also have a number of benefits: improving patient choice, self-care and shared decision-making (SDM) contributing to research partnerships and changes to service delivery and patient outcomes.19 22 23 The involvement process has also been seen as an important way in healthcare systems of enhancing democratic principles and accountability.24 25 However, PPI often appears to be trapped in a vicious cycle. Uncertainty exists about why and how to do involvement well and how to involve and support a diversity of patients and the public, rather than a few selected individuals. The reality of implementation is complex and yields suboptimal evidence of impact.22 26 This fuels the cycle of predictable and disappointing results and exposes PPI to criticisms of exclusivity and tokenism.16 27–29 This article presents our reflections on these issues and explains why we think changing the balance of power, promoting empowerment, diversity and equality, and strengthening evaluation of outcomes and impact are the neglected aspects of involvement that, if given due attention, can offer a way to break the cycle.

–29 This article presents our reflections on these issues and explains why we think changing the balance of power, promoting empowerment, diversity and equality, and strengthening evaluation of outcomes and impact are the neglected aspects of involvement that, if given due attention, can offer a way to break the cycle. The purpose and value of involvement Different words, theories and approaches have emerged from disparate social movements, policies and practices to describe the involvement process,30 for example, consultation, engagement, participation, partnership or co-production. These have sometimes been used to imply a greater or lesser level of involvement, power or influence in decision-making processes within an organisation. However, this language does not always reflect the underlying ethos of these involvement activities.31 In the absence of consensus on terminology,30 we define involvement as an activity that is done ‘with’ or ‘by’ patients or members of the public rather than ‘to’, ‘about’ or ‘for’ them.32 This definition sees the involvement process as a partnership between patients, the public and health professionals. This is important given major power differentials exist between those involved in a lay capacity and paid healthcare professionals.

nts or members of the public rather than ‘to’, ‘about’ or ‘for’ them.32 This definition sees the involvement process as a partnership between patients, the public and health professionals. This is important given major power differentials exist between those involved in a lay capacity and paid healthcare professionals. At its core, the purpose of any involvement activity should be to improve the health and the experience of services for patients, their relatives, carers and users of health and social care services as well as the wider public.31 Figure 133 provides a typical organising framework for involvement (the term engagement is used in this framework) that shows involvement can take place at multiple levels. Figure 1 A multidimensional framework for patient and family engagement in health and healthcare.33 This figure builds upon Arnstein's34 widely quoted ‘ladder of citizen participation’. This described “a continuum of public participation in governance ranging from limited participation, or degrees of tokenism, to a state of collaborative partnership in which citizens share leadership or control decisions”.33 Reproduced with permission of Project HOPE/Health Affairs from Carman et al.33

vel meetings were awash with clinical quality metrics, among which the ‘softer data’ emerging from the HEADS-UP briefings had a less certain place.96 As the organisation moved to merge with another institution, fewer resources were available for continuous QI in the interim. (Incentives and actions for senior managers) Engagement—a reciprocal commitment from staff and their organisation There is no universal definition of engagement: it may be an attitude, behaviour, an outcome—or all three.26 Schaufeli et al27 describe engagement as an employee's positive motivational state, characterised by ‘vigour, dedication and absorption’. A broader, more cooperative, position is that engagement is a two-way phenomenon, with an onus on the organisation to establish conditions encouraging engagement and the opportunities for it to be manifest.28 Clinical engagement, then, involves staff actively contributing ‘within their normal working roles to maintaining and enhancing the performance of the organisation, which itself recognises this commitment in supporting and encouraging high quality care’.28 This working definition makes it clear that real engagement is a very different entity to staff acquiescence, for which it is often confused.29 30

participation’. This described “a continuum of public participation in governance ranging from limited participation, or degrees of tokenism, to a state of collaborative partnership in which citizens share leadership or control decisions”.33 Reproduced with permission of Project HOPE/Health Affairs from Carman et al.33 It can range along a continuum, from consultation to partnership and shared leadership. At the lower end, patients are involved but have limited power or decision-making authority. At the higher end, involvement is characterised by shared power and responsibility, with patients as active partners in defining agendas and making decisions. PPI can also occur at the level of individual health behaviour or direct care, or can occur at the collective level in organisational design and governance and in policymaking. We argue other areas can include commissioning, monitoring, evaluation and research. Multiple factors affect the willingness and ability of patients to engage at these different levels, including patient beliefs about their role, health literacy, education, organisational policies and practices and culture, society and social norms, regulation and policy. We believe that issues to do with inequality, discrimination and social exclusion also play a strong role in preventing many individuals and groups, participating in the involvement process as indicated in table 1. Table 1 Summary of barriers and enablers for involvement27

It can range along a continuum, from consultation to partnership and shared leadership. At the lower end, patients are involved but have limited power or decision-making authority. At the higher end, involvement is characterised by shared power and responsibility, with patients as active partners in defining agendas and making decisions. PPI can also occur at the level of individual health behaviour or direct care, or can occur at the collective level in organisational design and governance and in policymaking. We argue other areas can include commissioning, monitoring, evaluation and research. Multiple factors affect the willingness and ability of patients to engage at these different levels, including patient beliefs about their role, health literacy, education, organisational policies and practices and culture, society and social norms, regulation and policy. We believe that issues to do with inequality, discrimination and social exclusion also play a strong role in preventing many individuals and groups, participating in the involvement process as indicated in table 1. Table 1 Summary of barriers and enablers for involvement27 Key exclusions Key barriers Overcoming barriers Equality and discrimination: barriers on the basis of gender, ethnicity, culture, belief, sexuality, age, disability and class. Devaluing people: not valuing or listening to what people say. Access: ensuring all participants have effective ways into organisations and decision-making structures to have a real say in them. Where people live: Homeless. In residential services. In prison and the penal system. Travellers/gypsies. Tokenism: asking for involvement but not taking it seriously or enabling it to be effective. Support: building confidence/skills, offering practical help/opportunities to get together to support people's empowerment and capacity.Use of advocacy: important for people who are disempowered and isolated. Communication issues: Deaf people. Blind/visually impaired people. People who do not communicate verbally. People for whom English is not their first language. Unwanted voices: Some points of views/experiences are more welcome than others (particularly those who agree or are less challenging of the system or services). People can also be excluded because they are seen as too expensive/difficult to include such as those with dementia. Stigma: stigmatising people for their identity or why they became involved or because they have had a poor experience of care35 and discouraging involvement on the basis of their identity. Confidence and self-esteem: making people feel they do not have much to contribute. Inadequate information about involvement: Lack of appropriate and accessible information about getting involved or about the involvement opportunities.36 Different forms of involvement: using innovative approaches that go beyond traditional methods: meetings, surveys, written and verbal skills. Different methods include: entertainment organised by lay participants, offering safe opportunities to explore ideas. Supportive activities, informal venues and encouraging networking.

ent forms of involvement: using innovative approaches that go beyond traditional methods: meetings, surveys, written and verbal skills. Different methods include: entertainment organised by lay participants, offering safe opportunities to explore ideas. Supportive activities, informal venues and encouraging networking. Outreach and development work: reaching out to those traditionally identified as ‘hard to reach’, going to them and community leaders, building trust, asking what works.Meetings where used: making them attractive, inclusive, enjoyable, with free refreshments that are culturally appropriate, safe, supportive environment, with access to key knowledge. Issues of poor health literacy: this can be an important determinant of access to healthcare, impacting upon patients ability to book, cancel/attend appointments, respond to an adverse error in their care or medication or a deterioration in their care.37 Gatekeepers/individuals who block the involvement process: individuals who obstruct the involvement process by their attitudes or actions and stop people getting involved. Tools to support patient empowerment38 Motivational interviewing: used by clinicians/non-clinicians, personal budgets, expert patient (self-management) programme, patient decision aids in shared decisions, helping people prepare for consultations, access to health records. Achieving greater health literacy in the population is integral to improving the health of disadvantaged populations and to tackling health inequalities.23 Financial barriers: not paying participants for their involvement (which is a widely accepted principle) and speedily can deter people with limited resources or high costs because of the nature of their situation or impairment from being involved. Good practice regarding health literacy37 Improving communication with all patients can include: ascertaining what the patient knows first to determine level of discussion. Speaking slowly, avoiding jargon, repeating points to improve comprehension, encourage and expect all patients to ask questions.Check understanding and recall. Ask patients to repeat back critical info (making clear this is about the health professionals’ ability to communicate clearly). Communicate in ways other than speech/printed material, eg, multimedia, translation services/materials.

ion, encourage and expect all patients to ask questions.Check understanding and recall. Ask patients to repeat back critical info (making clear this is about the health professionals’ ability to communicate clearly). Communicate in ways other than speech/printed material, eg, multimedia, translation services/materials. Understanding the role of power in involvement Many authors capture the history and evolution of involvement and demonstrate that multiple influences and ideas have shaped this activity.30 39–41 Global health social movements42 and disenfranchised groups, including black, disabled, mental health, lesbian and gay, and women's groups, can be seen as providing collective challenges to poor care and discriminatory or paternalistic services and medical policy and belief systems.42 These rights-based groups first emerged in healthcare and in other parts of the public sector in the UK as far back as the 1970s. Campaigns from patients who have been harmed during their care, and their relatives, are only the most recent manifestation of these challenges to paternalistic healthcare. These campaigns have been reinforced by wider arguments that also emerged in the late 1970s, emphasising the limitations of a biomedical model in promoting health and illness43–45 and calling for more shared models of treatment and decision-making.46 47 Other concerns during this period highlighted a need to address poorer and unequal access to healthcare for some groups48 49 as well as the role of the professions in healthcare iatrogenesis.50 51 These arguments are still relevant today as we discuss later in the paper. In practice, what they illustrate are various examples where the balance of power apparently favours the organisation or system, rather than partnership working with patients and the public.52

of the professions in healthcare iatrogenesis.50 51 These arguments are still relevant today as we discuss later in the paper. In practice, what they illustrate are various examples where the balance of power apparently favours the organisation or system, rather than partnership working with patients and the public.52 With current involvement practice, power imbalances frequently manifest themselves in different ways, starting with who to involve. PPI often involves a narrow group of individuals, with the handpicking of just one or two ‘appropriate’ or ‘acquiescent’29 patient representatives to be involved in committees or projects. Patient representatives are less commonly drawn from black and minority ethnic groups,27 53 54 and are often middle class.9 29 31 Yet in the UK,53 55 and “in most healthcare systems, it is acknowledged that black and minority ethnic (BME) populations have until now experienced poorer health and barriers to accessing certain services”.56 57 Many other different groups are also excluded from involvement processes (table 1). These groups may have particular or even greater healthcare needs than the wider population,58 59 yet their views are seldom heard or listened to.16 27 28 60 In reality, the capacity to be successfully involved is significantly affected by education level, income, cognitive skills and cultural differences, which can affect patients’ health beliefs and ability to use health services.12 61 The consequences of narrow PPI selection processes mean that those with most to gain are most excluded from healthcare decision-making. This restricts the pool of ideas for improvement and limits the opportunity to break cycles of suboptimal care and services.

ts’ health beliefs and ability to use health services.12 61 The consequences of narrow PPI selection processes mean that those with most to gain are most excluded from healthcare decision-making. This restricts the pool of ideas for improvement and limits the opportunity to break cycles of suboptimal care and services. At the organisational level, factors that can hinder PPI in service planning and decision-making include laypeople feeling unclear about their role and what is expected of them, a shortage of resources to support the process, concerns about representation, negative attitudes27 and resistance from healthcare staff and managers12 17 29 62 (table 1). In the UK, it has been suggested that the majority of involvement activity in healthcare has traditionally taken place at the level of feedback and information giving.63 Shared forms of decision-making, which have been found to have proven health benefits,64 are still not the norm.65 Internationally, there is evidence that shows that patient representatives are struggling to influence decisions and are largely expected to work within existing systems in improving quality and safety.61 Involvement at this level has been criticised as providing little opportunity to influence decision-making processes in any depth. This serves to maintain professional and system interests and power.10 34 47

influence decisions and are largely expected to work within existing systems in improving quality and safety.61 Involvement at this level has been criticised as providing little opportunity to influence decision-making processes in any depth. This serves to maintain professional and system interests and power.10 34 47 Current models of PPI are therefore too often rooted in a mechanistic, controlled and professionally dominated approach, based upon a very practical and atheoretical way to getting someone's input. This narrow ‘managerialist or consumerist’ model10 has its roots in market research and ‘improving the product’, which has typically come to dominate approaches to PPI. It mainly draws upon data collection methods and consultation and the reporting of patient survey data at board meetings. This contrasts with a wider democratic rights and values-based approach, which emphasises the need for the direct involvement and empowerment of users of services in the decision-making process and broader democratisation at a community level. Such an approach goes beyond just a focus on individuals as the source of the problem and recognises the systemic nature of health inequities and how different groups can be excluded. Democratic models focus on the need for change to take place within social systems as well as within individuals and services.66

level. Such an approach goes beyond just a focus on individuals as the source of the problem and recognises the systemic nature of health inequities and how different groups can be excluded. Democratic models focus on the need for change to take place within social systems as well as within individuals and services.66 Linking theory and evidence for empowerment and impact Broader frameworks and methods of involvement should be used that offer better ways to share power with healthcare professionals. Central goals of involvement should focus on issues of inclusivity and representation, equalities, non-discrimination and empowerment. It has been recognised that different levels of participation are appropriate in different circumstances.63 But it is clearly important to think about which level is important and how it will influence decision-making. In current PPI practice, there appears to be a considerable disconnection with much of this thinking and how it can be used to achieve clarity of purpose in much of mainstream healthcare. This situation is worse in areas such as patient safety, where PPI is largely atheoretical.61

how it will influence decision-making. In current PPI practice, there appears to be a considerable disconnection with much of this thinking and how it can be used to achieve clarity of purpose in much of mainstream healthcare. This situation is worse in areas such as patient safety, where PPI is largely atheoretical.61 At the organisational and community level, models of co-production are increasingly being seen as a way of addressing power imbalances by designing and delivering public services in more democratic, equal and reciprocal relationships between professionals, people using services, their families and their neighbours.67 68 There is no one ‘correct’ way of doing co-production, but there are six principles that help to underpin practice: assets: recognising people as assets; capabilities: building on people's existing strengths; mutuality: reciprocal relations with mutual responsibilities and expectations; networks: peer support and engaging a range of networks inside and outside services; blur roles: removing tightly defined boundaries between professionals and recipients to enable shared control and responsibility; catalysts: shift from delivering services to supporting things to happen.67 69 70

mutuality: reciprocal relations with mutual responsibilities and expectations; networks: peer support and engaging a range of networks inside and outside services; blur roles: removing tightly defined boundaries between professionals and recipients to enable shared control and responsibility; catalysts: shift from delivering services to supporting things to happen.67 69 70 This more collaborative framework can also support methods to empower patients at the individual level. Evidence on four key ways to empower patients has been identified: (a) empowering individuals in their own care; (b) reviving the revolution in decision-making tools as part of a systematic drive in shared decision-making (SDM) (SDM offers a process where patients and clinicians can work together to select tests, treatments, access personal health records and health budgets, care planning and decision aids, management or support packages based on clinical evidence and the patient's informed preferences).64 65 The latter is particularly important in preventing misdiagnosis and unwanted interventions.);71 (c) giving patients co-ownership of their records, not just access; and (d) encouraging patients to ask more questions and targeting a national campaign for people with long-term conditions and offering greater access to structured education on self-management. Tackling health literacy has also been found to be central to the empowerment of patients and reducing health inequalities.23 Table 1 provides examples of tools and ways to develop empowering practice with a diversity of groups at the individual and organisational level.

cess to structured education on self-management. Tackling health literacy has also been found to be central to the empowerment of patients and reducing health inequalities.23 Table 1 provides examples of tools and ways to develop empowering practice with a diversity of groups at the individual and organisational level. Empowering users and providers and supporting frontline staff to feel confident in sharing power and accepting user expertise will be crucial in developing these more shared and collaborative ways of working.72 This is important given that time, resources and funding17 and competing organisational priorities and a lack of training for clinical providers have been identified as key barriers in the implementation of PPI strategies.73

expertise will be crucial in developing these more shared and collaborative ways of working.72 This is important given that time, resources and funding17 and competing organisational priorities and a lack of training for clinical providers have been identified as key barriers in the implementation of PPI strategies.73 Finding effective ways to evaluate PPI processes are also important to ensure a wider range of expertise and experiences are included in PPI activities. There are a number of factors that make evaluation in this area challenging: the need for a shared understanding of PPI in practice and how it is conceptualised and measured, limited documentation of underpinning theory19 and the difficulty in isolating involvement from other factors that influence change.18 Within the democratic tradition, less emphasis has traditionally been placed on measuring PPI as involvement is seen as something that has intrinsic value in and of itself, over and above any attempt to measure it from an instrumental perspective. We suggest national involvement standards such as 4Pi31 (table 2) could provide a broader, inclusive framework by which to support good practice in the development of PPI approaches and interventions as well as to understand the effects of PPI. 4Pi draws on research evidence and user experience to identify the characteristics and attributes of meaningful involvement. This framework, in combination with a sound understanding of evaluation principles,74 could support the generation of better evidence. Table 2 4Pi national involvement standards: involvement for influence31

Finding effective ways to evaluate PPI processes are also important to ensure a wider range of expertise and experiences are included in PPI activities. There are a number of factors that make evaluation in this area challenging: the need for a shared understanding of PPI in practice and how it is conceptualised and measured, limited documentation of underpinning theory19 and the difficulty in isolating involvement from other factors that influence change.18 Within the democratic tradition, less emphasis has traditionally been placed on measuring PPI as involvement is seen as something that has intrinsic value in and of itself, over and above any attempt to measure it from an instrumental perspective. We suggest national involvement standards such as 4Pi31 (table 2) could provide a broader, inclusive framework by which to support good practice in the development of PPI approaches and interventions as well as to understand the effects of PPI. 4Pi draws on research evidence and user experience to identify the characteristics and attributes of meaningful involvement. This framework, in combination with a sound understanding of evaluation principles,74 could support the generation of better evidence. Table 2 4Pi national involvement standards: involvement for influence31 Component Supportive attributes for inclusion and empowerment Principles Involvement is underpinned by values, eg, inclusivity and non-discrimination, respect and transparency and being open-minded to cultural differences. Purpose The purpose should be clearly articulated so that everybody understands the goal of involvement and has the opportunity to shape and influence the process. Presence Who to involve will be determined by the purpose. This means an inclusive approach that seeks to address inequalities. Process Involvement in direct care for example, will require different approaches to involvement than at the collective/organisational level. Consultation methods will deliver different results by comparison with co-design or co-production approaches. Impact Impact can be considered in different ways, eg, on individual conduct as well as on organisational culture, policy/planning, outcomes, outputs, diversity and equality of opportunity and the experience of service users. Conclusions Our findings point to a need to re-evaluate methods and approaches for involving patients and the public in all aspects of healthcare and in healthcare improvement. Partnership working has long been the explicit stated goal of involvement. However, current involvement practices at a national and local level often involve a narrow group of individuals in involvement activities, with little consideration given to including a broader demographic of the population. Moving beyond this tokenistic, narrow and exclusive approach requires a critical appraisal of evidence and a debate about the focus and methods of involvement. Use of broader and more democratic models is important to address imbalances of power between patients, public and healthcare professionals and organisations. Evaluating these approaches, to understand the impact and effectiveness of chosen PPI methods, as well as how inclusive they are, is important. Developing greater partnership working will require key policy organisations and networks to take a lead in promoting this broader approach, disseminating good practice and evidence and building in requirements into funding streams.

d effectiveness of chosen PPI methods, as well as how inclusive they are, is important. Developing greater partnership working will require key policy organisations and networks to take a lead in promoting this broader approach, disseminating good practice and evidence and building in requirements into funding streams. Ultimately to deliver a broader and more effective approach to involvement, staff will need to be trained and supported within organisational contexts where partnership working with a diversity of patients and the public is clear, embedded and normal. Moving beyond tokenism to sharing power and decision-making more equitably will promote empowerment and help develop models of healthcare that are more co-designed and co-produced between all stakeholders, regardless of whether they are using or providing services. Contributors: Both authors contributed to all aspects of the article, with Josephine Ocloo leading as first author. Funding: The Health Foundation, National Institute for Health Research Imperial Patient Safety Translational Research Centre. Disclaimer: This article presents independent research commissioned by the National Institute for Health Research (NIHR) under the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) programme for North West London. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: None declared. Provenance and peer review: Commissioned; internally peer reviewed.

Disclaimer: This article presents independent research commissioned by the National Institute for Health Research (NIHR) under the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) programme for North West London. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: None declared. Provenance and peer review: Commissioned; internally peer reviewed. Data sharing statement: This is not an empirical study but a narrative review. This selective review scopes various literature on patient and public involvement across a range of disciplines.

Introduction Over the last decade there have been considerable efforts to evaluate and improve the quality of healthcare delivery. Three to six per cent of inpatient deaths may be preventable,1–4 and early attempts to foster better care invoked crew resource management in aviation,5 industrial quality assurance techniques6 and transparent outcome reporting.7 However, the pace of improvement remains sluggish,8 despite an awareness of how infrequently patients receive the best therapy already available.9 10 Quality improvement (QI) interventions have enormous potential to improve healthcare delivery, but well-publicised research successes have proved difficult to replicate outside the trial setting.11–17 Discrete QI interventions also struggle to outperform the secular trend towards system-wide improvement.17 18 The challenges of reproducing QI successes on a wider scale remain poorly understood, but two key factors are often cited: the engagement of clinical staff in the broader initiative, and the context in which it takes place. Neither clinical engagement19 nor context20 is well defined, yet the two have become de facto explanations for QI failure. Moreover, the numerous attempts to explain how context affects QI21–25 risk overwhelming researchers and clinicians,21 with limited ‘how-to’ support for those implementing change.20 Here, we discuss whether the focus on clinical staff is misplaced, and propose a novel, pragmatic model for the development and selection of effective, durable QI interventions.

ain how context affects QI21–25 risk overwhelming researchers and clinicians,21 with limited ‘how-to’ support for those implementing change.20 Here, we discuss whether the focus on clinical staff is misplaced, and propose a novel, pragmatic model for the development and selection of effective, durable QI interventions. In a narrative synthesis guided by insights from a recent trial (see box 1), we first explore the varying definitions of clinical engagement. We go on to discuss the specific challenges of clinical engagement in QI, and strategies shown to circumvent them. Next, we explore the role of managerial staff, whose importance in QI implementation has been underestimated. Bringing these concepts together, we outline a ‘model for alignment’, highlighting key factors of practical importance for successful QI. The model is then used to retrospectively describe the implementation of an interdisciplinary intervention (see box 2). The model for alignment emphasises that QI efforts must explicitly meet aligned clinical, managerial and organisational needs if they are to become ‘business as usual’. Box 1 Autoethnographic observations from the Hospital Event Analysis Describing Significant Unanticipated Problems (HEADS-UP) study83 The HEADS-UP study evaluated an interdisciplinary team intervention tailored for general medical wards: a structured, daily safety and quality briefing.83 92

become ‘business as usual’. Box 1 Autoethnographic observations from the Hospital Event Analysis Describing Significant Unanticipated Problems (HEADS-UP) study83 The HEADS-UP study evaluated an interdisciplinary team intervention tailored for general medical wards: a structured, daily safety and quality briefing.83 92 HEADS-UP briefings were designed to embed proactive organisational risk surveillance into routine ward care. Through structured discussion, clinical staff would identify risks to the delivery of high quality care on a daily basis, addressing them promptly through facilitated communication with senior clinicians and managers, before patient harm occurred. One of the authors of this paper (SP) was heavily involved in HEADS-UP implementation at two institutions, and in facilitating the use of the data arising from it—a form of participatory research.93 His observations, discussions with staff, reflections on clinical governance proceedings and implementation challenges were recorded in field notes over a 20-month period. These ‘autoethnographic’ insights, from researchers embedded within their host organisations, are widely used in organisational case study research, providing rich accounts of culture and practices.51 94 95 Observations from the HEADS-UP study informed the narrative synthesis of the literature described here, and the construction of the model of alignment for successful quality improvement interventions.

These ‘autoethnographic’ insights, from researchers embedded within their host organisations, are widely used in organisational case study research, providing rich accounts of culture and practices.51 94 95 Observations from the HEADS-UP study informed the narrative synthesis of the literature described here, and the construction of the model of alignment for successful quality improvement interventions. Box 2 Applying the model of alignment to analyse Hospital Event Analysis Describing Significant Unanticipated Problems (HEADS-UP) implementation at one study site Poor engagement with incident reporting was an organisational concern. However, there was little capacity to enact an improvement programme. No other goals were set aside to prioritise the HEADS-UP programme. As an experimental intervention, the efficacy of HEADS-UP and its implementation strategy was unknown. (Strategic selection of quality improvement (QI) target and intervention)

cern. However, there was little capacity to enact an improvement programme. No other goals were set aside to prioritise the HEADS-UP programme. As an experimental intervention, the efficacy of HEADS-UP and its implementation strategy was unknown. (Strategic selection of quality improvement (QI) target and intervention) Front-line staff expressed initiative fatigue at the beginning of the study, but the face validity of the HEADS-UP tool (and its co-design) mitigated much of their expected resistance, and HEADS-UP was incorporated into normal workflow. QI participation was not formally rewarded, but some junior clinical staff were able to exploit their involvement in HEADS-UP to help with career progression. Others reflected that they found HEADS-UP useful for their own practice and that it improved the quality of interdisciplinary care for their patients, which may have been perceived as a benefit or reward. (Incentives and actions for front-line clinical staff)

eir involvement in HEADS-UP to help with career progression. Others reflected that they found HEADS-UP useful for their own practice and that it improved the quality of interdisciplinary care for their patients, which may have been perceived as a benefit or reward. (Incentives and actions for front-line clinical staff) Middle managers were not personally incentivised to participate in HEADS-UP, although an endorsement by the Care Quality Commission during the study period prompted senior managers to designate a greater focus on the programme. Where HEADS-UP generated information to bolster middle managers’ existing business plans and develop new ones, those managers coordinated staff-identified opportunities for service development with linked organisational priorities. They gave HEADS-UP their personal backing, encouraging its use, although no additional resources were available to accompany this support. In contrast, middle managers for whom HEADS-UP was less directly useful did less to hold their service areas accountable for HEADS-UP performance. HEADS-UP piggybacked onto existing staff meetings and governance structures: no protected time was made available for involvement in dedicated training, analysis or feedback. Middle managers’ time and attention was strictly limited, and competing priorities (eg, the implementation of an electronic health record, or a forthcoming merger) often precluded meaningful progress with this QI intervention. Interestingly, resolution of the persistent issues raised in the HEADS-UP briefings appeared to depend less on each one's inherent safety threat, than on agreement between clinical staff and managers of the need for change. (Incentives and actions for middle managers)

this discussion (unless specified otherwise) we group clinical healthcare professionals together. QI interventions are typically interdisciplinary, and securing greater engagement of a single staff group is not an end in itself, only a step towards an ‘organisational culture where all staff feel valued and involved’.30 Clinical engagement can be improved by co-design, local modification and strategic selection of QI interventions Healthcare professionals have been reluctant to involve themselves in QI initiatives.31 This is especially apparent in periods of sustained organisational turbulence, but is a long-standing, multifactorial and international problem.31 Doctors are disproportionately hesitant to participate in safety behaviours like incident reporting,32 33 and active resistance from senior staff remains the most common barrier to the successful implementation of interdisciplinary safety checklists.34–38 The narrative of ‘automatic’ clinical resistance to new initiatives, or ‘change fatigue’,31 is seemingly widely accepted within the QI literature. Circumventing this fatigue is considered a major triumph, even fundamental to QI success. With few exceptions,24 25 local QI breakthroughs are attributed to good clinical engagement; conversely, failures are seen only through the prism of inadequate clinical buy-in.

s seemingly widely accepted within the QI literature. Circumventing this fatigue is considered a major triumph, even fundamental to QI success. With few exceptions,24 25 local QI breakthroughs are attributed to good clinical engagement; conversely, failures are seen only through the prism of inadequate clinical buy-in. Not all interventions are subject to the same clinical disengagement barrier, however: some programmes might have lower thresholds for participation.21 39 40 Conversely, problems with the introduction of a specific improvement strategy do not necessarily indicate a wider reluctance to change practice. The intervention's characteristics, at least in part, determine its reception. In our experience, iterative co-design of a structured quality and safety briefing intervention with physicians, to maximise its face validity, mitigated much of their expected resistance (see box 2). The process of co-design may also in itself improve staff ownership of the intervention. Other strategies may also have immediate appeal: interventions that used peer facilitation,41 wider reporting options and feedback,42–47 or engaging whole teams to identify problems48–50 all seemingly fell on fertile ground.

ox 2). The process of co-design may also in itself improve staff ownership of the intervention. Other strategies may also have immediate appeal: interventions that used peer facilitation,41 wider reporting options and feedback,42–47 or engaging whole teams to identify problems48–50 all seemingly fell on fertile ground. Even when QI strategies do not appeal intuitively to clinicians, generating clinical engagement need not prove an insurmountable challenge. Defining the ‘soft periphery’ of a QI programme—the elements that should be flexibly adapted to optimise the programme's acceptance, without invalidating the entire intervention—is key.51 Making the effort to appropriately modify QI tools for the context in which they will be applied (eg, creating separate versions of surgical safety checklists for different specialties) then makes those tools much more palatable for clinicians.34 In fact, local adaptation is the most commonly cited factor affecting checklist implementation, more so even than resistance from specific clinicians.34 This reflects each organisation's responsibility to create the opportunities for meaningful engagement: active clinical involvement is more likely when QI tools have been purposefully tailored, and when there is protected time for training to use them.31 34 In contrast, unmodified checklists are unlikely to be used as intended, nor improve patient outcomes—regardless of hospitals’ reported compliance.13 52

ul engagement: active clinical involvement is more likely when QI tools have been purposefully tailored, and when there is protected time for training to use them.31 34 In contrast, unmodified checklists are unlikely to be used as intended, nor improve patient outcomes—regardless of hospitals’ reported compliance.13 52 While co-design and local modification do improve the adoption of QI interventions, the financial and opportunity costs of pre-existing efforts represent a major challenge to any new initiative. Relentless organisational change, with little sense of an overall strategic direction, also contributes to a general ennui.31 Clinicians, believing that each ‘fad’ will soon be replaced with another, feel there is ‘little point in investing heavily in any one initiative’.31 The strategic selection of a limited number of QI interventions, appropriate to the organisation's capacity to implement them, is therefore crucial.23 53 Experts have identified 22 patient safety strategies with a sufficient evidence base to recommend their widespread adoption:54 55 organisations may choose to focus on these first, with a view to their specific local needs. However, if organisations are to select only the QI targets that they have the capacity and willpower to pursue, other potentially valuable initiatives need be deferred in the interests of preserving engagement and momentum. Regulatory bodies have an important role here: they should give institutions the time and space to develop these focused improvement strategies.56 It may seem odd to decry the slow pace of improvement, and yet advocate a more deliberate, institution-specific approach. With space for self-determination, however, organisations that strategically shape their QI attempts go on to see wider benefits, tackling deep cultural issues that go unaddressed with more haphazard approaches.56

to decry the slow pace of improvement, and yet advocate a more deliberate, institution-specific approach. With space for self-determination, however, organisations that strategically shape their QI attempts go on to see wider benefits, tackling deep cultural issues that go unaddressed with more haphazard approaches.56 Beyond clinical engagement: the role of managers in QI Improving clinical engagement is only part of the solution to ineffective QI: ‘administrative engagement is equally important, or disillusionment… ensues’.29 Quality of care is not a leading priority for many hospital boards, however.57–59 Although board-level attention to quality issues has been associated with clinical quality,57 58 60 how this commitment translates, in practice, into front-line action remains unclear.61 A recent survey study provided a key insight: board and middle management practices are linked, and correlate strongly with hospital performance on clinical quality metrics.62 Certain board characteristics were specifically linked to middle management styles: board attention to quality was associated with management practices that monitored it, and the use of quality metrics at board level corresponded to good operational management and target setting.62 If good management is truly associated with clinical quality, the role of managers in QI deserves further attention.

les: board attention to quality was associated with management practices that monitored it, and the use of quality metrics at board level corresponded to good operational management and target setting.62 If good management is truly associated with clinical quality, the role of managers in QI deserves further attention. Yet managerial participation in QI interventions is often assumed, rather than analysed in detail. Although senior hospital executives may participate constructively in collaborative safety programmes,14 63 more often, active managerial involvement goes no further than the ‘expressions of support’ described in many QI reports. More detailed evaluations describe difficulties recruiting executives to work with QI teams, even as part of major safety initiatives.25 When they do engage, managers have a different outlook on quality and safety programmes to clinical staff, perceiving different components of the programme to be valuable and holding more positive views of the overall results.64 Meaningful input from managers is important for the design, monitoring and evaluation of QI interventions;64 simply obtaining their permission to proceed is not enough.

programmes to clinical staff, perceiving different components of the programme to be valuable and holding more positive views of the overall results.64 Meaningful input from managers is important for the design, monitoring and evaluation of QI interventions;64 simply obtaining their permission to proceed is not enough. There is little published work on the role of managers in QI, the majority of which relates only to senior (board-level) managers, rather than the middle managers under their supervision.59 61 Importantly, most improvement initiatives fail to specify how they engage these middle managers, with whom front-line staff interact directly and regularly. Middle managers are a particularly heterogeneous group, with diverse professional backgrounds,61 often promoted on the basis of a technical skill set rather than any specific leadership or management ability.65 Many have ‘hybrid’ clinical and administrative duties,61 66 with an inherent tension between those roles;67 their decisions are necessarily ‘constrained, contested and political’,68 but favour knowledge drawn from experience rather than research findings.69 70 Little more is known about the cognitive biases that affect middle managers’ judgements, but enthusiasm for QI is not automatic. For example, they may feel the operating costs of QI programmes are not justified by any potential future benefits.71 There remains a pressing need for research into how healthcare managers balance their multiple fiscal, statutory and service responsibilities.59

gers’ judgements, but enthusiasm for QI is not automatic. For example, they may feel the operating costs of QI programmes are not justified by any potential future benefits.71 There remains a pressing need for research into how healthcare managers balance their multiple fiscal, statutory and service responsibilities.59 Influencing middle managers to facilitate effective QI: status, incentives and resources It appears, then, that managers contribute to organisational quality; that their active involvement in QI has been taken for granted rather than proven; and that their decision making relating to QI is likely to be complex, with conflicting priorities that are not easily resolved. Yet middle managers, in particular, are uniquely placed to facilitate effective QI. They have the power to accelerate or impede the implementation of innovations,61 mediating organisational messages for front-line staff, but also upwardly influencing their seniors to draw attention to the high-level support needed for specific QI programmes.72 73 Acting as information brokers, translating organisational strategy into actionable tasks, and promoting innovative practice, middle managers can convince clinical staff to prioritise QI implementation among numerous competing demands.61

to draw attention to the high-level support needed for specific QI programmes.72 73 Acting as information brokers, translating organisational strategy into actionable tasks, and promoting innovative practice, middle managers can convince clinical staff to prioritise QI implementation among numerous competing demands.61 Harnessing middle managers’ ability to broker organisational and front-line attention to a QI programme may prove essential to its success: proactive commitment from middle managers does influence effective QI implementation.74 Although many advocate a clinician-led, ‘bottom-up’ approach to improvement,75 staff-driven initiatives that do not align well with strategic priorities have only limited impact or longevity.71 76 77 Clear tensions emerge when QI efforts are delegated entirely to clinicians without support for their direction and goals.71 Without more senior support, front-line staff are unable to marshal the resources required to spread change,78 and managers have an important role to play in navigating cross-departmental obstacles.59 Managers who effectively facilitate QI, without micromanaging it, are well appreciated by front-line staff.78 The subsequent pace of change may be slow, but a combination of top-down and bottom-up implementation results in a lasting impact.78

have an important role to play in navigating cross-departmental obstacles.59 Managers who effectively facilitate QI, without micromanaging it, are well appreciated by front-line staff.78 The subsequent pace of change may be slow, but a combination of top-down and bottom-up implementation results in a lasting impact.78 Senior managers play a role in determining middle managers’ commitment to QI. These senior managers should directly emphasise QI as an organisational priority, incentivise QI commitment in performance reviews, and—vitally—make available the necessary resources.72 In addition, encouraging middle managers to leverage the human resources and performance reviews at their own disposal also improves their commitment to QI implementation.72 Interestingly, a performance-related human resources management framework for clinicians has recently been described and implemented, encountering little of the expected physician resistance.79 Transparent negotiation of QI goals at each organisational level may therefore be feasible and necessary for high quality implementation.

ly, a performance-related human resources management framework for clinicians has recently been described and implemented, encountering little of the expected physician resistance.79 Transparent negotiation of QI goals at each organisational level may therefore be feasible and necessary for high quality implementation. A model of alignment for successful QI We suggest that neither clinicians nor managers can make meaningful QI progress in isolation: their collaboration is fundamental to sustainably embedding practice innovations. The choice of a QI intervention, and its implementation model, both need coordination between clinical and managerial teams. We have discussed, in the preceding sections, how each group might be motivated to take part in this process. Yet their efforts need to be aligned, if QI is to form a significant part of their workload, and not be overwhelmed by other priorities.62 We highlight some examples of QI programmes in which the degree of collaboration between front-line and managerial staff may have contributed to the ultimate outcome (table 1). In trials reporting significant improvements, investigators ensured there was adequate managerial participation—or took on managerial roles themselves—to complement clinical involvement.11 12 15 80 Similarly, implementation and spread of a QI intervention in a real-world setting was best accomplished with the co-leadership of top-level administrators and front-line champions.78 Where managerial engagement was lacking, interventions did not improve outcomes significantly, or systems defects did not prove amenable to the efforts of clinical teams alone.13 17 81

tervention in a real-world setting was best accomplished with the co-leadership of top-level administrators and front-line champions.78 Where managerial engagement was lacking, interventions did not improve outcomes significantly, or systems defects did not prove amenable to the efforts of clinical teams alone.13 17 81 Table 1 Descriptions of managerial collaboration in selected quality improvement (QI) interventions

tervention in a real-world setting was best accomplished with the co-leadership of top-level administrators and front-line champions.78 Where managerial engagement was lacking, interventions did not improve outcomes significantly, or systems defects did not prove amenable to the efforts of clinical teams alone.13 17 81 Table 1 Descriptions of managerial collaboration in selected quality improvement (QI) interventions QI intervention Implementation phase (proof of concept/trial/scaling up) Managerial collaboration Outcome Surgical safety checklist Trial11 12 80 Systems changes facilitated by the local investigator—essentially fulfilling a dedicated managerial role.11 Hospital administration/management leaders required to ‘support the intervention’11 80 Reduced in-hospital complications Scaling up13 No assessment of managerial involvement in mandatory checklist implementation. Meaningful local implementation unlikely to have taken place52 No significant change in patient outcomes Program to reduce central line infections Trial15 Program targeted middle managers and senior hospital leaders as well as front-line staff.24 Chief executives wrote ‘commitment letter’ to the program team. Nurse manager led the project locally; project team also included a hospital executive advocate Reduced infection rates Scaling up17 Chief executives agreed organisations would participate, and that a director would join the local project team. In practice, most units struggled to involve executives25 No improvement compared with controls Program to detect and mitigate organisational weaknesses Proof of concept81 Executive sponsor for each site team. Managerial staff less often directly involved as project team members System defects not tractable to small clinical teams’ QI methodology Program to improve interprofessional coordination Scaling up78 Spectrum of managerial involvement. In ‘bottom-up’ hospitals, administrators delegated and served as resources. In ‘top-down’ hospitals, managers primarily drove the change effort Co-leadership of top-level administrators and front-line champions best facilitated implementation and spread of the intervention With this in mind, we propose a simplified model for successful QI interventions (figure 1). This model emphasises, foremost, that QI interventions aiming to change healthcare providers’ practice should aim to meet the aligned needs of staff at multiple levels in the organisation. Failing to coordinate these interests renders interventions susceptible to failure, regardless of enthusiasm and engagement at the other organisational levels.

ost, that QI interventions aiming to change healthcare providers’ practice should aim to meet the aligned needs of staff at multiple levels in the organisation. Failing to coordinate these interests renders interventions susceptible to failure, regardless of enthusiasm and engagement at the other organisational levels. In fact, the degree to which an intervention recognises, makes use of, or conflicts with existing staff priorities is fundamental to its success, and should not be considered in the accompanying implementation strategy only. This pre-emptive consideration of where an intervention is likely to garner support, and the conflicts that need to be resolved to allow wholehearted participation, reflect the ‘practical wisdom’ thought to be a critical element of successful QI.82 Throughout this narrative literature review, and reinforced by our recent practical experience (box 1), we identified specific facilitators that coordinate clinical, middle management and senior management participation in QI. To build a useful model, we then separated these factors into incentives (establishing each group's QI participation as a core expectation of their work) and actions (specific actions by that group that make QI implementation more effective). We also highlight important barriers to aligned QI, again identified from the narrative synthesis. For facilitators and barriers, we focused deliberately on modifiable factors, with a view to building a valid model that has immediate application in practice. Importantly, the inclusion of managers as core members of the QI team may augment what is actually ‘modifiable’: changes that remain frustratingly out of reach for clinical QI teams81 may fall within the remit of an expanded clinical-managerial group.

o building a valid model that has immediate application in practice. Importantly, the inclusion of managers as core members of the QI team may augment what is actually ‘modifiable’: changes that remain frustratingly out of reach for clinical QI teams81 may fall within the remit of an expanded clinical-managerial group. Figure 1 The model of alignment. Strategically selected quality improvement targets and interventions successfully align the interests of clinicians and non-clinicians at multiple levels within the organisation. At each level, staff engagement with these interventions is facilitated by deliberate incentives to prioritise it, the recognition of competing priorities and barriers to involvement, and actions to address them. How could this model be used in practice? Prospectively, clinicians and managers jointly establish and prioritise the challenges facing their service. Multidisciplinary tools systematically collect data relevant to front-line care delivery problems (from staff and patients).83 84 Teams then assess and rank the apparent safety threats, for example with streamlined versions of tools like Healthcare Failure Modes and Effects Analysis or Hierarchical Task Analysis.81 85 These tools inform the strategic selection of high priority targets for local improvement efforts.

(from staff and patients).83 84 Teams then assess and rank the apparent safety threats, for example with streamlined versions of tools like Healthcare Failure Modes and Effects Analysis or Hierarchical Task Analysis.81 85 These tools inform the strategic selection of high priority targets for local improvement efforts. Interventions are co-designed or adapted with clinicians, with a focus on the quality strategies for which effectiveness and implementation evidence is strongest. Clinicians’ participation is supported, recognised and rewarded, perhaps as part of a formal performance management process. Middle managers coordinate alignment of the improvement efforts with organisational goals, and are themselves heavily incentivised to see QI facilitation as a core role of their own. Influencing their supervisors to attract organisational support and resources for QI efforts, middle managers’ interest in quality is further reinforced by protected board time for quality issues, and board-level use of quality metrics. Board members may need to robustly engage with other stakeholders in the local healthcare economy to generate (and protect) an institution-specific quality strategy.

QI efforts, middle managers’ interest in quality is further reinforced by protected board time for quality issues, and board-level use of quality metrics. Board members may need to robustly engage with other stakeholders in the local healthcare economy to generate (and protect) an institution-specific quality strategy. The model can also be used retrospectively, to describe how interventions were implemented in practice, and to explain their effects. We use the model to explore a recent interdisciplinary intervention in our own institutions, describing mutable influences on staff over the course of a trial (see box 2). Definitive progress on a number of key issues occurred only when front-line staff and middle managers agreed the need for change.

ain their effects. We use the model to explore a recent interdisciplinary intervention in our own institutions, describing mutable influences on staff over the course of a trial (see box 2). Definitive progress on a number of key issues occurred only when front-line staff and middle managers agreed the need for change. We feel the proposed model of alignment is a useful, novel concept for a number of reasons. First, it emphasises the need to go beyond clinical engagement. Second, it highlights the role of non-clinical staff in sustaining effective QI. Most importantly, it integrates separate literature streams on clinical and managerial influences on QI, lending itself to the prospective design of interventions as well as the retrospective analysis of why they achieved their goals or not. Interestingly, multilevel interventions (explicitly addressing patient, professional and organisational factors, for example) show the most consistent improvements in process and clinical outcomes.86 The model of alignment suggests how we might incorporate a similarly multifaceted approach into the design of any new QI intervention. Other authors have recently raised concerns about ‘colliding’ QI interventions, conceived in isolation but effectively competing in a limited marketplace.87 Our model encourages a broader analysis of the environment into which any new QI intervention is launched.

ted approach into the design of any new QI intervention. Other authors have recently raised concerns about ‘colliding’ QI interventions, conceived in isolation but effectively competing in a limited marketplace.87 Our model encourages a broader analysis of the environment into which any new QI intervention is launched. We hope this model will prove useful for future interventions, which will need to more explicitly link their assumptions with underlying theory.88 However, it does not negate the value of existing models, which remain well placed to structure QI reporting and evaluation.21–23 The model of alignment was derived from a narrative review of the literature, drawing on prior systematic reviews,31 59 and tested retrospectively against the experience of implementing a single intervention. It requires more empirical validation, particularly in the prospective design of novel interventions. Lastly, though parsimony is necessary for a ‘good’ theory of context,88 the model of alignment may be too simplistic for some analyses.

sted retrospectively against the experience of implementing a single intervention. It requires more empirical validation, particularly in the prospective design of novel interventions. Lastly, though parsimony is necessary for a ‘good’ theory of context,88 the model of alignment may be too simplistic for some analyses. Conclusion Critics of implementation research argue that its theories are no more helpful than common sense.89 90 Proponents reply that these theories are open to questioning, while common sense—in itself an informal, ‘lay’ theory—relies on implicit assumptions that are difficult to challenge.20 What we have described here, we hope, might satisfy both camps. Sharing the concern that existing theories offer little to practitioners at the sharp end, and informed by our own recent experience of a complex QI intervention, we offer a focal point for the design and evaluation of future attempts to improve healthcare delivery. The extent to which QI aligns the interests of front-line staff and their managers has not previously been explored in this way.

the sharp end, and informed by our own recent experience of a complex QI intervention, we offer a focal point for the design and evaluation of future attempts to improve healthcare delivery. The extent to which QI aligns the interests of front-line staff and their managers has not previously been explored in this way. Perhaps, in truth, we have historically expected too much of clinicians in QI, and demanded too little of their managers. Previous assumptions that managers are well prepared to make meaningful contributions to QI interventions have not been substantiated.59 The next generation of guidelines for QI reporting will emphasise narrative understanding,91 and this should extend to fuller descriptions of whether clinical and managerial priorities coincided (or collided) within the context of the intervention. Although there are few hard barriers to either group's participation in QI, competing demands force clinicians and managers to rationalise their efforts, and in some cases consciously relinquish other priorities. Developing effective and sustainable QI interventions may depend on our ability to align the two groups’ divergent interests. Twitter: Follow Samuel Pannick at @sam_pannick Contributors: The first author conceived the paper and drafted the manuscript, and all authors subsequently revised it for important intellectual content.

Perhaps, in truth, we have historically expected too much of clinicians in QI, and demanded too little of their managers. Previous assumptions that managers are well prepared to make meaningful contributions to QI interventions have not been substantiated.59 The next generation of guidelines for QI reporting will emphasise narrative understanding,91 and this should extend to fuller descriptions of whether clinical and managerial priorities coincided (or collided) within the context of the intervention. Although there are few hard barriers to either group's participation in QI, competing demands force clinicians and managers to rationalise their efforts, and in some cases consciously relinquish other priorities. Developing effective and sustainable QI interventions may depend on our ability to align the two groups’ divergent interests. Twitter: Follow Samuel Pannick at @sam_pannick Contributors: The first author conceived the paper and drafted the manuscript, and all authors subsequently revised it for important intellectual content. Funding: This paper represents independent research supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre (grant RDPSC79560), the West Middlesex University Hospital NHS Trust and Imperial College Healthcare Charity (grant GG14/1022). Sevdalis' research was supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South London at King's College Hospital NHS Foundation Trust. Sevdalis is a member of King's Improvement Science, which is part of the NIHR CLAHRC South London and comprises a specialist team of improvement scientists and senior researchers based at King's College London. Its work is funded by King's Health Partners (Guy's and St Thomas' NHS Foundation Trust, King's College Hospital NHS Foundation Trust, King's College London and South London and Maudsley NHS Foundation Trust), Guy's and St Thomas' Charity, the Maudsley Charity and the Health Foundation. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

King's College Hospital NHS Foundation Trust, King's College London and South London and Maudsley NHS Foundation Trust), Guy's and St Thomas' Charity, the Maudsley Charity and the Health Foundation. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: Nick Sevdalis is the director of London Training & Safety Solutions Ltd, which delivers team assessment and training to hospitals on a consultancy basis. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction As England's independent quality regulator for health and social care, the Care Quality Commission (CQC) uses bespoke statistical surveillance tools to monitor care quality and ‘predict where there may be problems and make better decisions about when, where and what to inspect’.1 These tools exploit the wealth of administrative and outcome data generated by the NHS to identify those providers at greatest risk of providing poor-quality care so that the CQC can prioritise them for inspection and avoid using its limited resources to check up on trusts providing good-quality care.2–4 The UK government now requires all regulators to follow this ‘risk-based’ approach to targeting their inspections in order to ease administrative burdens on regulatees and ensure the proportionality of enforcement action.5–7 In addition to serving the government's ‘better-regulation’ goals, risk-based prioritisation is also important for the CQC itself, given the vast number of health and social care providers its c.1400 inspectors are responsible for overseeing; currently, some 30 261 registered providers operating at 49 528 different locations across England.8–10 With the government reportedly now contemplating funding cuts of up to 40% to NHS inspectorates, the CQC is going to become increasingly reliant on statistical surveillance tools to help it plan and prioritise its inspections according to risk.11 12

providers operating at 49 528 different locations across England.8–10 With the government reportedly now contemplating funding cuts of up to 40% to NHS inspectorates, the CQC is going to become increasingly reliant on statistical surveillance tools to help it plan and prioritise its inspections according to risk.11 12 The use of statistical surveillance tools to assess performance and prioritise NHS hospital trusts for inspection is well established. The first system of its kind developed by the CQC's predecessor, the Healthcare Commission, showed early signs of success.13 However, over the past decade inspections, the standards they assess, and the data available to prioritise them have changed significantly. Developed by the management consultants McKinsey & Company,14 the CQC's current statistical surveillance tool is called Intelligent Monitoring (IM) and is far simpler than earlier tools. IM generates a single trust-level ‘risk score’ based on approximately 150 performance indicators for each NHS trust.15 All of this information is published by the CQC but at present does not fully direct the CQC's inspection activity as it has committed to inspecting and rating all trusts by July 2016, regardless of the risk of them providing poor-quality care.9 Once this initial baseline round of inspection-based quality rating is complete, however, the CQC has said it then plans to use statistical surveillance to help it identify which hospital trusts should be targeted for further inspection and improvement.11 With funding cuts likely to reduce the number of inspections the CQC can conduct, accurate targeting will become even more important. Recently, however, concerns have been raised about the quality and reliability of the indicators used by the IM tool and how they have been combined to calculate the overall risk score.16–18 To date, however, there have been no peer-reviewed assessments of the predictive ability of IM.

will become even more important. Recently, however, concerns have been raised about the quality and reliability of the indicators used by the IM tool and how they have been combined to calculate the overall risk score.16–18 To date, however, there have been no peer-reviewed assessments of the predictive ability of IM. This study addresses that gap by assessing the ability of the CQC's IM statistical surveillance tool to predict the quality of care provided by NHS hospital trusts and thus identify those that should be prioritised for inspection. Our analysis focuses on the case of NHS hospital trusts because these providers offer the most complete and comprehensive set of performance data for statistical surveillance of any area of health and social care. If the IM tool cannot predict which hospital trusts are most likely to be found wanting by inspectors, then it is likely to face even greater challenges in other areas of health and social care that do not offer the same wealth of performance data for statistical surveillance. Method The aim of this study is to assess the ability of the CQC's IM statistical surveillance tool to predict the quality of care provided by NHS hospital trusts and subsequently prioritise them for inspection. To do so, our analysis measures the statistical relationship between IM risk scores and subsequent quality ratings awarded to NHS hospital trusts following detailed on-site inspections. The datasets and methods for this analysis are detailed below.

by NHS hospital trusts and subsequently prioritise them for inspection. To do so, our analysis measures the statistical relationship between IM risk scores and subsequent quality ratings awarded to NHS hospital trusts following detailed on-site inspections. The datasets and methods for this analysis are detailed below. Data Independent variables: quantitative risk scores and ordinal risk bandings Introduced in 2013, the IM tool uses a simple, unweighted method for aggregating approximately 150 indicators to produce a continuous risk score R. These indicators were selected by McKinsey & Company and the CQC following a broad consultation process to identify those indicators that are ‘most important for monitoring risks to the quality of care’.19 These indicators cover a range of areas including, inter alia, mortality rates, waiting times, whistle-blower reports, staff and patient surveys and ‘Healthcare Worker Flu vaccination uptake’.15 Each IM indicator has specific scoring criteria determined by the CQC. For the most part, these scoring criteria are based on a well-established system of Winsorised z-scores2 13 15 20 which defines indicator performance either in terms of statistical deviations from the norm or a predefined target.15 For each indicator, the IM assigns a trust with one of three ordinal scores: 0 (‘No evidence of risk’) 1 (‘Risk’) 2 (‘Elevated risk’)

Each IM indicator has specific scoring criteria determined by the CQC. For the most part, these scoring criteria are based on a well-established system of Winsorised z-scores2 13 15 20 which defines indicator performance either in terms of statistical deviations from the norm or a predefined target.15 For each indicator, the IM assigns a trust with one of three ordinal scores: 0 (‘No evidence of risk’) 1 (‘Risk’) 2 (‘Elevated risk’) Each trust's overall IM risk score, R, is then calculated as a percentage by dividing the sum of these individual scores by the worst possible score; that is, two (the score for ‘Elevated risk’) multiplied by the number of indicators that are relevant to the trust.15 The greater the value of the risk score R, the greater the risk of the trust performing poorly. Risk scores for all trusts were first published on 21 October 2013 and were then simultaneously updated on four subsequent occasions. The CQC also classifies trusts into one of six ordinal risk bands according to their risk score R (table 1). Table 1 Risk bandings applied by CQC to the continuous risk score R15

Each trust's overall IM risk score, R, is then calculated as a percentage by dividing the sum of these individual scores by the worst possible score; that is, two (the score for ‘Elevated risk’) multiplied by the number of indicators that are relevant to the trust.15 The greater the value of the risk score R, the greater the risk of the trust performing poorly. Risk scores for all trusts were first published on 21 October 2013 and were then simultaneously updated on four subsequent occasions. The CQC also classifies trusts into one of six ordinal risk bands according to their risk score R (table 1). Table 1 Risk bandings applied by CQC to the continuous risk score R15 Risk score: R Risk band R≥7.0% Band 1 (highest risk) 5.5%≤R<7.0% Band 2 4.5%≤R<5.5% Band 3 3.5%≤R<4.5% Band 4 2.5%≤R<3.5% Band 5 0≤R<2.5% Band 6 (lowest risk) There are only two circumstances in which the CQC will manually reclassify a trust from the banding to which it was automatically assigned on the basis of its published risk score R, which is itself never adjusted and always represents the pure, quantitative risk score generated by the IM tool. First, trusts placed in ‘special measures’ by the CQC or the economic regulator Monitor—that is, trusts identified as ‘not providing good and safe care to patients’ and whose ‘management cannot fix the problems by themselves’—are classified by the CQC as band 1* regardless of their underlying risk score R.15 21 Second, trusts that have recently been inspected are automatically banded as ‘Recently inspected’. These manual adjustments to the risk bandings are made by the CQC to avoid any dissonance arising between CQC's considered view, as reflected in the assignment of ‘special measures’ or the inspection-based quality rating, and the risk banding.22 Given these manual adjustments, our analysis uses the CQC's continuous risk score R and the associated unadjusted risk bandings generated by the IM tool as they represent the pure, forward-looking prediction of the risk of poor-quality care, rather than a post-hoc adjustment of that prediction to reflect a known historical outcome.

anual adjustments, our analysis uses the CQC's continuous risk score R and the associated unadjusted risk bandings generated by the IM tool as they represent the pure, forward-looking prediction of the risk of poor-quality care, rather than a post-hoc adjustment of that prediction to reflect a known historical outcome. Dependent variables: inspection-based quality ratings Inspection-based quality ratings are assigned to individual NHS hospital trusts by the CQC under a new regime of trust-wide inspection introduced in 2013. Under that regime, inspections are conducted by large specialist teams who assess individual hospital services against five ‘key questions’: is the service ‘safe’, ‘effective’, ‘caring’, ‘responsive to people's needs’ and ‘well led’?22 Based on their inspection teams’ on-site visits, the CQC then award one of four possible ordinally ranked ratings for each service within a hospital: Outstanding Good Requires improvement Inadequate The CQC then aggregates those service-level ratings using an algorithm to assign hospital-level ratings, which, in turn, are further aggregated, using a similar algorithm, to generate an overall trust-level rating. These aggregated trust-level ratings, but not the underlying rules for generating them, are publicly available on the CQC website.

hose service-level ratings using an algorithm to assign hospital-level ratings, which, in turn, are further aggregated, using a similar algorithm, to generate an overall trust-level rating. These aggregated trust-level ratings, but not the underlying rules for generating them, are publicly available on the CQC website. Between 17 October 2013, when the new style of inspection began, and 29 September 2015, the CQC completed and published the quality ratings resulting from 103 inspections of 90 different NHS hospital trusts. This represents 55% of all NHS hospital trusts in England. The CQC has committed to inspecting the remaining trusts at least once before July 20169 to provide a baseline against which to measure future performance. Accordingly, in scheduling this initial round of inspections, CQC senior management considered a host of factors in addition to the risk score including the geographical availability of expert inspection team members; the time since the last inspection; the need to evaluate the new inspection methodology; and the commitment to inspect all trusts regardless of risk. Visual inspection of figure 1 does not highlight any clear temporal trends in the risk scores R (left y-axis), risk bandings prior to any adjustment (right y-axis), time elapsed between risk score publication date and inspection start-date (distance from preceding vertical line) or inspection-based quality ratings (shape and colour), each of which might have been indicative of selection bias in our sample of trusts.

ft y-axis), risk bandings prior to any adjustment (right y-axis), time elapsed between risk score publication date and inspection start-date (distance from preceding vertical line) or inspection-based quality ratings (shape and colour), each of which might have been indicative of selection bias in our sample of trusts. Figure 1 Time series of inspection-based quality ratings (coloured shapes) charted by the most recently published continuous risk score R (left y-axis) and associated ordinal risk banding (right y-axis) at the start of the inspection. The publication date for each of the five sets of Intelligent Monitoring (IM) risk scores generated to date is shown with a white vertical line. The summary statistics in table 2 also support the assumption that our sample is representative. Although there is evidence for some initial prioritisation of trusts in high-risk bands, thereafter the distribution of trusts between high-risk and low-risk bands, and in terms of their subsequent inspection-based quality ratings, is fairly consistent over time. Moreover, there is little variation by risk banding or inspection outcome in the average elapsed time between the publication of the risk scores and the inspection start-date during which the quality of care has the potential to worsen or improve. Table 2 The number of inspections and the average elapsed time (in days) between the inspection start-date and the most recently published Intelligent Monitoring (IM) risk score aggregated by risk banding and inspection-based quality rating

The summary statistics in table 2 also support the assumption that our sample is representative. Although there is evidence for some initial prioritisation of trusts in high-risk bands, thereafter the distribution of trusts between high-risk and low-risk bands, and in terms of their subsequent inspection-based quality ratings, is fairly consistent over time. Moreover, there is little variation by risk banding or inspection outcome in the average elapsed time between the publication of the risk scores and the inspection start-date during which the quality of care has the potential to worsen or improve. Table 2 The number of inspections and the average elapsed time (in days) between the inspection start-date and the most recently published Intelligent Monitoring (IM) risk score aggregated by risk banding and inspection-based quality rating IM risk score publication date 21 October 2013 13 March 2014 24 July 2014 3 December 2014 29 May 2015 Total n Average time n Average time n Average time n Average time n Average time n Average time Risk banding High risk (bands 1–3) 11 92.3 13 66.7 11 83.7 12 91.1 3 32.3 50 79.9 Low risk (bands 4–6)  5 98.4 18 62.1  9 78.0 20 80.5 1 5.0 53 74.1 Total 16 94.2 31 64.1 20 81.2 32 84.5 4 25.5 103 76.9 Inspection rating Inadequate  3 119.0 3 59.3  4 69.0  3 129.7 0 – 13 92.3 Requires improvement  8 85.0 23 65.7 13 83.5 21 87.2 4 25.5 69 75.5 Good  5 94.0  4 44.3  3 87.3  7 63.3 0 – 19 71.2 Outstanding  0 −  1 119.0  0 −  1 41.0 0 –  2 80.0 Total 16 94.2 31 64.1 20 81.2 32 84.5 4 25.5 103 76.9 There are good reasons, therefore, to assume that the 55% sample of hospital trusts that have so far been inspected is broadly representative of the totality of NHS hospital trusts and provides a robust sample for analysis.

ing  0 −  1 119.0  0 −  1 41.0 0 –  2 80.0 Total 16 94.2 31 64.1 20 81.2 32 84.5 4 25.5 103 76.9 There are good reasons, therefore, to assume that the 55% sample of hospital trusts that have so far been inspected is broadly representative of the totality of NHS hospital trusts and provides a robust sample for analysis. Data linkage We paired each of the 103 trust-level inspection-based quality ratings in our dataset with the most recent risk score generated by the IM tool for the corresponding NHS trust prior to its inspection. Statistical methods We conducted a series of statistical tests to assess the ability of IM to predict the quality of care provided by NHS trusts and identify which ones should be prioritised for inspection. First, we used ordinal regression analysis to test the hypothesis (H1): (H1): IM risk scores can predict the subsequent inspection-based quality rating

Statistical methods We conducted a series of statistical tests to assess the ability of IM to predict the quality of care provided by NHS trusts and identify which ones should be prioritised for inspection. First, we used ordinal regression analysis to test the hypothesis (H1): (H1): IM risk scores can predict the subsequent inspection-based quality rating By testing the statistical relationship between the inspection-based quality ratings and the continuous risk score, R, ordinal regression makes full use of all the information to assess the ability of IM to predict the quality of care provided by NHS hospital trusts. This ability to predict the probability of any given inspection-based quality rating represents the ideal for a fully risk-based approach to prioritisation. If the CQC could predict the specific quality rating, it would then be able to set differentiated thresholds for triggering inspections that reflect how its tolerance for false positive errors (inspecting trusts performing well) and false negative errors (not inspecting trusts performing badly) may vary depending upon how bad performance is likely to be.

c quality rating, it would then be able to set differentiated thresholds for triggering inspections that reflect how its tolerance for false positive errors (inspecting trusts performing well) and false negative errors (not inspecting trusts performing badly) may vary depending upon how bad performance is likely to be. For the purposes of prioritisation, however, predicting the exact inspection-based quality rating may not be necessary. Statistical surveillance could still be useful so long as IM can distinguish trusts performing acceptably from those performing below some threshold of acceptability. To assess its ability at this less demanding task, we conducted two further sets of statistical tests. First, we conducted a series of binary logistic regressions to see whether the continuous risk score R could distinguish between different acceptability thresholds. Second, we used the CQC's unadjusted ordinal risk bandings, rather than the continuous risk score R, to divide trusts into ‘high-risk’ and ‘low-risk’ groups for which we then ran a series of χ2 tests of independence to see if there is any statistically significant association between those binary groupings and various possible thresholds of acceptability in the subsequent inspection-based quality ratings. This latter approach to assessing the value of statistical surveillance has been adopted by the CQC itself and National Audit Office (NAO).23 These two statistical tests were both run to test two further hypotheses about IM and its ability to identify trusts that should be prioritised for inspection:

For the purposes of prioritisation, however, predicting the exact inspection-based quality rating may not be necessary. Statistical surveillance could still be useful so long as IM can distinguish trusts performing acceptably from those performing below some threshold of acceptability. To assess its ability at this less demanding task, we conducted two further sets of statistical tests. First, we conducted a series of binary logistic regressions to see whether the continuous risk score R could distinguish between different acceptability thresholds. Second, we used the CQC's unadjusted ordinal risk bandings, rather than the continuous risk score R, to divide trusts into ‘high-risk’ and ‘low-risk’ groups for which we then ran a series of χ2 tests of independence to see if there is any statistically significant association between those binary groupings and various possible thresholds of acceptability in the subsequent inspection-based quality ratings. This latter approach to assessing the value of statistical surveillance has been adopted by the CQC itself and National Audit Office (NAO).23 These two statistical tests were both run to test two further hypotheses about IM and its ability to identify trusts that should be prioritised for inspection: (H2) IM can distinguish the trusts performing poorly—that is, those subsequently rated either ‘Requires improvement’ or ‘Inadequate’—from the trusts performing well—that is, those subsequently rated either ‘Good’ or ‘Outstanding’.

These two statistical tests were both run to test two further hypotheses about IM and its ability to identify trusts that should be prioritised for inspection: (H2) IM can distinguish the trusts performing poorly—that is, those subsequently rated either ‘Requires improvement’ or ‘Inadequate’—from the trusts performing well—that is, those subsequently rated either ‘Good’ or ‘Outstanding’. (H3) IM can distinguish the trusts rated ‘Inadequate’—the most serious category of poor performance—from all other trusts. Following this same logic, we might also have tested whether IM can successfully predict which trusts would be rated ‘Outstanding’, and should therefore be excluded from any inspections. However, with only two out of 103 trusts inspected to date having been rated ‘Outstanding’, this hypothesis could not be robustly assessed. For each of the three hypotheses, we report the key findings in the next section of the paper and confine the more technical statistical output to appendices. Since these analyses involve only one independent variable—either the continuous risk score, R, or some binary grouping of the data based upon it—there are no issues with multicollinearity. Results (H1): IM risk scores can predict the subsequent inspection-based quality rating

For each of the three hypotheses, we report the key findings in the next section of the paper and confine the more technical statistical output to appendices. Since these analyses involve only one independent variable—either the continuous risk score, R, or some binary grouping of the data based upon it—there are no issues with multicollinearity. Results (H1): IM risk scores can predict the subsequent inspection-based quality rating Figure 1 above does not suggest any clear relationship between the continuous risk scores and the subsequent inspection-based quality ratings. Visual comparison of the distribution of IM risk scores by subsequent rating does not suggest a clear relationship either (figure 2). While the mean risk score for the two trusts rated ‘Outstanding’ is much lower than those for the other three categories, a number of other trusts had similar, or even lower, risk scores than the two rated ‘Outstanding’, including one given the very lowest rating of ‘Inadequate’. Similarly, the predictive ability implied by the upward trend in the mean risk scores for each successively more negative category of inspection-based quality rating is belied by the substantial overlap between their risk score distributions. Indeed, the median risk score, which is less influenced by outliers and provides a better measure of central tendency than the mean risk score, was actually higher for those trusts subsequently rated ‘Good’ (R=4.74%), than for those trusts rated ‘Requires improvement’ (R=3.85%).

between their risk score distributions. Indeed, the median risk score, which is less influenced by outliers and provides a better measure of central tendency than the mean risk score, was actually higher for those trusts subsequently rated ‘Good’ (R=4.74%), than for those trusts rated ‘Requires improvement’ (R=3.85%). The ordinal logistic regression model confirms our initial visual assessment of the data. Table 3 above indicates that the risk score R is significant at the 10% level when differentiating between the ‘Good’ and two ‘Outstanding’ trusts—the least useful distinction in a risk-based approach—but not between any of the other ordinal quality ratings (see online supplementary appendix A for further details). Figure 2 Boxplot of Intelligent Monitoring (IM) risk scores by Care Quality Commission's (CQC's) subsequent inspection-based quality rating. The bottom line (or whisker) indicates the lowest value of R, the bottom edge of the white box indicates the first quartile, the line in the middle of the box the median and the line at the top of the white box the third quartile. The top line (or whisker) indicates the maximum value of R or 1.5 times the interquartile range, if that is less than the maximum value of R, with outlying values of R indicated by individual points. Table 3 The regression coefficients and associated SEs, ORs and associated 95% CIs and p values for the ordinal logistic regression

Figure 2 Boxplot of Intelligent Monitoring (IM) risk scores by Care Quality Commission's (CQC's) subsequent inspection-based quality rating. The bottom line (or whisker) indicates the lowest value of R, the bottom edge of the white box indicates the first quartile, the line in the middle of the box the median and the line at the top of the white box the third quartile. The top line (or whisker) indicates the maximum value of R or 1.5 times the interquartile range, if that is less than the maximum value of R, with outlying values of R indicated by individual points. Table 3 The regression coefficients and associated SEs, ORs and associated 95% CIs and p values for the ordinal logistic regression Beta (SE) 95% CI for OR 2.50% OR 97.50% Pr(>|z|) Outstanding/good (intercept) −1.190 (0.898) Good/requires improvement (intercept) −1.023 (0.468) Requires improvement/inadequate (intercept) 2.520 (0.600) Outstanding/good, R −0.965 (0.519) 0.138 0.381 1.053 0.063 Good/requires improvement, R −0.066 (0.081) 0.799 0.936 1.097 0.416 Requires improvement/inadequate, R −0.103 (0.085) 0.764 0.902 1.066 0.228 10.1136/bmjqs-2015-004687.supp1Supplementary data The IM risk scores therefore are not statistically significant predictors of inspection outcomes. This finding, coupled with the distribution of risk scores preceding each inspection rating, suggests that IM cannot predict inspection-based quality ratings.

Beta (SE) 95% CI for OR 2.50% OR 97.50% Pr(>|z|) Outstanding/good (intercept) −1.190 (0.898) Good/requires improvement (intercept) −1.023 (0.468) Requires improvement/inadequate (intercept) 2.520 (0.600) Outstanding/good, R −0.965 (0.519) 0.138 0.381 1.053 0.063 Good/requires improvement, R −0.066 (0.081) 0.799 0.936 1.097 0.416 Requires improvement/inadequate, R −0.103 (0.085) 0.764 0.902 1.066 0.228 10.1136/bmjqs-2015-004687.supp1Supplementary data The IM risk scores therefore are not statistically significant predictors of inspection outcomes. This finding, coupled with the distribution of risk scores preceding each inspection rating, suggests that IM cannot predict inspection-based quality ratings. (H2) IM can distinguish the trusts performing poorly—that is, those subsequently rated either ‘Requires improvement’ or ‘Inadequate’ from the trusts performing well—that is, those subsequently rated either ‘Good’ or ‘Outstanding’. If the IM tool is not able to predict specific inspection-based quality ratings, it might still be useful for prioritising inspections if it can at least predict which trusts are performing poorly, regardless of the severity of that poor performance, so that they can be prioritised for inspection. To test that hypothesis, we combined those trusts rated ‘Requires improvement’ and ‘Inadequate’ to form a new ‘performing poorly’ category and compared them against those rated ‘Outstanding’ and ‘Good’, which were combined to form a new ‘performing well’ category.

rmance, so that they can be prioritised for inspection. To test that hypothesis, we combined those trusts rated ‘Requires improvement’ and ‘Inadequate’ to form a new ‘performing poorly’ category and compared them against those rated ‘Outstanding’ and ‘Good’, which were combined to form a new ‘performing well’ category. Visual comparison of the distribution of the risk scores by these two new inspection-based quality categories suggests that the risk score R cannot distinguish those trusts performing poorly from those performing well (figure 3). While the mean risk score for trusts subsequently judged to be ‘performing poorly’ is greater than for trusts judged to be ‘performing well’, the median risk score is lower, indicating a lower probability of poor performance for trusts judged to be ‘performing poorly’ than for those judged to be ‘performing well’. Figure 3 Boxplot of Intelligent Monitoring risk scores by grouped inspection-based quality rating. Trusts rated by the Care Quality Commission (CQC) as ‘Outstanding’ or ‘Good’ are grouped here as ‘Performing well’, while those the CQC rated as ‘Requires improvement’ or ‘Inadequate’ are grouped as ‘Performing poorly’.

Visual comparison of the distribution of the risk scores by these two new inspection-based quality categories suggests that the risk score R cannot distinguish those trusts performing poorly from those performing well (figure 3). While the mean risk score for trusts subsequently judged to be ‘performing poorly’ is greater than for trusts judged to be ‘performing well’, the median risk score is lower, indicating a lower probability of poor performance for trusts judged to be ‘performing poorly’ than for those judged to be ‘performing well’. Figure 3 Boxplot of Intelligent Monitoring risk scores by grouped inspection-based quality rating. Trusts rated by the Care Quality Commission (CQC) as ‘Outstanding’ or ‘Good’ are grouped here as ‘Performing well’, while those the CQC rated as ‘Requires improvement’ or ‘Inadequate’ are grouped as ‘Performing poorly’. With a p value of 0.424 and an OR of 1.069 (95% CI 0.908 to 1.257), logistic regression analysis shows that the continuous risk score is not a statistically significant predictor of whether or not a trust is performing poorly (see online supplementary appendix B). The substantial variation in risk scores preceding each rating category resulted in a very ill-fitting model with a pseudo Nagelkerke R2 value of 0.026 and a receiver operating characteristic (ROC) area under the curve value of 0.540 (95% CI 0.321 to 0.599). The continuous risk score, therefore, does not provide a reliable basis for identifying those trusts that are performing poorly and should be prioritised for inspection.

with a pseudo Nagelkerke R2 value of 0.026 and a receiver operating characteristic (ROC) area under the curve value of 0.540 (95% CI 0.321 to 0.599). The continuous risk score, therefore, does not provide a reliable basis for identifying those trusts that are performing poorly and should be prioritised for inspection. If logistic regression shows that the continuous risk score R is of no use in predicting poor performance, an alternative approach would be to group the trusts using CQC's unadjusted risk bandings, so as to study the general signal from the IM tool rather than the precise risk scores. We can then use the χ2 test of independence to see if these banded risk scores can be used to distinguish trusts that are ‘performing poorly’ from those that are ‘performing well’. That is, can we identify a ‘high-risk’ group in which a significantly greater proportion of trusts are ‘performing poorly’ compared with the ‘low-risk’ group? We tried a number of approaches to using the risk scores to divide trusts into low-risk and high-risk categories. First, following the approach taken by the CQC and NAO, we categorised the CQC's risk bands 1, 2 and 3 as ‘high risk’ and bands 4, 5 and 6 as ‘low risk’ and then compared the subsequent inspection-based quality ratings awarded to trusts in each of our two risk categories (table 4). Table 4 A contingency table showing the number of trusts classified as ‘high risk’ (ie, having a risk score in bands 1, 2 or 3) and ‘low risk’, that is, having a risk score in bands 4, 5 or 6, against their subsequent inspection outcome

If logistic regression shows that the continuous risk score R is of no use in predicting poor performance, an alternative approach would be to group the trusts using CQC's unadjusted risk bandings, so as to study the general signal from the IM tool rather than the precise risk scores. We can then use the χ2 test of independence to see if these banded risk scores can be used to distinguish trusts that are ‘performing poorly’ from those that are ‘performing well’. That is, can we identify a ‘high-risk’ group in which a significantly greater proportion of trusts are ‘performing poorly’ compared with the ‘low-risk’ group? We tried a number of approaches to using the risk scores to divide trusts into low-risk and high-risk categories. First, following the approach taken by the CQC and NAO, we categorised the CQC's risk bands 1, 2 and 3 as ‘high risk’ and bands 4, 5 and 6 as ‘low risk’ and then compared the subsequent inspection-based quality ratings awarded to trusts in each of our two risk categories (table 4). Table 4 A contingency table showing the number of trusts classified as ‘high risk’ (ie, having a risk score in bands 1, 2 or 3) and ‘low risk’, that is, having a risk score in bands 4, 5 or 6, against their subsequent inspection outcome ‘Performing well’ ‘Performing poorly’ ‘High risk’ 11 39 ‘Low risk’ 10 43 The number of incorrect predictions (11 false negatives+43 false positives=54 errors, shaded in red) slightly exceeds the number of correct predictions (39+10=49, shaded in green), resulting in an overall accuracy rate of just 47.57% (49/103).

Table 4 A contingency table showing the number of trusts classified as ‘high risk’ (ie, having a risk score in bands 1, 2 or 3) and ‘low risk’, that is, having a risk score in bands 4, 5 or 6, against their subsequent inspection outcome ‘Performing well’ ‘Performing poorly’ ‘High risk’ 11 39 ‘Low risk’ 10 43 The number of incorrect predictions (11 false negatives+43 false positives=54 errors, shaded in red) slightly exceeds the number of correct predictions (39+10=49, shaded in green), resulting in an overall accuracy rate of just 47.57% (49/103). We tried the four other possible groupings and no cut-off point produces a significant result in the χ2 tests (full details are available in see online supplementary appendix B). The IM tool, therefore, cannot pass the less onerous test of predicting a high-risk group of trusts that are significantly more likely to be ‘performing poorly’ and that the CQC might be able to prioritise for inspection. (H3) IM can distinguish the trusts rated ‘Inadequate’—the most serious category of poor performance—from all other trusts.

We tried the four other possible groupings and no cut-off point produces a significant result in the χ2 tests (full details are available in see online supplementary appendix B). The IM tool, therefore, cannot pass the less onerous test of predicting a high-risk group of trusts that are significantly more likely to be ‘performing poorly’ and that the CQC might be able to prioritise for inspection. (H3) IM can distinguish the trusts rated ‘Inadequate’—the most serious category of poor performance—from all other trusts. If IM cannot distinguish between those trusts ‘performing poorly’ and those trusts ‘performing well’, it might still be useful for prioritisation if it can at least predict which trusts will be found ‘Inadequate’ in order to target them for immediate investigation. An initial visual assessment of the distribution of risk scores suggests that the risk score has somewhat more ability at predicting whether or not a trust is ‘Inadequate’ (figure 4) than it does for distinguishing the broader category of trusts that are ‘performing poorly’ (figure 3). Both the mean and median risk scores for trusts that were subsequently rated as ‘Inadequate’ are higher than for those trusts that were not. Figure 4 Boxplot of risk scores by grouped inspection-based quality rating. Trusts rated by Care Quality Commission as ‘Outstanding’, ‘Good’ or ‘Requires improvement’ are grouped together as ‘Not inadequate’ while trusts rated as ‘Inadequate’ remain as ‘Inadequate’.

If IM cannot distinguish between those trusts ‘performing poorly’ and those trusts ‘performing well’, it might still be useful for prioritisation if it can at least predict which trusts will be found ‘Inadequate’ in order to target them for immediate investigation. An initial visual assessment of the distribution of risk scores suggests that the risk score has somewhat more ability at predicting whether or not a trust is ‘Inadequate’ (figure 4) than it does for distinguishing the broader category of trusts that are ‘performing poorly’ (figure 3). Both the mean and median risk scores for trusts that were subsequently rated as ‘Inadequate’ are higher than for those trusts that were not. Figure 4 Boxplot of risk scores by grouped inspection-based quality rating. Trusts rated by Care Quality Commission as ‘Outstanding’, ‘Good’ or ‘Requires improvement’ are grouped together as ‘Not inadequate’ while trusts rated as ‘Inadequate’ remain as ‘Inadequate’. However, logistic regression for the risk score R as the sole independent variable produced an insignificant p value of 0.225 with an OR of 1.111 (95% CI 0.937 to 1.317) (see online supplementary appendix C). The logistic regression model is again a very poor fit with a Nagelkerke pseudo R2 score of 0.010 and an ROC area under the curve value of 0.616 (95% CI 0.450 to 0.783). There is so much variation in the risk scores for those trusts subsequently rated as ‘Inadequate’ and those trusts that were not, that no well-fitting binary logistic model exists.

is again a very poor fit with a Nagelkerke pseudo R2 score of 0.010 and an ROC area under the curve value of 0.616 (95% CI 0.450 to 0.783). There is so much variation in the risk scores for those trusts subsequently rated as ‘Inadequate’ and those trusts that were not, that no well-fitting binary logistic model exists. We also examined the banded risk scores to see whether a greater proportion of ‘Inadequate’ trusts could be categorised as ‘high risk’ rather than ‘low risk’. The most statistically significant result using CQC's bandings is achieved when CQC risk band 1 is classified as ‘high risk’ and all the other CQC risk bands are classified as ‘low risk’ (table 5). Table 5 A contingency table showing the number of trusts classified as ‘high risk’ (ie, having a risk score in band 1) and ‘low risk’ (ie, having a risk score in bands 2, 3, 4, 5 or 6) against their subsequent inspection outcome

We also examined the banded risk scores to see whether a greater proportion of ‘Inadequate’ trusts could be categorised as ‘high risk’ rather than ‘low risk’. The most statistically significant result using CQC's bandings is achieved when CQC risk band 1 is classified as ‘high risk’ and all the other CQC risk bands are classified as ‘low risk’ (table 5). Table 5 A contingency table showing the number of trusts classified as ‘high risk’ (ie, having a risk score in band 1) and ‘low risk’ (ie, having a risk score in bands 2, 3, 4, 5 or 6) against their subsequent inspection outcome ‘Not inadequate’ Inadequate High risk 21 6 Low risk 69 7 This time, the number of correct predictions (6+69=75, shaded in green) far exceeds the number of incorrect predictions (21 false positives+7 false negatives=28, shaded in red), for an overall accuracy rate of 72.82% (75/103). The χ2 value is significant at the 10% level (p=0.080). Since the aim of this prioritisation scheme is to focus CQC inspections on those trusts most likely to be ‘Inadequate’, the 20.38% false positive error rate may be less important than the false negative failures of detection. Unfortunately, however, this prioritisation scheme identified less than half (6 out of 13) of the trusts found to be ‘Inadequate’. The results of χ2 tests on three possible remaining, less statistically significant classification schemes are detailed in online supplementary appendix C.

e negative failures of detection. Unfortunately, however, this prioritisation scheme identified less than half (6 out of 13) of the trusts found to be ‘Inadequate’. The results of χ2 tests on three possible remaining, less statistically significant classification schemes are detailed in online supplementary appendix C. The IM tool, therefore, was able to categorise trusts as high or low risk with a significantly greater proportion of high-risk trusts subsequently being found ‘Inadequate’. The practical benefit of such a prioritisation method is questionable, however, as it successfully identifies fewer than half of the ‘Inadequate’ trusts, incorrectly identifies more than three times as many ‘Not Inadequate’ trusts and does not account for ‘Requires improvement’ trusts which are also performing poorly. The CQC, understandably, has a very limited risk appetite and no tolerance for ‘Inadequate’ care.24

identifies fewer than half of the ‘Inadequate’ trusts, incorrectly identifies more than three times as many ‘Not Inadequate’ trusts and does not account for ‘Requires improvement’ trusts which are also performing poorly. The CQC, understandably, has a very limited risk appetite and no tolerance for ‘Inadequate’ care.24 Discussion Statement of principal findings The IM tool failed three successively less demanding statistical tests of its ability to predict quality and identify which NHS hospitals should be prioritised for inspection. First, ordinal logistic regression confirmed that the IM tool's risk scores cannot predict inspection-based quality ratings of NHS hospital trusts. Statistical surveillance does not provide the CQC with a statistically significant basis for setting differentiated thresholds for triggering inspections that reflect how its tolerance for false positive errors (inspecting trusts performing well) and false negative errors (not inspecting trusts performing badly) may vary depending upon how bad performance is likely to be.

a statistically significant basis for setting differentiated thresholds for triggering inspections that reflect how its tolerance for false positive errors (inspecting trusts performing well) and false negative errors (not inspecting trusts performing badly) may vary depending upon how bad performance is likely to be. Second, the risk scores cannot be used more simply to distinguish the trusts performing poorly—that is, those subsequently rated by inspectors either ‘Requires improvement’ or ‘Inadequate’—from the trusts performing well—that is, those subsequently rated either ‘Good’ or ‘Outstanding’. The binary logistic regression results in a p value of 0.424 indicating no statistically significant relationship between the continuous risk score and the subsequent performance category. Furthermore, the risk scores cannot be used to perform the less onerous task of determining a high-risk group of trusts which contain a significantly greater proportion of trusts ‘performing poorly’ than in the low-risk group containing the remaining trusts.

ontinuous risk score and the subsequent performance category. Furthermore, the risk scores cannot be used to perform the less onerous task of determining a high-risk group of trusts which contain a significantly greater proportion of trusts ‘performing poorly’ than in the low-risk group containing the remaining trusts. Finally, the risk scores are of limited use for distinguishing the worst performing trusts—that is, those subsequently rated ‘Inadequate’—from the rest. For this test, the binary logistic regression results in an insignificant p value of 0.225. It is possible to use the risk scores to define a high-risk group of trusts of which 6 out of 27 are ‘Inadequate’; a significantly greater proportion than in the low-risk group of trusts of which 7 out of 76 are ‘Inadequate’. Although this classification is statistically significant at the 10% level (p=0.080), its practical value for prioritisation is limited as it successfully identifies fewer than half of the ‘Inadequate’ trusts that must be prioritised for inspection. Strengths and limitations of the study Our analysis includes all 103 inspections completed and published in the 2 years since the launch of the new IM statistical surveillance tool. This dataset includes inspections of just over half of all NHS hospital trusts in England. The sample showed little evidence of significant temporal bias or trend in terms of risk score, risk bandings, time elapsed between risk score publication date and inspection start-date, and inspection-based quality rating.

tool. This dataset includes inspections of just over half of all NHS hospital trusts in England. The sample showed little evidence of significant temporal bias or trend in terms of risk score, risk bandings, time elapsed between risk score publication date and inspection start-date, and inspection-based quality rating. One limitation of the study is the possibility that the CQC may have used accurate supplementary intelligence either to bring forwards the inspection of some trusts with low-risk scores that it suspects, for reasons not reflected by the IM, of delivering poor care or to delay the inspection of some trusts with high-risk scores that it suspects, for reasons outside of IM, of delivering good care. This would increase or decrease respectively the probability of an inaccurate risk score being included in our sample, and could understate or overstate the ability of IM. Inaccurate supplementary intelligence may have the opposite effect. It is not possible to determine the extent or direction of this possible bias.

. This would increase or decrease respectively the probability of an inaccurate risk score being included in our sample, and could understate or overstate the ability of IM. Inaccurate supplementary intelligence may have the opposite effect. It is not possible to determine the extent or direction of this possible bias. Policy implications This study highlights problems with the CQC's risk-based approach to making ‘sure health and social care services provide people with safe, effective, compassionate, high-quality care’.25 This regulatory strategy depends on the ability of the CQC to predict which providers are at greatest risk of performing poorly so that it can then prioritise them for inspection and improvement. Yet our analysis shows that the CQC's current surveillance tool—IM—is unable to predict which hospital trusts are most likely to be providing poor-quality care. By the CQC and NAO's measure, IM's predictions are wrong more often than they are right. Potential funding cuts and other pressures, however, have led the CQC to announce plans for greater prioritisation of its future inspection activities according to risk.11 If this is to be the case, then a new statistical surveillance approach will be required. Given the nature of our data and analysis, we cannot say why the risk scores produced by the CQC's IM statistical surveillance tool failed to predict the inspection-based quality ratings. There are, however, a number of possible explanations that should be investigated in developing an improved approach.

l be required. Given the nature of our data and analysis, we cannot say why the risk scores produced by the CQC's IM statistical surveillance tool failed to predict the inspection-based quality ratings. There are, however, a number of possible explanations that should be investigated in developing an improved approach. It may be that the IM tool is too simplistic. Some IM indicators may be more effective predictors of care quality than others, but their signal could be going undetected by the tool because it weights each indicator equally. Moreover, the NHS generates hundreds of available indicators that are not currently used by the IM tool, but which may be more effective predictors of care quality. An ideal next step, therefore, would be to conduct a machine learning exercise to determine whether there is any combination of indicators that could predict the outcome of historic inspections more reliably. The development of a successful model would offer hope for the continued use of a risk-based approach. Should it prove impossible to develop an improved model through machine learning, it may be because existing indicator data are inadequate. For example, IM can only produce trust-level risk scores as the majority of data is only available at trust level. This may be too coarse in scale to discern the localised pockets of poor quality detected by skilled inspectors. The collection of more hospital-level data might resolve this problem but the cost and practicality might be prohibitive.

uce trust-level risk scores as the majority of data is only available at trust level. This may be too coarse in scale to discern the localised pockets of poor quality detected by skilled inspectors. The collection of more hospital-level data might resolve this problem but the cost and practicality might be prohibitive. If gathering improved data and then using a machine learning approach does not work, there would then be two further possible explanations for why the current, or indeed any future improved statistical surveillance model, finds it difficult to predict the outcome of CQC inspections. First, it may be that statistical surveillance is effective at identifying poor-quality care, but that the quality ratings assigned by inspectors are unreliable.26 However, the current system of CQC's comprehensive trust-wide inspections by large teams of specialist inspectors, clinicians and ‘experts by experience’ have received widespread support and it would be challenging to expand them further.8 23 27 Second, it may be that data-driven statistical surveillance systems and inspectors are simply assessing different things. There are substantial challenges to comprehensively assessing the quality of care provided by large, complex, multisite NHS hospital trusts and their highly skilled workforce. Moreover, judgements of care quality are subjective, intangible and difficult to capture by indicators alone. Indeed, if it were simple, there would be little need for inspection.

comprehensively assessing the quality of care provided by large, complex, multisite NHS hospital trusts and their highly skilled workforce. Moreover, judgements of care quality are subjective, intangible and difficult to capture by indicators alone. Indeed, if it were simple, there would be little need for inspection. Regardless of the explanation, if a reliable method of statistical surveillance cannot be found, then the CQC cannot realise the efficiencies promised by prioritising its inspections according to risk. Moreover, given the prevalence of poor-quality care, risk-based targeting of inspections may provide little benefit. Of the 103 inspections in our sample, 81 resulted in an overall rating of either ‘Requires improvement’ or ‘Inadequate’ and a further 12 included at least one rating of ‘Requires improvement’ concerning whether the trust was ‘safe’, ‘caring’, ‘responsive’, ‘effective’ or ‘well led’. In that context, rather than trying and failing to target inspections, it may be preferable to continue systematically inspecting each trust or adopting a random approach to signal to managers and the public alike that every hospital matters.28 With reduced resources, the length of time between inspections is likely to increase, but this may still be more desirable than poor-quality care going undetected for even longer periods in trusts the CQC mistakenly believes to be low risk. Whatever strategy is adopted, there is clear value in empirically evaluating whether the purported benefits are achieved in practice.

pections is likely to increase, but this may still be more desirable than poor-quality care going undetected for even longer periods in trusts the CQC mistakenly believes to be low risk. Whatever strategy is adopted, there is clear value in empirically evaluating whether the purported benefits are achieved in practice. Unanswered questions and future research While statistical surveillance tools were first developed for monitoring hospitals, the CQC now uses them across the whole health and social care sector. Given the results of our analysis, it would be valuable to assess their effectiveness at identifying poor-quality care in other areas of health and social care and prioritising the CQC's associated inspection activities. Conclusions We have shown that the CQC's IM statistical surveillance tool cannot predict which NHS hospital trusts are at greatest risk of delivering poor-quality care and should be prioritised for inspection. A new approach to statistical surveillance and inspection planning is therefore required.

Unanswered questions and future research While statistical surveillance tools were first developed for monitoring hospitals, the CQC now uses them across the whole health and social care sector. Given the results of our analysis, it would be valuable to assess their effectiveness at identifying poor-quality care in other areas of health and social care and prioritising the CQC's associated inspection activities. Conclusions We have shown that the CQC's IM statistical surveillance tool cannot predict which NHS hospital trusts are at greatest risk of delivering poor-quality care and should be prioritised for inspection. A new approach to statistical surveillance and inspection planning is therefore required. Contributors: AG co-conceived the study, sourced the data, undertook the statistical analyses and co-wrote the initial manuscript. HR co-conceived the study and co-wrote the initial manuscript with DD who subsequently helped restructure the statistical analysis and revise the manuscript. A-LB provided original insight based on her own research into healthcare regulation and, along with AG, DD and HR, contributed to and approved the final manuscript. AG is the study's guarantor. This study was undertaken by AG, A-LB, DD and HR in their personal capacities. The opinions expressed in this article are the authors' own and do not reflect the view of CQC. Funding: Economic and Social Research Council (ES/K006169/1).

Contributors: AG co-conceived the study, sourced the data, undertook the statistical analyses and co-wrote the initial manuscript. HR co-conceived the study and co-wrote the initial manuscript with DD who subsequently helped restructure the statistical analysis and revise the manuscript. A-LB provided original insight based on her own research into healthcare regulation and, along with AG, DD and HR, contributed to and approved the final manuscript. AG is the study's guarantor. This study was undertaken by AG, A-LB, DD and HR in their personal capacities. The opinions expressed in this article are the authors' own and do not reflect the view of CQC. Funding: Economic and Social Research Council (ES/K006169/1). Competing interests: AG reports part-time employment with the Care Quality Commission (though the work was conceived, conducted and authored with complete independence) and funding from the ESRC and Quality Assurance Agency for Higher Education during the conduct of the study for his doctoral research; DD, HR and A-LB report a grant from the ESRC (detailed above) during the conduct of the study. Provenance and peer review: Not commissioned; internally peer reviewed. Data sharing statement: The Intelligent Monitoring risk scores and CQC's inspection ratings are public data.

Introduction Prescribing errors occur in 1%–15% of medication orders written for UK hospital inpatients, resulting in harm to an estimated 1%–2%.1 Local studies suggest a similar prevalence of error.2 While there have been relatively few studies of interventions to reduce them,3 a common theme in UK studies of the causes of prescribing errors2 4 5 is that hospital doctors are often unaware of their errors. Provision of explicit feedback to junior doctors and encouragement of help-seeking behaviours have been recommended to address this.2 4–6 In the improvement and implementation science literature, feedback has been widely studied as a means of improving outcomes and changing professional behaviour.7 Research suggests that a number of characteristics are important for effective feedback, including perceived usefulness, credibility, intensity, timeliness, relevance and the presence of linked quality improvement mechanisms.8–12

been widely studied as a means of improving outcomes and changing professional behaviour.7 Research suggests that a number of characteristics are important for effective feedback, including perceived usefulness, credibility, intensity, timeliness, relevance and the presence of linked quality improvement mechanisms.8–12 Previous work within our organisation confirmed that lack of feedback was a problem locally.2 We therefore wanted to conduct a quality improvement study aimed at reducing prescribing errors through provision of feedback. We undertook pilot work providing feedback to teams of doctors13 and ascertained the acceptability of pharmacists providing more formal feedback to junior doctors on an individual basis.14 However, a barrier was that individual prescribers could only be identified for about 10% of individual inpatient medication orders.15 Our basic logic model16 (figure 1) was that improved prescriber identification and better provision of feedback17 on prescribing errors would facilitate learning, reflection and changes to practice. This would in turn reduce prescribing errors. Drawing on prior research in the area of audit and feedback, our theory of change was that the implementation of timely, relevant feedback to prescribers would effect changes in professional behaviour and reduce prescribing errors.8–12 This paper describes the development, testing and evaluation of our approach, structured according to the SQUIRE guidelines.18 Our objectives were to improve prescriber identification on medication orders in order to facilitate individual feedback and to provide effective feedback to prescribers at both the individual level and group level with the aim of reducing prescribing errors.

r approach, structured according to the SQUIRE guidelines.18 Our objectives were to improve prescriber identification on medication orders in order to facilitate individual feedback and to provide effective feedback to prescribers at both the individual level and group level with the aim of reducing prescribing errors. Figure 1 Logic model: basic high-level model depicting the planned inputs and intended results.16

r approach, structured according to the SQUIRE guidelines.18 Our objectives were to improve prescriber identification on medication orders in order to facilitate individual feedback and to provide effective feedback to prescribers at both the individual level and group level with the aim of reducing prescribing errors. Figure 1 Logic model: basic high-level model depicting the planned inputs and intended results.16 Methods Setting The study took place at two large teaching hospitals within the same UK hospital organisation. We focused on junior doctors, specifically foundation year 1 doctors (those in their first year post-medical school). This group of doctors do a large proportion of prescribing,5 having a relatively high prescribing error rate,5 and good prescribing habits learnt early will hopefully be continued throughout their career. Online supplementary appendix S1 explains UK medical staff grades. The interventions were developed and piloted with existing junior doctors and pharmacists at the intervention hospital (site 1) between February and July 2013. The interventions were then implemented at this site concurrently with the new cohort of junior doctors starting in August 2013. A second hospital (site 2) acted as control for selected aspects of the evaluation. At any one time there were 30–34 junior doctors at site 1, mainly based on 12 wards, and 43–45 at site 2, again across 12 wards. Both sites used the same preformatted paper medication charts on which medication orders were handwritten for inpatients and electronic prescriptions at discharge. Junior doctors generally rotated between specialties within the same hospital every 4 months.

based on 12 wards, and 43–45 at site 2, again across 12 wards. Both sites used the same preformatted paper medication charts on which medication orders were handwritten for inpatients and electronic prescriptions at discharge. Junior doctors generally rotated between specialties within the same hospital every 4 months. 10.1136/bmjqs-2015-004717.supp1Supplementary appendix 1 Pharmacists provided a typical UK ward pharmacy service, with wards generally visited by a pharmacist for 1–3 h each weekday. Pharmacists checked inpatient medication orders and discharge prescriptions to ensure that they were clear, legal and clinically appropriate. Standard practice was that any prescribing errors identified were rectified following discussion with a prescriber, if necessary. If the original prescriber could not easily be identified and/or contacted, any available prescriber was asked to rectify the error.

e that they were clear, legal and clinically appropriate. Standard practice was that any prescribing errors identified were rectified following discussion with a prescriber, if necessary. If the original prescriber could not easily be identified and/or contacted, any available prescriber was asked to rectify the error. Developing the interventions Previous research suggests specific mechanisms that maximise the effectiveness of feedback and its capacity to elicit behavioural change. Feedback is most effective when provided more than once, using both verbal and written formats and including both explicit targets and an action plan.7 19 To develop interventions that incorporated this best practice, a panel of six existing junior doctors was recruited to contribute to initial development and testing using plan–do–study–act (PDSA) cycles between February and July 2013. We focused on three linked interventions: improving prescriber identification, developing an agreed approach for pharmacists to use when providing feedback on individual prescribing errors and a method of sharing learning among all junior doctors following common or serious errors.

ycles between February and July 2013. We focused on three linked interventions: improving prescriber identification, developing an agreed approach for pharmacists to use when providing feedback on individual prescribing errors and a method of sharing learning among all junior doctors following common or serious errors. Intervention 1: prescriber identification To facilitate prescriber identification on inpatient medication charts, we used a multifaceted approach. Junior doctors were given personalised name stamps and were reminded to write or stamp their name whenever prescribing on paper medication charts. They were emailed fortnightly run charts showing their compliance with prescriber identification as a group, established by a member of the project team using a signature log created for the project. Following suggestions from junior doctors in early PDSA cycles, we subsequently modified the emails to include comparison among the three main clinical specialities, which introduced an element of competition. The inpatient medication chart used on both hospital sites was also modified to clearly specify that the prescriber's surname was required alongside their signature and pager number.

quently modified the emails to include comparison among the three main clinical specialities, which introduced an element of competition. The inpatient medication chart used on both hospital sites was also modified to clearly specify that the prescriber's surname was required alongside their signature and pager number. Intervention 2: individual feedback We worked with our panel of six junior doctors to agree some principles of effective feedback plus a set of phrases that they felt would clearly communicate that an error had been made in a non-threatening manner (box 1). Training sessions were held with pharmacists to explain why and how to provide feedback on prescribing errors, incorporating quotes from junior doctors to highlight the importance of appropriate feedback. Where possible, a junior doctor representative attended to explain first hand why feedback was valued. The principles were also incorporated into local clinical pharmacy standard operating procedures. Box 1 Principles of effective feedback on individuals' prescribing errors and suggested phrases to use, developed from the literature8–10 and our panel of six junior doctors Principles of effective feedback Feedback should: be as soon as possible after the event; ensure the prescriber is aware that an error has been made; discuss possible solutions; highlight any relevant prescribing resources (eg, clinical guidelines) and be non-judgemental and blame-free. Suggested phrases ‘I want to highlight to you that there's an error on this drug chart. The correct way to prescribe it is…’.

Feedback should: be as soon as possible after the event; ensure the prescriber is aware that an error has been made; discuss possible solutions; highlight any relevant prescribing resources (eg, clinical guidelines) and be non-judgemental and blame-free. Suggested phrases ‘I want to highlight to you that there's an error on this drug chart. The correct way to prescribe it is…’. ‘This dose is incorrect for this patient: it should be……here's where you find the protocol’. Intervention 3: shared learning To support shared learning, ‘good prescribing tip’ emails were developed and sent fortnightly to site 1 junior doctors and pharmacists. Topics were based on local incident reports or suggestions from pharmacy or medical staff. Each prescribing tip (see online supplementary appendix S2) focused on one specific serious/common error and provided evidence-based solutions, aligned with local guidelines. PDSA cycles were used to develop materials perceived as relevant by the target audience and visually appealing on both desktops and smartphones. 10.1136/bmjqs-2015-004717.supp2Supplementary appendix 2 Evaluation of the interventions We used a mixed-methods approach with quantitative and qualitative elements to evaluate changes in key process and outcome measures as well as exploring the experiences of the prescribers and pharmacists involved.

Intervention 3: shared learning To support shared learning, ‘good prescribing tip’ emails were developed and sent fortnightly to site 1 junior doctors and pharmacists. Topics were based on local incident reports or suggestions from pharmacy or medical staff. Each prescribing tip (see online supplementary appendix S2) focused on one specific serious/common error and provided evidence-based solutions, aligned with local guidelines. PDSA cycles were used to develop materials perceived as relevant by the target audience and visually appealing on both desktops and smartphones. 10.1136/bmjqs-2015-004717.supp2Supplementary appendix 2 Evaluation of the interventions We used a mixed-methods approach with quantitative and qualitative elements to evaluate changes in key process and outcome measures as well as exploring the experiences of the prescribers and pharmacists involved. Process measures Prescriber identification A signature log was created for all junior doctors; this was used by a member of the project team to conduct a weekly audit on the 12 wards on each site where junior doctors were based. The first 10 available medication charts on each ward were examined for prescriber identification (presence of a signature, plus either a handwritten or stamped name) for junior doctors' medication orders. Prescriber identification rates were compared pre-intervention and post-intervention using a χ2 test.

were based. The first 10 available medication charts on each ward were examined for prescriber identification (presence of a signature, plus either a handwritten or stamped name) for junior doctors' medication orders. Prescriber identification rates were compared pre-intervention and post-intervention using a χ2 test. Pharmacists' provision of individual feedback We added three extra criteria to the standard set of 43 criteria used to assess pharmacists during routine peer-reviewed ward visits. Peer-reviewed visits were routinely conducted by senior clinical pharmacists employed at the study hospitals based on a validated approach20; the three extra criteria relating to feedback were added only at site 1. For any prescribing errors identified, these new criteria established whether pharmacists attempted to contact the initial prescriber, whether they gave appropriate feedback and whether they highlighted any relevant prescribing resources. Each criterion was assessed on a 4-point Likert scale (always, mostly, sometimes, never).20

escribing errors identified, these new criteria established whether pharmacists attempted to contact the initial prescriber, whether they gave appropriate feedback and whether they highlighted any relevant prescribing resources. Each criterion was assessed on a 4-point Likert scale (always, mostly, sometimes, never).20 Outcome measures Junior doctors' and pharmacists' perceptions of feedback Evaluative questionnaires (see online supplementary appendix S3) were developed for both junior doctors and pharmacists, including items based on characteristics of effective feedback: perceived usefulness, credibility, intensity, timeliness, relevance and accuracy of feedback.8–10 All measures were reported on an 8-point scale with higher values representing more positive responses. Items were additionally combined into a mean overall feedback effectiveness scale score (Cronbach's α 0.904 for pharmacists and 0.853 for junior doctors). The questionnaire was administered at site 1 during spring 2013 (pre-intervention) and again in spring 2014 (approximately 9 months post-intervention) to a different cohort of junior doctors at the same point in their training year, and to all pharmacists employed at each time point. Mean scores were compared pre-intervention and post-intervention using independent samples t-tests, with p<0.05 indicating statistical significance for the overall feedback effectiveness scales and p<0.01 for each of the individual items (to account for multiple testing). 10.1136/bmjqs-2015-004717.supp3Supplementary appendix 3

Outcome measures Junior doctors' and pharmacists' perceptions of feedback Evaluative questionnaires (see online supplementary appendix S3) were developed for both junior doctors and pharmacists, including items based on characteristics of effective feedback: perceived usefulness, credibility, intensity, timeliness, relevance and accuracy of feedback.8–10 All measures were reported on an 8-point scale with higher values representing more positive responses. Items were additionally combined into a mean overall feedback effectiveness scale score (Cronbach's α 0.904 for pharmacists and 0.853 for junior doctors). The questionnaire was administered at site 1 during spring 2013 (pre-intervention) and again in spring 2014 (approximately 9 months post-intervention) to a different cohort of junior doctors at the same point in their training year, and to all pharmacists employed at each time point. Mean scores were compared pre-intervention and post-intervention using independent samples t-tests, with p<0.05 indicating statistical significance for the overall feedback effectiveness scales and p<0.01 for each of the individual items (to account for multiple testing). 10.1136/bmjqs-2015-004717.supp3Supplementary appendix 3 Two focus groups were conducted at site 1, one for junior doctors and one for pharmacists, to explore views on our interventions in more detail. Four main topics were explored: prescriber identification and name stamps, prescriber identification run charts, individual feedback of prescribing errors and prescribing tip emails. Unintended consequences of the interventions were also explored. Discussions were recorded and professionally transcribed. We used thematic analysis21 with a deductive approach using an a priori framework of the perceived advantages, disadvantages, facilitators and barriers of each intervention.

prescribing tip emails. Unintended consequences of the interventions were also explored. Discussions were recorded and professionally transcribed. We used thematic analysis21 with a deductive approach using an a priori framework of the perceived advantages, disadvantages, facilitators and barriers of each intervention. Prescribing error prevalence We studied the prevalence of erroneous medication orders written by junior doctors at site 1 (intervention) and site 2 (control) using identical methods on each site. Following verbal and written briefings, including our definition of a prescribing error,22 ward pharmacists collected data once weekly from August to December 2013 on any prescribing errors identified in the first 10 junior doctors' medication orders encountered, using the signature logs produced by the project team. Errors were classified based on previous work2; each medication order could have more than one error. We used interrupted time series analysis to test for an effect of our intervention on weekly prescribing error rate at the intervention site, controlling for baseline variation in error rates and longitudinal variation in error rates at the control site.23 The time series components of the model were fitted as segmented regression with parameters representing baseline trend, intervention onset and the post-intervention change in trend. A significant reduction in weekly error rate and/or trend in error rate, as indicated by p<0.05 for the value of t associated with each parameter coefficient, would indicate a positive effect of the intervention.

with parameters representing baseline trend, intervention onset and the post-intervention change in trend. A significant reduction in weekly error rate and/or trend in error rate, as indicated by p<0.05 for the value of t associated with each parameter coefficient, would indicate a positive effect of the intervention. Figure 2 presents the study timeline. An action–effect diagram24 is presented in figure 3. Figure 2 Project timeline. Figure 3 Prescribing improvement action–effect diagram. Ethical issues The study was approved locally as a service evaluation; NHS ethics approval was not required. Data could not be attributed to individual patients or staff. Focus group participants provided written informed consent. Results Course of the intervention Following a series of four PDSA cycles, name stamps and supporting information were distributed to all junior doctors at site 1 during their induction in August 2013, before they started patient care. Pharmacists were briefed on providing feedback during July and August 2013. We estimate that around half of the department's pharmacists attended a face-to-face training session. Through regular publicity, including presentations at pharmacy staff meetings, posters and emails, we are confident that all pharmacists were aware of our aims. Following three PDSA cycles, prescribing tip emails were produced every fortnight starting August 2013. All interventions were subsequently rolled out during January 2014 to also include all junior doctors and pharmacists at site 2.

ings, posters and emails, we are confident that all pharmacists were aware of our aims. Following three PDSA cycles, prescribing tip emails were produced every fortnight starting August 2013. All interventions were subsequently rolled out during January 2014 to also include all junior doctors and pharmacists at site 2. Process measures Prescriber identification At site 1, 11 374 medication orders were examined over 29 weeks post-intervention (mean 392 per week). The prescriber's name was present for 5935 (52.2%), of which 3617 (60.9%) were stamped names. No comparative baseline data were available for site 1 as the interventions were introduced contemporaneously with the junior doctors starting employment. At site 2, where the interventions were introduced later, 789 medication orders were examined at baseline over 3 weeks (mean 263 per week) and 2332 during six post-intervention weeks (mean 389 per week). Pre-intervention, the prescriber's name was specified on 48 of 789 (6.1%) medication orders; this increased to 860 of 2332 (36.9%) post-intervention (p<0.0001; χ2 test), of which 501 (58.3%) were stamped.

mined at baseline over 3 weeks (mean 263 per week) and 2332 during six post-intervention weeks (mean 389 per week). Pre-intervention, the prescriber's name was specified on 48 of 789 (6.1%) medication orders; this increased to 860 of 2332 (36.9%) post-intervention (p<0.0001; χ2 test), of which 501 (58.3%) were stamped. Pharmacists' provision of individual feedback Senior clinical pharmacists recorded data for five peer-reviewed ward visits that took place at site 1 between August and December 2013. In three of the five visits, pharmacists ‘always’ attempted to contact the original prescriber and provided feedback in a professional manner and in the remaining two visits this was done ‘mostly’ as assessed by the peer reviewer. Signposting to suitable resources or solutions was done less consistently.

December 2013. In three of the five visits, pharmacists ‘always’ attempted to contact the original prescriber and provided feedback in a professional manner and in the remaining two visits this was done ‘mostly’ as assessed by the peer reviewer. Signposting to suitable resources or solutions was done less consistently. Outcome measures Junior doctors' and pharmacists' perceptions of feedback At baseline (Spring 2013), we received questionnaire responses from 25 of 37 (68%) site 1 pharmacists and 26 of 34 (76%) junior doctors. At the second time point (Spring 2014), we received 18 of 32 (56%) pharmacist responses and 21 of 30 (70%) from junior doctors. For the overall feedback effectiveness scale scores, the mean score for junior doctors increased slightly post-intervention (mean at baseline=6.15, SD=0.74; mean post-intervention=6.23; SD=0.86), but was not statistically significant (p=0.74; unpaired t-test). The mean score for pharmacists improved significantly post-intervention (mean at baseline=5.26, SD=0.88; mean post-intervention=5.84, SD=0.80; p=0.03). For the junior doctors' responses, improvement was observed in the mean response for five individual items, although none reached statistical significance (see online supplementary appendix S4). For pharmacists, there was improvement in 14 items. Two reached statistical significance at p<0.01: perceptions of the accuracy of feedback delivered to junior doctors (item 15; p=0.003) and perceptions of whether junior doctors found the data to be trustworthy (item 16; p=0.007), with improvements in mean scores of 1.4 and 0.87, respectively.

s improvement in 14 items. Two reached statistical significance at p<0.01: perceptions of the accuracy of feedback delivered to junior doctors (item 15; p=0.003) and perceptions of whether junior doctors found the data to be trustworthy (item 16; p=0.007), with improvements in mean scores of 1.4 and 0.87, respectively. 10.1136/bmjqs-2015-004717.supp4Supplementary appendix 4 A total of 14 doctors (42% of 33 at site 1) attended the junior doctors' focus group in October 2013, all of whom participated in the discussion. Participants expressed positive views and experiences of receiving feedback and reported increased understanding of the importance of ensuring their identity is known when prescribing. Prescriber identification run charts and prescribing tips were well received and thought to be useful, although some felt fortnightly run charts were too frequent. Quotes are presented in table 1. Table 1 Quotes from junior doctors' focus group Intervention Quotes Prescriber identification I think the fact that we'd been given the stamps highlighted the fact more than anything that we should be putting our names on the paperwork. Now even when I don't have my stamp I think, oh, I'd better write my name because I don't have my stamp on me. I find using the stamp makes me take a lot more ownership of it, I think, do I really know what I'm doing? I'm putting my name to that, like George Foreman grill, it'd better work.

Intervention Quotes Prescriber identification I think the fact that we'd been given the stamps highlighted the fact more than anything that we should be putting our names on the paperwork. Now even when I don't have my stamp I think, oh, I'd better write my name because I don't have my stamp on me. I find using the stamp makes me take a lot more ownership of it, I think, do I really know what I'm doing? I'm putting my name to that, like George Foreman grill, it'd better work. Individual feedback [The prescription] would need to be changed [because of] patient safety, and that can be done by anyone on the team, but I'd like to know personally that I'd made a mistake. …[I]t's hard to say what it would be like without [receiving feedback], it makes you feel safe because it feels like every error you may be making is being checked by somebody and that they are then feeding back. So it feels like I'm making less errors now because I've learnt from the errors I've made. Shared learning [The prescribing tips are] good because, there's often a picture of a drug chart so you can look at it quite quickly and…You don't have to read a lot of text, you can just look and read, oh, I can see just a gap, that … if you're just quickly checking it because you don't have time.

…[I]t's hard to say what it would be like without [receiving feedback], it makes you feel safe because it feels like every error you may be making is being checked by somebody and that they are then feeding back. So it feels like I'm making less errors now because I've learnt from the errors I've made. Shared learning [The prescribing tips are] good because, there's often a picture of a drug chart so you can look at it quite quickly and…You don't have to read a lot of text, you can just look and read, oh, I can see just a gap, that … if you're just quickly checking it because you don't have time. Four pharmacists attended the pharmacists' focus group in November 2013. Pharmacists felt that feedback was generally well received, although they felt uncomfortable referring explicitly to ‘errors’ and seemed protective of relationships with their junior doctor colleagues. They preferred to use the terms ‘mistake’ or ‘incorrect’. They felt that pharmacists were providing more individualised feedback due to increased awareness of the benefits and had additionally used the prescribing tips to facilitate discussions. No unintended negative consequences were identified by either group. An unintended positive consequence was junior doctors using the name stamps in the written health records as well as for prescribing.

due to increased awareness of the benefits and had additionally used the prescribing tips to facilitate discussions. No unintended negative consequences were identified by either group. An unintended positive consequence was junior doctors using the name stamps in the written health records as well as for prescribing. Prescribing error prevalence We identified 390 errors in 367 (15.2%) of 2410 medication orders at site 1 and 391 errors in 368 (15.1%) of 2432 medication orders at site 2 (table 2). Overall, the most common error types (see online supplementary appendix S5) were ‘incorrect dose’ (175 of 781 errors; 22%) and ‘medication omitted when clinically indicated’ (172; 22%), usually involving omission of patients' usual medication following hospital admission. Table 2 Error rates for intervention and control sites Site Pre-intervention erroneous medication orders (% of all medication orders) Post-intervention erroneous medication orders (% of all medication orders) Overall erroneous medication orders (% of all medication orders) 1 (intervention) 92 of 620 (14.8) 275 of 1790 (15.4) 367 of 2410 (15.2) 2 (control) 35 of 276 (12.7) 333 of 2156 (15.5) 368 of 2432 (15.1) 10.1136/bmjqs-2015-004717.supp5Supplementary appendix 5

ion erroneous medication orders (% of all medication orders) Overall erroneous medication orders (% of all medication orders) 1 (intervention) 92 of 620 (14.8) 275 of 1790 (15.4) 367 of 2410 (15.2) 2 (control) 35 of 276 (12.7) 333 of 2156 (15.5) 368 of 2432 (15.1) 10.1136/bmjqs-2015-004717.supp5Supplementary appendix 5 Over the course of the project, the mean weekly prescribing error rate at both sites increased between baseline and follow-up periods. When an interrupted time series model was fitted to examine the change in mean error rate, while controlling for baseline variation and compared with the control site, no significant effect of the intervention at site 1 was observed. The overall effect of intervention implementation (ie, the unique effect associated with the intervention while holding baseline trend and control constant) was a non-significant reduction in the error rate of 4% (β=−4.045; t=−0.638; p=0.532). Similarly the estimated trend in error rate was a non-significant decrease of 4% post-intervention (β=−4.440; t=−1.097; p=0.288). Discussion A set of interventions were developed, refined and implemented with the aim of reducing prescribing errors made by junior doctors. Process measures and qualitative findings indicate some benefits. Although the net controlled effects of the interventions were estimated decreases in both trend and level of weekly prescribing error rates at the intervention site, these did not reach statistical significance.

prescribing errors made by junior doctors. Process measures and qualitative findings indicate some benefits. Although the net controlled effects of the interventions were estimated decreases in both trend and level of weekly prescribing error rates at the intervention site, these did not reach statistical significance. Relationship to other evidence Our findings are in line with a Cochrane review and more recent update,7 19 which suggest that providing feedback results in small-to-moderate positive effects on professional practice and that process measures may be more sensitive to feedback initiatives than outcome measures. However, these reviews focus on studies of feedback on specific aspects of clinical practice, such as prescribing for a particular clinical condition; no studies of hospital prescribing errors were included. Our prescribing error rate was comparable with UK studies employing similar methodologies.2

an outcome measures. However, these reviews focus on studies of feedback on specific aspects of clinical practice, such as prescribing for a particular clinical condition; no studies of hospital prescribing errors were included. Our prescribing error rate was comparable with UK studies employing similar methodologies.2 Strengths and limitations In designing an evaluation based on mixed methods including a controlled interrupted time series analysis, we have applied a robust quasi-experimental design to a quality improvement initiative; such initiatives are rarely subjected to this level of evaluation. We were therefore able to examine any change in prescribing error rates above and beyond those predicted by a baseline trend which might, for example, describe natural improvement in junior doctors' prescribing as they gain experience. Importantly, the survey and qualitative components of this study allowed for generation of insights into the processes involved and maximisation of the transferable value of the study. The questionnaire had a good response rate and the junior doctor focus group was well attended—we believe the findings to be representative of the cohort. Since only four pharmacists participated in their focus group, these results may represent a more specific set of views.

tion of the transferable value of the study. The questionnaire had a good response rate and the junior doctor focus group was well attended—we believe the findings to be representative of the cohort. Since only four pharmacists participated in their focus group, these results may represent a more specific set of views. The main limitation was that the new junior doctor intake was contemporaneous with the initiation of our interventions at site 1, potentially confounding detection of any effect of our intervention on error rates. A longer pre-intervention baseline at both sites would have been preferable. These issues were mainly due to the dates for both project-funding and junior doctors' inductions being fixed. Other limitations were that senior pharmacists were only able to assess pharmacists' feedback provision on five occasions, and we recognise that pharmacists collecting prescribing error data may vary in their adherence to data collection procedures. Finally, we provided fortnightly run charts to the junior doctors showing prescriber identification rates as we were collecting these data as part of our evaluation. However this relied on manual data collection and was quite time-consuming and so is unlikely to be sustainable in routine practice.

ollection procedures. Finally, we provided fortnightly run charts to the junior doctors showing prescriber identification rates as we were collecting these data as part of our evaluation. However this relied on manual data collection and was quite time-consuming and so is unlikely to be sustainable in routine practice. Interpretation We achieved our first objective of improving prescriber identification, although it appears that the percentage of junior doctors' medication orders for which the prescriber could be identified hit a ceiling of around 50%. Possible reasons for this are that name stamps were lost or forgotten and for some sections of the medication chart the signature box is very small. It is also difficult to depress the stamp onto the chart without resting it on a firm surface (a particular problem on ward rounds). Introducing a change where junior staff may differ in practice to their senior role models is also likely to be difficult. We suspect that it will not be possible to achieve 100% prescriber identification until inpatient electronic prescribing is introduced. Routine use of signature logs could form part of a solution in the meantime. We also achieved our second objective of providing effective feedback to prescribers at both the individual level and group level, with our interventions well received by both pharmacists and junior doctors.

ectronic prescribing is introduced. Routine use of signature logs could form part of a solution in the meantime. We also achieved our second objective of providing effective feedback to prescribers at both the individual level and group level, with our interventions well received by both pharmacists and junior doctors. Critical analysis of our logic model (figure 1) suggests why we may not have detected a significant overall reduction in prescribing error rates. The model is based on the assumption that increasing prescriber identification would lead to increased feedback by pharmacists, leading to changes in prescribing behaviour and a reduction in prescribing error rates. A feedback initiative of this type therefore relies on reliable identification of potential ‘learners’, opportunities for the timely communication of relevant and credible feedback, interaction with individual psychology to produce ‘learning’ and an environment conducive with implementing that learning in practice. Our work highlights the need for a more comprehensive theory of change in this area, underpinned by broad synthesis of theory and experience across a range of application contexts, to understand how contextual and intervention factors affect the success of feedback initiatives. For example, we were only able to increase the identification of junior doctor prescribers to about 50%. Although a substantial increase from baseline, this meant that the prescriber still could not be identified for 50% of medication orders, thus limiting opportunities for individual feedback. We did not set out to assess the percentage of junior doctors' prescribing errors for which a pharmacist gave individual feedback, but assume this would not be 100% as time constraints, shift patterns and individual motivation are likely to prevent some opportunities for feedback.

opportunities for individual feedback. We did not set out to assess the percentage of junior doctors' prescribing errors for which a pharmacist gave individual feedback, but assume this would not be 100% as time constraints, shift patterns and individual motivation are likely to prevent some opportunities for feedback. Prescribing errors are multifactorial2 4 5 25 26 and it is likely that feedback would only prevent a subset, especially if feedback relates to drugs that are rarely prescribed. One of our most common error types related to medication reconciliation on admission, which may be largely a system problem and less ameliorable through individual feedback. Detection of effects of feedback on error rates is likely to require sustained monitoring over longer time periods, with an adequate baseline, comprehensive monitoring of compliance with the intervention and stable participant groups over baseline and intervention periods. Based on our experience here, we would recommend at least 3 months' weekly error rate data be collected both pre-intervention and post-intervention, with quasi-experimental control over potential confounders (such as changes in junior doctor cohort) through careful selection of a comparable control site. These methodological requirements may represent a significant challenge, but our study suggests that effective feedback on prescribing errors has the potential to support beneficial learning. The impact on prescribing errors, ideally in combination with other interventions, therefore warrants further investigation. Further work should also explore how information technology could be used to deliver individualised feedback around prescribing errors in the hospital setting.

ial to support beneficial learning. The impact on prescribing errors, ideally in combination with other interventions, therefore warrants further investigation. Further work should also explore how information technology could be used to deliver individualised feedback around prescribing errors in the hospital setting. Lessons learnt from the project's evolution We identified a number of learning points from our PDSA cycles. We found that the names some doctors used in practice differed from their given names (such as use of their middle name rather than their first name) and we subsequently attempted to establish and use preferred names on the name stamps. Some doctors initially believed that the stamp replaced a prescribing signature; we clarified that a signature was required alongside a stamp. The PDSA approach allowed us to identify and address such issues early on. Locally, we now issue name stamps to all junior doctors and non-medical prescribers. We audit prescriber identification annually and continue to send out fortnightly prescribing tips. The requirement of providing feedback to the initial prescriber has been incorporated into pharmacy standard operating procedures. Our interventions were subsequently introduced at two further large teaching hospitals within a different organisation and are now being rolled out across North West London.

escribing tips. The requirement of providing feedback to the initial prescriber has been incorporated into pharmacy standard operating procedures. Our interventions were subsequently introduced at two further large teaching hospitals within a different organisation and are now being rolled out across North West London. Conclusion Using a set of three linked interventions, we increased the percentage of medication orders for which junior doctor prescribers stated their name from about 10% to 50%. Principles of effective feedback were developed in conjunction with junior doctors and both quantitative and qualitative evaluation suggested an improvement in experiences around feedback. No change in prescribing error prevalence was detected. Attempts to produce a measureable reduction in prescribing errors are likely to need a multifaceted approach of which feedback should form part. Contributors: The study was conceived by BDF, AJ, MR, DI and JB. SJ, MR and BDF collected the data. JB, SJ, MR, BDF and IS conducted the data analysis. All authors contributed to paper writing and have approved the final version." Funding: This project is part of the Health Foundation's Shine 2012 programme. The Health Foundation is an independent charity working to continuously improve the quality of healthcare in the UK. This paper represents independent work supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

prove the quality of healthcare in the UK. This paper represents independent work supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. Competing interests: None declared. Provenance and peer review: Not commissioned; externally peer reviewed.

Introduction Methodical and detailed case review is commonly used as a strategy to improve care, including care of pregnant women1 through documenting the number and causes of morbidity and mortality, and through identifying preventable factors.2 3 Two approaches have been taken nationally in the UK to learning from adverse incidents in maternity care: external anonymised case review (Confidential Enquiries),4 defined as an independent systematic multidisciplinary anonymous investigation which identifies the causes and avoidable or remediable factors associated with them,5 and local (facility-based) reviews using different tools such as root cause analysis.5 Root cause analysis is a structured methodology used to identify the most likely underlying causes of incidents within an organisation. The aim of both is to formulate solutions to prevent the incident occurring again to protect the health and safety of the public and encourage a culture of openness.6 However, reports following recent high-profile systematic failures in maternity care have noted that local reviews of serious incidents have not always identified the key messages to improve care.7

prevent the incident occurring again to protect the health and safety of the public and encourage a culture of openness.6 However, reports following recent high-profile systematic failures in maternity care have noted that local reviews of serious incidents have not always identified the key messages to improve care.7 No national strategy exists in the UK for local learning from near-miss maternal morbidity, and little is known about the nature of approaches being used in different hospitals. The recent UK Confidential Enquiry into Maternal Deaths and Morbidity, which examined the care of women who died from sepsis, as well as women who survived septic shock, found that local reviews had not been undertaken for most cases, and on several occasions it was unclear whether, when undertaken, a local review had been multidisciplinary and what specific method had been used.4 The aim of this project was to describe the methods used for conducting local reviews of the care of women with severe morbidity in pregnancy, to compare the lessons identified for future care through external review (confidential enquiry) and local review, and to estimate the additional costs associated with external review, in order to inform development of a strategy for optimising review of serious incidents in maternity care.

vere morbidity in pregnancy, to compare the lessons identified for future care through external review (confidential enquiry) and local review, and to estimate the additional costs associated with external review, in order to inform development of a strategy for optimising review of serious incidents in maternity care. Methods Six sites were randomly selected following stratification from all National Health Service (NHS) consultant-led maternity units in England (two large teaching hospital units (5000+ deliveries per year), two medium units (2000–4999 deliveries per year) and two small units (up to 1999 deliveries per year)), and on the basis of their Clinical Negligence Scheme for Trusts (CNST) level at the time of selection8 (one of each size with CNST level 1, the basic risk management level, and one of each size with CNST level 3, the highest risk management level). Two of the initially selected sites declined to participate; resampling was therefore carried out to identify two further units which both agreed to participate. Sites were located in the North East, Midlands and South East of England.

t level, and one of each size with CNST level 3, the highest risk management level). Two of the initially selected sites declined to participate; resampling was therefore carried out to identify two further units which both agreed to participate. Sites were located in the North East, Midlands and South East of England. Each unit provided a list of all serious incidents in maternity care that had triggered the requirement for a local review process during the previous 6 months (listed in table 1). From these, incidents of direct (obstetric) near-miss maternal morbidity were identified (listed as included conditions in table 1). From these lists of near-miss maternal morbidity incidents, six cases were randomly selected from each unit, and full sets of clinical notes, together with their local review reports, were requested. In total, 33 sets of anonymised clinical records were obtained and reviewed; the remaining 3 sets of notes were unavailable. The methodology used for the local reviews was identified from the reports, in terms of any specific tools used, such as root cause analysis, the team members involved and their disciplines. The reports were analysed to determine whether an action plan was present, and whether this included plans to audit the results of any recommended actions. Table 1 Conditions leading to local review processes during the 6 months of the study

Each unit provided a list of all serious incidents in maternity care that had triggered the requirement for a local review process during the previous 6 months (listed in table 1). From these, incidents of direct (obstetric) near-miss maternal morbidity were identified (listed as included conditions in table 1). From these lists of near-miss maternal morbidity incidents, six cases were randomly selected from each unit, and full sets of clinical notes, together with their local review reports, were requested. In total, 33 sets of anonymised clinical records were obtained and reviewed; the remaining 3 sets of notes were unavailable. The methodology used for the local reviews was identified from the reports, in terms of any specific tools used, such as root cause analysis, the team members involved and their disciplines. The reports were analysed to determine whether an action plan was present, and whether this included plans to audit the results of any recommended actions. Table 1 Conditions leading to local review processes during the 6 months of the study Conditions included Conditions excluded Severe haemorrhage (>1500 mL) Shoulder dystocia Amniotic fluid embolism Other maternal incidents Cardiac arrest Violation of local protocol Eclampsia/pre-eclampsia Retained swab or instrument Uterine rupture Medication error Uterine inversion Organisational incidents Severe maternal sepsis Third-/fourth-degree tears Hysterectomy Unsuccessful forceps/ventouse Placental accreta Readmission of mother Placental praevia Pulmonary embolism Major anaesthetic complications Intensive care admission (maternal) Acute fatty liver Reviews undertaken for near-miss maternal morbidity conditions were included in the study; other incidents were excluded.

ctomy Unsuccessful forceps/ventouse Placental accreta Readmission of mother Placental praevia Pulmonary embolism Major anaesthetic complications Intensive care admission (maternal) Acute fatty liver Reviews undertaken for near-miss maternal morbidity conditions were included in the study; other incidents were excluded. External reviews were undertaken using a standard format as used by the Confidential Enquiry into Maternal Deaths.4 9 As part of this standard approach, assessors were required to refer to national evidence-based guidelines when commenting on care. All 11 external reviewers, comprising 5 obstetricians, 4 midwives and 2 anaesthetists, received an online training session prior to undertaking assessments. Two obstetricians, a midwife and two anaesthetists assessed the care of each woman; five reviewers thus examined each woman's care. Each primary assessor completed an independent review of the woman's care, highlighting the lessons to be learned to improve care in the future. This was checked by a second assessor in the relevant specialty. Expert assessors were located throughout England; to maintain anonymity, assessors were only asked to review the care of women who had been cared for outside their region. The assessment process and all individual findings were strictly confidential; all assessors were required to sign a confidentiality agreement. Upon completion of all case assessments and reports, each assessor was asked to complete a short questionnaire describing their experience of the process and estimating the time it required to assess each case. The administrative time required to liaise with units, photocopy, anonymise and scan each record and liaise with external assessors was also captured. The average administrative staff and assessor time required for each case were then calculated, and the cost estimated using published salary scales (national spinal point 18 for administrative staff, consultant medical salary scale, band 7 midwife scale).

Uterine inversion occurred within 9 min of delivery of the baby; so, almost certainly mismanagement of third stage. I am sure delivery was challenging for all concerned. There was immediate recognition that uterine inversion had occurred, and it was dealt with and managed appropriately. Query whether the patient was debriefed or whether the significance for future pregnancies was discussed. Costs of external reviews The administrative time spent on organising, photocopying, anonymising and scanning each case and liaising with external assessors was a mean of 17 h, which amounts to an average administrative cost of £300 per case. On average it took each external assessor 3 h to read and report on each case. The cost of five external assessors' time, based on the estimated consultant and senior midwife salary costs,13 was £1800 per case. Thus, the total cost of conducting an external review into a near-miss case of maternal morbidity per case was estimated as £2100. Feedback from external reviewers All 11 external reviewers completed the questionnaire to provide feedback on the process. The common challenges faced with conducting external reviews were the poor quality of the notes, working in isolation and having time to do the reviews, as well as meeting the requirements of NHS work.

ch record and liaise with external assessors was also captured. The average administrative staff and assessor time required for each case were then calculated, and the cost estimated using published salary scales (national spinal point 18 for administrative staff, consultant medical salary scale, band 7 midwife scale). Qualitative and quantitative approaches were used to compare the lessons learned from local and external reviews.10 11 Emerging themes among the lessons learned were initially identified through reading and re-reading the reviews, and a coding frame was constructed and agreed upon by four authors. The data were then coded by three authors and checked by a fourth author. NVIVO 10 software was used to facilitate the analysis.12 Once the themes had been identified, the number of reviews identifying each theme was quantified and compared using χ2 test or Fisher’s exact test as appropriate. Following completion of the analysis of local and external reviews, results of the study/comparison were presented to the staff at the six participating units, and local staff were asked to complete a questionnaire detailing their perspectives on the process of undertaking local reviews and the feedback from the external reviews. Results The care of 33 women with near-miss maternal morbidities was assessed by the confidential enquiry panel; the majority of women (73%) had severe haemorrhage (>1500 mL blood loss) (table 2). Table 2 Numbers of near-miss maternal morbidity conditions reviewed in the study*

Following completion of the analysis of local and external reviews, results of the study/comparison were presented to the staff at the six participating units, and local staff were asked to complete a questionnaire detailing their perspectives on the process of undertaking local reviews and the feedback from the external reviews. Results The care of 33 women with near-miss maternal morbidities was assessed by the confidential enquiry panel; the majority of women (73%) had severe haemorrhage (>1500 mL blood loss) (table 2). Table 2 Numbers of near-miss maternal morbidity conditions reviewed in the study* Conditions Numbers Severe haemorrhage (>1500 mL) 24 Amniotic fluid embolism 1 Cardiac arrest 1 Eclampsia/pre-eclampsia 3 Uterine rupture 3 Uterine inversion 2 Severe maternal sepsis 4 Hysterectomy 6 Placental accreta 1 Placental praevia 2 Pulmonary embolism 0 Major anaesthetic complications 1 Intensive care admission (maternal) 3 Acute fatty liver 1 *Several cases involved more than one type of near-miss maternal morbidity condition; thus, the total number of conditions exceeds 33, even though the actual number of cases was 33.

eta 1 Placental praevia 2 Pulmonary embolism 0 Major anaesthetic complications 1 Intensive care admission (maternal) 3 Acute fatty liver 1 *Several cases involved more than one type of near-miss maternal morbidity condition; thus, the total number of conditions exceeds 33, even though the actual number of cases was 33. Local review processes All the cases had triggered the requirement for a local review, that is, local guidance had indicated that a review should be carried out. However, in five cases, the records had either been reviewed by the individual responsible for conducting local reviews, who had felt that there were no lessons to be learned and a formal review was not required, or the staff responsible for conducting a local review had not been able to because they did not have time capacity to review all the cases which triggered a review requirement. Thus local reviews were carried out for only 28 of 33 women (85%). Methods and reports of the outcome of local reviews were very varied. A formal report of the local review process and outcome had been written in 11 cases; 4 of these local reviews used root cause analysis. For 12 cases, a local review group had discussed the incident, and a summary of the outcome had been noted in a spreadsheet. For an additional five cases the documented local review constituted a timeline of events with brief notes on evidence of good care and if issues had occurred.

l reviews used root cause analysis. For 12 cases, a local review group had discussed the incident, and a summary of the outcome had been noted in a spreadsheet. For an additional five cases the documented local review constituted a timeline of events with brief notes on evidence of good care and if issues had occurred. The categories of staff involved in conducting reviews varied between units. An individual midwife conducted four reviews, a group of midwives conducted two reviews, and five reviews involved both obstetricians and midwives. One of these five reviews included an anaesthetist, and one included midwifery trainees. It was not possible to identify the specialty or grade of reviewers in 17 local review reports nor was it possible to identify how many reviewers participated. There was patient involvement in five local reviews. In these cases it was documented that the woman concerned had asked the review group to consider specific questions about her care, and that she was later notified of the group's conclusions and action plans. In the local review reports, action plans had been written for 14 cases. The conclusions from three local review reports included a recommendation to audit the subsequent change to clinical practice.

The categories of staff involved in conducting reviews varied between units. An individual midwife conducted four reviews, a group of midwives conducted two reviews, and five reviews involved both obstetricians and midwives. One of these five reviews included an anaesthetist, and one included midwifery trainees. It was not possible to identify the specialty or grade of reviewers in 17 local review reports nor was it possible to identify how many reviewers participated. There was patient involvement in five local reviews. In these cases it was documented that the woman concerned had asked the review group to consider specific questions about her care, and that she was later notified of the group's conclusions and action plans. In the local review reports, action plans had been written for 14 cases. The conclusions from three local review reports included a recommendation to audit the subsequent change to clinical practice. Messages for future care In comparison to the local reviews, many additional, detailed messages for care were identified in the external reviews (table 3). Importantly, the external reviews highlighted aspects of good care in every case (n=33, 100%) compared with only 55% (n=18) of the local reviews which identified good care (p<0.0005). The local reviews included individual disciplinary actions, for example, four recommended individual supervisory action; a need for discipline of specific individuals was not identified in any external reviews (p=0.11). For seven of the cases, local review reports noted local factors affecting the situation, such as ‘staff were dealing with two emergencies simultaneously’ and ‘insufficient room for resuscitation.’ External reviewers, who were making their assessment solely on the basis of the medical records of individual cases, did not identify any local factors. External reviewers suggested an alternative clinical approach in 16 cases, whereas alternative clinical approaches were not mentioned in any local reviews.

scitation.’ External reviewers, who were making their assessment solely on the basis of the medical records of individual cases, did not identify any local factors. External reviewers suggested an alternative clinical approach in 16 cases, whereas alternative clinical approaches were not mentioned in any local reviews. Table 3 Number of cases where themes were noted in local (n=28) and external reviews (n=33) Theme Local reviews N=28 External reviews N=33 National guidelines or good practice points: Followed 13 28 Not followed 11 24 Senior review mentioned Consultant involvement noted 14 28 Lack of consultant involvement 7 12 Communication themes reported Good: with the patient and family 2 19 between staff members 4 14 Poor: with the patient and family 3 9 between staff members 4 11 Comments on documentation Some areas of good quality 8 13 Some areas of poor quality 10 29 Medication errors noted 2 11 Individual supervisory action recommended 4 0 Reported delays in: Diagnosis 5 22 Obtaining a second opinion 2 3 Providing treatment (excluding drugs) 7 9 Administering drugs (eg, antibiotics) 3 5 Requesting blood or clotting products 0 2 Gaining access to blood or clotting products 0 2 Transferring patients (eg, to theatre or the labour ward) 2 6 Suggestions for alternative clinical approach made 0 16 Note that several themes may have been identified for each woman, and therefore the total number of themes identified exceeds the actual number of cases examined.

ccess to blood or clotting products 0 2 Transferring patients (eg, to theatre or the labour ward) 2 6 Suggestions for alternative clinical approach made 0 16 Note that several themes may have been identified for each woman, and therefore the total number of themes identified exceeds the actual number of cases examined. While the external reviewers made a judgement that five (15%) women had received ‘good care,’ ‘improvements to care that would not have made a difference to outcome’ were identified in 17 cases (52%), and for eleven women (33%), ‘improvements to care were identified that may have made a difference to outcome.’ These improvements which might have changed the women's outcome had only been identified in the local reviews for four of these cases. Examples of lessons identified Overall, the lessons for care identified in the local reviews were briefer and less focused than those identified on external reviews. The following are examples of lessons identified from the local review of the care of a woman with a postpartum haemorrhage and hysterectomy: Syntometrine not indicated. Incorrect dose of syntocinon. Incorrect practice. [Long] timescale between delivery of placenta and examination. No notes from Registrar. No ongoing estimation of blood loss. Guidelines followed once gynae[cology] consultant arrived. ?Incorrect estimation of blood loss. Timely decision for hysterectomy. Appropriate place for care.

Examples of lessons identified Overall, the lessons for care identified in the local reviews were briefer and less focused than those identified on external reviews. The following are examples of lessons identified from the local review of the care of a woman with a postpartum haemorrhage and hysterectomy: Syntometrine not indicated. Incorrect dose of syntocinon. Incorrect practice. [Long] timescale between delivery of placenta and examination. No notes from Registrar. No ongoing estimation of blood loss. Guidelines followed once gynae[cology] consultant arrived. ?Incorrect estimation of blood loss. Timely decision for hysterectomy. Appropriate place for care. In contrast, the external assessor’s reports into the same case gave more detailed comments: Care provided to this woman appears to have been extremely well managed (National Institute for Health and Care Excellence (NICE) Guidance: Routine care for the healthy pregnant woman. http://www.nice.org.uk/Guidance/CG62). The multidisciplinary team appears to have responded quickly to the emergency situation. Consultant obstetricians and anaesthetists were actively involved in the care.

In contrast, the external assessor’s reports into the same case gave more detailed comments: Care provided to this woman appears to have been extremely well managed (National Institute for Health and Care Excellence (NICE) Guidance: Routine care for the healthy pregnant woman. http://www.nice.org.uk/Guidance/CG62). The multidisciplinary team appears to have responded quickly to the emergency situation. Consultant obstetricians and anaesthetists were actively involved in the care. In addition to the measures taken to control the haemorrhage, other procedures should have been considered at or before laparotomy to preserve the uterus. The notes do not reflect consideration of tying off the uterine arteries, calling for vascular surgical assistance or considering uterine artery embolisation (Prevention and Management of postpartum haemorrhage. http://www.rcog.org.uk/files/rcog-corp/GT52PostpartumHaemorrhage0411.pdf). I am unaware of the facilities available in the unit, and if any of these measures could have been taken but potentially, this would have prevented the need for hysterectomy. The resuscitation was excellent, including ensuring active warming and temperature monitoring. The medical notes are adequate but do not give a very clear picture of the events and the decision process at each stage. Lack of observation charts.

back from external reviewers All 11 external reviewers completed the questionnaire to provide feedback on the process. The common challenges faced with conducting external reviews were the poor quality of the notes, working in isolation and having time to do the reviews, as well as meeting the requirements of NHS work. The reviewers viewed their roles as external assessors in this study and as local assessors in their own maternity units to be different. An external assessor’s role was thought to be to provide an unbiased, objective opinion on the clinical care documented. As local assessors, they would evaluate skills among staff, training, protocols and organisational issues. Some found it quicker and easier to be involved in local reviews than external reviews because of being able to speak to the staff involved and having access to all the notes. The reported challenges of being involved in local reviews were being influenced by the people involved in an incident and giving constructive criticism to colleagues while avoiding a blame culture.

In addition to the measures taken to control the haemorrhage, other procedures should have been considered at or before laparotomy to preserve the uterus. The notes do not reflect consideration of tying off the uterine arteries, calling for vascular surgical assistance or considering uterine artery embolisation (Prevention and Management of postpartum haemorrhage. http://www.rcog.org.uk/files/rcog-corp/GT52PostpartumHaemorrhage0411.pdf). I am unaware of the facilities available in the unit, and if any of these measures could have been taken but potentially, this would have prevented the need for hysterectomy. The resuscitation was excellent, including ensuring active warming and temperature monitoring. The medical notes are adequate but do not give a very clear picture of the events and the decision process at each stage. Lack of observation charts. The lessons identified in a case of maternal sepsis illustrate the different focus observed among local compared with external reviews. The local review of the case noted the following points with respect to suggestions for local service improvements: Delay in the administration of antibiotics within the ‘golden hour’ [the hour immediately after sepsis is first suspected]. Consultant obstetrician informed who discussed the incident with consultant on [Accident and Emergency] (A&E). All pregnant women over 20 weeks gestation who attend A&E should be transferred to either the delivery suite or patient assessment unit (except those involved in major trauma when A&E should call the obs[tetrics] and gynae[cology] team to attend A&E).

Consultant obstetrician informed who discussed the incident with consultant on [Accident and Emergency] (A&E). All pregnant women over 20 weeks gestation who attend A&E should be transferred to either the delivery suite or patient assessment unit (except those involved in major trauma when A&E should call the obs[tetrics] and gynae[cology] team to attend A&E). Feedback to A&E staff and include in communications diary. Raise awareness with [Ambulance Service] advising that all women of 20 weeks and over should be brought straight to maternity. In contrast, the external reviews of the same case focused more on specific aspects of clinical care and noted wider actions in the whole care pathway (including the general practitioner (GP)): There were delays in recognition and management of this woman's sepsis. Only two of the six ‘sepsis six bundle’ measures were implemented by A&E staff—blood and oxygen therapy, and there was a significant delay in commencing IV antibiotics. The ‘Surviving Sepsis’ campaign highlights the importance of early antibiotic therapy in the ‘golden hour’ after presentation. Pregnancy should not be a reason to postpone antibiotic administration when sepsis is suspected, and neither is it a contraindication to chest X-ray. A [Modified Early Warning Score] (MEWS) of 7 should generate a more urgent medical assessment. There might have been some delay in recognising the severity of her illness by her GP, as it was documented that she had been ill for 4–6 weeks.

Pregnancy should not be a reason to postpone antibiotic administration when sepsis is suspected, and neither is it a contraindication to chest X-ray. A [Modified Early Warning Score] (MEWS) of 7 should generate a more urgent medical assessment. There might have been some delay in recognising the severity of her illness by her GP, as it was documented that she had been ill for 4–6 weeks. There were several cases in which local reviews identified no lessons for care but where external reviews considered that there were important lessons to be learned. For example, the following was noted from local assessment of the care of a woman with uterine inversion: All appropriate actions had been undertaken and no reason could be found for it to have occurred, and therefore, no formal report was completed. In contrast, the external reviewers noted clear lessons from reviewing the same woman's care: The management of this woman's labour and delivery was complicated by the fact that she was not suitable for regional anaesthesia. There seems to be little planning for analgesia in labour or if anaesthesia were required at delivery (the option of central neuraxial blockade having been dismissed). The anaesthetist however did not recommend any alternative options, nor did an anaesthetist review this woman in labour when pain management was clearly an issue. This led to a delay in the first stage of labour because there was reluctance to commence syntocinon (despite this being a primiparous woman's induction) as this would strengthen contractions and consequently pain.

e options, nor did an anaesthetist review this woman in labour when pain management was clearly an issue. This led to a delay in the first stage of labour because there was reluctance to commence syntocinon (despite this being a primiparous woman's induction) as this would strengthen contractions and consequently pain. There was also a delay in the second stage (over 4 h), as although earlier delivery was considered it was thought a better option to allow this woman to continue pushing as vaginal examination was so difficult due to her distress. The alternative at this time was a general anaesthetic for a forceps delivery in theatre. Uterine inversion occurred within 9 min of delivery of the baby; so, almost certainly mismanagement of third stage. I am sure delivery was challenging for all concerned. There was immediate recognition that uterine inversion had occurred, and it was dealt with and managed appropriately. Query whether the patient was debriefed or whether the significance for future pregnancies was discussed.

al reviews than external reviews because of being able to speak to the staff involved and having access to all the notes. The reported challenges of being involved in local reviews were being influenced by the people involved in an incident and giving constructive criticism to colleagues while avoiding a blame culture. Feedback from local staff Feedback was obtained in questionnaires from 75% (36 of 48) of the members of local staff from the six participating maternity units who attended the feedback sessions. The three key points to optimise local review that were most frequently recalled from the discussion were the importance of multidisciplinary review, the need to understand the local situation at the time of the incident, and ensuring lessons are learned and implemented in a constructive manner. Many members of staff reported appreciation of the educational benefits gained from reviewing incidents following presentation of the results of the study. Some improvements to local reviews had already been put in place following the presentation of the results of the study. Staff from one unit reported that anaesthetists were now invited to their incident review meetings, and several individuals reported that they were more conscious of improving their documentation in clinical notes and in the minutes of local review meetings. However, almost half of the local staff members (n=16) reported that no changes had been made to the clinical practice in their unit following the presentation of the results.

dividuals reported that they were more conscious of improving their documentation in clinical notes and in the minutes of local review meetings. However, almost half of the local staff members (n=16) reported that no changes had been made to the clinical practice in their unit following the presentation of the results. Discussion The local reviews of the care of women with severe pregnancy complications were very varied. A formal report was written for only one-third, and very few used a recognised methodology such as root cause analysis. Only one of the reviews included in the study was multidisciplinary, involving midwives, obstetricians and anaesthetists. Patients were involved in fewer than one in six reviews. An action plan was drawn up for fewer than half the cases examined; one-fifth of the action plans included recommendations for auditing changes in practice, as a result of the lessons for care identified in the review. External reviews of the same cases identified many additional messages for care; the most notable differences were that external reviews identified examples of good care in every woman's case, and that there were no external reviews that recommended individual disciplinary procedures for staff. External reviewers identified alternative recommended management approaches in half the cases.

for care; the most notable differences were that external reviews identified examples of good care in every woman's case, and that there were no external reviews that recommended individual disciplinary procedures for staff. External reviewers identified alternative recommended management approaches in half the cases. Recent reports of critical incidents in maternity units in the UK have identified similar themes of mixed purposes to reviews and lack of local critical review.7 External review (confidential enquiry) is carried out nationally using a standard approach.4 The observed differences between local and external reviews could be due to the uniform and standardised nature of the external reviews and the fact that the assessors received common online training prior to undertaking the assessments. Improvements in local reviews may thus be achieved by local assessors undertaking similar training or using national protocols. This would allow difficult areas such as cultural issues to be addressed, to allow unbiased, detailed positive and constructive feedback.

online training prior to undertaking the assessments. Improvements in local reviews may thus be achieved by local assessors undertaking similar training or using national protocols. This would allow difficult areas such as cultural issues to be addressed, to allow unbiased, detailed positive and constructive feedback. In this comparison of local and external reviews, we were unable to establish whether the external or the multiprofessional perspective added to the lessons learned and therefore the apparent advantages of external reviews. Both local and external review processes identified important messages to improve future care, although the number of specific messages identified was greater in the external reviews. Although there was little evidence of multidisciplinary review at a local level, it is important to note that it is possible that the review reports did not fully capture the multidisciplinary nature of discussions that might have occurred outside of the formal review meetings. It was also apparent that local review groups had a role to institute individual disciplinary procedures, where required, which may detract from the identification of generalised messages to improve clinical care.

he multidisciplinary nature of discussions that might have occurred outside of the formal review meetings. It was also apparent that local review groups had a role to institute individual disciplinary procedures, where required, which may detract from the identification of generalised messages to improve clinical care. However, the external review process is labour-intensive, requires administrative support and on average we estimate it would cost £2100 per case. We were not able to capture costs for the local review processes. Such additional costs are likely to prohibit the use of external reviews in current maternity practice and emphasise the importance of focusing on optimising local review processes. Nevertheless, while the costs appear high, if lessons are learned and implemented, future serious morbidity could be prevented. In addition to patient benefit, this would also potentially prevent future litigation costs. These costs were estimated by the National Audit Office in 2013 as £700 out of the £3700 spent on average on each birth in England.14

ar high, if lessons are learned and implemented, future serious morbidity could be prevented. In addition to patient benefit, this would also potentially prevent future litigation costs. These costs were estimated by the National Audit Office in 2013 as £700 out of the £3700 spent on average on each birth in England.14 This study included only six units, randomly selected from all units in England, stratified by size and risk management (CNST) level, and thus they may not be representative of maternity units in England as a whole. Nevertheless, we identified many differences in the review processes used at a local level even among this limited number of units, and we have no reason to suspect that this pattern of variation in methods used, and professional groups involved, would not be replicated across all maternity units. No copies of electronic records (eg, antenatal, postnatal and intensive care unit notes) or local protocols were available to the external assessors, and thus the external reviews could only be based on the documentation provided by the maternity units. The cases assessed were mostly cases of postpartum haemorrhage, since this represents the most common severe complication in maternity care,15 and thus it is possible that the differences we observed between local and external reviews would not be the same if the care of women with other morbidities were examined. However, the classification of care received by the women in this study, who predominantly had postpartum haemorrhage, was similar to the classification of care for the women who survived septic shock who were included in the recent MBRRACE-UK confidential enquiry,4 which gives some evidence that the results may be generalisable to other conditions.

care received by the women in this study, who predominantly had postpartum haemorrhage, was similar to the classification of care for the women who survived septic shock who were included in the recent MBRRACE-UK confidential enquiry,4 which gives some evidence that the results may be generalisable to other conditions. Substantial variability in local reviews is consistent with the findings of a national survey that documented considerable disparity in the types of incident listed for review by maternity units in the UK, including maternal sepsis only being listed by two-thirds of units, despite this being a current major concern within the NHS and internationally.4 16 17 A systematic appraisal of the quality of local guidelines for incident reviews in UK maternity units concluded that guidance was of good or high quality for 55%, and that 81% recommended that a multidisciplinary group of health professionals should review incidents.18 This guidance does not appear to be reflected in the actual practice identified in this study, since the majority of local reviews were not multidisciplinary. It was striking to note that only three local reports included mention of monitoring or audits of changes in clinical practice in response to review recommendations. Almost three quarters of local incident review guidelines recommend such auditing,18 which is essential to monitor the implementation and impact of recommendations.

iking to note that only three local reports included mention of monitoring or audits of changes in clinical practice in response to review recommendations. Almost three quarters of local incident review guidelines recommend such auditing,18 which is essential to monitor the implementation and impact of recommendations. Although this work focuses on incident reviews in maternity, following on from the high-profile failures documented in recent reports,7 other authors have highlighted problems with current local incident review processes throughout the healthcare system.19 20 Particular differences in the safety review and reporting processes which have grown up within the healthcare system, in comparison to the aviation industry from which they were derived, such as a focus on quantity rather than quality of incident review reports,19 are echoed in the findings of our study. Our observation that local reviewers appear to have a role for instituting individual disciplinary procedures is also at odds with the principle from other industries that incident reporting is coordinated by an operationally independent group.19 20 This, perhaps, argues for the more widespread use of the Confidential Enquiry process, as it is, by design, conducted by an independent group.5 Other criticisms of current healthcare incident reporting processes have focused on the fact that the focus is on reporting rather than in-depth investigation and driving improvement,19 a feature also reflected in the local reviews we examined, in which fewer than 1 in 10 planned to audit the recommended changes in practice.

criticisms of current healthcare incident reporting processes have focused on the fact that the focus is on reporting rather than in-depth investigation and driving improvement,19 a feature also reflected in the local reviews we examined, in which fewer than 1 in 10 planned to audit the recommended changes in practice. Conclusions The clear message from this study is that the quality of local reviews of the care of women with severe maternal morbidity can be improved. Very few of the reviews involved patients, despite recent initiatives to improve transparency in examining the quality of care. Local reviews should be multidisciplinary, and should be managed as a separate process to those that include individual disciplinary procedures. Local reviews should generate an action plan, and the implementation of recommendations should be audited to ensure that change has led to the desired improvement in outcomes. Improvements in local reviews may be achieved by local assessors undertaking standardised training or the development of national review protocols. Further evaluation is needed to establish whether there is added value to including an external perspective to local reviews once high-quality multidisciplinary reviews are fully implemented.

al reviews may be achieved by local assessors undertaking standardised training or the development of national review protocols. Further evaluation is needed to establish whether there is added value to including an external perspective to local reviews once high-quality multidisciplinary reviews are fully implemented. The authors would like to thank the obstetric, midwifery and anaesthetic assessors who reviewed cases, and the staff in the maternity units who assisted with obtaining ethics approvals and local research and development approvals for this study, for providing us with anonymised copies of clinical notes and local review reports and who acted as lead points of contact with respect to all issues arising during this project. Twitter: Follow Marian Knight at @Marianfknight Contributors: AS collected the data, conducted the analysis and drafted the paper; BK conducted the analysis and commented on the paper, SS and LH assisted with the analysis and commented on the paper; MC assisted with collecting the data and commented on the paper; PB and JK commented on the paper; MK designed the study, supervised the analysis and revised the paper. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

ed on the paper; PB and JK commented on the paper; MK designed the study, supervised the analysis and revised the paper. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. Funding: This paper summarises independent research funded by the National Institute for Health Research (NIHR) under the ‘Beyond maternal death: improving the quality of maternity care through national studies of “near-miss” maternal morbidity’ programme (programme grant RP-PG-0608-10038). The views expressed in this paper are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. All researchers are independent from the study funders. Competing interests: None. Ethics approval: The study was approved by the South East Scotland Research Ethics Committee (1) (study reference 13/SS/0059). Provenance and peer review: Not commissioned; externally peer reviewed. Data sharing statement: All data underlying the findings are available by request to the National Perinatal Epidemiology Unit Data Sharing Committee.

Video recording in the operating room Video recording technologies offer a powerful way to document what happens in clinical areas.1 Cameras, and to a lesser extent, microphones, can be found in a growing number of modern operating rooms in the USA, UK and other parts of the world. While they could be used to create a detailed record of what happens in and around the operating table, this is still rarely being done; the vast majority of operations are still only documented in written operation notes. When operations are being recorded, it is primarily for educational purposes: for instance, to broadcast a live feed of a surgical demonstration to a remote audience; to provide an ‘adjunct’ to live observation;2 to collect authentic footage for edited, instructional videos on a surgical technique or procedure; to facilitate video enhanced debriefing and coaching; or to formally assess surgical skills. Recently, Makary et al1 3 have proposed that video equipment in the operating room could be used as an auditing tool. They also argue that making video recording a routine occurrence would improve performance and make surgical care more transparent. They propose that a video archive of operations could prove useful for surgeons preparing to operate on a patient who had been operated on before—to check the anatomy and density of adhesions for example. It has also been suggested that when used routinely, video could be used to investigate adverse events.4–6

sparent. They propose that a video archive of operations could prove useful for surgeons preparing to operate on a patient who had been operated on before—to check the anatomy and density of adhesions for example. It has also been suggested that when used routinely, video could be used to investigate adverse events.4–6 What has gone relatively unnoticed in these recent discussions about the potential of video in the operating room is the possibilities it opens up for empirical research. While videos can only provide a partial representation of what happened and are always open to interpretation, they do provide a relatively objective, shareable point of reference. Using video, clinical events can be looked at from different angles. In suitably equipped operating rooms, the laparoscopic camera and light handle cameras provide a detailed view of the operative field, while wall-mounted cameras capture the entire multiprofessional team in action. In-built microphones and wireless microphones worn by members of the team can be used to record verbal communications.

quipped operating rooms, the laparoscopic camera and light handle cameras provide a detailed view of the operative field, while wall-mounted cameras capture the entire multiprofessional team in action. In-built microphones and wireless microphones worn by members of the team can be used to record verbal communications. Thus, video recording has now become a relatively cheap, accessible yet powerful method for collecting live data. Such data can provide alternative or complementary data to that gained through interviews, where team members provide a retrospective account of events based on their memory of what took place. Video enables different team members, external observers and analysts to review segments of an operation repeatedly, zooming in and slowing down where relevant, so as to verify each other's judgements and interpretations and develop a shared understanding of what took place.