MONITORING ICU PERFORMANCE: A CRITICAL ASSESSMENT OF COMMONLY USED TOOLS

Measuring healthcare outcomes and clinical performance is perceived as a fundamental strategy to rationalize the health system and better the quality of care. Critical care, with its high costs and high mortality rates, has been well scrutinized from the start, and severity scores or other acuity adjustments systems were first introduced in the 1980s. In fact, ICUs are a small portion of the health care system, but they still represent an important and disproportionate segment of medical care and costs. In order to evaluate clinical performance, almost three questions need to have an answer: (i) How to benchmark clinical performance? (ii) How to immediately identify a worsening in clinical performance? (iii) How to investigate the causes of worsening? Risk adjustment tools are devoted to manage the first issue. Observational studies that compare groups, services, facilities, providers and treatments must consider the possible differences already existing in their respective populations, and thus the differences in patient characteristics that can determine care outcomes. Particularly, mortality depends not only on the quality of care delivered but also on the patient’s underlying health status, physiologic reserve, and admitting diagnosis. Therefore, the goal of an investigator is to take into account the possible confounding effect of the different distribution (among groups, services, facilities, providers or treatments) of “a priori” characteristics which can influence the occurrence of the outcome. Risk adjustment procedures usually are based on the construction, by mean of a logistic model, of a prognostic scoring system used to define the “a priori" risk (patients’ severity). The application of this severity measure leads to obtain a direct estimate of the probability of the outcome for every patient. Therefore, the expected outcome for the study population can be calculated as the sum of the outcome probability for each patient. Finally, the expected outcome can be compared with the observed outcome. Given the possibility to calculate the number of events expected in a group, how should an ICU verify whether the number of actual deaths is greater than the number of expected deaths? Tools of statistical quality control finalized to detect shifts in performance levels have been borrowed from industry or expressly created to monitor clinical procedures; these tools are based on sequential monitoring of performance over time and indicate whether the overall performance is better or worse than expected on the basis of the predicted risk of failure, estimated using a statistical model for risk adjustment; they are asked to early recognize a worsening in quality of care and to distinguish between random variation and special-cause variation detecting chance contribution to the performance deterioration. Particularly, the Variable Life-Adjusted Display (VLAD), based on the cumulative sum (CUSUM) method, provides valuable visual aids to evaluate clinical performance giving an alarm when the observed performance is worse than the expected. However, when a signal occurs it does not prove that a problem exists, but it suggests the opportunity to investigate for an explanation of the observed- expected difference. During the PhD course three main issues have been investigated: 1. When differences between observed and expected clinical performance is observed, it is impossible to assume that patient population is adequately modeled by the risk adjustment. Particularly, in evaluating elective surgical patients, predictive models based on a standard logistic functional form seem to systematically overestimate the expected probability of death. Methodological aspects implicated in this behavior have been explored and approaches alternative to the logistic model have been proposed; 2. Assuming that confounding is correctly modeled, monitoring performance tools need to be able to early detect a change, therefore the ability of the Variable Life-Adjusted Display (VLAD) has been investigated and the delay in signaling quantified; 3. When a negative worsening in clinical performance is observed, the investigation for a possible explanation could be arduous: it requires an analysis technique able to model complex interactions or patterns that may exist in the data. Unfortunately, traditional modeling methodologies, as regression techniques, may not be able to identify these interactions, and the application of more complex techniques need to be experimented. Modeling techniques other than linear model have been applied in order to evaluate if they are more able to guide and support interpretation of covariates role in determining a change in clinical performance.