Using Data & Measures to Evaluate the Success of an Improvement Project

Loading...

From the course by Johns Hopkins University

Measuring the Success of a Patient Safety or Quality Improvement Project (Patient Safety VI)

18 ratings

Johns Hopkins University

Measuring the Success of a Patient Safety or Quality Improvement Project (Patient Safety VI)

18 ratings

Course 6 of 7 in the Specialization Patient Safety

How will you know if your patient safety and quality project is meeting its objectives? Peter Drucker once said “What gets measured, gets managed.” In this course, students will learn why measurement is critical to quality improvement work. Equally important, they will learn which data sources provide the most meaningful information and tools for how and where to locate them. Finally, students will learn how to interpret data from their patient safety and quality projects to guide and modify them during implementation to maximize their chances of making a difference for patients.

From the lesson

From Data to Action and Iteration

In this module, learners will be exposed to how mixed-methods research, which includes integrating qualitative and quantitative data together, can be used successfully in quality improvement projects.Learners will also become familiar with how data and measures are used to evaluate the success of a quality improvement project. Two case studies will be shared to demonstrate how these concepts have been used in real-life quality improvement projects.

A Mixed Methods Approach7:02

Use of Quantitative & Qualitative Data in QI Projects6:24

Case Study: Improving Family Engagement in FCR7:12

Using Data & Measures to Evaluate the Success of an Improvement Project12:32

Case Study: Improving Medication Reconciliation on Hospital Admission15:19

Meet the Instructors

Matt Austin
Assistant Professor
Anesthesiology and Critical Care Medicine

In this video, we will be reviewing how to

use data and measures to evaluate success of an improvement effort.

My name is Julia Kim and I'm a faculty member at

the Johns Hopkins University School of Medicine in

Pediatrics and the Armstrong Institute for Patient Safety and Quality.

I serve as the Associate Vice Chair of Ambulatory Quality and Safety and the

Co-Director of the Armstrong Institute Patient Safety and Quality Leadership Academy.

I have worked on both outpatient and inpatient quality improvement projects.

When discussing data and measures,

it's always good to go back to definitions.

What is quality improvement?

The Health Resources and Services Administration defines

quality improvement as systematic and continuous actions that lead

to measurable improvement in health care services and

measurable improvement in the health status of targeted patient groups.

So, we can see that measurement is the foundation and the basis for quality improvement.

There are several frameworks for quality improvement.

One, is the Lean Sigma framework,

where you define a problem,

measure, analyze, improve and then control and sustain your improvement.

There's also the Institute for Healthcare Improvement,

with the three big questions addressing the aim,

measures, and ideas for change and

then the plan to study act cycles for continuous improvement.

There's also the translating evidence into practice or TRIP model,

where one summarizes the evidence,

identifies local barriers and implementation,

measure performance and ensures that all patients receive interventions.

In each of these frameworks and other frameworks as well,

you can see that measurement plays an important role,

an important step in your approach to quality improvement.

In defining your problem and in defining your aim for your quality improvement project,

it is always good to consider the Institute of Medicine Aims for Improvement,

as outlined in the 2001 Crossing the Quality Chasm report,

that outlined the aims for Healthcare Improvement to be safe,

effective, patient-centered, timely, efficient, and equitable.

Similarly, it's also helpful to think about

your aim and couch it as a smart aim and for it to be specific,

measurable, achievable, realistic, and relevant, and timely.

So when we get to the measurement aspect and we ask ourselves a question,

how will we know that a change is an improvement?

It boils down to measurement.

Your key measure should be related to your aim.

Also, measurement over time is an essential aspect to quality improvement and you want to

ensure that your measure is feasible to

collect and that's able to show improvement quickly.

One tip is to select

multiple measures and to

choose different types of measures to accurately evaluate your changes.

Three types of measures include outcomes measures,

which are your clinical outcomes or the outcomes that reflect changes in the system.

There's also process measures,

which reflect the steps that need to be taken to achieve the outcome measures,

and these are often identified as some of

the key drivers to achieving your clinical or systems-based outcomes.

There are also balancing measures or

those measures that are not directly related to your quality improvement project,

but will reflect whether or not there are

unintended consequences of your interventions or ideas for change.

And ideally, you'd use a balanced set of measures for all improvement efforts.

When you think about selecting your measure,

you want to be able to define your measure to be able to plot it on a graph.

You want to be able to plot the measures on the graph to

show the effect of the changes you are testing.

You want to be able to select an appropriate metric and think about,

is it best to define your measure as a ratio or a percentage, and ideally,

you would be explicit with the numerator and denominator of your measure,

or is it best to track your measure as a rate versus account?

And if it's an event that you're trying to track that may not be as common,

you can also consider tracking the time between events.

Another tip would be to collect data as efficiently as possible.

And if resources are not available to collect

your measures on a daily basis for all patients,

then you can use sampling strategies.

For example, to collect data on 25 patients that present

during a week or to collect data for a week out of the month.

Also, you want to be able to plot the data often enough to

be able to provide your teams with timely feedback using the graphs.

This can often help to motivate the teams to continue with the improvement project.

And in thinking about data collection and how to present the data,

you can think about whether this is going to be monthly data,

weekly data or daily data and often

by collating the data monthly and presenting it in monthly graphs.

This can minimize the variability of day to day results.

Another tip would be to try to integrate the data collection into the daily workflow of

the staff to minimize

the burdens of collecting data for your quality improvement project.

And if resources are available,

think about incorporating automated data reports from an electronic health records system

or to include the information technologists on your quality improvement team.

One of the key aspects of measurement is to measure data over time,

and there are a couple of ways that the data can be presented.

And one of the most common and effective ways to

demonstrate that data over time is through a run chart or a time series graph.

And in this example of the run chart,

we can see time along

the horizontal X axis and we can see that your measurement, your metric,

would be along the Y vertical axis,

potentially as a percentage,

as in this example and you would collect baseline data.

And in this example,

there are 10 data points and you want to have

at least 10 data points in your baseline data.

And then, you would look at the baseline median and extend that line going forward.

And then, you would then track your interventions or ideas for change.

And you can see an intervention one demonstrated,

that it started right after Time point 10.

And then, you would continue to plot your time points and then

you would demonstrate when a second intervention potentially was implemented.

And you would also note your goal line,

and in this case, the goal was 70%.

So, there's a lot of information that can be conveyed through this run chart.

When interpreting the results of your run chart,

you want to look for several different patterns in the data.

And so, you can look for shifts in the data where there are

at least six points above or below the center median line.

You can try to look for astronomical data,

or points that are so far off compared to the other data points,

or you can look for trends in the data,

where there are at least five consecutive points that either run upwards or downwards.

So, these are some examples of signals for an effective change process taking place.

So another way to demonstrate measurement of

data over time is through the use of control charts.

Similarly, we can see the key metric plotted over time.

The difference with control charts are that they

include upper control limits and lower control limits.

And these control limits indicate three sigma from the mean.

And they help us to differentiate between

random common cause variation of a data versus special cause variations.

So, there are also special cause variation

tests that we can use to try to determine whether

or not we are impacting and having effective change.

And so, some key differences between the run chart and the control chart

are that the run chart plots the median while the control chart plots the mean.

The run chart does not include control limits while the control chart includes +/-

3 sigma upper control and

lower control limits which can help us

distinguish between common cause versus special cause variation.

The run chart requires less data points,

at least 10 baseline data points,

control chart requires more.

And we have signals in both run charts.

We can identify shifts, trends,

runs or astronomical points on the run chart or through the control charts,

we can also identify shifts,

trends, cycles or outliers.

And here are some references that you can refer to find out

some more information about how to use

data and measurement in your quality improvement process.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play