In January, we began a series of articles on health information
technology by noting that clinician adoption of new technology requires
a rigorous look at systems, people, processes and capabilities. To date,
we have discussed leadership and governance, change management and
process redesign and clinical adoption principles.
Now, let's take a look at the importance of capturing often
elusive clinical benefits measurements.
Measuring ROI
When the chief financial or operations officer asks, "What is
our return on investment from all of the health information technology
deployments we are making?" The answer is usually, "that
depends. ..."
While some authors have documented a financial return on investment
related to the "benefits" derived from deploying clinical
systems initiatives, most of the literature remains silent on the issue.
There is a sense that if a return on investment cannot be expressed
in terms of dollars and cents, hospital executives and boards of
directors will stop funding these initiatives, even though quality and
patient safety returns are frequently self-evident, especially to
physicians and other clinicians.
In fact, it was this very issue that prompted Congress to pass
legislation as part of the American Recovery and Reinvestment Act (ARRA)
in support of clinical information systems deployments that ultimately
resulted in the meaningful use guidelines.
Historically, the non-monetary returns on investment derived from
enhanced quality and patient safety have been insufficient to justify
the expense of a true benefits realization program. However, the
external pressures, such as the Institute of Medicines (IOM) "call
to action" (Too Err Is Human and Crossing the Quality Chasm) and
subsequent the federal legislation have created an imperative for moving
ahead with clinical information systems initiatives. In fact, the
investment incentives are moving the health care industry in a new
direction that will ultimately reduce costs, improve outcomes and
enhance service.
When it comes to measuring a return on investment related to the
benefits of clinical systems initiatives, the indicator most commonly
used is expressed in terms of financial gains or cost savings--as are
most expenditures, initiatives, or investments in the health care arena.
While valuable, measuring gains in terms of dollars and cents alone
fails to capture the value of increased quality or patient-safety. It is
these outcomes that are becoming increasingly important priorities in
today's health care arena and will gain momentum as we move from
volume to value in the new era of health reform.
Already, organizations are scrambling to meet the criteria for
patient-centered medical homes through the CMS and NCQA initiatives--all
of which focus on patient access, safety and quality of care. Meeting
core measurement requirements, decreased or zero reimbursement for
certain readmissions, and transparency all need to be taken into
consideration when determining a return on health information technology
investment.
Additionally, reporting meaningful use measures for eligible
providers and eligible hospitals is a new dynamic in measurement and
reporting. These elements will need to be incorporated into any benefit
realization program
Developing a measurement framework
Adaptations from measurement science (i.e., the use of
Shewhart's control charts and regression analysis) provide a
framework for implementation of a benefits realization solution in a
clinical environment. The use of a clear framework provides decision
makers with methods for more accurately creating and disseminating
statistical reports.
In most health care organizations, however, the limitations in
measurement practices where multiple operational variables play a factor
in results often obscures the ability of management to draw
cause-and-effect relationships from the reported results.
The result is that good leaders and providers arc left in a
quandary as to how best to achieve clinical and financial benefits
realization.
The answer is that benefits realization can be best accomplished
through the use of e-technology measurements and leadership-driven
process improvements. We have worked with a number of health systems to
solve this elusive benefits realization issue. Executive buy-in and the
delivery of timely, metric-driven data points are crucial elements for
fostering efforts to improve patient care. The integration clinical
technology infrastructure with clinical transformation is essential.
The benefits realization pyramid (see Figure 1) offers an
underlying framework for health care delivery organizations to better
understand the linkages between technology, clinical transformation, and
the benefits achieved. The deeper the breadth of technology and the
level of transformational effort, the more benefits can be realized.
There is only one way to optimize the impact of information
technology and transformational efforts within an organization and that
is through the use of health care measurement science. To be effective,
a benefits realization program must hardwire reliable measurement
processes into the ongoing operation of the health care system,
including efforts to:
[FIGURE 1 OMITTED]
* Link strategic goals and objectives to operational outcomes
* Create synergy between existing measurement programs
* Promote measurements that support course correction
* Enhance technology system design for clinicians
* Promote buy-in to the selection and deployment of measures
* Operationalize the assessment of value and opportunities for
improvement using a sustainable, reliable approach
The traditional viewpoint for measuring the clinical systems
benefits is to implement the system and then measure data to determine
the impact of the system on operational processes. Such an approach,
however, is shortsighted. Such a retrospective approach promotes a
"tail-wagging-the-dog" effect, where the benefits measured are
often selected because the data are readily available post go-live.
A second issue with the traditional retrospective approach is that
as post-go-live performance issues become evident, the attention of
health care executives is heightened. When this occurs, the
implementation team frequently moves into a firefighter mode by focusing
time and energy on resolving hot topic issues.
While resolving the hot topics is important, it frequently derails
efforts to truly measure benefits due to the lack of a formal benefits
realization program. Most often, the issues faced by the implementation
team will influence the benefits achieved. The issues represent
variables that should be proactively considered as part of a more
comprehensive benefits realization program.
For example, if one of the benefits to be measured is clinician
adoption, then the physician login issues must be resolved or adoption
will be unduly impacted.
An alternative approach to establishing a benefits realization
program is to adopt a measurement framework that has its origin in the
strategic and operational plans of the organization. Using such an
approach requires that organizational leadership consider the primary
attributes of enabled technology during the formative stages of the
decision making process related to investments in clinical information
systems (CIS).
Elements that can provide the approach toward evaluating the
investment include consideration of CIS capabilities related to:
* Information access and data use for all stakeholders
* Horizontal--across departments/business units
* Vertical--within departments/ business units
* Historical--data trending and actual results reporting
* Predictive--forecasting and future episode of care events
evaluations
* Information timeliness
* Efficiency of operations
* Process/decision support (clinical and operational)
* Evidence/protocols management
* Automation/alerts and decision support
Consideration of these information system attributes and how they
will impact the operations and processes within the health care system
should be incorporated into the CIS selection process. During the CIS
selection process, the selection team should also consider what
strategic and operational problems they want to solve by implementing
corresponding clinical technologies. These issues should drive the
benefits realization indicators selected for measurement.
For instance, if a hospital is purchasing a pharmacy system to
improve patient safety in the medication administration process, then
indicators that measure patient safety should be included in the
framework for the benefits realization program. For hospital systems, it
is important to ensure a common framework horizontally across the
system. This will enable intra-hospital comparisons that will be more
valid and reliable.
The authors participated in and helped lead the development of a
useful framework at Perot Systems that we believe provides a valuable
approach in implementing a benefits realization program. The SCORE
framework promotes an organized approach to benefits realization
indicator management. SCORE is an acronym that reflects five key
strategic or operational goals related to the deployment of clinical
technologies, including:
* S = Safety/quality
* C = Clinical automation
* O = Operational efficiency
* R = Return on investment
* E = Evidence-based utilization
Within each of these five domains, an unlimited numbers of
indicators can be developed to measure the outcomes of the systems
implementation and the impact they have on organizational efficiency and
effectiveness.
While the possibilities are "unlimited," our experience
suggests strongly that the health care organization select one or two
domains as the primary focus for benefits realization. By using such an
approach, key attributes or drivers can be monitored that create synergy
for the benefits expected from the investment based on the particular
needs of the health care organization.
As an example, some organizations for strategic purposes may be
focused on safety and quality while another organization really must
foster operational efficiency before moving toward a focus on safety and
quality. The use of the SCORE framework allows the health care
organization the flexibility for managing across the spectrum of
opportunities that have the potential for supporting greater efficiency
and effectiveness.
It is critical to define each domain within a set of SCORE domains
so there is a clear organizational understanding of what is intended to
be measured. Ensuring that the constituencies involved in the benefits
realization program have a clear understanding of the framework and the
areas to be measured will dramatically improve data consistency and,
ultimately, data integrity.
It is important to note that satisfaction is not included as a
domain in the SCORE framework, even though there is some interest in
measuring this area post-implementation of a clinical information
system. Both patient satisfaction and employee/ physician satisfaction
are measured using survey tools to gather data on an annual or
semi-annual basis. These survey tools do not usually include questions
that will help answer the question "What impact does the
implementation of clinical systems have on your work environment?"
(or, " ... your care?").
While these may be questions that should be asked, most patients
cannot relate the use of technologies as a tool for improving their
care. Proxies are often used, such as "Was the registration process
timely and efficient?"
The SCORE framework uses similar assumptions. If patients are safe
and have been given quality care, then their satisfaction will be high.
If clinicians and physicians are satisfied with the system, they will
embrace automated systems for clinical care management and will use the
system as it is designed to be used. Inclusion of patient, physician,
and clinician satisfaction should be performed as a balancing indicator
when those results become available.
Indicator development and selection
In an infinite universe of possible variables to measure. It is
helpful to define the scope of a benefits realization program. This will
ensure that the program is "doable" and that the organization
is not "collecting data for data's sake."
Every indicator included in the framework for benefits realization
should be a deliberate decision on the part of the organization's
leaders. Data collection and management take time and effort that is
then not able to be spent on other initiatives or patient care. Clear,
actionable indicators, within each of the SCORE domains, helps promote
improved measurement and comparability, both internally and externally
to an organization.
Accountability for consistency in data collection and reporting is
a critical step in ensuring comparability, especially within distributed
health care systems. Finally, it is important to recognize that the
specific domains used by the health care organization can change or
evolve over time.
For example, an organization may start with a focus on operational
efficiency and move toward one of the other domains as success is
realized. In our experience, use of the framework is an important
adjunct in the process because the pareto principle or the "80-20
rule" applies whereby 80 percent of the value is derived from 20
percent of the activity.
The same principles applies in benefit realization. Eighty percent
of the value derived from clinical information systems will come from
the 20 percent of domain areas targeted by the organization under the
SCORE framework.
The SCORE cards (see Figure 2) provide examples of tools that can
be used to report meaningful indicators. The first example shows a
scorecard that can be used for reporting results, while the second is a
tool to publish the criteria upon which the indicator level of
performance is scored.
Good information and good management
Understanding and using measurement terms and definitions
consistently across the organization will improve data management,
thereby enhancing the integrity of the results reported. This can be
accomplished, in part, by having clearly defined indicator statement for
each reported measure. Indicator statements describe in narrative form
what is to be measured, how it will be measured, and how it will be
reported.
The first step in writing an indicator statement is to clearly
identify a question you want to answer by conducting data analysis and
measurement. In a strong benefits realization program, the questions
asked are always linked back to the organization's strategic and
operational goals for clinical system implementation. Consider the
following guidelines when defining indicator statements:
* Specifically define the activity (or a piece of any given
process/activity) that you want to measure in order to answer the
question posed. This promotes the ability to identify what impact a
change in the process will have related to opportunities for
improvement. In addition, clarity on the purpose of collecting the data
will reduce the downstream challenges from colleagues and other
constituencies, such as "Why are we measuring that?"
* The process should include the generation of clear definitions of
the populations "or data sample" to be assessed and may also
include a list of those parts of the process or population that you need
to exclude (such as including only inpatients in length of stay
calculations or excluding those patients who are coming into the
emergency department for a 24-hour follow-up visit).
* Establishing data statement parameters very early in the process
helps to clearly define the scope of the indicator that is being
assessed and facilitates data analysis and interpretation.
Too frequently, health care organizations require too many metrics
in compiling evidence for a benefits realization program, which can
cause conflict between stakeholders and the organization. Leadership
must endeavor to ensure that a limited number of meaningful indicators
are included in the benefits realization program.
Data collection, analysis, and reporting Is a time-consuming
process, even with the advantage of automated systems. Also, it is
imperative that the same indicators be used across multi-system
organizations if comparisons over time arc to be included in the
reporting process. Selecting only three to five indicators for each of
the SCORE domains promotes a balanced analysis of the major impacts of
enabled technology and clinical systems in the health care environment.
And, finally, under the notion that repetition breeds
recognition--we highly recommend that only one or two domains serve as
the primary focus for the organization's benefits realization
program.
Establishing collaborative relationships between the clinicians,
physicians, information technologists, and quality management teams very
early in the process is critical to program management success. These
teams each possess unique viewpoints of the same system and together can
provide a balanced view of implementation challenges, data management
issues, and acceptance of results reporting.
Pushback from physicians, clinicians, and department leaders often
occurs when initial data reports are received. Leaders must establish a
process for dialogue to educate all interested parties about the
indicator definitions, (as outlined in the written indicator statements)
prior to publication of data.
Once the data have been disseminated, ongoing forums for discussion
should be encouraged, and legitimate reasons for trends and findings
must be explored and understood. Successful programs involve the
interested parties early in the process so that issues and concerns
related to potential indicators on the panel can be revealed and
discussed during the developmental phase of the program.
References
1. Thompson DI, Henry S, Lockwood L, Anderson B, and Atkinson S.
"Benefits Planning for Advanced Clinical Information Systems
Implementation at Allina Hospitals and Clinics." Journal of
Healthcare Information Management. Winter 2005. Vol. 19, No. 1.
2. Institute of Medicine- Too Err Is Human: Building a Safer Health
System. L. T. Kohn, J.M. Corrigan, and M.S. Donaldson, eds. Washington,
DC; National Academy Press, 2000.
3. Institute of Medicine - Crossing the Quality Chasm: A New Health
System for the 21st Century. Washington. DC; National Academy Press,
2001.
NOTE: The authors wish to acknowledge the contributions of Patricia
Bush, RN, and other members of the Perot Systems clinical transformation
team who assisted in developing many of the concepts and ideas outlined
in this paper.
[ILLUSTRATION OMITTED]
Kevin Fickensche, MD, CPE, FACPE. FAAFP, president and founder of
CREO Strategic Systems. kevin.fickenscher@ps.net
[ILLUSTRATION OMITTED]
Michael Bakerman, MD, MMM, CPE, FACPE, is CMIO of UMASS Memorial
Health System. Michael.bakerman@umassmemoriaL.org
Figure 2
Performance Scoring Levels
Meets/ At Risk, Requires New Sept Oct. Nov.
Exceeds Optimization Performance Initiative,
Standards Strategies Improvement Not Scored,
Recommended Strategies Not
Applicable
Medication
Verification
Nursing
Medication
Verification
Pharmacist
Lab Time to
Results
Routing
Lab Time to
Results Stat
Radiology
Time to
Results
Routine
Radiology
Time to
Results Stat
Meets/ Target
Exceeds
Standards
60
Minutes
60
Minutes
60
Minutes
30
Minutes
60
Minutes
30
MinutesFigure 2
Indicator Performance Performance Requires
Level within Level at Performance
Reach/Meets Risk/ Improvement
Target Optimization Plan/Strategies
Strategies
Recommended
Operational
Efficiency
Median Coding 3 days 4-5 days More than 5
Completion Time days
from Discharge
Cross Match to 2.0 3-5 More than 5
Transfusion
Ratio
Discharged Not 3 days or 4-5 days More than 5
Final Billed less days
Lab Duplicate 2% or less 3-40% More than 5%
Test
Cancellations as
a percent of
all canceled
tests
Radiology 2% or less 3-4% More than 5%
Duplicate Exam
Cancellations as
a percent of
all canceled
tests
ED Efficiency 210 min or 211-240 min More than 241
Time into ED to less min
Admission
ED Efficiency 120 min 121-251 min More than 252
Time into ED to min
Discharge
Productivity-% increase of Increase of Increase of less
Productive 5% or more 4-3% than 3%
Hrs/UOS
Time of 30-40 min 41-60 min More than 60
Medication min
Verification by
Nursing Order
to Nursing Sign
Off
Indicator New
Initiative/
Not
Scored/Not
Applicable
Operational
Efficiency
Median Coding
Completion Time
from Discharge
Cross Match to
Transfusion
Ratio
Discharged Not
Final Billed
Lab Duplicate
Test
Cancellations as
a percent of
all canceled
tests
Radiology
Duplicate Exam
Cancellations as
a percent of
all canceled
tests
ED Efficiency
Time into ED to
Admission
ED Efficiency
Time into ED to
Discharge
Productivity-%
Productive
Hrs/UOS
Time of
Medication
Verification by
Nursing Order
to Nursing Sign
Off