Co-authors on “If You’re Not Keeping Score, You’re Just Practicing: A Lean Health Care Program Evaluation Framework:” Martin L. Puterman, Yue Zhang, Sema K. Aydede, Becky Palmer, Stuart MacLeod, and Jennifer MacKenzie
Making the most of scarce resources has always been central to the health care industry, where increased costs and growing patient loads present current challenges in delivery. Many health care settings, including hospitals, pharmacies, and research laboratories, have been experimenting with “Lean methods” to help meet these challenges. Lean methods—strategies that help organizations target and eliminate sources of waste in daily operations—originated with the Toyota Production System and took off after World War II. Despite the adoption of Lean methods in health care, formal evaluations of its effectiveness have been lacking, and there is an urgent need for a comprehensive evaluation framework that is based on measurable indicators. To address this gap, my colleagues and I partnered with the Provincial Health Services Authority (PHSA) in British Columbia, Canada, to evaluate its flagship Lean initiative, imPROVE.
The imPROVE program was launched in 2007 across all nine agencies that operate under the PHSA, which include British Columbia’s Children’s Hospital, Women’s Hospital and Cancer Agency. Together, the PHSA employs about 8,500 people and has annual expenditures exceeding $2 billion, making it an ideal organization to benefit from Lean methods. The key component of the program involved the set-up of 75 Rapid Process Improvement Workshops (RPIWs) each year, which consisted of groups of PHSA staff members collaborating with trained program leaders and outside consultants to identify and solve relevant problems ranging from lowering post-partum patient wait times for physician appointments to reorganizing workspace. The imPROVE program staff were particularly trained in facilitating discussion and encouraging participation from agency staff, management and the executive team.
The goal of the imPROVE program was to enhance staff engagement, system sustainability and system efficiency. Evaluating the program’s effectiveness was difficult, however, because other system-wide initiatives confounded results, and no controlled studies were taking place.
When we were asked to evaluate imPROVE, we began by reviewing the literature, interviewing stakeholders, participating in RPIWs and creating an oversight committee. We were impressed by the level of “grassroots” program participation and quickly appreciated the challenge of evaluating the ambitious and multi-faceted program. Although the program staff was eager to gauge their effectiveness, we identified a number of issues that hindered their ability to do so: ambiguous terminology, poorly defined RPIW metrics that were not clearly aligned with RPIW goals, sample sizes that did not allow for rigorous statistical analysis, inaccessible data, poorly defined project status codes, and costs for staff involvement that were not easily obtainable.
Our team worked together with the imPROVE staff to develop a glossary of common terms and lay out the essential components of a Lean evaluation: short- and long-term efficiency gains, quality and safety improvements, staff engagement, resources and financial inputs. This framework allowed us to then systematically analyze performance metrics for Lean activities using system-level performance measures and qualitative information from staff interviews. We recommended evaluating individual RPIW outcomes; combining RPIW measures; evaluating system-level impacts; evaluating staff engagement; conducting post-RPIW evaluations and interviews and reviewing longitudinal surveys and employment data.
Our proposed evaluation framework for Lean interventions is based on a rigorous statistical foundation and considers key performance indicators that are easily measured. We hope that the specific evaluation framework, which is detailed in our publication and tailored to the health care setting, can be used broadly to benefit practitioners experimenting with Lean initiatives.
The title of the paper, “if you’re not keeping score, you’re just practicing,” is a quote from Vince Lombardi, former head coach of the Green Bay Packers.
Tags: