Independent External Evaluation
Read the full report below or download the pdf version.
This document summarizes the recent external evaluation of the Gordon and Betty Moore Foundation’s Betty Irene Moore Nursing Initiative (BIMNI). It is being circulated as an expression of the Foundation’s gratitude to the grantees, experts and other external stakeholders who provided valuable input for the evaluation and to share with interested stakeholders some of the lessons that have been learned. This external evaluation is one of several sources of information that assist BIMNI program staff in managing the Initiative. The recommendations highlighted in this summary report may or may not be adopted by the BIMNI program team in its next Initiative Strategic Refresh.
Five Evaluation MethodsFindings for this evaluation are from five primary data sources:
1. Document Review - Hundreds of pages of documentation on BIMNI were reviewed.
2. Grantee Survey - A web-based grantee survey was administered, and data from 73% of BIMNI’s grantees were analyzed.
3. Stakeholder Interviews - Data from interviews with 45 internal and external stakeholders were synthesized.
4. Site Visits/Focus Groups - Site visits and focus groups with program participants in both nursing education and quality improvement activities supported by BIMNI were conducted.
5. Expert Panel - An expert panel composed of six national leaders in the areas of nursing education, quality improvement and quality measurement was formed to provide strategic, path-forward recommendations on how BIMNI efforts might be enhanced.
The Gordon and Betty Moore Foundation commissioned The Lewin Group (Lewin) to conduct the first external evaluation of the Betty Irene Moore Nursing Initiative (BIMNI). Beyond ensuring accountability for the Foundation’s investment in BIMNI, this evaluation also was designed to identify opportunities to further align program activities with guiding goals and objectives and to inform strategic planning for future Foundation investments in this area.
Since 2004, BIMNI has been actively working in five San Francisco Bay Area counties to, “create a replicable model for improving nursing-related patient care through a regional system that expands the registered nurse (RN) workforce and implements hospital best practices.”1 A 2007 decision by the Foundation’s Board of Trustees has expanded BIMNI’s reach to Greater Sacramento, with the goal of improving nursing-related outcomes through partnerships with hospital systems and other health care organizations in the region.
This evaluation covers the first four years (2004-2007) of BIMNI’s activities. During this time, BIMNI awarded grants totaling more than $67 million to advance two main strategies:
developing a larger, better educated RN workforce; and
implementing more effective hospital practices to improve patient safety and quality. The remainder of this report provides an overview of the key evaluation findings and recommendations.
Key Evaluation Findings and Recommendations
Key evaluation findings and recommendations are summarized below. The evaluation found that BIMNI’s overarching strategic framework is effective and on course and that stated goals and objectives are being achieved. However, the evaluation identified several opportunities to leverage BIMNI’s current efforts and successes in the areas of: 1) Oversight Processes; 2) Initiative Outcomes; and 3) General Strategic Direction.
Overall, BIMNI program staff’s oversight of its grantees compares favorably with oversight from other funders. The majority of grantees rated BIMNI’s grant award process and administration and management of grants as effective and at least as or more efficient than other funders. Most grantees also rated BIMNI program staff as being appreciably more hands-on than other funders.
Other data collected from grantees indicate that almost 80% are either on-track, ahead of schedule or have fully achieved their expected grant outcomes. The majority of grantees indicated that their projects were attracting interest from others, and efforts were being taken to ensure their sustainability. Collectively, it appears that BIMNI’s processes for establishing and overseeing grants are working effectively.
While process measures are valuable, BIMNI’s commitment to identifying tangible outcomes for its grants and then monitoring progress toward these outcomes on a very comprehensive report card is highly rigorous. We found the BIMNI Report Card to be much more advanced than report cards used at other foundations. While the Report Card does provide objective data, several opportunities to strengthen the Report Card were identified.
Adjusting the Report Card measures should be a high priority for BIMNI program staff, and investments should be focused on determining which existing measures would strengthen the Report Card, not on developing new measures.
Recommendations to Strengthen the BIMNI Report Card
Incorporate data collected by the Bay Area Patient Safety Collaborative (Beacon) that reflect the impact of specific improvement efforts.
Focus on nurse-sensitive measures from national lists of such measures that are updated and expanded upon regularly.
Include nurse-focused, patient experience measures from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS), which is available for all hospitals.
Convene periodic expert panels to review and make recommendations for changes to the current measures based on best practices in the area.
BIMNI’s efforts to expand the size and improve the quality of the Bay Area’s nursing workforce and to improve safety and care quality in Bay Area hospitals are achieving their goals. Through 2007, BIMNI had added 266 new RNs at the bedside, 92 new faculty and helped to increase pre-licensure enrollment by 20%. In addition, BIMNI had 85% of Bay Area hospitals participating in Beacon, had saved about 500 lives through the 100K Lives Campaign and had produced 1,209 change agents.
Four key BIMNI programs were selected for comprehensive assessments to understand the programs’ change models and impact. Evaluation findings and recommendations for these programs are provided below.
Accelerated Doctoral Program (ADP)
Recommendations for Improving the ADP
If program is replicated, increase the length of the ADP to a minimum of 4 years.
Conduct more frequent assessments of the ADP.
If program is replicated, convert the current ADP stipend to a non-taxable, forgivable loan.
Consider sustaining and building upon the outcomes of the ADP by replicating them at the new Betty Irene Moore School of Nursing (BIMSON) after the current program ends in 2013.
As of the end of 2007, the ADP had produced 10 nurse educators who continue to hold nursing faculty positions in the Bay Area. Another 32 students are in the pipeline, which will achieve the target of 42 nurse educators trained in the ADP. ADP-projected grant outcomes also include training an additional 750 baccalaureate and higher degree-holding nurses in the Bay Area between 2007 and 2013 through the educational expansion enabled by these faculty; the program is on track for meeting this objective.
Centralized Clinical Placement System (CCPS)
The CCPS is an on-line resource to match students with local clinical placement opportunities. Currently, 102 schools of nursing and hospitals in the Bay Area are using the system. In 2007, 9,536 students were placed and, in 2008, that number increased to 12,529. Four additional regional markets have adopted the CCPS, and future plans include additional expansion to new markets to meet revenue targets.
Although the number of students placed by CCPS is large, it is challenging for CCPS to quantify its placement rates (i.e., number of students placed/number of students requesting to be placed) as students are matched to placements in cohorts. As a result, it is challenging to quantify whether the program is producing better placement rates per se than were obtained prior to its implementation. However, the ultimate outcome of the program is increased nursing school enrollment and an independent evaluation by UCSF of the impact of CCPS on total placements and nursing school enrollment indicates that CCPS has resulted in an increase in total placements and an incremental increase in Bay Area nursing school enrollment.
At this point, only hospitals are participating as clinical sites in CCPS, and there may be an opportunity for CCPS to broaden its impact by engaging other types of clinical sites. In addition, licensing of the program to other states and regions should continue to ensure the ongoing financial viability of the program. Even though additional funding for CCPS by the Foundation is not planned, it is recommended that BIMNI program staff encourage CCPS leadership to: 1) ensure that CCPS continues to deliver value to its markets; and 2) remains financially self-sustaining moving forward.
Beacon (Bay Area Patient Safety Collaborative)
BIMNI has invested substantially in creating a Bay Area umbrella organization to promote safety and quality improvement activities. Based on interviews and a review of relevant materials, we conclude that Beacon has been extremely successful in:
Acquiring an excellent reputation among Bay Area hospitals and among external groups familiar with its efforts.
Engaging with Bay Area hospitals to participate in improvement and educational efforts.
Improving clinical outcomes related to reducing blood stream infections, ventilator-acquired pneumonia and other targeted conditions.
Despite its achievements to date, Beacon’s transition to a viable, self-sustaining organization is not guaranteed. Beacon must evolve from an organization that has been providing most of its services for free to one that receives financial support from area hospitals and other sources.
Integrated Nurse Leadership Program (INLP)
The INLP was designed to develop the leadership and change management skills of nursing managers and frontline nursing staff. During Phase I of the INLP, frontline nursing teams applied these skills to reduce overall medication administration errors in seven Bay Area hospitals. Accuracy rates improved from 83.8% at the start of the program to 93.0% after 18 months in pilot units of these participating hospitals.
The recently awarded Phase II of INLP will broaden the program’s focus to address the reduction in inpatient mortality from severe sepsis. Although this is a very successful effort, there are opportunities to expand INLP’s impact through greater coordination with other BIMNI-sponsored safety and quality investments.
General Strategic Direction
There was strong support from the expert panel and other external stakeholders for BIMNI’s strategic framework and direction and the general strategies it is pursuing. Strengthening the nursing workforce and targeting nurse-sensitive outcomes to improve the quality and safety of patient care were judged to be important, achievable outcomes that BIMNI should continue to pursue. However, several additional opportunities were identified that have the potential to leverage efforts BIMNI already has underway.
Invest in workforce strategies that support nurse retention
Recommended Strategies for Beacon
Identify matching funds to support Beacon fundraising efforts.
Support an expansion of Beacon’s role to encompass efficiency, as well as quality, as part of its mission.
Consider an affiliation with an umbrella organization with a complementary mission that might provide financial support.
Clarify the roles of Beacon and BIMNI program staff to ensure maximum coordination.
Promote Beacon through the contacts that BIMNI staff possess.
With the exception of two programs, BIMNI primarily has focused on investing in RN workforce strategies to increase the supply of nurses. However, retaining both new and experienced nurses is another important component of addressing the nursing shortage in the Bay Area. BIMNI’s efforts to expand the workforce will have greater impact if investments in programs and strategies designed to retain RNs are made. These potential investments should be considered in light of any changes in the environment for nursing, such as changes in nursing retention and turnover.
Better link workforce programs with quality improvement activities
While there are logical links between BIMNI’s workforce and quality improvement strategies,
there are no formal mechanisms for using the two strategies to strengthen each other. BIMNI program staff should consider:
Encouraging an emphasis on patient safety and quality improvement at UC Davis BIMSON.
Formalizing efforts to link participants in BIMNI’s nursing training programs with its quality improvement grantees and Beacon.
Expand the scope of safety and quality efforts currently being targeted
BIMNI’s quality improvement efforts currently are focused on direct patient care and quality and patient safety activities in the acute care setting. In order to comprehensively address safety and quality improvement, it is recommended that BIMNI program staff:
Target multiple levels of the health care system. Because patient care is part of broader systems of care, BIMNI should support programs that seek to strengthen hospital leadership’s commitment and understanding of quality and safety issues. Moreover, because quality and safety goals often are not aligned with payment drivers and other incentives, efforts to change the care environment through public reporting, payment reforms and other strategies should be considered.
Target care beyond the acute care setting to include the important immediate post-discharge patient experience. The immediate post-discharge period is oftentimes a confusing, stressful and hazardous time for patients and their caregivers. BIMNI should continue its efforts to improve discharge planning and identify other opportunities to strengthen the quality of transitions between different settings of care.
Strengthen clinical and expert input on strategy design and measurement issues
Because BIMNI’s grants support areas that are complex, evolving and require unique subject matter expertise, mechanisms for gaining more external input should be considered.
Broadening the expertise of the BIMNI program team by adding individuals with requisite subject matter expertise in nursing, patient safety, quality improvement and measurement.
Establishing a National Advisory Committee that will provide a forum to regularly engage with outside experts in nursing, patient safety, quality and measurement to gather their input on strategy design and setting priorities for BIMNI.