In 2016, the Evaluation Sciences Unit at the Stanford University School of Medicine conducted a mixed-methods evaluation of the Patient Care Program’s work to redesign patient care in the intensive care unit. The evaluation aims were to 1) evaluate the achievements to date, 2) assess the design and implementation of the work, and 3) assess the potential for scale and spread of efforts both inside and outside the ICU setting. The evaluation produced important findings and lessons for the Patient Care Program to consider in its future work. Some of these lessons can be applied broadly and offer important considerations for other foundations and their funding efforts. We note these general lessons below:
- Elucidate root causes and measurement gaps when defining the scope of a problem. While the need for clear problem definition and an understanding of gaps may seem obvious, it is not always done in a timely or thorough fashion. Through this evaluation, we learned it is essential to fully investigate what factors are contributing to a given problem, to ensure that adequate measurement instruments are available to define the full scope of the problem and to conduct a baseline prior to embarking on the work. This will help ensure that the strategies for a given area of work are well-selected and prioritized effectively.
- Design programs to ensure the funding and timeframe are sufficient to achieve expected results. Decision makers – both on the foundation and grantee sides – should clearly outline and agree on expected results for the work within the problem scope. When a time frame or allocation will not be sufficient to achieve outcomes, it is important to articulate rationale for pursuing the work despite these known constraints, or to revisit and revise plans to address the time or resource needs.
- Differentiate grantmaking structure and outcome selection based on the stage of evidence. For example, if a foundation is still in the process of defining the area of work and understanding the scope of the problem the work will address, exploratory grants, coupled with the development and validation of measures, may prove to be a wiser approach. Likewise, when a program area is at the stage of disseminating established best practices, grants to support distribution networks or a best practice collaborative may be most appropriate.
- Rely on domain expertise and rigorous market analysis prior to funding work. If an area of work intends to bring new products to market, or scale a product for broader use, consulting experts within a given domain from the start of a project is essential. In the case of the ICU work, tech-related products created with the intent of broader use could have benefited from earlier input from experts outside the health care sector. It is also important to identify and deploy mechanisms that can de-risk the investment such as co-funding with an experienced industry partner.
- Incorporate and align on a robust and standardized strategy for monitoring and evaluation at the outset. It is important to plan for monitoring and evaluation in the design phase of the work, to identify clear measures of success and to engage relevant stakeholders in this planning. It can be challenging to conduct rigorous or conclusive evaluations of portfolios that lack complete or consistent monitoring data, or portfolios that do not require consistent or common metrics from grantees. In the case of the ICU work, there was variation in site level evaluation design and a lack of shared metrics across the grantee sites. The lack of standardized data collection and measures was challenging for assessing impact or gleaning shared insights.
- Define evaluation methods a priori, especially when tackling complex systems. When working in complex systems like that of an ICU, it’s important to define evaluation methods a priori. The variation in the site-level evaluation design and the lack of shared metrics across sites created barriers for conducting a comprehensive external evaluation of the work, and the evaluation was conducted while some areas of work were wrapping up. To mitigate these limitations, evaluations should be designed using innovative or flexible methodologies. In the case of the ICU evaluation, a rapid appraisal method was employed that allowed for efficient and real-time data collection.