Join us in tracking the evolving landscape of social innovation.
POST
Moving Toward a Better Future for Measurement By Rhonda Evans, Gabriel Kasper, & Tony Siesfeld / February 2017 / Leave a comment

How can monitoring, evaluation, and learning become even more powerful tools for social sector leaders?

Take a minute to consider where you think monitoring, evaluation, and learning (MEL) in philanthropy will be in 10 years.

It is possible that the future will be a continuation of what we see today. There is increasing demand for monitoring and evaluation, but effectively integrating results into decision-making remains a challenge. While there has been a flowering of new data methods, tools, and analytics, funders and grantees struggle with the complex landscape and limited resources. And despite a growing number of funders seeking to incorporate the voices of those they aim to help into their evaluation activities, MEL too often benefits foundations more than it does their grantees or the communities they serve.

In this “expected future”, most foundations continue to hold evaluation results close due to reputational concerns, or fears about regulation and grantee exposure. More and more organizations use data analytics, but subscale datasets and a lack of systematic data sharing limit their helpfulness.

But one can also imagine a more positive future for MEL, where continuous learning becomes a core management tool; where foundations, as commentator Van Jones once put it, “stop giving grants and start funding experiments”; where foundations and grantees share data, learning, and knowledge openly and widely; and where constituent feedback about what is needed and what success looks like is central to strategy development and review. In this “better future”, MEL becomes an integral part of funder and nonprofit strategy development that guides better, more informed decision making and social action.

What will it take to move MEL from the future we expect to the future we hope to see?

That’s what Monitor Institute, a part of Deloitte Consulting, has set out to answer with our year-long, multi-funder initiative to “re-imagine measurement.” The project—supported by the David and Lucile Packard, James Irvine, Robert Wood Johnson, Rockefeller, S.D. Bechtel Jr., W.K. Kellogg, and Wallace foundations aims to highlight existing “bright spots” worth spreading and inspire experimentation with a range of “next practices” for the future.

Our goal is to hold up a mirror to the field—not to endorse any one approach, nor the views of any single institution or project sponsor—and in the process, help individual organizations and the social sector as a whole explore and influence possible futures for MEL. To date, we have spoken with more than 125 stakeholders, including foundation executives and program staff, evaluation experts, nonprofit leaders, and data wonks. We also have immersed ourselves in literature review and used specialized content analysis software to complement our investigations.

Three Promising Areas for MEL Innovation

Through our interviews and research, we have identified three defining pillars of a better future for MEL:

  1. Putting decision-making at the center of the MEL system
  2. Empowering constituents and promoting diversity, equity, and inclusion in choice-making processes
  3. Catalyzing learning at scale

None of these areas are particularly new to the field, but in the exploration of how we might develop each of them, we can begin to see what more effective MEL systems might look like.

1. Putting decision-making at the center. One of the most pivotal areas for innovation is to find and promote better ways to put decision-making at the center of MEL—to make sure we collect data and choose methods in a way that is driven by the decisions that foundations, nonprofits, and other social sector leaders need to make. Too often the starting point for measurement is understanding reporting requirements and on what metrics and methods to use, rather than on deeper questions about what decision-makers need to know to make smarter choices about creating impact. As one expert told us, “Instead of evidence-based decision-making, we need decision-based evidence-making.” Strategy and decision-making require more agile and continuous feedback loops that link decisions with the right data, the development of persuasive analysis, and the integration of these insights into decision-making processes.

2. Empowering constituents and promoting diversity, equity, and inclusion (DEI) in decision-making. Diversity brings unique perspectives and life experiences to the decision-making table; equity focuses on promoting justice, impartiality, and fairness within decision-making procedures; and inclusion is the degree to which individuals with diverse perspectives and backgrounds are able to participate fully in decision-making processes. All these forces will drive innovation in MEL. Who participates in MEL processes (and who doesn’t) changes what gets asked, what gets measured, and how information gets used. Embracing a DEI lens can be critical to correctly identifying social and environmental needs, making connections across interventions, and engaging stakeholders in addressing their own challenges. Taking this stance is also crucial to building feedback that puts constituent perspectives about what is needed and what is effective at the center of social impact work.

3. Catalyzing learning at scale. The third area that is ripe for innovation is how funders and nonprofits learn together. We must get better at learning from and with each other—the good, bad, and inconclusive—to better match the scale and complexity of today’s social and environmental problems. This requires the development of collective knowledge efforts that promote actionable learning and balance the desire for standardization with specific organizational needs. Learning together and using new data analytics approaches also requires investment in infrastructure and tools that allow the aggregation of information and insight across organizations at scale.

In essence, putting decisions at the center of the MEL system will allow for greater meaning and parsimony; promoting DEI in MEL will broaden the perspectives for interpreting information, improving the likelihood of more effective outcomes; and combining efforts to learn at scale will improve the efficiency of the entire system.

Yet there remain real barriers to achieving these three characteristics of a better future. We are working with funders, nonprofits, and others to identify levers that can help overcome important obstacles, including:

  • Missing and misaligned incentives that can result in low-quality information about efficacy, a focus on the wrong information, and a lack of feedback, all of which make it difficult for funders and nonprofits to make the right decisions and learn from mistakes.
  • Insufficient individual and collective capacity that limits implementation. Nonprofit leaders have made it clear that they do not have sufficient capacity, and that foundations consistently underinvest (in both time and money) in the data systems, human capital, and technical support required to collect and use information effectively.
  • Complex or expensive practices, tools, and technologies that inhibit adoption. Even with innovations in data collection methods and analytics, there remains a sense that many available solutions are too costly or difficult to implement.

Bright Spots

Despite these barriers, lots of good work is already underway. For example, the online philanthropy platform GlobalGiving uses a points-based incentive structure to promote organizational learning among organizations that participate on its site. When organizations solicit constituent feedback—positive or negative—they earn benefits such as website positioning on the platform that can lead to increased fundraising.

Family Independence Initiative (FII) meanwhile uses human-centered design practices and new technology tools to build trust with and gain insight into low-income communities. To start, families that participate in FII journal about their lives via an online platform. FII then uses this data to tailor its resources to meet the priorities of those families, as defined by the families themselves. It also shares the data it collects with the families, encouraging them to be transparent about their activities without fear of penalty, and empowering them to use their personal data to guide and improve their own choices.

And the Open Society Foundations (OSF), the international philanthropic network founded by George Soros, has begun to separate conversations about learning from conversations about resource allocation. Every two years, it conducts a “portfolio review” of each area of work; program staff and board members to self-critique their activities and assess what has worked and what has not. Program allocation decisions then occur separately as part of a strategy review up to two years later. This strategy review reflects not just grantee performance, but also refinements to OSF’s approach that emerged from the learning-focused portfolio review.

Through our project, we will continue to document hundreds of emerging bright spots like these; identify orthodoxies that may be getting in the way of progress; glean lessons from important parallels like sports analytics, patient-centered health care, and behavioral economics; and surface emerging trends that may be disruptive and create new challenges and opportunities for MEL. Our hope is to develop a catalog of experiments that funders and nonprofit leaders can test—both on their own and together—as a way of shifting mindsets and behaviors that can tilt the field toward a more positive MEL future in the years ahead.

Originally published here in the Stanford Social Innovation Review.

Leave a comment, question, or resource