Keystone’s Chief Executive David Bonbright delivered the talk below at IMPCON 2016 in Atlanta.
Despite the striking resemblance, this is not an engraving of Jeremy Nicholls. It is William Blake’s depiction of Isaac Newton. Staying with this theme of artistic perspectives on measurement, I have borrowed from the writer Anais Nin for the title of my talk, a spy in the house of love.
I say “a spy” because I am not formally trained in the usual social sciences associated with evaluation and social value measurement. To aggravate matters, I am trained as a lawyer and now I am going to move into prosecutor mode. As I do, I don’t want you to feel persecuted. There are a lot more than just evaluators in this dock. Alongside you sit funders, researchers, nonprofit managers, and many other leaders in our field.
The charge is that a single-minded obsession about social scientific rigor has caused a great imbalance, it has made the powerful more powerful, and crushed the life out of many other valuable ways of measuring and understanding the work of social change.
Let me tell you where I am coming from. For the first 25 years of my career, as a grant maker and a social entrepreneur, I commissioned many evaluations, read many scores more that were commissioned by others, and was even included in the subject of a few. Overall, I have to say that at best they were interesting and occasionally insightful. I wish I could say that I or anyone else used them well.
By the mid 1990s, I came to be sure that the social change sector had to do a much better job of measurement. In 2004, I banded with a group of other measurement discontents, and set off on a journey that led to starting Keystone Accountability in 2006. For the past 10 years Keystone has been generating practitioner-friendly measurement practices. Here is some of what we have learned along the way.
First, to become more effective and useful, measurement needs to grasp the principal-agent problem at the heart of the social sector. Paying customers quickly and directly come to know the value of what they purchase. But those who pay for social change are not the persons directly affected by their “purchases”. Funders are the principals. Grantees mere agents that must cater to the principals before they can pursue their missions.
Principal-agent dynamics in social change generate several pathologies. For one thing, there are unacceptable time lags between the work to be measured and the emergence of usable measures of that work. For another, the primary constituents’ perspectives are routinely neglected.
I turn now to our early scratchings towards a solution. But as I do I would sum up what I have said so far as this. Social change measurement as practiced today has failed and will continue to fail to realize its potential until ensures that principals and agents are able to use those measures, in real time; to see what people want, whether they are getting it, and, if they are not, how to improve. It is high time for a phrase that began among mental health and disability activists in the 80s to arrive at the door of social change measurement, “nothing about us, without us”.
When we reached this conclusion, about ten years ago, we asked ourselves whether somebody out there had cracked this nut. The answer we found surprised us — customer satisfaction. Despite the fact that businesses have the clear summative metric of the “bottom line”, they attach great importance to future-looking measures of consumer perception, trust and loyalty. In business, in turns out, relationship is the best predictor of future value.
Today, the cutting edge customer satisfaction tools like Revoo combine social media, with open feedback like Yelp or TripAdvisor, with sophisticated public responses from the company, in ways that start to blur the customer with company and create the impression of co-constituents in a common enterprise. And if that starts to sound like the kinds of constituent relationships we aim for in the mission-driven social sector, you would be exactly right.
So we shamelessly borrowed from the customer satisfaction folks, blended it with our own grounding in participatory methodologies, to produce an approach that we call Constituent Voice. Like customer loyalty metrics, Constituent Voice metrics can predict outcomes. CV is a five-step, cyclical process that the implementer leads. (Remember, grantmakers are implementers too, and they can practice CV by listening to their grantees as well as those that they and their grantees ultimately seek to help.)
First, ask the right questions. If you ask dog owners if they will pay 40% more for a healthier dog food, most say yes but still reach for the lower price product when they shop. To find those who really will pay more, you ask where the dog sleeps. The right answer is “in the bed”. If you want to know if a teacher is effective, you don’t ask students if they like the teacher. You ask if they treat the teacher with respect.
Step 2: Use frequent, micro-surveys surveys to create a steady signal from all your constituents. In our model, the survey is used to create a signal that responds immediately and reliably to changes in the organization’s performance. It doesn’t burden you with data…just what you need to learn more about to get better results. It genuinely perplexes me that most organizations don’t have this reliable signal of precisely where you need to learn more, and don’t appreciate how important it is to have.
Which brings me to Step 3. To learn more, you analyze the data to understand the correlations with other objective measures, and also to segment your constituents into three – those who are fully with you, those who are ready to be convinced, and those who are not with you. Disaggregate by the two or three most important personal characteristics for your work, such as gender, age, wealth, length of relationship, and so on.
Step 4 is the second and most important part of “learning more”: Design reports that are meaningful and can be understood by your constituents in order to share with them the findings, analyses, and questions arising from all your evidence. Use your Constituent Voice practice to turn your frontline staff into co-creators of improvements with your constituents. Ask constituents why they gave you the scores they did, and pay attention to what they tell you. Draw them into a search for a solution that you can act on with them.
Step 5: Course correct. And make sure everyone knows that you have changed stuff so that when they get the next micro-survey why it is worth their while to respond.
That’s it. The rest is, as they say, just detail.
But just in case, I would like to make it clear something that I have not said. I have not said that we don’t need rigorous impact evaluations. They have their place. But what we need more…What we need before, is a way to turn measures (including those deriving from impact studies) into learning and action. That way must begin with the primary constituents, and ensure that all actors in the ecosystem understand their experience. Constituent Voice is one such dialogic learning model. We need more.
To conclude, I will risk a prediction. As we take this path of Constituent Voice we are going to meet strong resistance. To quote Fred Reichheld and Rob Markey, the inventors of the Net Promoter system: “An army of adversaries will rise to oppose a system of accountability that really works.”