Feedback as democracy in social change practice

This lead article was first published in Alliance magazine, as part of the June special feature – Beyond Accountability: Feedback as Transformation – co-edited by Keystone’s David Bonbright.

A huge shift is happening in the social sector. Dennis Whittle traces it all the way back to the birth of democracy in Greece circa 594 BCE. The people are sovereign! In titling our special theme ‘Beyond accountability: feedback as transformation’ we are signalling that the long march of democracy has arrived at our doors. When we say ‘feedback’, we think of a systematic process of listening and responding to an organization’s constituents that goes beyond accountability in ways that are transformative for organization and constituents.

Take the Ground Truth surveys of frontline health workers and citizens during the height of the Ebola outbreak in Sierra Leone. Weekly citizen surveys highlighted significant gaps in the Ebola response, gaps that biweekly health worker surveys corroborated. The surveys produced two immediate results. The higher authorities took corrective actions that frontline health workers had been advocating all along. And the health workers experienced a deserved boost in status, confidence, voice and agency within the overall Ebola response.

You might well be thinking, ‘Hmm, can feedback claim to be that important in the scheme of things? Just how transformative can formal surveys be?’ You might well have a point. Even a superficial scan of the feedback landscape suggests something more akin to James’ ‘blooming, buzzing confusion’ than to a coherent set of actionable practices.

In response, we would say that big changes are always hard and messy at the beginning. So it is immensely helpful to hear from others who are trying to make sense of it all. Experience suggests that it is exactly through these conversations that the way opens to new explorations, new applications and new norms. The articles in this issue are presented in this spirit, as purposeful experimentation. They introduce pathways that could lead to deeper investigation and the co-creation of breakthrough solutions. We hope that you choose to follow some of those pathways and even forge new ones of your own! We also hope that you will let us know what you find when you do.

‘The days are running out for implementers and funders to say, without shame, that they cannot demonstrate credible evidence of what the people they aim to help actually think about their work!’

How does one ensure that feedback data is accurate and representative? How can we afford to collect feedback from the most marginalized? What about survey fatigue? How can technology help us? What is the relative value of feedback from different constituent groups? How does feedback apply across the spectrum of intervention types, from service delivery to Satyagraha? How does feedback practice relate to the monitoring of commonly tracked ‘key performance indicators’? How does feedback data combine with other evidence of programme effects? As daunting as this list of questions is, we maintain that we, collectively, are getting better and better answers. Many of our special feature articles help to put us on the right track.

First, a clutch of case studies from feedback practitioners sparkling with creative problem solving. Our research for this issue suggests that a global wave of innovation in feedback practices is under way. Second, a group of articles looking at how the systemic or structural constraints are being charted and beginning to be addressed. The risk-to-reward calculus is being revised as new incentives, resources and infrastructure are created. Third, we look at the future of the feedback field, and the growing sense of urgency. The days are running out for implementers and funders to say, without shame, that they cannot demonstrate credible evidence of what the people they aim to help actually think about their work!

Feedback in practice
How does one use primary constituent feedback to set priorities before an intervention? As with so many things, we can find powerful answers by looking outside our own backyards. In this case, Katherine Cowan explains how the UK-based James Lind Alliance has transformed the field of medical research through a process that better aligns researchers with the priorities of patients. In so doing, the JLA is scourging the considerable investment in medical research that is avoidably wasted. One shudders to think how much philanthropy and aid is avoidably wasted due to failing to listen to those it is meant to benefit.

‘The UK-based James Lind Alliance has transformed the field of medical research through a process that better aligns researchers with the priorities of patients.’

Turning to a world where new communications technology ubiquity has not yet empowered the crowd, another of us guest editors, Fadel Ndiame, explores how farmer feedback can help to reform the field of smallholder agriculture – a field dominated by scientists and technicians, where the dominant paradigm for intervention is still very much top down. His article shows when and how the conditions for farmer voice must include old-fashioned organizing. To reform smallholder agriculture, feedback systems are best understood as a tool within a wider empowerment process grounded in strengthening farmer self-organization. Agriculture development is not the only domain where this insight applies.

Staying with agriculture development, a case study from Digital Green in India shows how it uses farmer voice, literally, as a tool to communicate innovation and best practice. Through its training videos, Digital Green allows farmers to learn from other farmers. And the organization relies heavily on the farmers to assess and improve those videos. The results from this reality TV-type innovation are blowing the socks off the traditional training video.

Many of us in philanthropy and social change shy away from the terms ‘client’ and ‘customer’. A few outspoken leaders, like anti-poverty activist and MacArthur ‘genius award’ winner Mauricio Lim Miller, actively embrace it as a lesson in the centrality of choice and voice when tackling poverty. For those in the social investment and social enterprise movement the importance of the voice of the customer is axiomatic. The Root Capital case study starts from that light bulb moment when the organization realized that despite having invested in 530 agricultural businesses representing 1.1 million smallholder farmers they did not listen to their clients systematically. It goes on to chronicle a four-year journey that uses the Constituent Voice (CV) method to carry out systematic feedback data collection, learn how to take corrective actions, and ultimately change the structure of the organization. One fascinating insight is that as Root Capital gets better at feedback, it also gets better at using other types of data. The same organizational muscles come into play, just as in yoga certain poses create the muscle awareness to master other poses.

One in five children under the age of 18 live in poverty in the US. Another case study shows how LIFT is using feedback to address poverty in America. LIFT’s use of CV shows how feedback data can guide and validate programme effectiveness. Relationships matter in social change. The more they matter, the more important feedback is. LIFT has found that those members who give LIFT high scores for the question ‘I am more connected to the community and community resources thanks to LIFT’ make five times as much progress as those who give low scores. ‘These early findings give preliminary support for LIFT’s core belief that strengthening members’ social capital is key to achieving results.’ While it may seem intuitive and even obvious that the lack of social connectedness works to trap people in poverty, this insight is largely ignored in anti-poverty programmes. Armed with careful feedback data analysis, LIFT can and will change the way America tackles poverty. You read it first here!

Overcoming the challenges
As in any field, conceptual clarity is crucial – we need to be clear what we’re talking about. Genevieve Maitland Hudson’s entertaining ‘epistemology of feedback’ sets out some important distinctions. She shows us how cognitive science’s prototype theory helps us to make sense of the blooming, buzzing confusion out there – and why a kettle is not a chicken!

Caroline Fiennes tackles another key set of distinctions. Over the past 20 years the big public donors have led the charge to do rigorous impact evaluations. This has spawned what have been termed the ‘evaluation wars’ between those promoting methodologies that aim to prove causality and those advocating evaluation activities geared more towards improving. Her article parses the respective dominions of impact evaluation using RCT experimental studies and CV.

We have a growing feedback literature. Two of the three books reviewed by Dennis Whittle are addressed to a business audience. This is probably representative of the wider literature but even in the business books, the authors are starting to contemplate the feedback opportunities in the social sector.

The social sector’s feedback innovators are purposefully learning from the business sector. The competitive business sector is good at cutting costs. The Net Promoter Score (NPS) methodology is a leading cost-saving innovation in how companies understand customer preferences. The  CV methodology that is mentioned by several of our authors was developed in part by adapting NPS to the different ways in which constituent decision-making plays out in social theories of change.^[1]

Technology also has a role to play. Elizabeth Christopherson, also one of us guest editors, writes in her article on civic solutions that a global wave of technology-driven innovation in civic engagement – so-called civic tech – is making feedback cheaper, easier, and more comprehensive. We are on a steep, well-caffeinated learning curve here and – thanks to efforts to share and build transparency norms for the movement – the lessons are widely available through the internet.

A growing infrastructure
Capacity is perhaps the biggest challenge. When LIFT started its work on CV it did not have a full-time staff position dedicated to measurement and data analysis. After four years of grappling with feedback data, Root Capital created new positions in Africa and Latin America – regional marketing manager – to lead and drive ‘client communication and feedback’.

The good news is that there are a growing number of intermediary organizations dedicated to supporting good feedback practice. In 2013 several of them came together to form Feedback Labs. In 2015, Feedback Labs released a free self-diagnosis Quiz that organizations can take in a few minutes to get an instant reading on their feedback capabilities, and pointers to resources to improve. Feedback Labs also partnered with the World Bank to create a searchable online catalogue of feedback-related apps, tools and services, the Feedback Store.

‘If I can see that my organization is in the bottom 20th percentile among organizations in my peer group, I am much more likely to take action than if I had no idea how poorly I was performing. This absence of what are commonly referred to as benchmarks has been a significant problem for feedback.’

When evidence can be compared reliably across organizations it gains utility. If I can see that my organization is in the bottom 20th percentile among organizations in my peer group, I am much more likely to take action than if I had no idea how poorly I was performing. This absence of what are commonly referred to as benchmarks has been a significant problem for feedback. While lots of organizations survey their stakeholders, they do so using different questions and approaches. So I don’t know if my mean score of 7 on user experience is a good or bad score.

To solve this problem, a number of Feedback Labs members have collaborated to launch the Feedback Commons. In essence, this enables organizations to share and compare their feedback data.

What is the Feedback Commons?
It starts with a survey builder that allows you to choose from a menu of questions that have been proved to be effective. Each question is tagged with its history (what types of organization have used it), ratings (how useful organizations found it) and available benchmarks (how many and what types of organizations have uploaded answers to it). Once an organization has used a Feedback Commons question in a survey, and returns to the commons to upload the resulting answers, the commons generates a benchmark report. The commons also has a space for learning and exchange among members and it incorporates the Feedback Store.

Why is feedback so rare?
If collecting, analysing and acting on feedback has never been easier, why aren’t more organizations doing it? Charity Navigator’s newest rating criteria – which assess charities on their results reporting – provide the world’s first large-scale external review of non-profit feedback practices. After reviewing 1,250 charities, Charity Navigator has found that less than 7 per cent publish beneficiary feedback of any kind, and only a fraction of this 7 per cent provide evidence to suggest how honest or representative that feedback may be. Why is rigorous feedback practice so rare?

Part of the answer is that we are just getting there now. Feedback is ‘next in line’ for adoption. Given the growing availability of support and tools, we can be optimistic that capacity will not be a continuing constraint. In fact, the remaining knotty constraints may have to do with the structure of the philanthropic marketplace, that is to say the way incentives, rewards and risks are understood and experienced. Keystone Accountability, home base for one of us guest editors, David Bonbright, helps other organizations build and implement feedback systems. The sales cycle at Keystone is anywhere from six to 24 months, with gestation often exceeding the lifespan of the resulting work! What Keystone often hears is the concern that an organization will lose funding if its funders get wind of negative feedback about it.

Which brings us to Tris Lumley’s seminal question, ‘Why aren’t we learning?’ In his look at why the past decade’s push to measure impact is not, well, measuring up, he points to purpose. When implementers measure to meet funder demands, when funders allow measurement to be used as a grade rather than as a means to improve, the higher purpose of learning gets lost.

‘If we want to see social organizations adopting feedback practices in large numbers, then funders must set an example. Unlike venture capitalists, who hope they are investing in winners, foundations are cultivating learners.’

This is a salutary lesson. As a late 2014 Center for Effective Philanthropy study showed, non-profits say that organized philanthropy is not doing its part to support feedback.^[2] If we want to see social organizations adopting feedback practices in large numbers, then funders must set an example. Unlike venture capitalists, who hope they are investing in winners, foundations are cultivating learners.

Fortunately, some major actors are leading the field in this direction. Charity Navigator influences the giving decisions of millions of individuals, so its new CV rating criterion is a strong signal to non-profits. It has yet to announce how the CV criterion will be factored into its star ratings, so this needs close watching. The World Bank president has decreed that the Bank will require robust feedback from beneficiaries on all projects for which there is an identifiable beneficiary. Even the US Congress has imposed new reporting requirements for foreign aid. Henceforth, USAID is required to report to Congress what the ultimate beneficiaries think about US humanitarian assistance; intriguingly, USAID must also report what it is doing in response to beneficiary views.^[3]

Perhaps the most promising funder effort comes from a collective of US foundations. The Hewlett, Ford, Packard, Rita Allen, Kellogg and JPB Foundations and LiquidNet for Good came together in early 2014 to create the Fund for Shared Insight to try to catalyse a new feedback culture within the philanthropy sector. This issue features an interview with one of the Shared Insight foundation founders, Kathy Reich, and an article from Shared Insight project manager Melinda Tuan. Both highlight the importance of beneficiary feedback for foundation effectiveness. When funders value and use beneficiary feedback to inform their own work, they are on the critical path to reframing the way their grantees see it as well.

Volker Then and Martin Hölz’s article illustrates how the idea of grantee feedback, now increasingly accepted in the US, is beginning to take root in Europe. But, as the collection of articles on foundation accountability shows, lack of accountability remains a challenge for foundations the world over.

The future of feedback
Why now? First, feedback is sprouting up everywhere. This issue could easily have included three times as many great examples. We could have told the story of User Voice, the UK prison reform programme started by ex-offender, Ashoka fellow and recent recipient of the Order of the British Empire, Mark Johnson. Or the story of how child psychologist Scott Miller has dramatically improved the results of youth counselling through what he calls Feedback-Informed Treatment (FIT).^[4]

Both Michele Jolin’s article and the interview with Jay Naidoo throw a fascinating light on the future of feedback. Michele Jolin sees beneficiary feedback in social programmes as a force to create political will to make government more effective. ‘It can both elevate the voices of those who are beneficiaries of government programmes and pressure government leaders to seek better outcomes.’

Caroline Hartnell’s interview with South Africa’s Jay Naidoo reports a familiar post-colonial cycle. ‘With the establishment of a democratic constitution in South Africa, the state to some extent usurped civil society’s role, believing that “the main instrument for delivering the better life that we promised our people in 1994 was the developmental state. I think we all missed the plot there of rethinking civil society. In a sense, it was the passion and activism and energy of people that paralysed the apartheid state and forced it into negotiations with us. We should have harnessed that dynamism and energy, and made them partners in the way we delivered it”.’ For Naidoo, the key message is that we can now apply CV across all sectors. ‘So you could build this feedback loop, based on constituency voice, into many projects. This should not be a pilot thing, this should be a standard, the starting point for any entity – company, government, bilateral agency, NGO, foundation. We need to make it part of our daily practice.’

The time is now
With feedback bubbling up everywhere, the pressure towards a tipping point is building. This is about something deeper than tools and methods, deeper than capacity gaps, deeper even than structure and systems. It points to what Ashoka calls framework or mindset change – changes in the way we think, in our values and norms, in culture. The tipping point will come when enough of us understand that making the voices of those who are meant to benefit truly matter is – to borrow a phrase from two of our articles – not only the right thing to do, but also the smart thing to do.

‘The tipping point will come when enough of us understand that making the voices of those who are meant to benefit truly matter is – to borrow a phrase from two of our articles – not only the right thing to do, but also the smart thing to do.’

‘How close are we to the tipping point? Kai Hopkins and Natalia Kiryttopoulou use Diffusion of Innovation theory to answer this question, and conclude that we are beyond the innovators and starting to move beyond the early adopters as we head for an early majority. When you think how long it has taken democracy to arrive at our doors, this may seem a bit optimistic. But it brings to mind a stock answer from days of the anti-apartheid struggle. When asked how long before apartheid was overthrown, it was often said, ‘Soon, very soon. The alternative is too ghastly to contemplate’.

We think it will not be long before credible published feedback metrics will be seen as elemental to social organizations as audited financial accounts. What do you think?

Guest editors for this Alliance special feature
David Bonbright is co-founder and CEO of Keystone Accountability. Email davidb@keystoneaccountability.org
Elizabeth Christopherson is president and CEO of the Rita Allen Foundation. Email ec@ritaallen.org
Fadel Ndiame is FOSCA Lead Coordinator at AGRA. Email FNdiame@agra.org

Electronic copies of the June issue of Alliance are now available free of charge on the Alliance website.  For paper copies of the issue, please contact Alliance magazine.