Turning high scores into low scores, or overcoming courtesy bias

Overcoming courtesy bias – the tendency of people to tell you what they think you want to hear – is challenging. Especially in situations with a power imbalance. But for feedback data to be useful it has to be honest. One of our clients came up with various ways to overcome this courtesy bias and find ways to persuade constituents that being frank is in everyone’s best interest.

The Center for Employment Opportunities (CEO) provides employment services to people with recent criminal convictions. It prepares and supports them to find a job and stay connected to the labor force. In the last decade, CEO has placed nearly 25,000 formerly incarcerated people in full-time, unsubsidized employment.

With Keystone’s support CEO has been using the Constituent Voice™ methodology with participants in the program. The survey used the Net Performance Score (NPS). NPS works on a single question where respondents give a rating from 0-10. Those who answer 9 or 10 are considered promoters, passives give 7 or 8 and detractors give 0-6. The net performance score is generated by subtracting the percentage of detractors from the percentage of promoters.

jobcoachAfter contact with CEO staff, such as job coaches, participants answer a survey on their phones. In the first round of data CEO received very high scores – mostly 9s and 10s. Through focus groups CEO discovered that participants were unwilling to give low scores in case it affected their chances of getting a job or led to other negative consequences for themselves or their job coach.

It’s like back talking to your parent. How can I tell a [CEO staff member] they’re wrong? It’s like biting the hand that feeds you.” – a CEO participant in a focus group

For the second round of questions CEO made some changes. Rather than focusing on CEO’s performance, the questions shifted to asking participants how they were doing. For example, in the first phase of the pilot questions asked participants to say whether their job coach encourages them and helps them to find a job. In the second phase, when the focus shifted away from CEO towards the participants themselves, it asks ‘Do you have a role model who encourages you to find work?’. The theory was that people would be happier to be honest about their personal situation rather than CEO’s work.

In the third phase of surveys, CEO used questions that focused on how participants were doing in and outside of work, as well as how CEO can better work with them to help them attain their goals. Questions focused more on the overall process of finding employment. Numerical questions used a 1-5 scale to make answering simpler. Other questions also asked for only a ‘yes’ or ‘no’ response.

pastedimage0-1Another very important change to the process was that for the third phase the responses were made confidential. CEO limited staff’s ability to access the data on an individual level making feedback less personal. CEO has also introduced a tablet in its NYC office allowing participants to give anonymous feedback as they leave, or are waiting to be seen.

In explaining the process CEO staff stress that they value honest feedback. Staff also strongly encouraged participants to open up in focus groups and to both solicit and provide feedback as a matter of course in all meetings they have with participants.

The results of these changes? Slightly lower scores. CEO’s net performance score was 79 in phase one, 65 in phase two and 56 in the confidential phase three. From the anonymous tablet, albeit with a still small sample size, the NPS is 34.

“I think that the data shows we’re getting better at getting honest feedback and we now have a strong mix of big-picture changes that participants would like to see (that are more complicated or cost-intensive to implement), and lower-hanging fruit that we think will result in easier, quicker improvements.” – Nathan Mandel Program Innovation Analyst of CEO

To find out more about how CEO uses Constituent Voice to improve its performance read the full case study.

Leave a Reply