Today’s product and design leaders often rely heavily on the word of their customers when building their product road maps; whether it’s a customer survey or a phone interview, loads of qualitative data through these methods is being collected and used to dictate how to design and develop products. Seems like a foolproof plan, right?

Actually, no—a sole reliance on customer input and feedback is built on an antiquated model of human decision making that assumes humans are rational.

Let’s take a look at why.

The standard advice for getting inside users’ brains is, “talk with them directly.” As the Ideo.org design guide says: “There’s no better way to understand the hopes, desires, and aspirations of those you’re designing for than by talking with them directly.”
In some cases, this works. In many others, however, it fails fantastically. Why? Much of our behavior, as proven time and time again in psychological research, is shaped by mental gymnastics we’re not even aware of. In order to truly understand user motivation, we need to often go beyond the interview and dig deep into observation and environment.

When asking questions fails

Imagine you’re a researcher at a company. You’ve been tasked with increasing employee participation in retirement savings programs. Your company already tried doing in-depth financial education to no avail; only 47% of their employees are signing up for sponsored retirement plans, which is a big deal when you consider similar companies boast 90% participation. Your company wants to do away with this disparity, and it’s your job to figure out what’s causing it.

If you’re following the typical PM and researcher script, you’d start with the customer. You’d ask people why they are or are not saving for retirement.
We did this. We asked over 900 people nationwide why they were or weren’t saving for retirement.¹ We asked in different ways to make sure we represented how a typical research would ask the question.. For example, we gave some participants multiple-choice questions (a quantitative, survey-based research method) and others open-ended text responses (a qualitative, interview-based method).

Surprisingly, regardless of how we asked the question, their answers did not reflect reality. The answers were not only misleading, they completely missed the core insight on what actually drove them to save or not save for retirement.

What people said:

86%² of people who were saving for retirement cited how important it was to have money for the future, saying things like “I want to be able to have something to fall back on” and “I want to be prepared and live comfortably.” Whereas people who were not saving for retirement said things like: “I can barely afford to pay bills.” 64% of this group blamed not having enough money.

These seem like reasonable, intuitive responses. However, they are false.

The real reason:

The real reason people save for retirement or (don’t save) is because they’ve been defaulted into a savings plan. Companies who automatically enroll employees in a retirement savings plan increase savings rates by 50 percentage points.³
It has nothing to do with a preference for the future or not having enough money.

But some people probably got it correct, right? Right?! Not really.

How many employees were able to articulate why they choose to save or not save for retirement? Across all conditions, only 6.2% of participants mentioned anything close to enrollment form design or default enrollment. Getting granular, while 12.3% of multiple choice respondents cited default enrollment as the reason they’re saving, only 0.2% of those answering open-ended questions mentioned it. That’s shockingly low.

If we relied solely on data from these traditional qualitative and quantitative research methods, we would likely spend a lot of time and money building solutions that would miss the mark, and for sure be less effective than just designing a default enrollment.
Put as a more general takeaway, people aren’t generally aware why they really make the decisions they do. They place far too much emphasis on their own attitudes and beliefs when they should really be giving credit to context and environment.

Imagine you could only trust 6% of the people you brought in for research. What’s Going On?

Decades of work in psychology show that much of human behavior is guided by mental processes outside our conscious awareness.⁵⁶ Simply put, we are unaware of what goes on in our brain, our susceptibility to biases, and how we make decisions.⁷

Given we’re typically incognizant of our decision-making processes, you’d imagine questions about our decision-making should be met with answers like, “I don’t know” or “I’m not sure” or “I’m not all that confident.” What happens, however, is the exact opposite.

In a study we conducted with 496 people, we explicitly asked whether participants thought a default setting would influence their choices for organ donation and retirement savings, and how confident they were about this.

If people were aware of how much the environment impacts their decision making, we’d expect people to have lower confidence in their answers. Not the case. 71%⁸ of people⁹ who said the default would not impact their retirement or organ-donation decision were “very” or “completely” confident in their answer.

Not only do people avoid answering questions with uncertainty, they’re actually confident in their incorrect answers.

This in mind, things don’t look good for traditional approaches to studying people’s behavior. When you ask people questions about why they did something, it seems they unwittingly make up stories and justifications. And perhaps more concerning for researchers, they’re confident in the story they made up!

This may be hard to believe. But researcher Petter Johansson and his colleagues got a little more extreme to prove the point.¹⁰ They showed participants two female faces and asked which face was more attractive. After the participant pointed to a face, the researcher secretly swapped photos to the one they didn’t choose (think David Cooperfield). Then, the participant was asked to explain why they chose the face (which they didn’t actually pick). A majority of participants didn’t even notice the face swap. And furthermore, they went on to explain in great detail why they preferred that face.Most participants made up the reasons—good reasons—for liking a face that they didn’t actually choose. Only a third of people realized they had been tricked.

Jonathan Haidt, Dan Dennet and Robert Kurzban give this phenomenon a name — they call it the “Press Secretary.”¹¹ The brain’s Press Secretary must, like a president or prime ministers’ press secretary, explain our actions to other people. “I don’t know” is simply not an acceptable answer. Many times the brain’s Press Secretary is correct, but other times, it’s just guessing, piecing together clues in order to give a plausible answer. The tricky thing is, we don’t know when our Press Secretary is using facts and when it’s just guessing.

This is not a new theory. In 1931, Norman Maier asked participants in his University of Michigan lab solve a puzzle most people were stumped by. When participants were out of ideas on how to solve the problem, Maier would ‘accidentally’ brush a rope, part of the problem’s solution, and set it swinging. Maier’s seemingly innocent gesture triggered participants to find the right answer. When asked, however, how they ended up solving the riddle, they failed to mention Maier’s move.¹² In 1977, the Press Secretary concept continued to popularize when Nisbett and Wilson put forth a seminal paper that argued people may be “unaware of the existence of a stimulus that importantly influenced a response”, or plainly put, people have no idea what causes them to act in a certain way.

Conclusion? We just can’t ask “Why”

Your customers (scratch that, all of us!) are not aware of all the forces at work on us and thus answering questions about them is almost laughable. One more example that should hit home.

When buying a new home, people don’t realize how much they’re influenced by the price of houses in the city they are moving from.¹⁴ Regardless of income, if someone moves from San Francisco (a super expensive city) to another city with a lower cost of living, research shows that they will have a high reference point on what a house actually costs. This high reference drives them to buy a more expensive home than if they’d moved from a cheaper city.

However, when asked why you bought your home, our studies show people cite their income, style aesthetic or commute length. People do not say: “because I was anchored high.” In this case, it’s easier to see the hilarity of asking people to articulate their motivations—it’s unlikely people even know what a reference price is, thus how could they say it’s the primary influence?

We’re blind to how much our environment affects us, making answering questions about it nearly impossible.

Let’s review:

1. Your customers tend to make up stories to explain their behavior. This is the Press Secretary.
2. These stories tend to attribute behavior to attitudes, beliefs and preferences (e.g. “I care about my future”).
3. The stories tend to discount the choice environment’s influence on behavior (e.g. we don’t say, “the default made me do it”)¹⁵
4. We’re not aware that any of this is going on. If anything, we are overconfident in our ability to reflect accurately on our behavior.

When you are seeking to understand customer behavior, you cannot rely solely on what people tell you. In fact, you may be misled if you do. Instead, you need to study what people actually do – vs what they say they will do.

About the Authors

Kristen Berman co-founded Irrational Labs, a behavioral product design company, with Dan Ariely and was on the founding team for the behavioral economics group at Google. Before designing, testing and scaling products that use behavioral economics, Kristen was a Sr. Product Manager at Intuit and camera startup, Lytro.

Dr. June Park John has worked with a range of organizations, from startups to international NGOs, on quantitative research projects including surveys, experiments and statistical analyses. She holds a Ph.D from Stanford University and has authored publications in behavioral economics and race/gender gaps in technology.

References

[1] Multiple Choice Survey demographics: (454 respondents: open response) 48% female, 72% White, 46% between 18–34, 41% less than $35,000 income, 60% employed full time. Open Text Survey demographics: (446 respondents: multiple choice): 36% female, 75% White, 57% between 18-34, 40% less than $35,000 income, 69% employed full time, Combined:(900 respondents): 42% female, 73% White, 51% between 18-34, 41% less than $35,000 income, 65% full time.

[2] 88% said “Having money in the future is important” for open text and 85% said this for multiple choice. 60% said “Not enough money to save” for open text and 67% said this for multiple choice.

[3] Beshears, J., Choi, J. J., Laibson, D., & Madrian, B. C. (2009). “The Importance of Default Options for Retirement Saving Outcomes: Evidence from the United States.” In Social Security Policy in a Changing Environment (pp. 167–195). University of Chicago Press.

[4] This represents the combined:(900 respondents Mturk survey: 42% female, 73% White, 51% between 18-34, 41% less than $35,000 income, 65% full time. “It’s the right thing to do” was 88% for open text and 84% for multiple choice (N= 474) “I don’t feel comfortable” was 18% for open text and 41% for multiple choice (N= 405). Filtered out “I don’t know”.

[5] Tversky, A., & Kahneman, D. (1974). “Judgment Under Uncertainty: Heuristics and Biases.” Science, 185(4157), 1124–1131.

[6] Morewedge, C. K., & Kahneman, D. (2010). “Associative Processes in Intuitive Judgment.” Trends in Cognitive Sciences, 14(10), 435–440.

[7] Nisbett, R. E., & Wilson, T. D. (1977). “Telling More than We Can Know: Verbal Reports on Mental Processes.” Psychological Review, 84(3), 231.

[8] Response breakdown: For the question on organ donation, 11.49% say they are “Not at all confident” or “Slightly confident” about whether a default would impact their decision. For the question on retirement savings, 13.51% say they are “Not at all confident” or “Slightly confident” about the impact of a default.

[9] Response breakdown: 73% for organ donation and 67% for retirement

[10] Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). “Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task.” Science, 310(5745), 116–119.

[11] The Righteous Mind: Why Good People Are Divided by Politics and Religion, https://righteousmind.com/author/jonathan-haidt-2/page/11/

[12] Maier, 1937. “Reasoning in Humans. II. The Solution of a Problem and Its Appearance in Consciousness.”

[13] Festinger, Leon, 1919–1989 (1962). A Theory of Cognitive Dissonance. Stanford, California. ISBN 0804709114. OCLC 921356

[14] Simonsohn, U. and Loewenstein, G. (2006), “Mistake #37: The Effect of Previously Encountered Prices on Current Housing Demand*.” The Economic Journal, 116: 175–199. doi:10.1111/j.1468–0297.2006.

[15] Ross, L., & Nisbett, R. E. (1991). McGraw-Hill Series in Social Psychology. The Person and the Situation: Perspectives of Social Psychology. New York, NY, England: Mcgraw-Hill Book Company.