Why forced choices reveal more customer insights than ratings

When it comes to surveys, standard rating scales often fail to predict actual behavior. People may say they are “extremely likely” to do something, but their actions tell a different story. 

Forcing individuals to make trade-offs — rather than simply rating options — leads to far more accurate predictions. Learn why forced choice works and how it can improve survey design and decision-making.

The problem with ratings: Why they fail to predict behavior

Several years ago, I inherited a media attitudes study. It was a standard type of questionnaire run among the general population to understand their attitudes toward the upcoming weekend’s movie releases. The study was on behalf of a media organization to allocate advertising spend to meet its target audiences.

That, at least, was the theory. But in practice, it just plain didn’t work. That was why I had “inherited” the study. As it stood, the predictive power of the attempt was almost zero — the company might as well have flipped a coin with their spend and the study added nothing to their insight.

Was this just a poorly designed study? In a way, yes, but not for the reasons you think. The core of the survey was a series of questions, like “How likely are you to go and see [Movie X]?”, answered on a five-point scale from “Not at all likely” up to “Extremely likely.”

Those with experience with these types of studies will know the problem immediately. These questions are very poor predictors of actual behavior. People are complex, and while anyone may have good intentions to go and see the latest Marvel movie, whether they do or not depends on a hundred factors, including the weather and how much sleep they get Friday night.

Is there a better way to get closer to the true likely outcome? It turns out there is a lot of research on exactly this problem. 

The power of forced choice: Making trade-offs matter

To vastly oversimplify the research: If you want more accurate predictions, you need to force a choice. You need the respondents to weigh their options. You need to make the choice, in one way or another, hurt.

Checking a box saying, “I am extremely likely to see this movie,” costs me nothing. It occurs in a hypothetical world where time and energy are infinite and my choices match my good intentions. But that is not the world we live in — in some ways, it instead mirrors the stereotypical physicist’s world of “perfectly smooth spherical cows.”

How do we solve this? It turns out there are many ways, but in our case, we simply reworded the question to the following: 

• “Is seeing [Movie X] on your list of the top three things you plan to do this weekend?”

The respondent must now weigh what their weekend looks like and whether seeing the movie “makes the cut.” To make the top three means demoting other options (that only they know). They have to make a trade-off.

At first, our client was skeptical, especially given the vastly lower number of people who answered yes to the new question. Where a typical survey may have shown about 15% of respondents saying they were “extremely likely” to see the movie, we now see 1% to 3% putting it in their top three. 

But guess what? That figure was predictive — movies that did well on that question did well at the weekend box office (and vice versa). All because we had forced a choice.

Dig deeper: Are your CX metrics hurting your customer experience?

Beyond movies: How forced choice applies to market research and CX

This approach — forcing a choice rather than rating — applies everywhere. Surveys always want us to “rate a purchase, ” a restaurant, or an experience. Often, these ratings are not as predictive of future behavior as we would expect them to be.

An extreme example is ratings of polarizing brands — like Apple — where you are either a fan or a hater. Looking at the average star ratings is meaningless on its own, and people are often surprised by low ratings compared to the products’ success.

Instead, you must force a choice, such as “My next phone will be an iPhone.” That will be far more predictive than “How many stars would you give Apple products?”

Dig deeper: How to augment market research and glean customer insights with AI

Designing better surveys: How to ask the right questions

This isn’t only true of predicting behaviors. Sometimes, asking ratings questions will not give meaningful answers. Consider the following pair of questions from a hospital choice study:

• “On a scale from 1 to 10, how important is compassionate health care to you?”
• “On a scale from 1 to 10, how important is a hospital’s technology to you?”

Here’s the thing — it isn’t reasonable to answer anything but a 10 to both questions! Compassionate health care and hospital technology are critical to everyone, so why answer anything but 10? There’s no trade-off or cost to choosing the highest rating for both.

Consider the following question:

• “Is receiving compassionate health care more important to you than having the latest technology?”

Now, they have to make a choice — and, most importantly, it is reasonable to answer either yes or no. For some, the presence of the latest technology will trump bedside manner. For others, the opposite is true. We can split people into segments likely to make very different choices.

When these two-sided questions are presented on a continuum they are called semantic differential questions. Well-designed semantic differential questions are extremely good at segmenting users and predicting behavior.

There are whole categories of approaches generally called discrete choice models, which include the powerful MaxDiff and conjoint techniques. All of these work by forcing a choice and then mathematically analyzing what drives that choice.

Before asking for a star rating or a scale, ask yourself, “Is there a choice I can have this person make that will better reflect the realities of future behavior?” Getting the question right can be the key to truly valuable insight.

Dig deeper: How to make the most of your market research data