MaxDiff Analysis: Survey Design, Examples

MaxDiff analysis is a survey-based research technique used to quantify preferences. A MaxDiff question shows respondents a set of items, asking them to choose what is most and least important. When the results are displayed, each item is scored, indicating the order of preference. MaxDiff, short for maximum difference scaling, is sometimes referred to as best-worst scaling.

A MaxDiff survey goes beyond a standard rating question. It forces respondents to pick the most and least important item from a list, helping to identify what your audience truly values. MaxDiff items are sometimes to referred to as features or attributes.

MaxDiff is used to identify preferences. For example, a real estate developer could use the above sample question to determine the most preferred resort features (attributes) for an upcoming project. To maximize the budget, the company should focus on areas that are most important to potential guests. When respondents evaluate this question, features are compared against one another, and a researcher can identify a preference.

MaxDiff is useful for single-level preference data, like in this real estate example. When collecting multi-level preference data, a cousin to MaxDiff, conjoint analysis, is the preffred research tool.

Stand survey question types like rank order, matrix, or rating scales lack this ability. There are three main reasons why MaxDiff is a great research tool.

It's simple for respondents to evaluate a subset of 4 or 5 attributes and pick what is least and most preferred. For most market research surveys, a ranking question is inefficient for respondents to evaluate. When there are ten attributes to rank, it becomes increasingly difficult to rank items accurately, leading to fatigue and lower quality data.

MaxDiff Forces a respondent to pick what is least important and most important. Respondents will likely rate all features as important if separate rating questions or a matrix are used instead of MaxDiff. This drawback could be referred to as "List Order Bias." In that scenario, an organization wouldn't have the data needed to maximize the budget; the company would use resources in areas that aren't truly important.

The output of a MaxDiff question is perfectly suited to create statistical models, something other question types lack. Statistical models make it possible to quantify preferences and understand what your audience truly values easily.

To create a MaxDiff survey, create a survey as normal, and then add a MaxDiff question where you see fit. You can add an unlimited amount of attributes for respondents to evaluate. You can display up to fifty sets (50) or display all attributes inside one set. The more sets you show, the more times individual features will be compared against one another.

Additional Options:

• Require an answer (can not proceed until answered)
• You can edit the data labels. For example, instead of "Most Important," you could use "Most Likely" or "Highest Priority." The custom labels will show throughout the survey and in your survey results.
• Question reset - To ensure respondents evaluate attributes against those in the given set, we do not include a button to go back. If you would like to give respondents the ability to start the question over, enable the "Reset Button" option inside the editor. This button will remove all answers from the MaxDiff question and reset the display back to the first set.

To avoid fatigue, a MaxDiff survey should be designed to show roughly five attributes per set. To ensure attributes are evaluated evenly, you would want to show each attribute roughly two to three times per question.

The MaxDiff calculator below will help you determine how many sets to show:

The above calculator uses the following equation to come up with the number of sets required.

You would want to collect a minimum of two hundred (200) responses as more responses would ensure more variation in the sets and attribute combinations. If you wanted to filter your MaxDiff results by a subgroup, for example, by gender, you would want to collect a minimum of two hundred (200) responses for both males and females. The response requirements would be the same for each additional sub-group you wish to study.

SurveyKing randomizes attributes when showing multiple sets and has a system to ensure attributes display as evenly as possible.

Some research projects require you to define the attributes in each set. For example, maybe you want to compare “Mattress Comfort,” “Room Cleanliness,” and “Hotel Gym” in the first set.

A feature unique to SurveyKing, is the ability to define the attributes displayed for each set. To access this feature, click “Define set attributes” within the question editor. The editor will show the attributes you want to display in the top section, and the editor will show the attributes to choose from in the bottom section. Drag from the bottom section to the top section to define the attributes displayed in the set. You can use the “Next” and “Previous” buttons to toggle the sets.

Unique to the SurveyKing Platform is the ability to ask a follow-up question to respondents based on the previous answers they submitted. This follow-up question is crucial to understanding why a respondent values a particular attribute. Once all sets are evaluated, we compute a ranking score for each attribute and display an input question asking, "What makes {the top-ranked attribute} so appealing".

The follow-up question default wording can be customized as needed. The follow-up data is available for export in a simple Excel format; one column with the attributes, one column with the follow-up answer.

Anchored MaxDiff provides a way to even out your data and offers higher quality insights. We have an additional resource page for Anchored MaxDiff. The basic concept is adding a single multiple-choice question asking, "Which of the following attributes do you think are truly valuable." This data is used in the statistical model to balance out the regression, giving you higher quality data.

A good example of Anchored MaxDiff would be a sports franchise doing market research on a new stadium. Two respondents could give the same answers for what they find least and most important for stadium features. One respondent might be very passionate about sports, while another respondent is only a casual sports fan. Even though their answer choices were the same, the value in these answers is much different.

In that example, Anchored MaxDiff would place higher importance on the answers submitted by the passionate fan, and less importance on the casual fan.

Anchored MaxDiff should only be used when your audience is broad. For example, a SaaS company doing research wouldn't gain any benefits here; all their customers are generally the same - they are using a single service to accomplish a general goal.

The analysis for a MaxDiff question is broken down into four separate categories:

1) A simple count analysis

2) Using a statistical model to compute the share of preference and probability

3) Segment reporting; e.g. seperating your MaxDiff data into male vs. female

4) Using a statistical model for latent class analysis

5) Further Analysis - Time Spent

A count analysis ranks answers based on a simple score, which is computed using the below formula:

• The higher the score, the more appealing an attribute is to your audience
• A positive score means that the attribute was selected as MOST appealing more often than LEAST appealing
• A negative score means that the attribute was selected as LEAST appealing more often than MOST appealing
• A score of zero means that the attribute was chosen as MOST and LEAST appealing an equal number of times OR it has never been chosen

A statistical model is the most powerful benefit fo using MaxDiff. SurveyKing uses a Multinomial Logit Model to compute coefficients (commonly referred to as utilities) for each attribute. This multinomial model uses a method called "empirical Bayes regression" which uses a Bayes estimation technique to solve the regression equation. Utilities are the output of that regression equation meaning, odds, the share of preference, probability, and a p-value for statistical significance can all be computed.

*Summarizing the odds of all coefficients is not valuable for statistical modeling. It's simply a way to attach a percentage to survey data - e.g. how much "weight" does this answer carry.

If you know the odds of an event occurring, you can also compute the probability with following equation:

Probability is the most powerful and easy-to-use data point with a MaxDiff output. Decision-makers can easily interpret "Feature X" as an 80% chance of being selected as most important; no deep statistical knowledge is needed to understand that statement.

a Maxdiff segment report is useful to drill down into the data to find hidden relationships. For example, you might include a question in your survey that asks for the respondents' gender. You could then create a segment report (or a cross-tabulation report) by gender. The results would include a MaxDiff regression model for both "Male" and "Female." You may notice "Males" prefer a particular attribute that females do not prefer or vice versa. Probability is the most helpful metric when comparing segments. For example, males could have 80% chance of "Feature X" being selected as most important, while females might only have a 40% chance. This type of analysis is easy for decision-makers to interpret without being a statistical expert.

Respondents might fall into categories that are only uncovered with additional statistical methods. Latent class analysis groups similar MaxDiff responses together in what are called "classes." Latent class analysis is similar to cluster analysis. sticking with e real estate example above, Class #1 might contain, on average are respondents who ranked attributes in roughly this order: "Mattress comfort" > "Room cleanliness" > "All-inclusive package." The ">" symbol here means greater than.

Latant class analysis is a useful tool to get a broad overview of your data set. While "Mattress comfort" might be a clear favorite, small classes of respondents might value this with less importance. The latent class analysis would lower the weighted average of "Mattress comfort" while also showing what other classes value and in what order. The respondents that are part of each class can be exported and used spot driving factors. E.g, One specific demographic might fall exclusively into a certain class.

Latent class analysis can give you up to 10 classes per MaxDiff question. Once the classes are created, each attribute will display a regression coefficient/utility for the class. Just like in a simple logit model, the coefficients are used to calculate odds and probability. The below table would be an example output.

The time spent on each MaxDiff set is also included in the results. This data is useful to eliminate low-quality responses. Responses that answered a set too fast (under 2 seconds) should generally be eliminated from the results. The time respondents spend on each survey page is also included in the results. It is also beneficial to remove any respondents who sped through other sections of the survey.

The sample report below is based on the sample real estate MaxDiff question. "Mattress Comfort" has a 51% preference share and a 75% probability of being selected as the most important resort feature. "Mattress Comfort" has a larger share of preference than "Customer service" and "Room cleanliness" combined. "All-inclusive package" and "Hotel gym" have the lowest percentage of preference and lowest probability of being selected as most important.

While this sample data is straightforward, it demonstrates how MaxDiff goes beyond other standard question types to identify what your audience truly values. This type of data is what should drive decisions by your organization.

There are multiple MaxDiff downloads available. The default survey export will contain a column header for most important and least important per set and will be displayed and then the respondent's selection below that. The follow-up Export will show follow-up answers. The anchor export will show a slit of attributes respondents found most valuable. Lastly, the statistical export will contain columns and rows to model the least and most choices to be used in a program such a R.

This segment report has the MaxDiff responses separated by gender. "Mattress comfort" has the highest probability among all customers, but "Customer service" has a higher probability for females. A marketing executive could use this information to help guide product activities; ads highlighting great customer service should be weighted towards the female demographic, making them more likely to book a trip.