Rank order survey questions (and how to use them)

Rank order survey questions (also called Ranking questions) enable participants to compare items and rank them according to their preference.

The Benefit of Rank Order Survey Questions

The benefit of this over other survey question types like multiple answer survey questions or rating questions is that you gain critical data about a participant’s preference of one item as compared to another—versus choosing multiple answers without ranking them or rating on an individual basis.

Ranking Survey Questions vs Rating Survey Questions

Ranking survey questions differ from rating questions in that participants are forced into providing a preference for one item over another—whereas in a rating question, two items may both compare similarly. For example:

“Rank your preference on the hotel amenities during your stay”

• Pool
• Gym
• Bar
• Fast Check-in

Will provide different information than

“On a scale of 1-5, please rate the following hotel amenities”

• Pool
• Gym
• Bar
• Restaurant

In the latter, all categories may receive a “5” – which will tell you how satisfied a guest is, whereas survey ranking questions may provide insight on what the most appealing amenity is.

How To Write Ranking Survey Questions

Unlike other types of survey questions, which may tell you how much a user enjoyed your product or service, Ranking questions are often used to determine what about the experience was most valuable.

For example, if you ran a Mexican restaurant, you may want to know the most popular Mexican dish among Mexican food fans in your area. But let’s say you are going to invest in certain parts of your business–new menu, top-shelf ingredients, renovation. But you only have money to invest the money into three projects. It might be helpful also to know which parts of your business customers already love and which they feel could use some improvement.

Here’s where ranked-choice survey questions are a must.

Just a few examples for your Mexican restaurant:

Please rank the following in order of importance

• Great cocktails
• Vegetarian options
• A delicious dessert menu
• A top-shelf tequila list
• Ample parking
• Speedy pick-up service

Please rank your favorite Mexican restaurants in the area

• Sleepy Joes
• Cantina Dos Segundos
• Casa Taqueria
• Rosa Mexicana
• Jose Pistolas
• Chipotle

Best Practices for Ranking Survey Questions

Don’t read too much into the lower rankings

Once people have selected their favorites or their least favorites, results can get a little noisy. Most people clearly know what they like and don’t like, and may just be selecting in the middle to get the question over with. Make sure to take this into account.

Aim for six items

Six items in a ranking question allows for a full range, without creating imprecise answering that longer lists can create. Six items allows you to determine a top 3 very clearly, or get a high end and a low end, with enough options in between. In our experience, between 6-10 items usually work well.

Shuffle items

If your survey platform allows it, set items to shuffle regularly. This will reduce survey bias. In some cases, the top result may be ranked higher because it is first on the list.

Split longer lists into categories

If you have a very long list of different product features that you want ranked, doing a long ranking list will provide messy data that is hard to use. Try splitting your items into categories, keeping lists short and creating multiple questions. It will increase response rates and give you a better dataset.