October’s General Meeting: Card Sorting 101

by Ted Edwins

When I heard the subject of the October General Meeting was to be Card Sorting, I thought, “Finally, something that can help my budding career as an online Texas Hold ‘Em Poker player!” But when I read the meeting description and discovered that Sherry McMenemy, manager of RIM's Knowledge Operations team, was going to teach us a quick and cheap method to improve the usability of websites information products, I thought, “Finally, something that can help my budding career as a Usability Expert!”

What is card sorting?

Card sorting is a venerable technique for improving the usability of anything from a website to an online help file to a software GUI to an index. Sherry assured us that card sorting is easy and that it quickly becomes addictive because of the great results you can attain.

Basically, you write examples of things that are put into categories or “buckets”, such as “Monthly Newsletter” or “Forgot my password”, onto sticky notes or 3×5” note cards. Give the cards to a representative user and let them sort the cards into buckets that make sense to them. Gather the cards at the end of the session and record the results in a spreadsheet. Repeat with as many users as makes sense for your application. At the end you have a road map of how users view your application.

Why use card sorting?

The reasons for using card sorting are many.

It is a quick and cheap method to test usability. You don’t need a fancy lab with mirrors and video cameras.
Card sorting is best done “offline” in personal interactive sessions, which allows for the “human” factor.
You can do it on paper or using a computer (though paper is usually the better approach as you tend to get more honest answers)
It’s very flexibility enables you to do a combination of the two methods so that you can do card sorting remotely or person-to-person sessions.

There are a couple of side benefits to card sorting. Firstly, the informal nature of card sorting provides a good way to gather information about your application, which identifies potential problems or challenges. Secondly, it is often possible to construct personas by watching and listening to users during the session. Sherry said that card sorting often leads to “aha” moments as you watch users go through the process.

Planning a session

Before you can conduct a session, a little forethought is required.

Determine the audience and “job one” of the system to be tested. Who is going to use to use the application? What work are the users really trying accomplish? How is your application supposed to help them achieve that goal?

If you are working on an existing system, document its current structure by preparing a record sheet. A record sheet can be either a spreadsheet that captures the hierarchy of terms or it can be a flowchart that captures the relationships between tasks or functions. It’s not the format that you use that is important; it’s that you capture the exact naming conventions as they are being used. Don’t try to correct terms, even if you think they need correcting. For example, if the user refers to “signing into” the computer, don’t change it to “logging into” the computer.
Line up your participants.Ideally, the session should be done in person so that you and the users are in the same room. This will impact the number of people to include in the session. “Bribery, such as small gifts, candy, or chocolate help to increase the attendance rate,” said Sherry.

How many participants are enough? Jakob Nielsen says 7-9. Boxes and Arrows, a usability blog, says 30-100. Typically, Sherry says her team at RIM uses 20-30. The people that do participate should be representative of the users who will actually be interacting with the product.

It is important to ask people who might not normally volunteer. Beware of the “professional” volunteers who want to help out on every project. You must find people who are “fresh” and will give untainted feedback.
Select your topics. There is a limit to how much work a participant can do in a session, so you must limit the number of topics each participant will process. Boxes and Arrows recommend 30-100. Sherry said that she tries to limit it to 25. The topics selected should be representative of what already exists, or what will be created. If you have time, you could do one round of tests with existing terms, then another round of tests with terms that will be used in the future.
Create at least two sets of cards. Sherry suggested that you neatly handwrite two sets of cards and number them discreetly. Let the participants know that the number on the card is for identification only and is no suggestion of order. One set of cards is ordered sequentially (for example, 1-26) and the second set of cards is ordered backwards (26-1) to counterbalance the context.

Conducting a Session

Now that you have your card sets made up and your record sheet ready, it is time to actually conduct a session with actual participants. To help participants feel comfortable, tell them there are no wrong answers and there are no “right” number of piles. Also ask them to tell you what they are thinking while they are placing each card in a pile.

Give the participant a pile of cards and tell them to put similar things together. As participants state why they were ordering the cards a certain way, she likes to write down their thought processes right on the record sheet. Use one sheet for each session. Write down exactly what the participant says, don’t try to interpret or correct the terminology that the person uses—be a “scientist” and only observe.

Keep an eye on how the participant shuffles and organizes the cards. Sherry said that it is very interesting to see how different personality types handle the cards. Some people read all the cards, and then sort them. Other people read and arrange each card as it comes up. Still others sort the cards in their hands, then put the piles on the table.

Sherry suggested that you could also experiment with conducting the session in one of two ways. You could have an “open” session, where there are no defined “buckets” for users to sort the cards. Instead, users define the buckets by putting the cards into as many piles as they want. The number of piles determines the number of buckets. When users are done sorting, they print the name of each group on a sticky and put it on that pile.

In a “closed” session, you define the buckets and then ask users to put the cards in each bucket. Each group is already named, and you want to see where the users put the topic on each card.

No matter which method you choose, at the end of the session, or some other convenient time, write the results of the session on the record sheet, and don’t forget to write a thank-you note (or email) to each participant!

Collecting and Analyzing Results

Now that you have all the session results captured on record sheets, you must decide what it all means. Sherry recommends transferring the data from the record sheets to a spreadsheet. Create a column for each category suggested by the participants, if they were allowed to do so. Group category names in a single column if you are highly confident that the meaning is the same, even if there is different wording for the categories.

As you enter the data on the spreadsheet, look for patterns and things you might not have expected. Analyze the results for number and type of categories. Are the categories named using an object- or task-oriented style?

Implementing Results

Using the analysis of your results, map out the information architecture and the navigation structure of your product. Sherry said that at this time you could run a closed card-sorting session to test the buckets or categories.

You must now write a report to provide stakeholders with compelling numbers and trends that were unearthed during the sessions. In the report, sort your inventory of items according to the results of the test. You could also provide a summary of findings to the participants and tell them the next steps of the process.

An important part of the report would be to tell the stakeholders when you’re going to start implementing the changes in the product. All the efforts of you and your participants will be wasted if you don’t make a concrete effort to actually implement the changes you’ve recommended.

Measuring Success

After a few months, conduct a different type of user testing on the interface to determine if the both the categories and the items in each category are correct. You can gather this information either through a scenario-based test or a user feedback/focus group. In either case, ask the participants and the stakeholders for feedback on the process. You want to know what went right as well as what went wrong.

Resources

Sherry recommends the following sites for more information about card sorting:

About Ted Edwins

Ted Edwins is an independent consultant specializing in software documentation. He is a Past-President of the chapter. In his spare time he continues his futile quest to lower his golf handicap.

In this issue: