What is Online Card Sorting?

Online Card Sorting

“Card sorts help improve user access to information by making site interaction more efficient.”

All too often information on websites is organised according to stakeholder perceptions.

As User Experience professionals, our role is to bring focus back onto the perceptions and desires of the users to provide an optimal balance between the two.

Card sorting is an established technique often used to help structure, organize, and generally improve the “findability of content or functionality” on a website, (Rubin and Chisnell, 2008, p. 18).

Users come to a website with expectations of how they will interact with a site and the language and terminology they will find there. Card sorts help improve the way information is structured on the site so that it matches users’ mental models.

A poor match can result in an unsatisfactory user experience and visitor drop off. A good match can reduce bounce rate and increase visitor satisfaction. Card sorting studies provide insights during a new design or redesign of a site.

What is Online Unmoderated Card Sorting?

Traditional individual card sorting involves a researcher meeting a participant in a structured environment, often a usability lab, presenting them with a deck of 3X5 cards and watching them place those cards into groups according to the design of the study.

While traditional card sorting is a tried and true tool in the UX toolkit, it can be limited by budgetary and time constraints.

Today User Experience professionals operate in a faster paced research environment. We work with more complex information architecture. We often feel the need to conduct increasing numbers of studies.

As such, robust technology has been created to run high quality online card sorts and to collect and analyze increasing amounts of data.

What are the benefits of Online Card Sorting?

  • More sorts in less time: Conduct more sorts in less time, because your online studies are up and running quickly. These are less resource intensive and provide easier, automated data collection and analysis.
  • More participants: You can test higher numbers of participants for increased confidence level in data. While resource restraints often limit lab-based card sorts to roughly fifteen participants, online card sorts easily and typically include 50 to 60 participants.
  • Reduce Cost: There is no need to rent lab space or compensate participants for travel expenses. Researchers aren’t required to moderate sessions.
  • Faster data analysis: Analyze your data is faster. Heavy statistical analysis and data representation is conducted instantly. Save yourself the time of manually analyzing data.

Online Card Sorting Techniques: Open Sort and Closed Sort

Open and closed card sort techniques have different purposes:

Open cards sorts are generative. They help to develop information organization, site structures, terminology etc.

Closed card sorts are evaluative. They are used to help verify or confirm information organization and site structures in brand new design and redesign efforts.

How does an open card sort work?

  • In a remote open card sorting activity, participants are presented with a list of items.
  • Participants are asked to sort the list into groups according to how they feel the items should be organized and by their relation.
  • Items that participants feel do not belong in any group are identified as well.
  • Participants are then asked to provide appropriate category titles for the groups they have made, as well as, describe these groupings.

What can you learn from an open card sort?

  • How your target customers expect to see information organized and navigation structured on your site.
  • Ideas for new or more appropriate categories and naming conventions.
  • How users see the relationship between items.

How does a closed card sort work?

Closed card sorts are conducted for many reasons including:

To help confirm the appropriateness of categories derived from an open card sort, to evaluate the vocabulary, organization, and structure of an existing site, and to determine the best way to add more content into an existing category structure.

How does a closed card sort work?

  • In a remote closed card sorting activity, participants are presented with a list of items and a list of categories (menu items).
  • Participants are asked to sort the list into one of the provided categories according to how they feel the items should be organized.
  • An additional category ‘Does not belong in any group’ may be provided so that items that participants feel do not fit within any group are identified as well.
  • Participants are also asked to identify which items they didn’t understand and why.

What can you learn from a closed card sort?

Closed Card Sort

  • How your target customers expect to see information organized and navigation structured on your site
  • How users see the relationship between items
  • What items, if any, participants feel are outliers and do not fit into established categories.

How do you use open and closed card sorting in tandem?

Using open and closed sorts in tandem is beneficial e.g. if you want to gain insight into how visitors perceive the information architecture on your site as part of a redesign project.

Start with an open sort to determine user created categories. Later, let them conduct a closed sort to evaluate redesign changes made to the site (some of which were suggested by the initial open card sort).

The closed card sort might reveal a high overlap between users expectations and the redesign organization. In this way, the closed sort supports the open sort results.

“Additionally, following an open card sort – multiple closed card sorts (with different sets of users) can be conducted in parallel to test and compare multiple ideas of reorganizing etc information on a site (Tullis 2007, cited in Tullis and Albert, 2008, p. 225).”

What metrics can be collected with card sorts?

Two important metrics that can be collected from closed (evaluative) card sorts include:

Frequency by categoryFrequency by Category: Match between users mental model and a site structure

  • The percentage of items that participants place into the appropriate category (Tullis and Albert 2008, p. 51). This metric is indicative of the degree of match between users mental model and a site’s structure and organization.

    Frequency by itemFrequency by Item: How items are categorised

    • The highest percentage for each item. In other words, “this is an indicator of how well the ‘winning’ group pulled the appropriate cards to it” (Tullis and Albert, 2008, p. 51). Was the item placed almost solely in one category (ideal), split evenly among few or more categories (not ideal) or not placed into any category at all?

      Making sense of card sort data

      Card sort analytical tools characterize ‘perceived differences’ among pairs of all cards in [a] study” (Tullis & Albert, 2008, p. 218).

      Besides the metrics mentioned above (item frequency by category, frequency by item), results can also be generated as visual diagram data in the form of a dendrogram.


      • Dendrograms, or tree diagrams, can be generated from card sorting data (using hierarchical cluster analysis) to visually show the groups of items based on participant’s perceptions of their relationships.
      • A dendrogram is a branching diagram illustrating the strength of relationships between items and between groups of items.
      • When interpreting dendrograms to identify potential new groupings of information, look for clusters of items that are both distinct and compact.
      • Compactness refers to the observation that the shorter the distance between two items or groups of items, the more similarity they share. Distinctness refers to the observation that the longer the distance between items or groups of items, the more distinct they are perceived to be from one another.

        Tools to conduct Online Card Sorting

        There are various Card Sorting tools available in the market.

        Make sure your Card Sorting tool offers the following options:

        • Easily test large numbers of users (we recommend 50)
        • Optionally recruit actual visitors to your site (or certain pages of your site) in real time through site intercepts a.k.a. Voice of the Customer
        • Create a study with up to 100 items to sort out & up to 12 categories to group
        • Present participants with an intuitive and easy to use drag and drop interface
        • Monitor results as they come in
        • Conduct additional data analysis with SPSS and other compatible tools

        Tips for running an Online Card Sorting Test

        Make sure to ask participants questions and additionally ask follow up questions depending on their responses. This allows participants to answer the important question: “Why?”

        1. One perception of online unmoderated card sort studies is that they produce only quantitative data at the expense of qualitative data.
        2. You can also include open-ended questions so that users can explain their category selections and their rationale for sorting items.
        3. They can also identify terms/categories that they did not understand. Additionally, users can evaluate ease or difficulty of each sorting activity.
        4. In open sorts, participants are requested to write descriptions of the categories they have created. Some tools allow you to select items from a checkboxed list.
        5. Depending on which items you select, you will see category descriptions written by participants who placed those items into the same category. This feature saves many hours of effort that would be required to do this manually.