What is a diary study and why is it useful for UX testing?

What is a diary study?

A diary study gathers together information about a user experience over an extended period of time (or longitudinally as you may hear it described in other places where they like big words).

Participants write about their experiences with a particular product or service in a diary.

They may also take photos or perform other activities to record their experiences. Once the study period is over, the researcher analyses the findings.

Why would you use a diary study?

A diary study is a qualitative research method, so the findings will be based on thoughts, feelings and observable behaviour rather than numerical data.

The longer period of time and the changing context of the user differentiates this method from other tests like surveys or labs, which record behaviours based on one specific moment in time, within a single context.

Diary studies remove the influence of both the researcher and the unnatural out-of-home setting, but they’re also useful for understanding long-term behaviour.

According to NN/g, these long-term behaviours can include:

• What time of day do users engage with a product?
• How do they share content with others?
• In what capacity do users engage with a product?
• What are their primary tasks?
• What are their workflows for completing longer-term tasks?
• What motivates people to perform specific tasks?
• How learnable is a system?
• How loyal are customers over time?
• How do they perceive a brand after engaging with the corresponding organization?
• What is the typical customer journey and cross-channel user experience as customers interact with your organization using different devices and channels?
• What is the cumulative effect of multiple service touchpoints?

All really helpful, and best of all, in-depth and genuinely contextual findings.

How long does a diary study last for?

A diary study can last as long as your project needs. So anywhere from two days to two months. Basically whatever best fits the purpose of the product you’re testing.

How is a diary study carried out?

During the test period, participants are asked to keep a diary and write down specific information about the activities they’ve been asked to carry out.

UX mastery recommends the user takes photos to explain their activities and highlight things that stood out to them across the course of their day.

According to Carine Lallemand, writing on UPXA, there are three categories by which entries can be collected:

• Interval-contingent protocol, in which participants have to report their experience at regular predetermined intervals (for example, every two hours or every day).
• Signal-contingent protocol, which uses a signalling device to prompt the participants to make an entry.
• Event-contingent protocol, which requires participants to report each time a specific event occurs.

The rate and timing of how people ‘self-report’ should be set up according to research needs. Lallemand suggests not being too demanding or the diary will become a “burden to your participants” – a maximum of two to three entries per day should be enough.

The structure of the diary can be open (the tester records entries in their own words) or highly structured (where closed-ended questions are predetermined). It’s up to you how precise you want the information to be.

Are there any special tools you need for a diary study?

There are electronic tools available for people to record their findings in a variety of multimedia ways (text, image, voice recordings), but it’s a good idea to discuss with your participants what method they’d be most comfortable with based on your own testing needs.

UX manager and columnist Peter Hornsby recommended a simple pen and paper approach in his recent AMA:

“I’m not aware of many technology platforms for diary studies; I’ve tended to use just written feedback in whatever format the participant is most comfortable.” Also bear in mind that “if you have to introduce a platform which itself has a learning curve, it can add an additional level of complexity to the process and can colour the client’s experience.”

Disadvantages of diary studies

• People can be full of good intentions, but then laziness/distractions/Netflix slips in. Hence why it’s possible to set up prompts to notify participants daily to fill in their diary. This however could potentially be quite annoying.
• The person has to be fully committed to the project. They also need to be comfortable and competent in writing down their findings in clear, legible language and terms.
• This also means that training sessions for the participants can be lengthy, and you may take a while to whittle down a pool of volunteers.
• NN/g refers to it as a “poor man’s field study.” A diary study is unlikely to provide observations that are as rich or detailed as a true field study, but they can serve as a decent approximation.
• Analysing diary studies can also be time-consuming and complex. As we’ll find out below…

How is the research analysed?

Post-study interview: here you can plan a follow-up conversation with the participant and discuss the specific details of the study. You’ll be able to clarify any ambiguous details (illegible handwriting issues, for instance). You can also ask for feedback from the user, which you can use to improve the process for future participants.

Data analysis: NN/g recommends “taking a deep breath” before diving into the vast amount of qualitative data and think about the following questions when evaluating their behaviours: how do they evolve and change over time? What influences these behaviours?

Then you can build a customer journey map to help understand the entire user experience from the perspective of your participant.