5 rules for creating a good research hypothesis

A good hypothesis is critical to creating a measurable study with successful outcomes. Without one, you’re stumbling through the fog and merely guessing which direction to travel in. It’s an especially critical step in A/B and Multivariate testing. 

Every user research study needs clear goals and objectives. Writing a good hypothesis stands in the middle of that process, which looks like this:

1: Problem: Think about the problem you’re trying to solve and what you know about it.

2: Question: Consider which questions you want to answer. 

3: Hypothesis: Write your research hypothesis.

4: Goal: State one or two SMART goals for your project (specific, measurable, achievable, relevant, time-bound).

5: Objective: Draft a measurable objective that aligns directly with each goal.

In this article, we will focus on writing your hypothesis.

Five rules for a good hypothesis

1: A hypothesis is your best guess about what will happen.
A good hypothesis says, "this change will result in this outcome."
The "change" is a variation on an element—a label, color, text, etc.
The "outcome" is the measure of success, the metric—click-through, conversion, etc.

2: Your hypothesis may be right or wrong, rather than ‘what you want’—just learn from it.
The initial hypothesis might be quite bold, such as “Variation B will result in 40% conversion over variation A”. If the conversion uptick is only 35% then your hypothesis is false. But you can still learn from it. 

3: It must be specific.
Stated values are important. Be bold while not being ridiculous. Believe that what you suggest is indeed possible. When possible, be specific and assign numeric values to your predictions.

4: It must be measurable.
The hypothesis must lead to concrete success metrics for the key measure. If click through, then measure clicks, if conversion, then measure conversion, even if on a subsequent page. If measuring both, also state in the study design which is more important, click through or conversion.

5: It should be repeatable
You should be able to run multiple different experiments testing different variants, and be able to re-test to get the same results. If you find that your conversion went down, then back up to a prior version and try a different direction. 

How to structure your hypothesis

Any good hypothesis has two key parts, the variant and the result. 

First state which variant will be affected. Only state one (A-B-C) or the recipe if multivariate. 
Be sure that you’ve included screenshots of each version in your testing documentation for clarity, or detailed descriptions of flows or processes. 

Next, state the expected outcome. “Variant B will result in a higher rate of course completion.” After the hypothesis, be sure to specifically document the metric that will measure the result - in this case, completion. Leave no ambiguity in your metric. 

Quick tips for creating a good hypothesis

• Keep it short—just one clear sentence
• State the variant you believe will “win” (include screenshots in your doc background)
• State the metric that will define your winner (a download, purchase, sign-up … )
• Avoid adding attitudinal metrics with words like “because” or “since” 

Hypothesis examples

A good hypothesis has its birth in data, whether the data is from web analytics, user research, competitive analysis, or your gut.

It should make sense, be easy to read without ambiguity, and be based on reality rather than pie-in-the-sky thinking or simply shooting for a company KPI (key performance indicator) or OKR (objectives and key results). The data that result is incremental and yields small insights to be built over time. 

The images below are for A, B, and C variants. The ‘control’ is the orange box, while green and grey are variants B and C (Always state a control, which is generally the current design in use).

Hypothesis: Variant B will result in the highest click rate.

Let's look at some examples of hypotheses in the real world. Read the examples below and ask yourself how the hypothesis could be improved.

Example 1:  Variant designs for a call-to-action button (CTA) on a web page

Background: It has been noted through web analytics that…

• Only 30% of page visitors scroll past the first screen.
• 6% of all page visitors click on the CTA button.
• Of those users that click, 12% purchase (note:  conversion can often be shown with a specific monetary value of $xx per year).

Example 2:  Variant designs for text narrative in a call-out/ad on a web page

Background:  It has been noted through web analytics that:

• Same metrics as in example 1 above, AND …
• 60% of all users are technical - i.e. IT professionals
• Basic usability has shown that technical users don’t like “marketing speak”.

NOTE:  In your background, it’s best to link to actual studies that show your insights. 

Ultimately, creating a solid hypothesis is about following a process. By thinking about the problem, your prior data, your experience, plus the design options you’ve created, you already have everything you need to write a great hypothesis.