Why big data and UX need each other

There is no doubt that big data analytics is useful and necessary to build and scale products, but can big data tell us everything we need to know?

An outside observer looking at the market might believe so. According to Forrester’s prediction, the big data market is worth $31B in 2018, up 14% from the previous year.

But in 2017, NewVantage Partners found that only 37% of companies who are trying to be data-driven have been successful.

Given this, we must ask if big data alone is sufficient to create disruptive innovations or solve some of the most wicked problems we face today? I would argue no, and that as a discipline big data can benefit from what UX can contribute, and similarly, UX can benefit from big data.

The myth of big data

Big data is obviously hot in the market. Few would argue that. But many people would also argue it has grown to mythic proportions beyond its true potential as a stand-alone discipline. Yet the devotees of big data feel otherwise.

For many in the industry who may not have been fortunate enough to work with a UX team, or get close to their users, they often feel they have the power to unlock insights that can answer our toughest business problems. This worldview may have been best stated by Chris Anderson, the one-time Editor-in-Chief of Wired, in his now famous 2008 article titled The Petabyte Age. In that article Anderson stated:

“Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.”

But can numbers really speak for themselves, and isn’t it important to understand why people do what they do?

In the UX world we would certainly argue that numbers alone can’t speak for themselves and that it is critical to understand why a user does something. And quite frankly, the ‘why’ is something that big data usually can’t tell us. It can tell us something is happening, but to understand the messy human experience behind that data point, we need UX.

How UX can help

UX is particularly suited to dig deep and qualitatively understand the lived experience of our users and their context. This is something that big data alone cannot do. That is because it can’t tell us what is going on offline or in the user’s mind, and it is exactly this type of rich data that helps us to color our understanding of the black and white quantitative data we collect from systems.

As Christian Madsbjerg of ReD Associates has said:

“The ability to truly understand someone other than you is not something that can be broken down into ones and zeroes.”

So if we are to begin to deeply understand the motivations of our users, as we so often talk about in product management, then we need to look beyond big data. That does not mean we should discredit big data insights, but we do need to move beyond big data as a primary source of insights to our toughest questions.

If we are to truly understand the fuzzier data points that often produce those most nuanced and illuminating understanding of people, then we need to observe and speak with users, and that is how UX can help.

A mixed methods model

It should also be reiterated that UX alone is not sufficient, and we too need to sometimes step back and look at the limits of our own quantitative methods such as clickstreams and heatmaps. That is not to say that these methods have no value. Of course, they do, but all methods, be that qualitative or quantitative lack some vantage point, and so both UX and data science need to reach across the table and begin working together to create a shared richer understanding of users.

For it’s this type of rich understanding that will not only create the types of disruptive product innovations we need but also the types of services and business strategies we need for an increasingly complex global market.

So in practice, how can we do this?

First, we need to get data science and UX teams working more closely together. Much like the idea that we need to get close to our users has proven to be true, the same goes for partnering with internal teams. We need to be close together to create a community of praxis where we can participate in crafting the questions we ask, analyzing the data, and sharing in the generation of insights.

Next, we need to understand how the two world-views are not opposed, but reinforcing. Taking a lead from scholarly research on mixed methods research, we should look to find the synergies between quantitative and qualitative data, and leverage the work that has come before us. Two excellent models can be found in the exploratory sequential design and the explanatory sequential design. These two models are essentially a logical framework for validating one type of data source with the other.

• Exploratory sequential design is great for the fuzzy front end of projects where you are looking to define the problem space, create disruptive innovations, or define a blue ocean strategy. In this method, researchers use qualitative studies to first understand the problem, and define and prototype some possible solutions. They then use quantitative studies to validate the potential solutions at scale.
• Explanatory sequential design is terrific for evaluation and optimization where there is already an existing product, and a business needs more data to inform its business strategy decisions or to improve an existing design. In this method, researchers use quantitative studies to understand the current performance and any anomalies in the data. Then they use qualitative studies to color their understanding of the quantitative data and explain the anomalies which they can’t make sense of.

Finally, we need to share and have critical conversations. It is not enough to have a community of practice within our organizations for the sake of competitive advantage. We also need to share our experiences and learnings from the trenches with the broader business community.

We need to foster approachable dialogue that brings us together so that we make sure we are building the right products, ethically. If Facebook has taught us anything lately, it is that an overly-engineering focused approach towards product management may not have the best social outcomes, even if it is commercially successful.

So let us start looking beyond our siloed disciplines to a broader concept of data, that aims to produce, knowledge and wisdom far beyond the reaches of the information any single discipline produces today.