Website analytics such as A/B testing and data mining tools are becoming more widespread, allowing companies to gain ever-deeper insights into their users and customers.

At the same time, users are getting more sophisticated as exposure to technology and the internet gets more commonplace.

As part of our series of interviews with Jakob Nielsen, I posed the following lengthy question…

Lee: People have really sophisticated analytics on their websites and on their apps, lots of people are running A/B testing and multivariate testing as well. Website owners use lots and lots of tools to capture user insights, so you can implement pop-ups on your site, you can get an understanding of intent, and you can mine lots of data. Big Data continues to be a theme that just won’t go away, and appears to be a really important part of the makeup of organisations, (larger ones moving forward).

So… is this testing with users where you just observe a number of users interacting with a site or an app, is it really still relevant? 

Jakob: I think first of all I want to modify the phrasing of this question and say increased sophistication of some users, because there are still a lot of users who are not sophisticated, and I think there will always be, because people just vary dramatically in so many aspects.

A lot of people are not very interested in computers or technology — they just want to get by. A lot of people don’t have a very advanced education, or, to be honest, are not that intelligent. So there’s a lot of people, a range, and those of us who work in any form of technology field (even if you are in marketing, but you are in digital marketing you are still kind of a semi-geek), we tend to be with other people who are also interested in computers and who are also highly educated and also smart and all of that, so we easily overestimate the sophistication of the broader population.

So that’s the first thing I really want to point out, that there’s really a very broad spectrum and we see a lot of people still struggling with even simple things like, let’s say, opening a new window on a browser tab (“what happened to the previous one!”) there’s a lot of people who have that problem.  

That said, there are definitely a lot of people who are better able to do the kind of advanced behaviour, like dealing with multiple browser tabs or dealing with the taskbar in Windows. Those I call ‘advanced behaviours,’ and some people listening to this might think those are basic behaviours, but actually they should be considered to be advanced behaviours and signifiers of somebody who is in higher-end of users.

But anyway, we do see more of that, we actually right now have been doing a lot of research on what is sometimes called the Millennial generation or, as I like to call them young people (younger than me anyway!), so people in their 20’s. And you see for those people we do see many of them doing well just like opening lots of tabs in the browsers and things like that. So yes, there is among some users increased sophistication, but there are also many who are not, let’s remember that.  

But for the sophisticated users, all the usability issues  still apply, because what you also see is increased impatience, particularly among these millennials, the young users, they don’t suffer bad design gladly. So if a site is too difficult to use, they will be out of there — they don’t want to sit and wait and struggle.

So it continues to be as important because it is really a competitive differentiator, and people can go anywhere they want to. In the old days we liked to say that on the web your competitor is only one click away, but with the Millennial users who are using what we call ‘Page Parking,’ (where they kind of park multiple different sites and different tabs), actually with Page Parking the competitor is already loaded in the browser and is only a tab away, and is  already there, waiting for the user to turn away from you and go elsewhere.

So definitely it’s completely, utterly still highly, highly relevant. Now whether the question is methodology-wise whether we should do qualitative studies or quantitative studies, I don’t think it’s an either/or, because I think the two will supplement each other quite well.

Qualitative has the benefit of first of all being fast and cheap, because you only need a handful of users, and secondly it provides you with a lot of that kind of deeper insight as to why people are doing things. So the analytics will show that a certain feature is not used very much but like why is that the case, but there are many explanations. One could be that it’s a useless feature and people don’t want it. The other one is it is a useful feature, but they can’t find it, and of course you would have very different reactions in those two cases. Or there is also the possibility that people can’t understand it — they could find it, yes, but they can’t understand it, or it could be a confusing layout… I mean, there are so many different possible explanations, that you will find usually with a just a few qualitative user test stations.

A/B testing or MV testing says: “here are two alternative designs, let me see which one is better.” The real traffic, well that’s all very good, but how do you come up with the two different designs  to test? You’ve got to have some insight and some hypothesis for why it would be that one design might be better than another design, and how are you going to get those hypothesis for user behaviour? Well, from watching users, so I think that all these methods play quite well together and supplement each other. And I think that the vast majority of companies today are doing too little user research in whatever method, probably all of the methods they should do more! But for sure I would definitely recommend to continue to usability testing — if they are already doing it, then bravo, keep doing it! And if they’ve not done it I would say try just once, try just once with 5 users and see what you get, I guarantee you, you will be astounded and want to do more.

Lee: Yes, it’s fascinating isn’t it, we find that amongst our client base the most successful are the ones that are using a really broad range of quantitative and qualitative tools and as you say it boils down to, slightly simplistic but it’s completely true, quantitative is what and how many and qualitative is why.

Check out the rest of our series of interviews with Jakob Nielsen