The Difference Between Qualitative and Quantitative Testing

At Lycos, back when we were #2 after Yahoo!, we did a lot of user testing and it was a humbling experience. What seemed obvious to the UI/UX group and the web designers was clearly not intuitive to the users. There were days when I ranted to the team that there has to be a better way. We would spend a week performing user testing for a new service and then literally have to start all over. The process was simple. Generate a set of paid volunteers to come into a classroom and ask them to perform a series of tasks, all the while talking their thoughts out loud so we could record what they were thinking. This is an example of qualitative research.

I was also responsible for driving traffic which started off at 12 million page views a day. In 1999, there was no Google Analytics, Web Trends, or any sort of analytics software – just log files. Lots and lots of log files. I would take a days worth of traffic and export it into Excel. I would have numerous tabs to assist in selecting and deleting data in order for me to establish trends. The first data sort was pretty much a full day option of not touching the machine until the sort completed. I was searching for the most trafficked pages. I would then take the top one hundred thousand and put them in a second tab. I would then sort based on traffic and category (we had an excellent taxonomy team). I would then take each traffic and category and put them on separate tabs. This was painfully time consuming and remember, I am only looking at one day’s worth of data. At this point I could establish trends and realized the top visited sites that day were typically music fan pages – Britney Spears, Backstreet Boys, ‘Nsync, DMX, J-Pop, K-Pop, etc. Then while performing this exercise, our CTO, Tech Director and I were speaking about this when the CTO said that it looked like more than 90% of our traffic were coming from maybe 5% of our users, all focused on these topics. This type of research is an example of quantitative research.

It’s important to understand the difference between qualitative and quantitative research. Qualitative and quantitative research serve incredibly different purposes. Each method for marketing research involves a different process, and reveals different information. However, it is wise to conduct both quantitative and qualitative market research whenever possible. In the two scenarios above, one focused on user experience of a web-ring technology we were testing to see if it made the site more “sticky” (kept them on the site longer), while the other was used to identify existing traffic patterns and how to increase traffic within the site by beta testing our new web-ring technology only on those high trafficked sites, since the traffic patterns of the top 5% of pages would be massive and if the new tech did increase traffic, it could significantly increase our pages overall by shifting traffic to other pages with the same or similar content.

Quantitative Market Research

Quantitative research generates data that can be transformed into useable statistics. It is used to measure trends in attitudes, behaviors, and other defined variables, from a large sample population. Quantitative research measures data to uncover patterns in research.

Quantitative data collection methods are much more structured; they can include various forms of surveys, telephone interviews, longitudinal studies, website interceptors, online polls, and systematic observations. They can also be used on the back end by analyzing incoming traffic where users go, when they leave, how long they’re on site, and more. In my opinion, this data is far more reliable as humans do not always remember the facts as well as user logs or data analytics – even your browser history. Think about it. How many web sites did you visit yesterday? Go to your browser history and count them. I guarantee you were way off and the actual number was much higher than you expected.

Quantitative market research is the collection of data to analyze trends in the data. As a result of the standardized questions when doing any form of surveys, quantitative market research can involve a larger number of respondents to participate in the research. One caveat to surveys is that there can be significant bias set up in the creation of the survey questions. There are numerous studies that document this confirmation bias effect in market researchers, especially when the study plays a huge stake in whether the project moves forward or is closed down.

Qualitative Market Research

Qualitative research is often used to explore improvements to existing products or services or to introduce new products and services. It helps researchers gain an understanding of what works, what doesn’t, why they would purchase or not, whether they would recommend it to a colleague or friend… It provides insights into any issues or pain points that the product or service still does not solve.

Qualitative research methodologies vary from focus groups, individual interviews, activity observations or immersions, and diary studies. The sample size is typically small, and respondents are highly targeted to match market personas that the company assumes will be primary and secondary consumers of their product or service.

Qualitative market research helps marketers understand why a consumer has acted and purchased in a certain way. It does not follow a predetermined set of questions but provides topic or discussion guides to ensure that the research remains consistent and the same questions are asked to each participant. During the research, the researcher is able to explore responses in more detail allowing for longer discussions that could reveal a vast amount of information the research never expected or accounted for. By understanding consumer viewpoints, emotional responses, and purchasing behavior, it can allow companies to alter and adapt their ideas to ensure consumer satisfaction and competitiveness within the market or alter a product or service to align with customer needs, allowing them to be competitive.

Selecting a Testing Methodology

Both quantitative and qualitative market research can be conducted first. The method to choose first is dependent on the following;

Qualitative market research should be conducted before quantitative market research if the project concept has not previously been researched. In this situation qualitative market research will enable the researcher to understand the consumer’s initial and unbiased reaction and opinions to the new concept with no external influencers such as past experience with similar products. It is important with a new concept to first understand areas of improvement, modification before moving forward towards validating the final concept through quantitative market research.

Quantitative market research should be conducted before qualitative market research if the project concept has been previously researched to some extent and some initial information from previous research has been absorbed. By conducting quantitative market research first, it allows an entrepreneur to understand the current feasibility of a project before understanding why the results read as they do. Quantitative market research highlights areas of further investigation before exploring the reasons through qualitative market research. Further to this quantitative market research allows the researcher to gauge a general understanding of the market before taking the time to adapt their research into a more specific and customized survey as part of qualitative market research.

Learning from the Data

In user research, quantitative data tells you what users did, and qualitative data helps you learn why they did it.

Let’s go back to my experience at Lycos, where we used qualitative user interface testing through human users completing tasks. If you were to measure their behavior on the website in order to keep them on the site longer, you might learn that 25% of people clicked on this button, then another button, and so on. This data is good, and you can run split tests (otherwise known as ‘A/B’ or ‘multivariate’ testing) to try out different versions of your prototype to see if you can change people’s behaviors. But we also asked people to think out loud in order to capture their thoughts, feelings and asked them why they did what they did.

So, using the same example — we used quantitative methods by sorting traffic logs to understand how many people were visiting what pages, for how long, in what category, and what they did next.

In your research, consider using both qualitative and quantitative methods together to be better equipped to solve the problem at hand. In this example, adding traffic meant that we had to run both, first to set a benchmark and second to measure changes in traffic. We selected the top performing topics and created a subset of these pages, calling them BetaPods (I don’t know why). We could not put advertising on users’ home pages as most companies weren’t comfortable with potentially harmful content that would reflect poorly on the brand. Out CTO and Tech director had found a snippet of code in Netscape and Microsoft Explorer that enabled us to exploit the ability to open a new browser window. Yep. That’s right. We invented the scourge of the internet in order to place advertising in a different window than the user’s home page.

Our first browser popup had one 468 x 60 pixel banner ad, and two smaller text ads to the right. We used a text link as well under the banner ad, since previous testing had shown that the link got 78% of the traffic over the banner ad itself. This original popup menu generated less than .003% click-through (aggregate – there were 17 individual links in the popup), or an additional 36,000 page views on 12 million page views a day.

Original Popup Menu:
Generated less than .003% click-through (aggregate).

Our first test changed the popup to include a Lycos search bar, but that sent traffic to Lycos, which although it was part of the network, we really wanted more traffic to our own pages. We also added automated category pages that listed all the members’ pages that matched the taxonomy of the current page. The new popup generated more than a .01% click through rate on an aggregate of seven links, which added an additional 120,000 page views per day.

New Popup version 1.0:
Generated more than a .01% click-through (aggregate). Doubled traffic to commerce partners and generated more than 10,000 referrals per day to the network.

Our final popup used both the quantitative data and the qualitative data. If a person was reading a Britney Spears page, it was likely that they would be interested in more Britney Spears pages, or may also be interested in other pop music bands such as ‘Nsync and Backstreet Boys. Using web-ring technology and changing the taxonomy and algorithm, the new popup now generated more than a .05% clickthrough rate, generating 600,000 additional page views per day.

New Popup version 2.0:
Generated more than a .05% click-through (aggregate).

The new popup menu design enabled the team to reach our target of 16 million page views in nine months rather than in one year as expected. So give both methodologies a try – you never know what will work. Human beings are unpredictable, fickle and very different from each other. Testing will humble you. Testing will frustrate and infuriate you. But you learn something new every time that will make what you make better.

Do you want to outsource this type of work so that you can focus on higher level activities? Subscribe today to learn more about building your business and receive a free PDF “Process Plan for Creating Your Own Innovation Program”. Feel free to email us to learn more about how we can help you grow your business.