Web Usability Case Study BBC vs CNN

Study Goals

Web Site Brand

The goal of this study is to test brand communication of news web sites. We compare BBC and CNN.
Each respondent is exposed to one OR the other, and is not informed beforehand which web site will appear.

Navigation

Furthermore we want to investigate navigation. From the front page we ask respondents to find the Sports section. From the Sport section they are asked to locate the Golf section.

Banner Advertisement Effectiveness

After the browsing session a number of survey slides are shown that tests the respondents memory on which site did they visit, and did they notice any of the advertisements.

Study Specifics

  • The study was conducted in a mall, respondents were recruited on the floor.
  • The segment is on males and females broad age range.
  • Data was collected using Attention Tool Standard Licence with Usability Module and Survey Module.
  • All browsing sessions were done in Internet Explorer 9.
  • Eye tracking hardware was Mirametrix S2 and Tobii X1. Beyond using the study to create this case, it was also an opportunity to test these eye tracking hardwares in a realistic setting.

Flow of Stimuli

Test Plan

A Test Plan was used to implement the study design. CNN and BBC websites can appear as stimuli in the same study but one respondent only sees one of them. I.e. The Test Plan implements a cell design.

 

Instructions / Tasks

The instructions were shown to the respondents as the first slide in the slide show.

The instruction slide was created using the Survey Module.

 

Front Page Heatmaps

On the front page the respondents had the task to locate the Sport section. As can be seen the links that connect to the Sport section receive Attention.

 

Sports Page Heatmaps

On the Sport page the respondents had the task to locate the Golf section. Again visual attention is concentrated around the areas that contain relevant information to solve the task.

 

Golf Page Heatmaps

On the Golf section there was no task. The respondent could freely look around for a half to a whole minute, after which the survey slides appear.

As can be seen the visual attention is much more spread out when there is no task. The respondents look based on whatever personal preference they may have.

In the survey that follows the browsing session, there are questions on whether respondents remember the advertisement on the CNN Golf page. Respondents are not beforehand instructed to notice advertisements, only to freely look around.

 

Summary of the recall/recognition task.

The BBC logo is clearly not recognised, while CNN gets a relatively good score.

 

How much time is spent answering the recall questions seems to be correlated with correct/incorrect answers.

Time on task to find the Sports section on the front page, and find the Golf Section on the Sports page.
There is no clear winner.

Question

Rolex Banners on Golf Page

Results

The respondents who saw CNN were exposed to Rolex advertisements on the Golf page. 6 Respondents correctly recall having seen Rolex (1 BBC respondent incorrectly recalled Rolex, there were no Rolex ads on BBC).

 

 

 

 

Heatmaps on Rolex Banners Ads

 

From the heatmaps we can clearly see that the segment of those who accurately remembered having seen the Rolex advertisement actually also have had their eyes on the banners. However the heatmaps do not tells us how many looked at the advertisements.

By using Areas of Interest we can get the actual number of respondents having looked at the advertisements. Furthermore we can see how much time is spent looking.

  • In general the ”Recallers” segment has a higher Ratio, 83% (5/6) looked at either of the Rolex banners on the CNN page.
  • For Non Recallers the Ratio is generally lower.
  • Time spent is generally higher for ”Recallers”, which could indicate that those who recall the advertisement actually spent some time investigating what the ad says, who is the sender. This is also confirmed by the heatmaps on previous page.
  •  It seems that when a respondent recalls the Rolex ad, it is because he notices not just one of the banners but two or three of the Rolex banners – in the same session.