Beantin

James Royal-Lawson

testing

Eye tracking Google Instant

This week I did held a number of presentations and demonstrations of eye tracking user testing as part of Per Axbom’s course at Jönköping University. One of the tasks we tested was intended to show how people search using Google with Google Instant enabled.

We tested 6 people, all but one of them were students in their early 20s. The test environment wasn’t exactly how I’d normally set up eye tracking tests (being in a lecture room with 20-30 people watching on a projector screen while you try to solve the task will of course have an impact on the results!).

It’s always fascinating to watch people search – which is why I wanted to show the students a test involving a search engine and information foraging. This though was the first time I’d done any testing with Google Instant enabled.

No-one looked at the ads

Here are some quick findings.

  • No-one paid any attention to any adverts. Not once. Not a single fixation on an advert during about 7 minutes worth of Googling by 6 people. What does that mean for paid search?
  • When someone did stop typing for a moment to look at the Google Instant SERPs, they looked at the first result.
  • Those that looked at the keyboard whilst typing didn’t see the Google Instant results at all.
  • Results in positions 1-4 were, by and large, the only ones that received any attention.

It would be nice to do some more digging into the data and pull out a few more findings for you, but time is limited and I want to make sure this post makes the light of day this week rather than next year!

Google Instant Heat map showing concentration of fixations around the search box, suggestions, and first snippet

Pulling out heap maps and the like from an Eye tracking test of Google Instant is awkward as the positions of each item in the SERPs varies depending on how many suggestions appear directly below the search box. Sometimes position 1 is where position 2 would be on a normal Google search without Instant enabled.

First result wins

You can see from the heat map above that there is a concentration of fixations around the search box, the instant suggestions, and then the first result. The first result can at times be an advert – but during this test no advert appeared in that position. Note the lack of fixations in the right hand column. Adverts regularly appeared there throughout the tests.

It’s worth noting that Google chose not to do any eye tracking testing of Google Instant before launch, claiming that they didn’t have time. Others have since found the time and published their findings.

The SERPs are more and more complicated

This was by no means a controlled test, and the sample size was just 6 people – but all 6 searched in a way that was clearly their normal and ‘natural’ way of searching. All of them had to solve the same task, and the task had a specific answer that they were unlikely to know before searching. Most of them hadn’t used Google Instant before, so perhaps their behaviour will change as they adjust their search techniques over time.

Nevertheless, Google Instant creates a whole load of issues, and has a varying impact on behaviour. Google search results were already complicated, but the addition of new features such as Instant and Preview during 2010 have pushed this complexity up to a whole new level.

12 Articles worth reading… (Spotted: Week 42-43, 2010)

Information Architecture 101: Techniques and Best Practices

A great “briefing paper” from Six Revisions, intending to raise awareness of the discipline with designers.

Mental Models

The latest Alertbox post covers what users think they know and how that affects their behaviour. Users, designers and developers all have differing mental models. You need to work with conformity not against it, and gently adjust the mental models of your visitors and users.

10 essential usability guidelines for websites

If every website followed the ten points in this list, i’d suddenly have hardly anything to complain about any more. Well, perhaps not *hardly* anything – more like “less”.

How Yammer Won Over 80% of the Fortune 500

Yammer boasts that 80% of the Fortune 500 use Yammer. I wonder how many of those 400 companies have adopted Yammer as their official, or main, collaboration platform?

How I learnt to stop worrying and love enterprise microblogging

Nice case describing how Yammer took flight at AXA Australia in just the few months since August.

Information flow part 3: Why persistent links are important

I’m really enjoying Kristian’s series of posts about aspects of his work over the last couple with the intranet at Region Västra Götland. This particular post goes into the details of how they’ve tried to deal with managing URLs and links across multiple systems.

How we improved our intranet search experience

Luke describes how they implemented and tweaked their Intranet search (using a Google Search Appliance). Some good lessons-learned and insights that anyone dealing with intranet search can make use of.

Report: iPad Is an Enterprise IT Triple-Threat

The headline talks about iPad (and thereby iSO) but the report concludes that Android and HTML5 should also be prioritised. I’m going one step further than Forrester and saying prioritise HTML5 and Android. Plan for mobile/wireless working and plan soon.

How Google tested Google Instant

An insight into how Google tests it’s products (before launch in this case). What I find interesting is that normally Google uses eye tracking whilst testing. It gives them real data to work with. In this case, Google Instant, they chose not to. Why? I suspect they did try, but realised that far too many people were spending too much time looking down at the keyboard whilst typing – and not looking at their instantly-changing search results.

How Facebook Decides What To Put In Your News Feed – These 10 Secrets Reveal All

Some useful testing into how Facebook decides what to display into your news feed. Would be interested though to know their source for saying “Top News is how a vast majority of Facebook users get their information”

20 Real Tips for Hiring a Social Media Consultant

A good list to help you separate the wheat from the chaff in the world of social media consulting. You don’t need to hold yourself religiously to all 20 points, but there is some really good, honest, to-the-point advice in Pam Moore’s post.

How Google dominates the Web

If you had any doubts about just how dominant Google are in our World of Web Stuff, then this Royal Pingdom posts shows you the stats in easy to consume pie-charts.

Tested: Java midlet QR code readers

QR Code readers are as easy as pie on smartphones such as the iPhone and Android-based phones. The ZXing barcode app for those platforms does a great job of decoding almost everything you can throw at it. The story isn’t quite as happy for owners of other types of mobiles who have to use J2ME/JavaME applications.

I’ve tested 7 different Java QR Code readers using a number of QR Codes (both on screen and printed) on my SonyEricsson C905. The C905 is a CyberShot telephone with an auto-focus 8MP Carl Zeiss lens. The camera is one of the best I’ve seen on a mobile. That said, scanning QR Codes with Java apps has, by and large, been an awful experience.

i-Nigma logo

i-Nigma – Best in test

Of the 7 free apps I tested i-Nigma was the only one that I can genuinely call useful. When I say useful, I mean it actually decoded every code I threw at it! It makes use of the auto focus, decodes quite quickly & keeps a history of what you’ve decoded. It didn’t made use of the phone’s flash, but it did manage to decode my low-light test code after a few attempts.

It also understood a number of different types of encoded data.

  • Contact details – offering to save the vCard to contacts.
  • Telephone numbers – offering to dial the number.
  • Email addresses – offering to send an email.
  • SMS – offering to send the SMS to the specifed number.

Everything it didn’t understand it displayed as text.

The application also offers a auto-power saving mode and the ability to recode the data back into a QR-code. You can also reasonably easily share the decode info on Facebook or Twitter. It is all in all a nice application to use.

Neoreader

It Auto-focused, and didn’t use the camera’s flash. It decoded as many codes as the test-winning i-Nigma, but what stopped Neoreader from being a joint-winner is that the application isn’t as nice to use, and it considered most things it decoded as a URLs and offered to open them – even if it was a vCard for example. It did though correctly deal with SMSes and telephone numbers.

BarcodeReader (ZXing)

This reader is the most common barcode scanner on Android phones and works really well on that platform. It auto-focuses and uses the built-in flash. There is a noticeable delay though from when you press the button to scan to when it focuses, and then to when it flashes, and then again to when it makes the shutter noise – and then finally, after what seems like a lifetime, it says if it found a code or not. A lot of time it said “no code found”. The way the user interface is designed makes you wonder whether it’s you that’s doing a bad job of taking the photo or the application doing a bad job of dealing with it.

I didn’t mange to decode any of my on-screen test codes, only the printed ones worked. This is probably due to the flash; but as there is no option to disable it, I couldn’t test that theory.

BeeTagg

Auto-focus, bit sluggish at times. Decoded most larger codes, but failed with smaller ones. It showed all types of codes as decoded text, scrolling across the screen – apart from URLs which it gave you the option of opening.

Kaywa Reader

First problem was a really long drop down list with phone models that was really awkward & slow to navigate, the second and ultimate problem was that I received an error when trying to download the software. Total failure.

Scanlife

No auto-focus. Never managed to get an in-focus scan, subsequently I never managed to get it do decode anything. Total failure.

UpCodeJava

Auto-focused but was incredibly slow at processing the picture. Most significantly it didn’t manage to decode any QR Codes at all. Total failure.

The Test Codes

I used 7 codes in my testing (although I have over a period of time tested all of the applications with more QR codes). 6 of them were generated using ZXing’s QR Code generator. 1 of them was my business card pined to the fridge in our kitchen. This code in this environment was used as my low-light test. I also printed out the MeCard contact details code.

Here are the codes, so you can test yourself:

Printed code in Low-light

Picture of a QR code in poor light

MeCard

MeCard QR Code

vEvent

MeCard

URL

MeCard QR Code

Geo-location

MeCard QR Code

SMS

MeCard QR Code

Phone number

MeCard QR Code

2 of 2
12
Reload this page with responsive web design DISABLED