you want to know when new interviews go online,
subscribe to the WebWord.com
Usability of Usability
An interview with Jared
Spool, Founding Principal of User Interface
Conducted via email by John S. Rhodes
last WebWord interview, from about two years ago, we talked a lot about
data. What usability data have shocked you the most over the last two years?
I think the thing
that has shocked me the most is how wrong we've been about our base
assumptions. For example, it is often stated as if it was almost a law
of nature that the faster pages download, the more usable the site was.
But when we actually compared the usability of sites to their download
times, we didn't see any correlations. None, zero, zip. If this
"fact" was true, we should've seen something.
To go farther, we
found that when we asked users to rate the speed of a site, that
didn't correlate to the actual download time either. Instead, the
perceived speed of the site correlated strongly to whether they completed
their tasks! This tells us that, when users are complaining about
download time, they probably aren't actually talking about the download
time, but about their ability to complete tasks.
We've had similar
findings with the number of users needed for testing. Again, common
belief in the usability profession is that with 8 users, you'll find 85% of
the problems. In a recent column, Jakob Nielsen went so far as to say
that 5 users was all you really needed and that you'll spot all the serious
problems almost immediately.
Yet in our recent
research, we found that we didn't even get above the 50% mark after 18 users
and were seeing serious problems all throughout the testing. If the
"85% theory" was true, we should've have learned everything after
the first 8 users, with the remaining 10 only repeating problems we'd seen
before. But, it didn't happen as it was supposed to.
The problem is
that when our base assumptions, like download time or number of users needed
for testing, are put into questions, we loose everything we have to hold
onto. It's like finding out that gravity doesn't really exist --
what's actually holding us on the planet?
So, to answer your
question about what has shocked me the most, I'd say it is the realization
of how much we really don't know.
Obviously you can't change the
past but looking back over the last 4-5 years, what would you change about
your usability work? What would you change about the usability profession?
(Actually, I can
change the past. It's a little trick I learned as a kid. Well, I think
I learned it as a kid -- I'm not sure. The
past is sorta fuzzy that way. :)
I'm not one for
regrets, because I believe that people will usually make reasonable
decisions with the available information. If, in retrospect, it seems
like a decision was made poorly, it is often because the parties involved
didn't have all the information they needed.
philosophy, I'd say that the biggest thing I'd try to do differently going
forward is look more closely at our assumptions and where we get these
"maxims of good usability." In the early days (I'm talking
1977-1990 now), we spent a lot more time on solid empirical research.
Then, with the success of the Mac, we got on this evangelism bent.
So, now the entire
field finds itself in the role of preachers of a religion of sorts versus
researchers of best practices. And we're now beginning to see
My sense is that
people don't want to be lectured to. They want to get their problems
solved. Repeating the same mantras over and over again doesn't help them
solve their problems. Real data on what works and what doesn't would
help them solve their problems.
I take a lot of
responsibility for where we are today. We spent years promoting the
"Well, it depends" attitude without ever answering the questions
we were asked. While it seems helpful at first, it quickly turns out
to be a negative -- people don't want to just be told "it
depends"; they want to know what to do differently.
example of how wrong we've been falls in our response to business.
Recently, a reporter interviewed me for a story she was doing on why we
haven't really seen any real improvements in the usability of web sites over
the last few years. She had interviewed the "usual suspects" of
gurus and consultants in the industry by the time she'd gotten around to
The first thing I
said to her was: "No matter what anyone else has told you, it's not the
fault of the company." Her response was "That's funny!
Everyone else said it was the fault of the company" She
went on to say that this cadre of intellect had told her that if only these
silly companies would wake up and understand that usability was important,
the world would be a better place. That's our [the usability
profession's] position: all these companies are idiots and need to change.
It's ironic: For
years, usability folks have cringed every time a designer says "the
users need to make major changes to their behaviors if they want to use my
product." Yet, what do usability professionals do? They
make businesses undergo major changes to their behaviors if they want to
produce usable products!
Usability is not
usable! It doesn't work. It doesn't produce the results we
promise it will. And we get angry when people stop paying attention to
As a profession,
we need to spend a lot more resources on basic research. We need to
stop thinking that there are pat, one-size-fits-all solutions to every
problem. And we need to align ourselves with the business goals more
directly. We need to make our own work usable.
What things were done right
over the last several years? What things were done wrong? Which web sites
capitalized on usability?
As far as I can
see, the things that were done right in making web sites more usable didn't
happen in the usability community. It's interesting to note that the
most usable sites in our studies, such as Dell, eBay, Amazon, and Edmunds
have no real usability efforts to speak of. eBay has a relatively new team
that is only a handful of folks, Dell also has a small team compared to the
size and impact of their site. The others: practically no usability
Now contrast that
with the organizations which have the largest usability groups, such as
Microsoft and IBM. From a usability perspective, their sites don't do nearly
as well. There are substantial usability problems that permeate both
companies sites seemingly without resolution, even though the teams work
So, what does this
mean when the most usable sites don't have large, high-powered usability
groups and those companies with big groups don't seem to have an impact on
their sites? To me, it means two things:
You don't need
to have a usability group to make a usable site. In fact, there is
no evidence to suggest you even need a single usability professional on
site to make a difference.
What we know
about managing a usability group apparently doesn't work. If
anyone could make it work, it would be folks like Microsoft and IBM. I
know those folks -- they are really smart people. I believe that
they could do a bang-up job, if we knew what we were doing. But we
don't, so they don't.
So, what was done
wrong is understanding what it really takes to produce usable web sites.
We still don't really know. In our studies, the best sites still fail
users 57% of the time.
(Is 57% an
acceptable number? Well, if your car failed you 57% of the time, you
wouldn't get to work 3 out of every 5 days. Is that acceptable to you?
Complacency doesn't get us very far.)
So, which web site
capitalized on usability? None really. We've never seen a site that
succeeds more than 50% of the time, let alone 60% or 70% of the time.
We don't know what one looks like. We certainly don't know what it
takes to build one. Oh, we can pretend we do. But we really
don't, because we never have seen one.
And what's worse
is that the sites that have come closest didn't use what we've been
preaching. What does that say about what we know?
What are your
hot research projects right now? What do people want? What do they still
We've got a couple
of things we're working on. It's not our policy to discuss
research-in-progress, but I can give you some ideas.
We're very excited
about our work in e-commerce. From a usability perspective, e-commerce
is one of the simplest types of web sites. This is primarily because
both the goals of the user and the goals of the business are easily
identified and are met at the same time: when a product is purchased,
everyone is happy.
So, we see
e-commerce as a "laboratory rat" -- a place where we can do some
in-depth work, looking for cause and effect. Everything we learn about
e-commerce turns out to be valuable for many non-e-commerce sites.
For example, our
recent work on categories vs. search showed us that customers are more
likely to keep shopping when they use categories and purchase more when they
use categories. We also learned that the most likely thing that causes
people to use search instead of categories is that the categories are poorly
designed. On sites where the categories were well designed (and we've
compiled a list of things that make "well designed categories"),
users never used search. (This happened on 21% of the sites we
tested.) This means that e-commerce designers can change their designs
and get more revenue from their site.
particular piece of research could also be relevant to non-e-commerce sites.
While the point where users meet their goals is harder to identify in
non-e-commerce sites (What is the goal of someone coming to CancerNet?
How do you accurately measure if they've achieved it?), we can see how
better categories might lead people to find content they wouldn't otherwise
find on the site. Since lots of our clients are telling us that they
want people to "discover the value of their site", this seems like
a hot direction to follow.
So, we're looking
at all the data we've collected from our e-commerce research and trying to
mine it for all the juicy nuggets. (If you want to know more about our
e-commerce research, look at our
latest whitepapers. We'll also be presenting our latest results during
the UIE Research Forum on October 4, 2001 at the User
Interface 6 East conference in Cambridge.)
We've also been
looking into the world of advertising-paid sites. Here, there's a constant
tension between bringing in revenue for the site, keeping the users happy,
and keeping the advertisers happy. It's a 3-way constraint system that
has to be managed.
For example, many
advertiser-paid sites get paid when the ad is retrieved from the server and
displayed on the screen. Interestingly, they are not paid based on
users actually seeing the ad -- just when it is retrieved from the server.
If the users never scroll down to see it, the advertiser still pays the site
even though they didn't get any value from the ad. They are will to pay
more, however, if the site can guarantee that the ad will be seen.
Now, a site could
generate a lot of ad revenue by having 10 pop-up ads appear after every
click, but that would disrupt and annoy the customer into never returning to
the site. Or the site could make a lot of money by displaying a dozen
or so ads at the bottom of each page, underneath the content in a place
where users would be unlikely to scroll. But that would quickly
piss-off the advertisers and cause them to put their money elsewhere.
So, the question
then becomes, is there a balance that can be reached? How do designers
measure whether a specific design is within the tolerable limits of both the
users and the advertisers, while still generating the required revenue?
And what is the
role of the content in this situation? We know that if you have
content that is highly desirable and unavailable from any other source,
users are more likely to put up with annoyances than for content that is
commonly available. How do you measure where your content lies on this
spectrum, so you know what the boundaries of tolerability are?
These are the
primary questions we're interested in around the advertiser-paid
And we've got lots
more stuff happening in the areas of branding and web-based applications.
We'll have something to talk about soon.
Privately you told me that you
can now directly tie design elements to bottom line effects in certain
contexts. Can you explain this? What implications does this have on web
Yes, well, this is
really exciting stuff. While we were conducting our e-commerce
research, we noticed a cool phenomenon which we could take serious advantage
of. To understand this phenomenon, I need to give you a little
background on the experiment the was the basis of the research -- something
we call the 7-11 Milk Experiment.
Imagine we had a
way to instantly identify when someone has run out of milk. We pick
them up in our car and drive them to the nearest 7-11. And just to
make sure everything goes smoothly, we give them the money to buy milk. How
likely is it that that 7-11 will sell milk in that context? Probably
close to 100%, right?
Well, that's what
we did online. We identified people who needed products, we brought
them to sites that had those products, then we gave them the money to buy
the products. The result: people only purchased 30% of the time.
Much worse than the 100% we would expect to see.
important here is that we've subsetted the shopping experience. We're
not trying to measure every possible combination of shopping, just a single
subset: people who know what they want and have the cash to buy it. We
should see very high purchase rates in this scenario. In fact, this subset
represents the most likely scenario of purchase.
Now, here's where
the real breakthrough comes in: we found that, during the study, people made
purchases that weren't on their shopping list. That fact alone isn't that
interesting, what's interesting is the reason: it was because of the design
of the site.
It turns out that
only 8% of the impulse purchases we saw were because of price or special
promotions. The rest were because of the design of the site. And
it wasn't small potatoes either. The impulse purchases averaged 42% of
the revenue from those sites. (To put this in perspective, Staples.com
generated $451 million last year. 42% of that revenue is $189 million. I
wouldn't mind having that kind of revenue.)
How did the design
of the site affect impulse purchases? Well, as I mentioned before, the
categories on the site played a huge role. On the home page of a
retail site, you basically have 3 choices when shopping: 1) use the search
engine, 2) choose a featured product, or 3) use the category links provided.
In our study, if you chose option 3, you were three times more likely to
make an impulse purchase than if you didn't.
(Oh, by the way,
not a single impulse purchase came from choosing option #2. This
finding puts into question the practice of dedicating lots of home page real
estate to featured products.)
We found that
subtle differences in the design of the home page would be a huge predictor
in the impulse sales on the site. Designs that worked hard to guide
users into the categories were far more successful in generating this
revenue than sites that essentially scared people into using the search
engine. (Our whitepaper on impulse purchases goes into far more detail
Well, it turns out
that we've identified about 40 of these "predictors" of revenue
for retail sites. And this gives us a huge new capability: we can now
analyze sites for the presence or absence of these predictors and use that
to help measure and diagnose problems on the site.
To that end, we've
just launched a new service, which we call our Compelled-Shopping Analysis
(CSA). This service, which is currently available for retailers
(though we're working on expanding it to other types of e-commerce and
non-e-commerce sites), allows us to benchmark the design of the site based
on the revenue it produces and how the users react to it.
CSA is different
from other types of measures, like SUMMI, QUIS, or WAMMI because it doesn't
base it's metrics on the opinions of users -- it actually measures the
success people have at shopping. We don't care if someone loves the site,
but doesn't buy. We're only interested in the shopping
So, with CSA we're
now measuring how sites are actually doing. And we're building up a
library of data that allows us to identify best practices amongst retailers
in a variety of contexts. This is extremely exciting because we really
get the full advantage of our research capabilities.
(If you are
interested in these best practices, you might check out our course, Designing
for Dollars: Discover How People Buy Online. We're offering this course
September 24-25 in San Francisco, MA.)
Challenging the Usability
I have made it
clear that I think usability will "go away" (see Trouble
in Paradise: Problems Facing the Usability Community). I don't literally
think that the tools, methods, and techniques will go away, but I do think
the profession is facing some very serious challenges. How do you feel about
the usability community? What would you change? What is your open message to
the usability profession?
As I've indicated
above, I agree that there are some real challenges. I think a huge problem
with the Usability Community is that we're trying to be a
"community". Usability is an attribute -- some things are
more usable, some things are less.
There are other
attributes, but they don't have communities. Fun doesn't have a
community. Integrity doesn't have a community. Why do we insist
on having a community?
The companies that
are doing the best at usability don't seem to have a community.
Instead, they have a culture. A culture where people perform the
activities needed to create usable sites just because the culture dictates
that's how things are done.
We need to
understand how to build more cultures like these. We need to
understand how they've come about, how they flourish, and how they become
interwoven into the patch quilt of the business. This is where the advances
will come from -- not by promoting a community.
In light of the previous question, why weren't you at the (Usability
Professionals' Association) UPA conference this year? You are well known,
you write well, and people seem to like your work. What is the scoop? People
did submit a couple of proposals. They were rejected. No reason
was given for the rejection, so I couldn't tell you why. It's possible
that I'm a crummy author or don't have anything new or novel to say to the
UPA community, despite the fact that I've spoken almost every year in the
past and I'm often one of the highest rated speakers at the
After I complained
to the conference chairs about my rejection, they did offer me a speaking
slot -- but unfortunately it was too late in the process and I was unable to
Despite what I've
written above, I'm a big supporter of UPA. My company was a Gold Sponsor
this year, and a Silver sponsor the previous year. My staff serve on
UPA committees. We recommend the UPA to our clients.
It's too bad.
I would've like to have been there. I guess there's always next year.
Wrap Up, Wrap Down
tetrahydrofuran? How did you figure it out? What tools did you use? How did
your usability knowledge and experience influence how you figured it
Long day, huh
Most people I know are trying
to lose weight and get in better shape. Does this have any impact on the
usability profession? Does it impact web design? When you think about it,
these questions aren't so strange. We are human. Should web designers seek
to understand human emotions and desires? What is your perspective?
Definitely, a long
What is your favorite usability
I've grown very fond of this quote lately, for whom I don't know who to
attribute it to:
comes from Experience. Experience comes from Bad Judgment.
Comments? Shameless plugs? Wisdom?
Man, don't you
think I've said enough? I've certainly plugged almost everything I
can. And I doubt I can impart any more wisdom.
I say we're done
- o - o - o -
Jared M. Spool
User Interface Engineering
242 Neck Road
Bradford, MA 01835 USA
(978) 374-8300 fax: (978) 374-9175
else can I say?
-- John S.