Vision: An Eye on User Data
What have been your most striking findings? What usability data have shocked you the most? Why?
I think our most striking finding is how bad web sites are in general. We have yet to find a site where, if you choose questions at random based on information the developers have placed on the site, users can find the answers more than 50% of the time. (The best we've found is 42% of the time.)
We think this means that, as designers, we still don't know what it takes to design a good web site. We now know that information design is an important part of web site design. How the information is distributed through out the site plays a huge role. For example, we've found that, in general, longer pages, packed with more information help users complete their mission better than shorter pages designed to prevent scrolling.
We also have found that graphic design is not as important as people think. The time it takes to generate nice graphics that download fast doesn't pay off in terms of the increased usability of the site. Some of the most popular sites are very text intensive.
We have also found that users, when trying to accomplish their mission, tend to follow "a scent" and when they loose the scent, they fall into predictable behaviors, such as randomly clicking on links or hitting the back button. Sites where the target information has a strong scent are the ones that perform the best in our studies.
Recently, we've been studying how marketing and design interact. On the web, you have a strange mix that never really happens in software or documentation design, the other two areas we specialize in. When people use software, it's usually already been purchased, whereas when people use a web site, it's often in a pre-sales situation. Designers now have to take marketing into account.
But, even armed with all this new knowledge about design, marketing, and the web, we don't know how to build a truly great site. Our quest is to find out how to build great sites.
They love it. We can't get enough data to them fast enough. The more we share, they more they want to know.
I give some details later in the interview about our popular usability course Web Sites that Work: Designing With Your Eyes Open. It has been selling out every time we offer it. Companies are calling us every day to bring it on site. We're hiring as quickly as possible to allow us to get the information out as we uncover it.
Two words: Team awareness. Once the team is aware of the problems, they are usually in a much better position to determine the right course of action. They know what their resources are and how their technology works. We've yet to encounter a team that makes a "bad" decision, once given all the information needed.
So, our job is to deliver information to the team. We do that through our research and through the techniques we've developed and refined that help teams collect their own info.
Design is gambling. When you take an existing design and make a change, there are only three possible outcomes: either you improve it, you make it worse, or you don't really have any affect. If you make changes at random, the odds of improving it are not very good (unless it was really bad to begin with).
The name of the game is controlling the odds. Designers can control the odds when they make changes based on information about the users and how they will use the system.
For example, what if you could predict the outcome of a given change?
As I said, the solutions are easy. It's knowing the problems that is difficult. That's where user testing comes in. That's the best tool we have for determining if our design is better or worse than before we made the changes. And with techniques such as paper mockups, we can determine the affects of our design in hours, instead of days.
What is User Interface Engineering? What is your role?
User Interface Engineering is the largest usability firm in North America. I'm the founding principal. (Which means my office is "The Principal's Office," which is very comfortable for me -- I've spent a lot of time there as a child.)
I started as a software developer, many years ago. I wrote one of the first voice mail systems (it wasn't my fault, really!), some of the first email clients, a couple of word processors, some spreadsheets and lots of system software.
Initially, I wrote what I was told. It was all really complicated and krufty. I could tell that it would be hard to use and kept asking the designers and my managers as what we would do if people couldn't figure it out. Their response: "the users will fill out bug reports, like they always do." This didn't seem satisfactory.
When I was at Digital Equipment Corporation, I connected up with the Human Factors group there. (The group contained such folks as John Whiteside, Dennis Wixon, Bill Zimmer, and Sandy Jones -- names which many old-timers will recognize.) They were focused on researching human factors issues, I was interested in building products that were usable. We taught each other a lot -- I taught them about getting products shipped, they taught me about user-centered design.
I guess I'm different from many of the key usability players in that I'm not a zealot trying to make the world more usable. Usability is an offshoot of what my interests are.
Instead, my interest is in design. What makes a design good? Certainly, if it meets the user's needs, it's good. But how does a designer integrate the user's needs into the design process?
If so many products that come out today are not well designed, what is it about our design processes that prevent good design?
Our initial work in this area says that the #1 thing that prevents good design is that the designers are missing key information. They are missing information about who the users are, what they will do with the product, how the product fits into the user's lifestyle and goals, how users are different from each other and how they are similar.
Most importantly, we don't have any mechanism that easily lets a developer tell if the wonderful design that they just imagined will actually make the product better or worse. We've all upgraded to new releases of products that were no better, just different. How do we prevent that from happening to our products?
Why should Web developers collect user data? Similarly, why do you think it is important to understand users?
The goal isn't to collect user data. The goal is to build great web sites. Right now, the only tool we have for determining if a web site is usable is to watch users use it. If they can use it, great. If not, fix the problem -- try again.
We've found that developers find it easy to generate multiple solutions to problems. However, knowing what problems need solving -- that's the hard part. The traditional development processes don't have any good way to identify the problems. So, developers end up solving things that don't need solving and leaving important problems unsolved.
Web developers need user data to find out if the hard work they are about to invest will make it any easier for users to accomplish their mission. If not, why bother?
Yogi Berra once said "You can observe a lot just by watching." That's our motto. An experienced observer can capture everything they need on their notepad.
In our course, Web Sites that Work: Designing with Your Eyes Open, we have several exercises where the class participants conduct simple usability tests with their classmates. The results are astounding -- within a few minutes, these designers have a list of changes that they are ready to make to the site, based on just those simple tests.
Imagine what would happen if we had real users in front of our designs every day. For our clients, we show them how to do that. And we don't have any special techniques or procedures. Just sit quietly and watch.
That said, we often videotape -- then we never watch the videotapes. We treat video like backup tapes -- 99% of the time you never look at them; 1% of the time you'll be very glad you made them. Sometimes they help when the development team can't physically make it to a test. There are few things more boring in the world than watching videotapes of usability tests -- unless it's your site that's being tested.
One technique that we use a lot is paper mockups. This technique allows us to have a full working site in just hours. After each test, we identify the key problems and change the design (in minutes) to try to rectify the issues we saw. Then we try it again with new users. It's amazing how quickly we can evolve to a design that really addresses what the users need.
The most important data collection tool is the human brain. Everything else is just to augment it. In most projects, we don't use any fancy tools -- just paper and pencil. We write our observations on post-its or index cards, one per card. Then we can put all our observations on the wall and sort and compare.
In some of our web studies, we've put together some fancy logging software for very specific purposes. For example, in one study, we were interested in why users chose each link to click on and what made it the best choice, in their minds. We built proprietary logging tools that allowed us to capture specific information about each click.
We redesign these tools for every study, because we're always looking for new things. Tools like this are research tools, not design tools. Designers don't need anything this fancy. It's just a distraction and doesn't yield any interesting value.
You can do usability testing on a shoestring (less than $100 per test) -- we've been doing it for years. You can also spend $200,000 on a project with us. How much you spend depends on what you are trying to do.
For most designers, they just need to find users and watch them. Preferably, you want to watch them do things they are interested in doing. We've learned that users act differently when the topic is of deep interest to them than when it is only of passing interest or they are doing it as a favor to you.
The expensive projects come when there is a huge investment into a site that no-one really understands what the user's missions are.
So, we're helping the team understand the core issues about their design. It turns out, after extensive testing with a variety of techniques, including using an eye-tracker to see exactly where the user gazes on each page, we've found that the problems come from inherent flaws in the designs of the pages.
Since the site makes its money through advertisements, this is a big concern for them. So, we're studying what it takes to make their site more valuable for the people who sign up, so they will return and use the site on a regular basis.
Again, this varies depending on the project. Sometimes we recruit people right off the street. We have a database of several hundred potential users that we collect in a variety of ways. When we get a project, we start by matching the target profile with what's in our database.
If that doesn't work, we have all sorts of tricks for recruiting users. For example, a favorite is to use real estate brokers. Real estate brokers know everything about everyone. They are constantly walking through people's most personal belongings, so they know who has kids, who has disposable income, who has what occupation, who uses computers, and many other fascinating little details. If we're looking for people with a specific profile, the brokers are a good place to start.
As far as the number, that also depends on the project. We've had projects where four users provided more information than we needed. We did a project this summer where we tested 91 users in 3 days.
For most of our client work, we test until we reach "the point of least astonishment" -- that point where the team is no longer surprised by what they are seeing. Usually that happens between 6 and 10 users. We typically schedule 8 for that reason. Sometimes we feel we had one too many tests, sometimes we wished we had just one more when we were done.
What are your favorite books? Favorite Web sites?
Clement Mok's Designing Business is a great book about the importance of design and what it really is all about. I'm also a fan of Edward Tufte's books -- great demonstrations on what information design is all about.
But then again, maybe I shouldn't. After all, would you notice a really great voice mail program? Probably not, as it should be so good that you don't even realize it's there. My guess is that really good web sites are the ones that you don't realize helped you. But I don't know for sure, because I've never met one.
It's our first attempt at trying to publish some of the wealth of information we've collected by studying users working with web sites. While we've come a long way in our understanding of how the web seems to work since we first wrote the book, it still holds up pretty well. We still haven't found many instances where content should play second fiddle to navigation or graphics, despite what the designers want to think. And we've found that many of our original findings, such as less whitespace increases user success at finding their target information, are still true, several hundred users later.
It's now being published by Morgan Kaufman Publishers and it's been selling really well. We're very happy with it.
This is one of our greatest secrets. Eye for Design is now in its 5th volume. Every 2 months we publish our latest results from our consulting and research work. Recent issues have talked about seducible moments, designing for branding, different ways to test documentation, how to recruit difficult-to-find users, and other pearls of wisdom we've acquired through the years.
But don't tell anyone. If everyone were to get a copy, then they'd all know how to do better design -- we wouldn't want it to get out.
The list price is $79/year. On our web site, there's a page where you can get the next issue for free without any obligation.
What, the previous two questions weren't shameless enough? Well, let's see. . . We've got our conference, User Interface 2000, that will be November 8-10 in Cambridge, MA. We've got our courses on paper prototyping, documentation, and web design. We've got UIEtips, our free e-zine that you can also sign up for on our web site. But, I won't mention those things, 'cause I don't want to be pushy and commercial.
As to final wisdom, one of my favorite quotes is from Woody Allen:
That pretty much says it for me.
If you want to know when
new interviews go online,
© 1999 by John S. Rhodes. All rights
Do not reproduce or redistribute any material from this document,
in whole or in part, without explicit written permission from John S. Rhodes.