|
WebWord Weblog Posting Posting Date: March 24, 2003 Usability Myths Need Reality Checks -- "You have undoubtedly heard that users give up because pages take too long to download. This is also a myth. Testing shows no correlation between page download time and users giving up. How does this myth continue to defy gravity?" (Comments: Usability Testing: Myths, Misconceptions and Misuses, Debunking the myths of UI design, A Collection of Usability Myths and Maxims, Linux Usability Myths, Top 5 Most Common Myths About Usability)
Reader Comments...
Awesome collection of links - thanks! Posted by: Lyle, Lyle - Croc O' Lyle on March 25, 2003 12:21 AM
Another myth to add to that list: 1) Jared Spool and UIE understand what usability means Yes, it's a myth. :) Posted by: Jared Spoof on March 25, 2003 01:33 AM
You only need to test with 5 users. I have 'discovered' the formula for proving how many users you need to test with, in a particular situation: NUMBER_OF_USERS_TO_TEST = (BUDGET / NUMBER_OF_ITERATIONS)
You can change the number of iterations to 3 and then have 3.3 users x = (10,000 / 3) / 1,000 If you are analysing real time usage automatically, your cost per user will go down and you can dramatically increase the number of users involved to 1,000. NUMBER_OF_TEST_ITERATIONS = 5 x = (10,000 / 5) / 2 As a simple rule of thumb, the more you charge for usability testing, the smaller the group of users. If you are using 5 or less users then you are 'Doing a Nielsen'. If you are using between 10 and 20 users then you have 'Hit the Spool'. Posted by: Mac - Stop the War on March 25, 2003 05:55 AM
Should have been: NUMBER_OF_USERS_TO_TEST = (BUDGET / NUMBER_OF_ITERATIONS) / COST_PER_USER Posted by: Mac - Stop the War on March 25, 2003 05:57 AM
Should have been: NUMBER_OF_USERS_TO_TEST = (BUDGET / NUMBER_OF_ITERATIONS) / COST_PER_USER Posted by: Mac - Stop the War on March 25, 2003 05:57 AM
Should have been: NUMBER_OF_USERS_TO_TEST = (BUDGET / NUMBER_OF_ITERATIONS) / COST_PER_USER Posted by: Mac - Stop the War on March 25, 2003 06:02 AM
Users may not bail out if a page is slow to download, but I suspect that slow download discourages further clicking. It's a bad experience, and users are in constant trade-offs... I thought there was a good POV (untested) on this at shorewalker.com, but I can't find it now... Basically, the idea is that a long download time makes a user more hesitant to click other links. It stands to reason, and I have heard it come up spontaneously in focus groups. Posted by: Frank on March 25, 2003 08:55 AM
Does he take into account that users won't give up because pages take too long to download and that users won't leave a site if they don't find what they want after three clicks, is because they're performing a test in a testing environment? I know if I were testing a site, whether in a testing facility or in my house on my own time, I'd probably wait longer for the page to download or click through a few more links to find what I'm supposed to find because it's a test and there's an expectation to complete the test.
The nice things about these myths are that, as long as they are believed, they will encourage web developers to create smaller, more efficient pages, better navigation schemes and more content-driven sites. If a site is the only source for certain data, I'm sure that users would wait for long downloads and deep navigation schemes. But it doesn't enhance the user experience Posted by: Kevin on March 25, 2003 09:06 AM
Darin, what you're suggesting is something that can be verified by looking at log files. If users bail out, one should see less click-through to deeper pages when a page is heavier, and more when it's lightened. Your query reminded me that Nielsen cites results from Claire Amundsen's (?) personal experience as a web master. It's in Nielsen's book "Designing Web Usability : The Practice of Simplicity." (Sorry, my copy is many floors away, and I can't provide the page reference.) Amundsen improved download speed by condensing graphics; there were no other changes to the pages. In examining pre-post log files, she saw substantially improved click-through. Nielsen himself has had a challenge out there (SFAIK not accepted or results published, but maybe I'm wrong) suggesting that companies experiment with the impact by degrading the download time using a script, and measuring the impact on deeper click-through. Posted by: Frank on March 25, 2003 09:53 AM
To differentiate between myth and reality, first we must impose minimal standards of research quality. UIE is unable to hold themselves to even the standards of a high school science fair, so they have absolutely no credibility from my perspective. Worse, few in the usability community seem to care. Guess everyone prefers myths to reality. Posted by: Ron Zeno on March 25, 2003 11:41 AM
Like all things Nielsen, it all depends. I think it depends on the content. If it's exclusive (can't be found anywhere else), then of course a user is gonna wait it out. However, if the content is a dime a dozen (info on a particular CD or movie), then of course a user's gonna switch. I think it also depends on the user's level of browsing experience 'cause an experienced user (such as I) may open a new window and try to find the same information on two different sites simultaneously. So yeah... it all depends... Posted by: Francis Wu on March 25, 2003 12:22 PM
Just as an aside, an additional cost associated with uncompressed images is in the bandwidth. One service I used to use (OptiView) would calculate the savings dollars associated with compressing any single graphic, based on your input of hits and bandwidth costs. I always found the service wonderfully simple to use, and dollars are something marketing people can understand. Posted by: Frank on March 25, 2003 12:37 PM
Darin's point is an important one. A testing environment doesn't necessarily mirror everyday typical use. If I'm in a usability test, what do I do after closing the browser? It's not my PC. I don't have a sales meeting to go to. I don't have any emails to read. Let me tell you a personal example. As I type this, I have at least 10 browser windows open for reading various things. OK, I may not be your average surfer. But I also have an email client which keeps receiving mail. I have an instant messenger where I'm chatting with a friend from USA. If some site loads too slowly, I'll close it and either read the rest of my emails, read the other open browser windows, or type more messages to my friend on Yahoo messenger. In a usability lab, what else is there for me to do? I'll have to grin and bear it. Another thing that these tests don't account for is the anticipation factor. For instance, John links to some story and says, "Great article". I just became more eager to read it. I'll wait a little longer for it to load if required. On the other hand, if I followed a link from some site's main page, I might be less inclined to wait. Put me in a usability lab and say, "read this", and I'll not be terribly interested in it. But again, what do I do next if I don't wait for the page to load? Check my email? I also want to know HOW UIE conducted these tests, what sites they used, what profiles they used for tested users, and what instructions they gave them. Otherwise, it's just a "because I say so" rule. Posted by: MadMan on March 25, 2003 12:44 PM
Damn, forgot to add this too. Another thing that bugs me is when these companies make extremely generalised statements like "users don't do xyz". Who are these users? Is my father the same user as I am? Is John Rhodes the same type of user that Ron Zeno is? Surely extrapolating alleged research to the millions of people worldwide who use the Net is a bit stupid? Ditto with sites. Don't tell me ALL sites are identical in the way people interact with them and react to them. If you're going to state a research finding, you should specify what kind of users you tested, and which sites you tested. And even then, it's not particularly reliable. Just because people didn't do [xyz] at Amazon.com doesn't mean they won't do it at BarnesAndNoble.com I repeat: it is not safe to extrapolate research data from certain sites and apply it to your own without questioning it. BTW, all this article says is "You've heard of this guideline? Well, it's a myth. Don't listen to that guru; listen to us." Why? How? Who knows? Hmmm... maybe Jared Spoof (comment #2) has a point. :) Posted by: MadMan on March 25, 2003 12:55 PM
I much prefer pre-project research where I can sit with a user in their own environment and observe their daily activity. The behavior of someone in this environment is so different from what I have encountered in arranged testing of a working program. Paid testers, especially, have an unusual amount of patience and will try to do their best to give you your money's worth while they are testing your application. Therefore, I find that traditional usability testing gives me excellent feedback as far as the impression of the product, the feel of it, and the high-level usage, but I never expect to uncover everything or even get much beneath the first couple of layers. (Perhaps this is why I am continually amazed at what I find, and things seem so interesting!) Whenever possible (and this has been regrettably sparse) I try to go observe usage in the user's environment when it is released. As for the "myth" of users giving up due to long download times? I see this a lot, but it is usually in situations where the user is going to a website that they are not incentivized to visit; they are just checking it out. If it takes a long time to load, it's just as easy to close the browser. If they are going to their favorite clothing store to order something or going to their banking site to look at their account, though, they'll wait. If I had to make a stab at numbers from my observations, the amount of time spent on an "unproven" site is approximately 10 seconds and the time spent waiting for a "proven" site is 30 seconds to a minute. None of it is scientific, but why not go with my figures? They're just as good. ;) Posted by: Lydia on March 25, 2003 01:35 PM
Lydia, you used a word like "incentivized". Congratulations, you just became eligible for management. :) None of it is scientific, but why not go with my figures? They're just as good. I hereby appoint you CEO of User Interface Engineering. Posted by: MadMan on March 25, 2003 01:46 PM
I thought you'd like that one, MadMan. ;) I think my first act as CEO will be to appoint a "Secretary of Interface Dogma" who will sit around and make up things to brainwash new recruits with. And we will post inspriational messages about the office that say things like "Remember: Three Clicks!" and "The Formula is Your Friend" and "Good Rules make for Good Usability" My first column in the employee newsletter will be about how to get programmers off your back when they want to make one of their "suggestions." Everyone knows they don't know usability and wouldn't have anything useful to say. Oh, and during "Free Lunch Friday" everyone will be expected to review the latest Alertbox. There will be a quiz on Monday. [Yes, I am joking. I post this disclaimer in case the sarcasm isn't traveling well over the ether.] Posted by: Lydia on March 25, 2003 03:08 PM
The WHAT to test is particularly important given that most projects have aggressive timelines - this perhaps can be broken down into 1. Critical tasks 2. Number of users and 3. Impact on business. This is all closely related to understanding NOT just the user needs but balancing this with the BUSINESS needs as well :) i.e. How does the business measure the ROI? Posted by: daniel szuc on March 25, 2003 08:42 PM
Also a fan of breaking 'Usability Testing' into as many chunks as the Customer can afford. Test with 5-8 and then re-test to note iterative improvements. This also allows the 'development team' to see if they can make changes on the go rather than at the very end. All very good reads - thanks John! Posted by: daniel szuc on March 25, 2003 09:21 PM
I think there's a grain of truth in many of the 'myths' or maxims: The fact is that people blindly follow maxims too much. If you tested with 5 users and saw no duplication of issues found in test sessions, then you should realize that you didn't find your "70%" of isses - you've just dipped your toe into a quagmire of crappy usability. Misapplication of Nielsen's 5 users maxim isn't Nielsen's fault -- it's the practitioner's. Is UIE saying to test at least 200 users? - I don't think so. Lord knows prioritizing findings from a small qualitative study takes long enough. I can't imagine spending weeks and weeks sorting through thousands of usability issues - and I also know most companies can't afford to pay for that level of scrutiny, let alone pay to implement the actual fixes. Also, for those of you bashing UIE, let me just say that I sometimes have questions about their research, and I've given some suggestions to Jared in the past. I think they can do a better job of explaining their methodology - sometimes it feels like qualitative research presented as quantitative research. Having said that... There is clearly a huge need for more basic research. I'm glad UIE and others are trying to provide at least some research. If you don't like their findings, then go disprove them with your own studies. BUT, if you won't disprove their research, then all you have is a THEORY - and how is that any more reliable than qualitative research? We need more verifiable published research, but recognize that just because you don't like someone's methods, reporting style or whatever, doesn't mean their findings are necessarily inaccurate. Posted by: Lyle, Lyle - Croc O' Lyle on March 25, 2003 11:34 PM
I'm glad UIE and others are trying to provide at least some research. If you don't like their findings, then go disprove them with your own studies. No need when those in question cannot conduct valid research. As I said, UIE is unable to hold themselves to even the standards of a high school science fair. If researchers cannot work to the standards of at least a Master's thesis, then it's junk. UIE publishes junk: Evolution Trumps Usability Guidelines - Shows they don't understand how to choose a valid hypothesis. (My comments) Testing Web Sites: Five Users Is Nowhere Near Enough - Shows they don't understand the consequences of variables in a study, and that at least some CHI reviewers don't either. Their book is far worse: Book Review: Web Site Usability: A Designer’s Guide
I don't have to be a chef (which I am, incidentally) to know when the fish at a restaurant is past its prime. If you don't like their findings, then go disprove them with your own studies Lyle, what you're saying is a logical fallacy: Burden of proof. The burden of proof is on the person making the assertion. In plain English, I don't have to prove you're wrong; you have to prove you're right. if you won't disprove their research, then all you have is a THEORY - and how is that any more reliable than qualitative research? I'm not advancing any theories at all. I'm merely questioning their research methods. As I have said in my earlier comments... The article makes an assertion: You have undoubtedly heard that users give up because pages take too long to download. This is also a myth. Testing shows no correlation between page download time and users giving up. And why is it a myth? No idea. The article sure doesn't tell me squat. Is it a myth because they say it's a myth? These tests... did they account for any of the factors I and others have mentioned in our earlier comments? but recognize that just because you don't like someone's methods, reporting style or whatever, doesn't mean their findings are necessarily inaccurate Actually if their research methods are questionable, their findings are probably more likely to be inaccurate. Lastly, I'm not the type to conduct specific research with specific user profiles on specific sites and then use that to make general guidelines for all sites and users. Or do you still believe users don't read on the Web? Posted by: MadMan on March 26, 2003 03:16 PM
Home | Moving WebWord | Cool Books | Hot Web Sites
URL: http://webword.com/weblog/ ©1998-2005 by WebWord.com. All rights reserved. |