|
WebWord Weblog Posting Posting Date: August 10, 2003 Common Mistakes -- "There are several common mistakes made by those conducting usability tests. Most of these, once identified, are easy to avoid."
Reader Comments...
"Failure to run a pilot test..." That alone covers so very much. Posted by: Ron Zeno on August 10, 2003 11:12 AM
I like the 'Accidental Revelation' part. In my opinion, most, if not all, user-testing being done out in the real-world is being done by people without the necessary skills (this means IAs, 'usability experts,' a whole slew of people who never had anything but a glossing on field research techniques, concentrating instead on deliverables, or statistics, etc.). The whole point of testing is to gather reliable data, yet, unfortunately, to most real-world practitioners, the idea of testing is just to 'run a test.' It's all song-n-dance; a check-list routine. Let's face it, field research is a social science skill. It is not taught or prevalent in the number-crunching 'usability' or IA disciplines, or in any of their off-shoots. Unfortunately it's a deliverable kind-of-world out there -- a mentality which will let you get away with sloppy research as long as there is a pretty picture to pass around in the end. The best user-testers in today's online businesses are the graphic artists. They have the right skills for what's being asked for. I can't blame anyone though, as this is what is being taught and what we can observe successful industry leaders doing and receiving accolades for. Posted by: Derek R on August 11, 2003 02:31 AM
Graphic artists?! Sure, and the best doctors are veterinarians, and the best software developers are marketing people... Not sure what alternate plane of existence you live in, but it's nothing like my world. Clearly you're extrapolating from an exceptional anecdote. Posted by: Lyle, Lyle, Croc O' Lyle on August 11, 2003 09:10 AM
One thing I would add to their list is the creation of accurate and meaningful usability objectives (metrics). This is, of course, depending on what type of usability test you are doing. I am not a big fan of objectives if I am just doing exploratory testing. Posted by: fajalar on August 11, 2003 11:33 AM
>| Not sure what alternate plane of existence you live in The title of the article is "Common Mistakes" Lyle. These mistakes are common because attention is being placed elsewhere. In my analysis, the attention is being misplaced into creating pretty end-product deliverables where the 'the look-n-feel' of a research document is more valuable than the actual research. Thus, my graphic artist quip. Nevertheless, I understand the aggression which denies anything is being done back-ass-wards or wrong. This is the entire point I wish to draw attention to. My contention still stands that user-testing is a social science skill that cannot be adequately adopted by Joe or Jill self-proclaimed usability-expert, or simple because the testing duty falls on your lap from your employer. Real user observation and testing must be done by social scientists if you wish to ensure impartial and reliable data. Only the social scientists are properly trained in this discipline. I think it behooves anyone, or any business, which claims to employ 'user-centered methodologies' to actually be trained in the social sciences. Who are we trying to kid? Posted by: Derek R on August 11, 2003 01:01 PM
uhhh....but graphic artists aren't social scientists Posted by: on August 11, 2003 02:09 PM
Fajalar's comment on my failure to list 'creation of accurate and meaningful usability objectives' is absolutley correct! Posted by: Gerry Gaffney on August 11, 2003 07:16 PM
Gerry, Just to let you know I have known of your site for a while and have passed it on informally to many people. Good stuff. I have even tired using Classified for validation of card sort results. Thanks! Posted by: fajalar on August 11, 2003 07:26 PM
"uhhh....but graphic artists aren't social scientists" Once again Derek is too eager to ram ahead with his rhetoric and fails to properly deliver a clear and meaningful communication. The irony ;-) When Derek says "The best user-testers in today's online businesses are the graphic artists." he's not saying they actually are, only that in a business environment which prioritises clear communication then it's the graphic designers that have the edge. Unfortunately for Derek it's not an either/or situation though. The quality of research is *orthogonal* to the quality of the presentation. It is entirely possible to do both, and a failure to accomodate that is a failure to understand people and their needs. Oh, the irony, again. Posted by: on August 11, 2003 10:52 PM
I obviously don't have a problem with presentation. Please present your user-testing results in the most accurate and communicative way possible -- likely with visual aids (ie. graphs and other visualizations of information). The point I have made, which I think you are skipping over, is that the mistakes mentioned in the article are 'common' because more attention is paid to the presentation of the testing results rather than to the actual reliable acquisition of data. In short, today's business would rather have a fancy colorful report than accurate results! A good example is the above anonymous coward's comment regarding 'clear communication.' You can see how the 'communication' of the usability-testing results receives the privileged position -- "a business environment which prioritizes clear communication..." There's no hiding here this blatant statement which places more importance on the 'communication' rather than the acquisition of accurate results. Surely there will be back-peddling about what this anonymous coward really meant after this message -- but it's too late to be convincing and these 'mistakes' which skew testing data continue to be 'common' (read the article) while we polish and hand out accolades for our communication skills. In the real professional world of user-testing and field research any mistake ruins the entire project. You don't make 'mistakes' in social science research and still have research. This is a big difference to what is going on in Web testing labs today.
>| The best user-testers in today's online businesses Re-reading this page I see several of us are talking past one another. To clarify: I was being sarcastic in suggesting 'the best user-testers are graphic artists' -- it only seems so with today's Web businesses adopting backwards priorities regarding user-research (ie. placing premium value on the presentation of results over the results themselves).
Again, research has nothing to do with presentation. Research has to do with research. If you are charged with both duties then you should know enough that presenting lies is not at all helpful -- the research must be there 100%. How well you 'present' is only relevant once actual, reliable and accurate research has been collected. I'm tired now and must sleep. Posted by: Derek R on August 12, 2003 04:18 AM
Derek wrote: I wouldn't say any mistake ruins the entire project. A mistake may discount the results you get from a single participant, and yes and entire lab may get "ruined." But as with all mistakes that occur while observing human behavior, you mark it down and move on. Later you can decide if the "mistake" has ruined everything. I've been in labs where we have mistakes happen. Almost always it is isolated to a single participant. That session stays part of the report and we may (or may not) get another participant. One the "make it pretty note" see if this NIST report format would work. Standardized reporting takes the guess work out of what to put in the report, and how to format/display it. Posted by: fajalar on August 12, 2003 09:22 AM
"the mistakes mentioned in the article are 'common' because more attention is paid to the presentation of the testing results rather than to the actual reliable acquisition of data." ummm ... the article describes six "common errors", all of occur either before or during testing, and none after testing. Unless you're arguing that those errors occur because the testing phase is rushed or under-resourced, as compared to the analysis/presentation phase. The article doesn't address the post-testing activities at all, including any related errors, so I don't buy your argument. If anything, I've seen extensive testing done and then the presentation be skimpy and unexciting, cutting to the chase and presenting only the major conclusions. Posted by: Eric on August 12, 2003 09:56 AM
Being impartial and neutral in a test is particularly difficult. As I think its a very human instinct to want help. Posted by: Daniel Szuc on August 12, 2003 10:45 AM
>| Unless you're arguing that those errors occur because Yes, that's what I'm indicating -- but more so that the importance and political emphasis is placed on results-presentation because that is where, as a usability-tester, more-or-less your job performance is evaluated. The 'boss' likes your presentation, or not, and that's your 'job review' so to speak. However, the boss doesn't know, and is not in the position to appreciate, the work done to obtain reliable and accurate results. So -- what ends up naturally happening is all the emphasis and worry and hand-wringing goes into the presentation phase because this is the 'make-or-break' portion of your usefulness to the company. This is backwards and wrong. Business needs to realize the importance of usability-testing without having it become a competition where one person can be seen as performing better than another (ie. the 'graphical design competition' that is today's usability-testing). Fajalar's NIST report format resource is a good step in this the right direction towards realizing this. Real professional user-testers (ie. social scientists trained in research techniques) have scientific standards which ensure impartial results -- that's why it's called a science! Today's design competition in the deliverable-happy Web business world misses the boat on what usability-testing aims to achieve. It's just a lot of bubble-worship (ie. wishful thinking that will get you in trouble). Listen, I think it's great to have sexy, colorful usability-results to grab attention for the field of usability-testing, to give it exposure as a legitimate and necessary business process, and, to demonstrate the value of the people performing this work -- but -- it is contradictory to the research that 'entertainment value' is the driving motivation behind the scenes in usability professions. We need to be doing things for the right reasons and not engage ourselves in blowing bubbles. Keep your feet on the ground, etc. You may just find that building from a solid foundation (a rock) proves more benefit than making ourselves into superheroes who fantastically arrive to 'save-the-day' in product development.
Yes, well I'm specifically referring to project mistakes, like the ones referenced in the article. As you indicate, individual participant mistakes are not really mistakes, just spoiled participants. Posted by: Derek R on August 12, 2003 05:11 PM
While I've not seen the problems Derek is discussing, it's certainly possible given how many companies have presentation-oriented management. The problems I've seen are more commonly like Derek's other point, that the testers don't have adequate experience and education in social sciences. Relate to Derek's presentation-orientation problem, is the problem of testers having to make politically-correct test reports (having to understate problem severity and/or likelihood, or being unable to test the most important aspects of a product). I once had a very big client that asked me to rewrite my test report to understate the problems we found. (I refused.) The worst part of it was how resistant they were to directly stating what they wanted and why. (They asked for a great deal of other changes which I was happy to oblige). Posted by: Ron Zeno on August 12, 2003 11:27 PM
Home | Moving WebWord | Cool Books | Hot Web Sites
URL: http://webword.com/weblog/ ©1998-2005 by WebWord.com. All rights reserved. |