Embarrassing. A very poor straw-man argument. Certainly doesn't give much credibility to the author or his company.

Information quality:   Very Low - Invalid tests prove nothing.
Propaganda quality:  High -      Entertaining self-promotion targeting the credulous.
Propaganda level:     Very High - Self-promotion and pseudoscience.

Posted by: Ron Zeno on September 9, 2002 07:28 PM

 

Ron,

I think Spool's article was pretty good. I'll admit it contained a fair amount of self-promotion and clearly didn't point out initiatives like that of the NCI where they are trying to rank guidelines based on research. Even so, Spool raised some good points -- even if he could have provided more detail. Not sure from your comment if you disagree with everythin he said or just the way he said it. Do you think guidelines are a Good Idea or what?

I don't think Spool did a good job of proving the "evolution" theory he proposes. I put more stock in processes like UCD. Evolution is a variation on a "iterative development" theme, but in reality I don't think companies have the resources to just "try stuff" and see what works -- they can't even figure out what success looks like in many cases -- but that's a bigger non-usability problem.

I agree with John's article on evolution -- he reminds me that guidelines limit creativity, and sometimes we need to encourage new ideas (and other times we need to stick with "proven" ideas). Best line: "Let your ideas have sex; you want offspring."

Anyway, I posted some of my thoughts on problems with usability guidelines. Let me know what you think.

Posted by: Lyle - Son of a Usability Guru on September 10, 2002 02:27 AM

 

Let your ideas have sex; you want offspring.

"Have sex" is so... pedestrian. You should say, "Let your ideas make love..." :))

Lyle, I'll get around to reading your stuff, but my own view of guidelines is that when they become dictatorial, you should abandon them. Something like "Make error messages helpful - suggest a way out" is acceptable, but telling me that there is only one way to create good error messages is not acceptable to me.

I agree on the UCD part. I'm amused when I see something posted to a UCD-related list like "when we tested 20 users, 18 didn't know that the logo was also a link to the home page." (Recent discussion on SIGIA-l.) Oh sure, if the audience consisted of people like my mum, that probably would be a valid assertion. If the audience were people like you and I, I dare say 18 out of 20 people would know that the logo had a hyperlink to the home page.

Er, the point of my rambling is that rigid guidelines without context are mostly useless. And when guidelines become rigid, they cease to be "guidelines" because they no longer guide, they dictate. (Remember how the Brits on this site pointed out that "shopping trolley" is more appropriate for British e-commerce sites than "shopping cart"? That's what I'm talking about.)

And we all know that despite Nielsen's competence and experience, his uncompromising "guidelines" are why he's hated by so many.

Posted by: MadMan on September 10, 2002 09:59 AM

 

(Nice article, Lyle!)

Spool selects poor guidelines to attack, and even then his (fatally flawed) arguments show a lack of understanding of the guidelines and how to properly test them.

Nielsen promotes his guidelines without (freely, if at all) giving the pertinent information for people to determine the soundness of the guidelines, their limitations, and how to properly apply them.

"Guidelines limit creativity" and this is a very good thing to do. By focusing creativity, by properly constraining it, we are more likely to produce better results. Good guidelines, properly applied, allow us to build upon the knowledge and experience of others.

Posted by: Ron Zeno on September 10, 2002 11:30 AM

 

(Nice article, Lyle!)

Spool selects poor guidelines to attack, and even then his (fatally flawed) arguments show a lack of understanding of the guidelines and how to properly test them.

Nielsen promotes his guidelines without (freely, if at all) giving the pertinent information for people to determine the soundness of the guidelines, their limitations, and how to properly apply them.

"Guidelines limit creativity" and this is a very good thing to do. By focusing creativity, by properly constraining it, we are more likely to produce better results. Good guidelines, properly applied, allow us to build upon the knowledge and experience of others.

Posted by: Ron Zeno on September 10, 2002 11:30 AM

 

Selecting a guideline which tests as generally correct to prove that {sometimes guidelines can be wrong} would be non-sensical.

The point of Spool's article, imho, is that some guidelines are good, most need testing, and some can be harmful if untested. The fact that some guidelines are written in a bogus form (ie. vague or subjective, rather than specific) is also noted as a black mark against guidelines as typically encountered.

In conclusion Spool recommends either *testing* the guideline, or if you don't have the resources/opportunity to do so effectively to use an *alternative* approach, one which is driven bottom-up rather than top-down.

Posted by: Eric Scheid on September 10, 2002 02:09 PM

 

Testing guidelines is hard. Spool himself doesn't appear to know how to do it properly. (At least he doesn't demonstrate it in his article.)

Instead of testing guidelines, critique them to determine if they will be valuable to you. Read Lyle's commentary to learn about some of the limitations of guidelines.

Don't reinvent the wheel if you don't have to, and don't throw wheels away just because you don't understand how to use them or properly evaluate them.

Posted by: Ron Zeno on September 10, 2002 07:50 PM

 

Critiquing a guideline still boils down to an untested opinion. Sure you could winnow out a fair bunch of dubious guidelines, but you'd still be left with the ones you hope, untested, will actually deliver the goods.

Yes, testing is hard to do right, and would need an article in its own right. I wasn't expecting his article to be a tutorial on how to do the tests. Did you?

Posted by: Eric Scheid on September 11, 2002 05:40 AM

 

...you'd still be left with the ones you hope, untested, will actually deliver the goods

I assert that a practitioner with some basic critical thinking and research skills can do far better just by examing the guideline and the supporting information. Otherwise, you are just arguing that you don't have these skills and at the same time will not rely upon others who do have them.

I wasn't expecting his article to be a tutorial on how to do the tests. Did you?

No, but I expect the author to give credible evidence that he knows how to properly test guidelines. He did not do so. Instead, by choosing a completely invalid hypothesis to test, the author shows he does not understand the guideline in question and shows very poor judgement on how to test guidelines. (Perfect example of why not to test guidelines, or at least how not to.)

Further the guideline in question is based upon rigorous testing - far better than any practitioner has the time, budget, (and perhaps ability) to do. Why waste time to test it again? (Why embarrass yourself by showing that you do not know how to test it properly?)

Until guidelines improve, practitioners have a number of options:
1) Demand better guidelines.
2) Stop using guidelines.
3) Become researchers and then test current guidelines and research new ones.
4) Critique current guidelines before deciding to use them.
5) Help others critique guidelines.
6) Share critiques of guidelines.

Posted by: Ron Zeno on September 11, 2002 02:16 PM

 

Ron,

Can you be more specific about what hypothesis was invalid (and why) and how exactly Spool tested things incorrectly. Not saying I disagree - just not following your comments.

Thanks.

Posted by: Lyle - Son of a Usability Guru on September 11, 2002 07:07 PM

 

just not following your comments

Fair enough. I had commented about the article in my weblog here. I should have noted it with my first comment.

The only hypothesis that can be checked for validity is the one in the case study, "if sites place these common elements where users expect, we should see users purchasing more from the sites."

It's not a valid hypothesis to test the guideline. The author of the guideline didn't even mention that it would effect purchasing, and even gave a caveat on applying the guideline. Spool introduced purchasing and ignored the caveat.

You can appear to prove anything if you can get people to overlook an invalid hypothesis...

Posted by: Ron Zeno on September 11, 2002 08:01 PM

 

Ron, I am finding it difficult to read your weblog with that flickery woman in the corner. I have been to Las Vegas once and I daren't go near the slots again.

Posted by: Mac on September 13, 2002 03:15 PM