As a usability person myself, mostly doing UI design, I've had the recent opportunity to work with external usability consultants conducting usability testing of our products.

All in all, I think the work has been good and the product owners (with whom I work) have been pleased with the results. The interaction with users has been conducted professionally and the reporting has been thorough with some useful recommendations.

There are a couple of things that have definitely displeased me though.

1) One consultant seems to feel comfortable enough to constantly make political commentary (always irrelevant) whenever possible. I have personally been offended by a number of these comments and have bitten my tongue on several occasions. I don't know how all the product owners who attend the testing feel about this but I think it is unprofessional in the extreme.

2) Consultants who don't seem to listen to our suggestions for conducting the usability tests or who seem to ignore comments pertaining to the methodology used. I've asked a consultant on a few occasions to make a change only to see it ignored. (I've forgiven this consultant or not said anything as I realize that the consultant is extremely busy). In any case, this has rubbed me the wrong way.

I think there might be a tendency on the part of consultants to become defensive and to appear less than fully-informed when suggestions for how they might recruit or run a test are put forward.

Another thing I've become aware of is how useless our videotaping of sessions with VHS tapes are. Though a couple of developers actually watch some tapes of the sessions, I've seen in the past how powerful 'highlight' videos of tests can be and the fact that these are so time-consuming and expensive to do make it difficult to justify or pay for someone to put these together.

Does anyone have suggestions for better video-capture equipment and software (oh, and relatively inexpensive and easy-to-use) for testing? Perhaps DVD? This is the sort of thing that is a bit difficult to justify but I know from my own experience in the past that such videos can really sell the benefits of usability or the horror of the lack thereof.

Posted by: Nevermind on August 20, 2002 08:54 AM

 

1) One consultant seems to feel comfortable enough to constantly make political commentary (always irrelevant) whenever possible.

I've seen similar (and worse) unprofessional conduct in a number of consultants. Unfortunately, it's not uncommon in the world of self-proclaimed consultants.

2) Consultants who don't seem to listen to our suggestions for conducting the usability tests or who seem to ignore comments pertaining to the methodology used.

This is to be expected. Consultants must do this to meet their goals of cost, time, scope, and quality.

Does anyone have suggestions for better video-capture equipment and software (oh, and relatively inexpensive and easy-to-use) for testing?

No, I don’t think such a solution exists (yet). This topic comes up every year or so on the relevant discussion lists. Many consultants join in to promote themselves, but they are unwilling to divulge the specifics on how their solutions are better.

Posted by: Ron Zeno on August 20, 2002 10:41 AM

 

My concern, from the business perspective, is that there is so much research out here today that paints a picture of confusion about the level of expertise and accuracy of the calls that UI/HE/HF consultants make.

I guess this is because, we the business people, don't ask for ROI based results. We just say look at my site and provide feedback.

The problem is when I ask for a cause and effect kind of approach I get the standard waffle about how this is an arcane art born of evil spells and black magic..... so where does it end...or where does it start……

So do I feel the output is valuable...yes, but! - I take it with a grain of salt and never use the consultants as the only guide for making decision.

Posted by: JB on August 20, 2002 12:27 PM

 

Okay, looks like so far I'm the only HCI person who is willing to respond in the JOS Discussion Forum. I didn't really think it was a fair question, as you'll see in my response. (Scroll down a ways or do a "Find" for 'Lyle', no way to link directly to my comments.)

Joel on Software rocks.
I particularly liked a post he wrote on bloatware and usability.

Posted by: Lyle - Usability Guru on August 21, 2002 01:13 AM

 

Lyle - good point. Perhaps the question did invite the responses I received. It was a spur of the moment question, which perhaps required more consideration. However, sometimes I like to post in haste and just see what happens. This may seem impulsive, but if you see some of the other dross on Joel's formum you'll see I'm far from being the only one (I'm not slagging Joel's site here - I too believe it rocks)

Sherlock

Posted by: sherlock_yoda on August 21, 2002 04:02 AM

 

It's fairly obvious where the usability people came into the conversation: my comment is just above Lyle's, and someone else followed on.

It would be interesting if some of the first posters replied, but it seems unlikely because I couldn't find the thread when going through the site. I had to come back to webword to find it!

Posted by: AlastairC on August 21, 2002 07:01 AM

 

I would like to see more open discussions between HCI folk and programmers, as it's the only way we're going to produce better systems.

nevermind as an alternative to expensive highlight tapes of usability sessions you could think about "performance highlights" where two team members who have seen the sessions put together a five minute re-enactment that illustrates the main problems (and successes). I have used this method a number of times, and if the performers get into their roles it can be very useful and entertaining.

Posted by: Mac on August 21, 2002 09:02 AM

 

looks like so far I'm the only HCI person (from Lyle)

Other than the two just before Lyle's. I post now and then. Are there complete archives somewhere?

I like how you turned the question back on the programmers, Lyle!

I particularly liked a post he wrote on bloatware and usability.

Where Joel cites "20%" without any reference to where it comes from? You're falling for his propaganda! I've actually run large-scale usage studies on word processors which show the exact opposite of what Joel cites.

I've emailed Joel and posted to his discussion group before, "over both the short-term and long-term studies less than 50% of the features of the word processor were used total. More importantly, a common set of less than 35% of the features were all that over 80% of the people used." (Joel has never responded.)

And then Joel generalizes it beyond word processors as well...

It's not completely Joel's fault. He's just repeating propaganda from Microsoft, masters of propaganda. As far as anyone knows, Microsoft is supporting their "20%" by misinterpretating research on the usage of features in UNIX operating system. (Wonderfully ironic!)

Posted by: Ron Zeno on August 21, 2002 10:19 AM

 

Mac, that is an interesting idea. In my mind at least, the power of the highlights tape is that these are actual users. "Actors" are at least one step removed and therefore, I think the project managers would give this less weight.

Maybe not, though. Interesting idea...

Thanks.

Posted by: nevermind on August 21, 2002 12:40 PM

 

We once did a 3 minute sketch of a user who just couldn't get past the first screen of an app.

No words, just sat there looking worried and fretful. The developers accused us of exaggerating and were incredulous when we said the user actually sat there for 10 minutes before giving up and then rating the app as "excellent".

Design as Theatre in action...

Posted by: Mac on August 21, 2002 03:49 PM

 

Hmm...while I should have maybe recognized a few folks names/handles, I guess I didn't think the responses reflected a very "pro usability" attitude. Re-reading them now I can hear a different tone, but on first read they seemed largely negative.

Question: As a group, how do "usability folks" engage in online debate without negative impact to the general public/business perception of usability practitioners? If we're airing our dirty laundry, doesn't that just help people take issue with us in general?

My reaction to Sherlock's question was to defend usability as a field, rather than highlight issues experienced working with individuals in that field... The same overall rationale is why I try not to throw many barbs at Nielsen and other very visible usability folk -- even if they deserve it at times.

Hmm...maybe I should change my moniker to "Lyle - Usability PR Department"

Posted by: Lyle - Usability Guru on August 21, 2002 06:57 PM

 

You could be right there Lyle, I sometimes find it difficult to switch from an 'inter-usability' mode (like here, and on usability lists), to an 'extra-usability' mode (to everyone else).

Perhaps it's a reaction to usability as a buzzword: I tend to think that there is no one panacea to web/product development, just methods and skills. Some more useful than others depending on the situation.

Posted by: Alastair on August 21, 2002 07:09 PM

 

how do "usability folks" engage in online debate without negative impact to the general public/business perception of usability practitioners? - Lyle

Good questions. There is at least one private discussion group specifically for this purpose.

As for the negative impact, it's a matter of perspective and intent...

Posted by: Ron Zeno on August 21, 2002 08:27 PM