Posting Date: January 16, 2003

The User Experience of Government Online: Recommendations for a Citizen-Centric Future -- "The study focused primarily on the user experience of government portal sites, comparing cross-agency portal sites that provide access to multiple departments within a single tier of government (such as fed.gov.au and vic.gov.au) with the new generation of intergovernmental portal sites that provide access to multiple tiers of government (such as Service Tasmania and Service South Australia). The study consisted of 12 usability test sessions with representative citizens, who carried out typical everyday tasks on the sites such as paying council rates and renewing vehicle registration." (Comments: I'm starting to get confused about the actual value of usability reports. The 33 page report referenced above is free, yet 37signals is offering a report on e-commerce search for $99. What are these types of reports really worth?)

Reader Comments...

Initial results from my report comparison study:

NNGroup Design Guidelines for Search - Cost:$45, Value To Me:$25
UIE Getting Them to What They Want - Cost:$24.99, Value To Me:$20
37signals E-Commerce Search Report - Cost:$99, Value To Me:$15

If anyone has an example of a good 'free' report on search then I will add it to my study.

Posted by: Mac on January 17, 2003 04:34 AM

Mac, could you please write a review of the 37Signals report? I don't buy any report but I still would like to know if they live up to the hype.

Posted by: MadMan on January 17, 2003 12:33 PM

Also, I'd like to know how you calculated "value".

As an aside, I wonder how the search usability of somesite.com makes any difference in the work we have to do, other than give us some figures to spout to someone who wants an "ROI calculation".

For instance, say Webword.com was an e-commerce site and we were working on the search engine. Would knowing that Amazon.com's search is rated 90% good give you anything useful to go on? Perhaps some "heuristics" and "best practices" (but we all know the limitations of heuristics.)

Oh, and I also want to know about 37Signals' testing methods. Is that mentioned in the report? How many users did they test? How did they build the profiles of these users? Or did they just test among the four of them?

Addendum: As I was typing the above paragraph, I glanced at their report page, and I found this little nugget:

Who conducted the tests in this report?
Each test featured in this report was conducted by 37signals. We did not use third parties or external testers.

I see...
Ah yes, a real world usability test. Wonderful. I'm sure the four of them are representative of the typical users. Seems about as useful as Uncle Jakob's heuristics. Comments?

Posted by: MadMan on January 17, 2003 12:42 PM

MadMan, here is my one word report on the 37signals report: Disappointing

It would make a very good powerpoint presentation. I really wanted to like this report, but it left me feeling cold. It looks very good, the layout is very tasteful, and it proves that 37signals are a competent outfit who I would be happy to work with, but......

Jakobs e-commerce search report is a better example of this type of report. The 37signals report boasts that it contains 150+ screenshots, in 45 pages! They are sceenshots in the same way that this is a screenshot of WebWord

I would rather have seen a lot less screenbites, and a lot more words.

What do I mean by Value To Me ?
Well this is how much I would pay for the report after knowing what was in it. This means that by my measurement of Value For Money the UIE Report is about 5 times better than the 37signals report.

I have involved no 3rd parties in my report of reports and have carried out all my testing with the aid of a dumb animal. All the views and opinions are based on my own feelings and prejudices and cannot be construed in any way as a robust review. I will get it finished in the next 10 days and will see if John wants to post it on WebWord.

Oh and does anyone know, how much of the reports I can include in a review without breaking copyright? Is a screendump of a third or a page too much?

Posted by: Mac on January 17, 2003 01:31 PM

It comes down to "fair use" for review. A page is probably OK if you're critiquing most of the page. Otherwise, restrict it to the parts you're reviewing.

"Fair use" has no clear legal boundary. It's a subjective test. If you reproduced 30% of the report, you may no longer be using it fairly.

Posted by: MadMan on January 17, 2003 03:38 PM

What are these types of reports really worth?

Consider: Do the authors have the qualifications to write such a report? Have the authors demonstrated they are capable of producing research at a certain level of quality? Have the authors demonstrated they understand the topic on which they report? Are the authors targeting people who will ask these questions?

UIE: No, no, no, no. (Recent example here)
NNGroup: Yes, yes, yes, no.
37signals: ?, ?, ?, no. (Though I've liked some of what I've seen from them, they're unproven and don't disclose their qualifications.)

Each test featured in this report was conducted by 37signals. We did not use third parties or external testers.
I interpret this to mean that the work is solely their own, rather than a comment on what type of evaluation they did.

Posted by: Ron Zeno on January 18, 2003 05:39 PM

I think "We did not use third parties or external testers" is pretty clear. They didn't test with any other users than themselves. Maybe someone from 37Signals can clarify it here for us? (I know at least one of them reads this site)

Posted by: Charlie Xavier on January 19, 2003 09:41 AM

Subscribe to Webword.com
Receive the best free usability newsletter on the Internet.

URL: http://webword.com/weblog/