I read my server logs daily in raw form, and they show the user agents (and when it says "Googlebot" it's pretty obvious!), but frequently the field includes a url where you can learn more about the agent and determine whether or not you want it to have access. For instance, I don't generally like users to make local copies of my web site; but there is an academic resource out there that indexes sites to better check for student plagiarism, in which case I'm happy to let them in.

But additionally, knowing the IP addresses of the search engines can help also because you may want content to be behind a firewall (with access only for those who pay a fee), but not want to charge search engines, so they can alert users to the existence of your content. (And yes, you can stop Google from caching your content.)