In my opinion, Web-site font sizing is broken in browsers and not in the CSS spec. There's nothing wrong with setting a definite px size in your stylesheet (but not pt, unless it's for printing). That represents your personal armamentarium in the battle of Type Display on the Web. Then the visitor comes along and uses his or her personal armamentarium, the browser, to accept or reject your setting.
Browsers should increase and decrease font size irrespective of units specified, which should be used for *initial unaltered display*.
Also, the implicationsof em sizing are very poorly understood by designers, which is surprising, since em is exactly the same as % in CSS sizing, IIRC, and % is readily understood. In other words, if I set up 12px as my default in my browser and you use 0.7 em, you are condemning me to 8px type. Not cricket, I don't think.
The entire issue is horrible on all counts and it is not fair or necessary to blame anyone for the current state of things, except for browser makers. It's a bit much to leave a site just because you can't increase its font size. I mean, try reading it in Lynx with big fonts, then.

You can blame the browser makers for being the cause, but I blame designers who do not adjust to the reality of the situation. Few people specify a default font size in their browser. Even low vision users I've observed will use the browser's +/- on-the-fly resizing, instead of configuring their browser because configuring is a much more complex task from their perspective.