FWIW, leaving a little earlier than normal seems to keep the critics at bay.
I think you will find that most people don't really care, but there will always be someone who comes up to your desk and asks, in a very loud voice, why you have wet hair. Another might run up to you looking stressed and ask where you were b/c they looked everywhere for you. I have had my share of these people, and they are usually the fat smokers who always act like they are overworked and underpaid.
Anyhow, I find that leaving at 11:30, for instance, catches people off guard, and when you are back at work at 12:45 - 1:00 (and they are still out eating or whatever), they are less likely to notice you were out for more than an hour.
It is the getting back at 2:15 crap that usually ignites a flame.
Anyway, try that if you can.
Another thing, talk to the CEO about it - or whoever is top dog around there. They will usually love the idea, and then you have their buy-in. It is like being "made."