For some people it can have a significant performance-enhancing effect (especially if, like you say, they don't normally consume it or abstain for a while then take it prior to a race).
One thing I was surprised about in a way is that coffee is the world's biggest source of antioxidants.
Apparently it provides more antioxidants than any other food or drink in the US diet. Other foods have more per g but it's the overall amount consumed that swings it.
The second link is pretty interesting.
Anyway... OP - what do you think? A total ban? Or a different level?
Personally I think it's OK and trying to ban it would create a load of problems that just aren't worth it. We have bigger problems. The limit should be somewhere that basically indicates that normal use - a few cups - is fine, but say having eight or twenty isn't.