Golfers are athletes - I tested a bunch, and physical abilities and coordination separate the pros from the good amateurs from the recreational people. It is a different type of fitness. Some of those are pretty strong and most are very flexible. The long driver competitors are actually pretty amazing.
We did a cycling VO2max test on one internationally famous golfer over the age of 50, and his VO2max was around a 62. I think he was among the top - he was ridiculous fit. The only reason we did a VO2max test on him is because he wanted it done - it was not part of our standard golf test. If memory serves me correctly, he was <10% body fat also.
It's completely different than running or a lot of other sports out there, but elite golf players should fall into the realm of athletes. I don't think a recreational golfer wouldn't be considered an athlete any more than somebody that jogs for fitness.