If you are happy with your fitness level in general don't worry about it. The only person you should be comparing your heart rate to is yourself. Everyone has their own range of heart rates.
An imperfect analogy would be a fit guy who is 6' being jealous of the body weight of a guy who is 5'6", even though they have the same BMI. The same fitness level for both of them will correspond to different numbers.