I'm not exactly elite or sub-elite, but I feel the same way. I found that by slowing down recovery runs I was able to do a lot more mileage and get a lot more out of the quality runs. In my case I was doing recovery runs at 8:00-8:10 to start and sometimes getting faster, maybe up to 7:45 depending on how it felt. Even if I felt good I made sure to hold back. Marathon pace was 5:55.
Generally I did this one day a week, between hard workouts and also a couple times a week for the easy side of a double.
I hear a lot of guys bragging about how fast their easy pace is. If you are pushing (consciously or subconsciously) to do recovery runs faster you are not giving your body the rest it needs. If you can train well while doing this you probably aren't pushing your quality days hard enough.
It is hard to swallow the pride, especially when people see you out slogging, or ask about your recovery pace. For me I had to run by myself, away for my usual groups to keep from picking up the pace. Just remember to let your races do the talking.