I don't think there's any aspect of training that is trickier to negotiate than easy days--and I include the experiences of others as well as myself when I write this.
When I was in college, on our easy days, someone invariably found themselves pushing the pace in a way that everyone could handle, but that just took a tole after a while. After a long period off, I recently built my mileage back up, and allowed myself similar indulgences on easy days--for 5-6 miles, I would start out easy only to end at a pace that was realistically speaking close to or at tempo pace.
This kind of thing adds up. Throw in an *actual* tempo run one day, an interval workout another, and a brisk long run another, and these "easy" days stop being easy. More importantly, they don't let you recover properly.
I should be up front and say that I'm not--and never have been--a real mileage hog. 50 mpw is plenty for me. But the long story short is that after getting into decent shape, I more or less burned out. The "easy" runs became harder, and the workouts/tempo runs were more muscling through rather than reaping any benefits. It could be that I was trying to extend my level of fitness for too long, but there's no way that my easy runs were helping.
So I would really like to echo the poster who said to give yourself 20 minutes easy. Even if it's just once a week rather than twice, it'll do some real good for you.