What workouts you do on the non-easy days ties into what pace you should be doing on your easy days. Training as a whole is a system, one piece can't be looked at in isolation.
I would suggest that you talk to your coach about this privately. Tell him you feel like you are going to slow, that it feels far to easy (if that is the case). Tell him that 6:40 pace still feels like a recovery pace for you and that is how fast you run alone in the off season and see what he says. He would hopefully be able to give you a good response.
If he gives you the green light to run faster, just tell your teammates you want to run faster and they're welcome to tag along. If you have a good relationship with them and say it in a polite and respectful way, I would hope that it won't cause a weird rift in the team, even if some don't want to pick it up that fast. Definitely give them an out with something like "if you feel your easy pace should be slower, don't let me force you to run too fast." Not everybody needs to do easy days at the same pace.