With my cycling background, I can answer that. Most of the people you see out there running, cycling, swimming, etc are NOT competitive racers.
1) 90% of people - don't race at all, just jog or bike because it's fun and to stay in shape.
2) 9% of people - they might do a few bike races or some 5k/10k races for fun a year, but don't tryhard or follow a training plan.
3) 1% of people might be actually interested in improving their performance and follow a training plan.
So 99% of people out there don't race much, don't need to taper or anything. So when they get passed, they hang on, they wanna experience the crazy speed of someone running 8 min/mile pace or cycling 23 mph. For them it's a race-like effort, which isn't bad, they go out of their comfort zone and end up being much faster than usual. Sure they might be toast for a few days after, but they are having fun.
If you are annoyed by someone hanging on you, there are enough ways to get rid of them. Just a few examples:
1) Constantly increase your pace, testing what their limits are. 7:50.. 7:30... 7:10... 6:50... 6:20... most will stop here already, but feel free to go faster if someone is really dedicated. By the time you run 5:20 you will have dropped just about anyone. Once they are gone you can get back to your 8 min pace or whatever you ran, they won't show up anymore.
2) Insert frequent surges. This destroys them mentally... They hang on and are so thankful when you finally slow down, letting them recover. But then you already pick up the pace again. Then you can insert a longer recovery interval, so they think you are finally done before adding another surge. At that point they are probably completely broken and will have lost their will to hang on.
3) Turn to a hilly route and keep your pace - 8 min mile pace - on the steepest hills you can find. They will be gone in an instant.
4) Go to a downhill trail. They won't have the mechanics to keep up with you and either risk tripping or need to give up.