First of all, you are slow. Maybe you meant "very, very slow." There is a difference. Freshman girls and 50+ masters men routinely run your 5km time without being super talented or doing big mileage, so you are indeed slow.
But yes, I understand your post. You are wondering why you can run 20-21 minutes for a 5km and others who are "trying their hardest" can't. It is because you are lean, healthy, male who used to run a lot more than most people do. You also ran during a formative time of your development. Your "baseline" is always going to be different from someone who is getting off the couch and getting into running.
Finally, I swam like two years in high school and was terrible. But guess what, because of the skills I learned then, I am still better than most everyone who was never a swimmer. It is just a skill-set that we learned at the right age. Among swimmers, I am the worst. Among people who aren't swimmers, I am pretty good. It sounds like you are in the same situation as a runner.
p.s. 4:40 (barely sub-5 minute mile) is also TERRIBLE for a competitive runner but AMAZING for a non-runner. So you slot nicely into the no-mans-land of ability.