I basically quit at 20 with mediocre PR's of 1:58.8y and 15:45 for 5k. Ran very little for 10-15 years and gained 25 pounds. Started doing a little to get in shape in my early 30's, and started racing a bit at around 35. I'm 41 now and have improved every year since, but missed most of this year with an achilles problem. My goal has been to get to the same level I was for 5k, and since I seem to be much stronger with age, to be even better at longer distances.
The biggest problem is injuries. Every time I think I am about to get to a new level, something gives out. I've missed several months virtually every year -- achilles, IT band, hamstring all have given me fits. It then takes longer each time to get back to where I was.
Anyway, I have figured that I have maybe 8-10 years from when I started competing again to improve, so the next year or two may be the peak. Would still like to get under 16:00 again for 5k and in the 2:30's for a marathon before I'm too old.
If you were better than me to begin with, your upside is probably greater now as well.
Good luck.