I came to the sport late as a walk on my freshman year of college and ran good enough times (14:19, 30:22) by my senior year to think that OT in the marathon would be a reasonable (although not necessarily attainable) goal to shoot for, and maybe even in the 5000 (this was back in the 90s, the world of sub 14's was a lot smaller) given that I was still developing as a relatively new runner.
But life got in the way and I didn't really pursue running beyond my first year after college despite showing some improvement that year (taking 3000m time down from 8:32 to 8:20).
On the one hand, I really don't regret not moving to a more running friendly place and seeing what my potential was - I kind of already know my best case upside was OT in the marathon, with an outside, but pretty damn unlikely, shot at an OT in the 5000 or 10,000. I certainly wasn't making any national teams and I wasn't ever going to earn more than a couple of grand a year in prize money.
No, I don't regret it from a running perspective at all. But, as a 45 year old entrenched in a career that provides pretty good financial security but only marginal joy, I will say that there is absolutely no rush to get here as long as you have a plan in place to get here eventually should that be your desire.
Taking time off to chase your dreams is worthwhile, but don't completely forfeit your future. Think about what kind of employment you will be suited for if you take 3 or 4 years working part time at a running store or waiting tables while you chase your dreams. Will you be able to get your career on track at that point? If you want to go to law school or grad school in the future and are comfortable that you can get in, or if you have connections that can get you into an entry level professional position in the future, that is one thing. But if you are going to be putting together a resume at age 28 or 29, and you have a degree in history and then several years of intermittent, inapplicable work experience, that is another. There is nothing wrong with waiting tables (my dad did it for years), but you need to factor the challenge of getting into the professional world (if that is the end goal) into the calculation when you do that math today.