In the 60s, 70s, and 80s there were a lot more serious runners (people with 40 hour per week jobs) who trained and raced for fun and the love of running.
I am not sure when it has dropped off so dramatically in the past 20 years or so. Many of the races have now become a social event and people just run to finish.
Also with the elites making money for the top finishers, many good runners just are not willing to do the work or do not have the time to train to get the good results.
I have experienced the same things you stated. I used to run on the low 2:20s in the Marathon and could not crack the top 100 in the big races like Boston for example or NYC. Now my times would put me in the top 30 or so.