It is easy to pick out the true middle-aged people who are married with kids in this thread. They speak the truth. Most of it comes down to a lack of "me" time and the evolution of life that creeps up on you.
It is super easy to say, "well I would never let my work impact my health," but then you find yourself with a mortgage, kids' sports fees, a wife asking for a vacation, a car that needs replacing, and the list goes on.... It's not like you can wake up at 40 and decide to be a free spirit one day. It would be extremely selfish and narcissistic to tell your family they have to go without so that you can prioritize working out (at middle age, especially).
I ran in college for an ACC school and did pretty well (not great), but enough to carry it into a serious marathon focus after graduation. I had a good job that was demanding, but as a young, single guy it was easy to knock out the 100 mpw I was running at the time. I got married relatively young (24) and my wife liked to work out as well. She wasn't a runner runner, but she ran or went to the gym every day. Still, she didn't understand why I needed to double or run long when I was no longer on a team. 
There was friction at times, but we talked about it a lot and her brother inadvertently helped me out and honestly changed my life for the better with one offhand statement. We were sitting around one afternoon at her parents' house during the holidays and I said, "Well, I am going to go run so I can be ready for dinner tonight." I had run that morning, and my wife wanted me to skip my afternoon run. My brother-in-law looked at my wife and said, "If there is one thing I know is true in life, it is that smokers are gonna smoke, and runners are gonna run. Be glad you got a runner." Most women are going to listen to their older brothers and respect their opinions, especially when they're still young. My brother isn't a runner at all, but he is hyper successful and just a great dude. His words meant a lot to my wife.
Now fast-forward 25 years and I am no longer running 100 mpw but I wish I could. I love running and working out. Love it. But my work phone starts blowing up at 7am and goes non-stop until 10pm. I travel for work at least one week each quarter (not a lot, I know). And all three of my kids are highly athletic. As any parent of sporty teens knows, this means spending a bunch of time on sports fields, tracks, gyms, and - worst of all - in the car. Even if you're not doing overnight sports travel, there are practices and games all over the place. Add in school sports and it is a constant game of carpools, logistics, etc. I have one that drives herself, but we still want to watch her play. She goes to college next year and I am going to miss the soccer games, track meets, and tennis matches. I WANT to be there to watch her. So even on the weekends when work slows way down, it is really hard to find more than 45 minutes to knock out a run. I just gets hard, even if the will and the drive and the physical ability remain. In short, your own fitness becomes less of a virtue and more of a vice, at least for me. I stay fit, but not as fit as I would like. 
Somehow, my wife remains super fit without working out a ton. Lots of running with the dog and combining activity with general chores.