The job didn't hurt me until I had kids. Once you have kids, it really, really makes an impact.
Even just married, things were fine because I would run at 6am and get home around the time my wife was getting out of bed. We'd have coffee and talk before getting ready. After work I would run again while she worked out and we'd chill at night. No big deal.
Once a kid arrives, things change. You no longer have control over your free time, so it becomes precious and you begin to realize that a chill morning holding the baby so mommy can sleep is much more important that a run. Fast forward to having 3 kids and you've got meals to make, soccer games to attend (coach), diapers to change, dogs to feed, and all kinds of crap that just take up your time.
Plus, who are you kidding with the 8-5 business...? Are you doing shift work? No job is 8-5 these days.