My response will be disagreed upon by many, but that's ok.
Some bodies are just less durable than others. I think this mainly has to do with recovery ability (ability to inflame, repair, and subsequently reduce inflammation) of muscles and connective tissues. That can be trained to improve, but is at least partially genetic. I used to do EVERYTHING right--exercise, sleep, nutrition, recovery like foam rolling, etc, and anything over 10 would just zonk me. This is after being a runner for 14 years. I could do longer runs--but I had to book several days of recovery time afterwards.
Now I am more into indoor climbing. I am running into the same thing. My muscle and connective tissue recovery is just slow. Hard workouts linger in my tissues longer than they seem to for others.
I would imagine part of it may be differing natural levels of HGH and testosterone. There are reasons those drugs work so well to help athletes recover and train.