Downhill Training?

It feels like you are secretly an ultrarunner asking about your 50K prep. But, yes, downhill training on proper terrain is beneficial.

Yes, it's trainable. You lack specific downhill eccentric leg strength to control your legs and thus not put your knees in danger.

I'm good at downhills, sometimes literally twice as fast as the runners holding back like you describe for yourself on steeper stuff, but level with others who are also fast downhill. But I will be bad at it/not confident without specific strength after it gets detrained (such as after skiing and flat treadmill all winter). There's technique too, but specific strength is necessary and probably most of it.

It really shouldn't take that much work if people otherwise at your level are blowing by you on xc downhills. My first uphill/downhill mountain training run every season, I take the downhill as slow as possible to not destroy my legs (typically~2,000-3500 ft. descending). They will still be sore for a week, but after they recover from just that one session, they are noticably hardened (and won't be as sore or not sore at all 2-3 sessions like that afterwards). You can and probably should progress it in a different way for cross country, such as by blasting shorter sections on less steep downhills on rolling training runs. My example is to show that you need to get sore at least once to build that downhill leg strength, and it does make a difference.