I think that cross country skiing could be the hardest. I xc ski and run at a high level. Both sports are equally demanding like any endurance sport (i.e. triathlon, swimming). Cross country requires three factors: major aerobic engine (similar to marathon running), high power/strength (similar to 400/800m runners) and flawless technique (i.e. swimming).
The part that makes xc skiing "easier" to make Olympics than running is that less of the world's population is on skis, smaller chance of finding a freak like Bolt or Kimetto. The top level of performance is not equivalent to a 2:03 marathon, but it is still extremely high. From my racing experience in both sports, I would guess 2:06/7 on a relative scale.
The main factor is money. For example, the US has a half dozen amazing skiers that can turn in top 10 world cup, Olympic, WC performances at peak form (level of Rupp, Ritz, Jenny Simpson, Simmonds, Hall). They make practically no money. Very little sponsorship and without the help of the USST, they would be completely unable to compete.
Our best is Kikkan Randall. She is ranked as number one sprinter in the world and is a recent world champion. She might earn as much as Ritz through sponsorship (Subway restaurant), but she doesn't get free world class facilities, support and coaching like he does.
This makes reaching the Olympics in XC really hard. You have to commit to being completely broke for over a decade and even if you make it to top 10 finishes at the Olympics, you still wont be making a six figure salary.
In running, you can usually pull down a good living if you can even qualify for the Olympics or finish top 10 in a major race (i.e. marathon). Except for Hartman!
This is different if you are Norweigian, Swedish or German. These countries love their skiers and their sponsorship deals are significant.
Peace.