You gotta respect the distance...marathon, 50k, 100k, 100 miles.
I ran my last 100k and totally crashed and burned. If you can imagine the bonk in a marathon after 20 miles you can imagine extending that for hours and hours longer! In the mountain/ultra trail races I've done you will get reduced to the point of walking...Max King (2:14 marathoner, 8:30 3k SC) walks. Rob Krar (3:44 1500m runner) walks. You have to stop and walk through the aid stations to eat and drink! You have to either slow down or stop to piss! Those things factor into the overall pace....
There are many variables at play when running 4, 6, 8, or even 12 or 16 hours. I haven't run 100 miles but I can see that the the chance of something going wrong increasing exponentially with distance.
That being said I can see a difference of running an ultra on a road vs a track (and there are all different records for these things). A lot of the ultra world doesn't care about Josh Cox running a fast 50km on a marathon road + 8km track course. But of course the numbers aren't the same for Olympic distance events. How many people go after a fast marathon time on a track?
But every distance event is hard to do and requires a lot of suffering. Making the USATF 100km team is hard because the qualification standards always favor returning runners (ie top 4 from the top 10 each year). In ultra running it's a totally different world than the xc, track and road racing scene events of 26.2 and shorter. I've seen both sides of these worlds and there are probably always going to be some strong opinions/disagreements on performances.