I don't think it matters one bit how we measure road speeds, volumes of gasoline, or the weight of a bag of potato chips. Things like that are what most people associate with the imperial/metric situation, but it only really matters in the sciences where you're actually performing calculations with these numbers. And a lot of science and engineering work in the US is done using the SI system. Of course there are a lot of imperial based standards and other pain in the ass to change things that are still used, and certain fields stick to imperial units just due to tradition. But I think we're getting there slowly.
And the whole math skills thing is silly. For anyone who will be going into a scientific field (i.e. the only people who really need to have a strong grasp on the use of units systems), learning, using, and converting between unit systems is trivial. Yeah so a spacecraft crashed, shit happens, millions of other projects involving multiple unit systems end up just fine.