Rojo, I pretty much agree with everything you wrote.
I think the strongest argument for the other side goes something like this:
1. There are too few XY women to be an existential threat to women's athletics.
2. Although women's sport has to be defined by biological sex, we don't have a perfect definition of biological sex, and we don't have 100% certainty regarding what aspects of sex actually confer advantages and to what degree.
3. Therefore, before causing possible social collateral damage (this seems to be what the reporter is concerned about), we should err on the side of overinclusiveness in women's athletics.
4. And finally, as you note, competing in the Olympics is not a human right. Accordingly, the people who miss out on opportunities to medal are not having their rights violated. And although sport itself is a human right, the effect of XY women on sport below the elite level is likely to be even more minimal.
On the other hand, I'm of the opinion that the integrity of our sport is very fragile, and that we should never knowingly make a decision that is contrary to the ethos of the sport simply because it serves some other worthy goal. But a lot of people won't agree with that, because a lot of people truly don't understand what athletics is about.