Arbitration is a private dispute resolution mechanism governed primarily by the agreement of the parties. Although the law governing these agreements varies by jurisdiction, the common thread throughout the world is that it is exceedingly difficult to get an arbitration decision overturned in a court. It's not like appealing a decision in a trial court, where the court of appeals will decide for itself whether the trial court was correct. Ignoring the Swiss court isn't a realistic option because most jurisdictions respect the terms of foreign arbitration agreements. If the IAAF tried to ban Semenya from competing outside of Switzerland, a court in the country hosting the meet would almost certainly grant her an injunction.
CAS decisions can only be overturned on the following grounds: (i) the arbitral tribunal was not properly constituted, (ii) the arbitral tribunal wrongly accepted or declined jurisdiction, (iii) the arbitral tribunal ruled beyond the claims submitted to it, or failed to decide one or more of the claims submitted to it, (iv) the parties’ fundamental procedural rights were breached, or (v) the award is incompatible with public policy.
All of these categories have been fleshed out through many, many court decisions, so they aren't nearly as vague as they sound. For instance, the public policy ground sounds like the court has broad permission to reverse any decision that's unfair, but it's actually an extraordinarily high bar. Basically, a decision violates public policy only if it clearly violates widely accepted legal norms. Frankly, I have a very hard time seeing how such grounds could be demonstrated in this case. If there's one thing that everyone who has talked about this case seems to agree on, it's that "this is a very difficult issue." That's exactly the kind of situation where a court should not be second guessing the merits of the arbitral panel's decision.
If the Swiss court vacates the CAS decision, then the CAS gets another bite at the apple. The court will not actually make a binding decision on the parties. Though, as a practical matter, the court's decision vacating an award may leave the CAS with very few options on remand.