Turner Syndrome, AIS, "hermaphroditism", all sorts of conditions with supporting scientific literature that concern individuals who do not neatly conform to the traditional male/female binary. In the case of AIS, there are complete, mild, and partial forms, all with varying degrees of masculinization. In other words, not all intersex individuals with XY are created equal; there are traits AIS individuals might have that could give an advantage, but there are some that don't. Also, there are certainly XX individuals with elevated testosterone, what about them?
You acknowledge that there are individuals with intersex traits that make it difficult to identify them as male or female and yet you reject the spectrum view of gender? Look, you don't have to understand gender as a spectrum, but I don't see how anyone can reasonably say that there are males and females and the rest is more or less conspiracy.