Mixed-sex sports have special rules specifically for them. Once you introduce a male into a women's league, it is no longer a women's league. It's mixed-sex, and instantly becomes less fair for female players. If the sport involves contact or combat, introducing a male instantly makes it less physically safe for female players too - and this in turn changes the situation in terms of liability and insurance coverage. The whole tenor, dynamic and spirit of the game often changes, too.
Plus, most girls over 8 or so and women don't want to share locker rooms, showers and toilets with males regardless of what kind of "gender identity" the males say they have or how feminine or masculine their "gender expression" is.
Who are you to say, "Nobody should get told they can't do the women's community soccer league because they're trans lol. Who cares?" Have you been involved in doing the hard work of founding or running a women's community sports league? Do you know anything about the duty of sports organizers to insure all participants are safe? Have you done risk assessments showing that males can be introduced into female leagues without endangering the females? Where do you get off thinking it's up to you to decide that women should no longer be allowed to have our own leagues? Your blithe "Who cares?" is everyday causal misogyny writ large.
Even in causal sports and community league sports, I don't agree with you that physiology as determined by sex, age and disability do not matter much or at all really because I dispute your contention that "the overwhelming majority of sports is not done in a performance oriented lens." Sure, when lobbing a tennis ball back and forth, going for a run, riding bikes, playing pickup basketball in the driveway, going kayaking etc, winning isn't always the point. Sometimes the aim is just fun, exercise and social activity. But when playing sports where scores are kept, even in casual settings like driveway hoops, and especially in community league competition, most people still want to play well and perform their best, and most people get a kick out of doing well, pulling off a particularly good move, and very much enjoy winning.
Also, safety always matters - and in collisions, accidental hits, intentional tackles and combat between adolescent or adult males and females, the females bear a disproportionate risk by far.
BTW, here is a thread of women discussing what happens when a couple of trans-identified males joined a field hockey league that was set up to be single-sex for females only. It shows that the answer to your question, "Who cares about the incursion of some males into women's community league sports?" is "many women do,"
https://www.mumsnet.com/Talk/womens_rights/4407004-Transwomen-in-womens-sport-advice-sought