Relate to that feeling big time. Can only share what I know. First order of business that helped me: I had to work through why random people's judgments and opinions mattered so much to me. I was a skinny runner in high school, and for a variety of other reasons people often assumed I was gay. Rather than confront my emotions around that, I ran the other way. Gained weight in college and lifted weights purely to look more like a "typical" guy. Was satisfied because people started seeing me the way I wanted to be seen, but I felt hollow going to the gym purely because I was worried about people making the wrong conclusion about me.
Post-college: It all crumbled. Gave up on the aesthetic pursuits. Started focusing on what I WANTED out of life, rather than what I was hoping to avoid, or what was expected of me. Stopped doing strength training except as it relates to the sports I want to do - running and hiking. Embraced being skinny and looking like a runner. Definitely went through some weird body image shifts. Felt alien in my own skin for a while. Now I don't have any notion of being skinnier than average, because average is defined by desk jockeys who go to the gym 3-4 times a week to add a little definition beneath their flab, and are afraid of looking stylish for fear they will violate the norm. Striving to look like the societally-normal template for any demographic is generally a waste of time - if it benefits you, it's because you're fitting in for its own sake, and you'll only be validated by fellow status quo-followers. I realized I didn't want that in the long term, because I had it for a while and it felt so pointless.
Once I realized that I didn't want to spend time and energy to fit in most of the time, I figured it was more important to look and act t he way I really am. And there will always be some macho man available to insinuate that I'm gay. I laugh in their faces now. I don't accept their viewpoint as "right" anymore. I don't even worry about what it means if he concludes I am gay - oh no, a stranger mistook a piece of my identity! I just laugh in his (likely closeted) face. When you're a prisoner of society's expectations, it's infuriating to see people walking around doing whatever they want.