I try and avoid threads on the vaccine but sometimes hard.
First, a good point on listening to Joe speak on something you yourself know well. On many things it suggests he has crammed for the podcast by reading everything he could find on Wikipedia. I listen and usually enjoy the physiology/sports guests - or did when on YouTube. Entertaining.
On the OP take: 18-49 year old females with comorbidities are not who he is talking about. Anyone with comorbidities cannot fit the bill of not having to worry, and 18-49 is too large a range to be meaningful. He specifically said 20s, and healthy. He clearly has in mind active people with zero health issues.
You can tell he also does believe that it's safe to vaccinate, he just never wants to tie himself to anything and is used to couching things in terms like "for the most part". So I don't think there's anything controversial at all on his view - just that in his personal opinion, a 20-year old (say an athlete) could maybe go without. And tbh, they probably could, but they'd probably be fine with getting it too.
On safety: there is a world of difference between being "on a drug", and taking a once-off vaccine. The mRNA technology is not 2019-new as some make out, and the vaccine technology for respiratory viruses, on which Covid-19 vaccines is based, is not new. It is all well understood and numerous primate studies over past decades. The barriers to most normal research studies is always 1) funding 2) back and forth with phraseology and formatting of ethical applications - months often 3) recruitment phases and barriers due to lack of public interest/knowledge/incentive 4) running consecutive cohorts 5) time spent in writing journal studies and barriers to dissemination and publication - for those who aren't involved and who don't know, it can cost thousands for journal publication in some prestigious cases 6) KTP or knowledge transfer from study completion to industry.
For Covid-19, #2 probably still took a little time and was closest to normal processes of all of these, ironically as this is what most are concerned with, #3 was initially uncertain but actually surprisingly good, #4 was overcome by allowance for staggered cohorts - with all adverse events monitored. Thankfully there were few. Everything else as a barrier was removed.
Re: public side-effect fears etc, read up on the initial response by the public to the smallpox/cowpox vaccine. Very similar. And very quickly addressed for the level of knowledge at the time.