I realize it's increasingly popular to attribute all differences between groups to "racism" and the like, but this is the most intellectually lazy analysis possible. Others can play "name that residual" too--many people say all differences in average group outcomes are genetic. Their analysis is no lazier than yours, regardless of how much encouragement you get from your echo chamber.
There are people who devote careers to studying these things, and the factors behind these differences are complicated and manifold. If you genuinely want to learn about them, the economic literature is out there, and largely free to access. But the reasoning you are applying is really poor. The more people who think this way, the farther we as a society will be from solving our problems.