You're right, we'll disagree. That absence of negative forces is quite a leg up, relative to the situation of many in this country. I'll also disagree that your privilege was not active. You were encouraged to read. You were taught to value college. I'm a teacher, and I can tell you that teaching is active.
Your construction implies that we are born knowing what is right and what is wrong, or that going to college is a virtue, or that reading is important: it takes some kind of "impediment" for us to miss this knowledge. This is incorrect. You are the beneficiary of your parents' wisdom, and the teaching, however subtle, they provided it to you. This is as much--or, I think you might agree--more of a privilege than simply being born with money.
I live in "Indian Country." This past weekend their was a powwow in our regional city (about 150 miles away). All weekend long, at the edges of the reservation, there was a noticeable increase in the number of speed traps and state highwaymen. This makes some sense: there is no extradition agreement between the tribe and state, and pulling over and checking the ID's of Indians for minor infractions is an effective way to find and collect wanted criminals. However statistically valid this practice is, it is also racist.
This is anecdotal and proves nothing, but: Once I was returning to my home on the reservation, and was stopped at a roadblock on the state side of the border. I was asked two questions ("Where are you going?" and "Do you have anything in the car we should know about?"), and then sent on my way without so much as having my ID checked. I obviously can't prove that had I been Indian, my ID would have been checked, but I am pretty certain that it would have been.
I am not suggesting the whole racist nation is against them, or that success in the current system is possible. I wouldn't be teaching where I teach if I did.
Let's go back to what I said about "parents' wisdom" above. That wisdom--that education is important, and that success if possible, and that following the rules is important--HAS to be taught somewhere. It doesn't appear magically. THAT is the privilege you have received and others haven't. At the moment there are many that are caught in a cycle where no one around them has that wisdom, and they never learn it.
Then you'd better lock everyone up with a life sentence. I have to punish students in every day. In my experience, punishment alone will not cause someone to change their actions--in fact, it will likely only aggravate them. They need to be taught.
If everyone continues to focus on "theirs," then the cycle continues. Privilege stays in one spot.
At the very least, it is a financial drain on you as a tax burden. I think it is my problem in less superficial ways, but that's just me.
Right: you won't do anything to fix the problem.
I say again: not always an option. If you have no money, you cannot move. If your parents don't want to move, you cannot move (okay, you CAN, but let's stay reasonable here). And if moving has never been shown to you as a reasonable option, you will not move.