The biggest difference between a 401k and a pension is who assumes the risk and what happens when the person dies.
Take two scenarios where you put the exact same amount of money into a pension vs a 401k for the exact same amount of years:
The pension fund would calculate a lifetime annuity based on what is expected to go in and how the fund can grow the money. You know what you are going to get.
The pension fund also pools several funds together to mitigate risk.
If the person dies early or markets are better than expected, the pension would pay out less than it took in plus market gains on that money. The excess goes into the fund.
If the person lives longer than expected or markets don't do as well, the pension would pay out more for that person than it took in (plus market gains). It would use the dead guy's money to pay for the shortage.
In the 401k, you have to budget and manage it yourself. If you live longer than expected or markets are bad, tough luck, you run out of money.
If markets do better, you gain.
If you die young then good news! You didn't outlive your money and your estate gets to keep the overage.
Pensions are better on a macro level but you still have the risk of them being mismanaged and affecting a large number of people.
People like the 401k because they don't trust others with their money.
Companies like the 401k because they are simple and don't have long term risk to worry about.
Social Security acts like a pension plan and is a comforting safety net so we as a society don't have as much to worry about with elders running out of money.