I first started running when I was around 14, and ran in Brooks, Asics, and some Nike shoes. When I was 16, I decided to transition to minimalist shoes.
In my experience, some parts of my calf and my Achilles were really sore. I adjusted by first wearing them just during warm-ups and cooldowns, then eventually by doing workouts, and then recovery runs in them. This was for when I was using one of those New Balance Minimus trail shoes. The whole process took me about 2 months to go from no experience in the shoes to running in only minimalist shoes.
I also had some experience running with FiveFingers. It's like you said, probably the hardest thing about them was the impact of running on pavement all the time (I was still in high school). Also, you might have had some issues regarding the use (or lack of) socks with these shoes. I only ran in them for a season.
Altras are definitely a good shoe. I've been running in Saucony Kinvara's for the last few years, and was thinking about going back to Altras when my current pair is worn down.
All that being said, I had some friends who jumped straight into minimalist shoes with no transition who didn't get hurt. So, it might all just depend on your form and experience.