I am indeed an old-timer. I have began a fan of the Velvets, but it did take awhile for their impact to be felt. They were well outside the mainstream in their time, to say the least, and it wasn't really until Lou Reed hooked up with David Bowie and put out "Walk on the Wild Side" that he had huge commercial impact. I mean...the Velvets weren't exactly part of the countercultural mainstream (itself a contradiction in terms, I guess) in their time, not even in New York City.
As much as I like them, I do consider them over-rated. Their body of work is not very large. As far as "influence" goes, yes, Reed's songwriting had some and so did John Cale's sonics, but there don't seem to be any prominent Nico imitators... which is just as well.
Music history as conveyed by the Rock 'n' Roll Hall of Fame is misleading and even dishonest. Many important acts are ignored while others are deified by a small cadre of rock critics and industry people who now dictate such things. You're going to have to explain to me why Arthur Lee and Love, whose "Forever Changes" is probably the best album of that late '60s - early '70s era, and who influenced the likes of Robert Plant, are not in the Hall.
There's also no getting around the fact that the Warhol scene was extremely, extremely creepy, though the truth about that is also being rewritten after that fact.