You're 100% right, but its cause they are older. Thats the only reason.
You know The Band had this song called "Rockin Chair" where Richard Manuel used to sing about how he was "pushing age 73." It was a great song when they started out, but by the time of the Last Waltz, Richard Manuels voice had gone to shit from all of his hard living, and it actually made the song better because he really sounded like he was pushing age 73 when he sang it.
Also, I would say The Allman Brothers Band is an exception to this rule, they are probably as good as they have ever been.
Also, most rockstars live really hard lives, and it causes them to age MUCH faster, so bands like the Stones seem older than they really are. I mean its a miracle that Keith Richards is even still alive.