You may be right. He was such a great swimmer that, even if he had not made it to the 2008 Olympics for some reason, he was already the greatest swimmer of all time even before that triumph.
You see the greatest athlete of all time? You could make that case. Most non-swimmers, which is almost everyone, cannot really fathom how difficult it is to do what he did, and how unprecedented it really was. Swimming is probably the most difficult sport there is, with the possible exception of combat sports where your opponent is trying to kill you. Swimming is a high conditioning and high skill sport, and those are very rare.