I have no idea who is the "toughest." What does that even me? Each branch specializes in different things. They aren't necessarily comparable. Using something like a knife fight to compare seems silly.
I've never been in the military, but have had coworkers (I am a firefighter now) who did quite a bit of time in the service. My one friend, a former Marine, said that ANGLICO impressed him the most (note: I don't even know what ANGLICO is). Another friend, a badass in his own right was passed up for Recon (not quite the level of "Force Recon").
Does anyone know if it is possible to get into a special forces unit directly as a civilian? As in, when you enlist you know you are going straight to special forces training after your basic training?