He's been accused of attempted rape and murder ("I thought he would accidentally kill me" --in Ford's mousy up-speak voice). If people take these accusations seriously, he can't just go back to being a judge on the DC court. The Dems raised the stakes so high that it was basically make or break for Kavanuagh. This wasn't just joking about a pubic hair in a coke. The Dems where using full scale political terrorist tactics here. If you don't get what the big deal was, you weren't paying attention.
Further, if people took this accusation seriously and it prevented him from getting the job, then it means anyone can be accused of anything WITHOUT ANY EVIDENCE at anytime without any protection of due process, and the mere presence of the accusation can accomplish it's purpose.
Here's an analogy. Let's say you are applying for a promotion at work. But then a female coworker says that 2 years ago you followed her into the restroom and groped her and forcibly kissed her. She says 2 men and a woman where in the hallway and saw you when you followed her into the restroom. She can't remember the exact date, but she knows it happened about 2 years ago. Unfortunately, your companies' security camera footage is only stored for 1 year, so there's no way to confirm or disprove her allegation. Also, the 2 men and 1 woman that she named as seeing you follow her into the restroom all say they don't remember seeing this incident at all. These 3 people are all called into HR and vow repeatedly that they never saw this happen. And the woman is a friend of the accuser. Further, let's say the accuser has already indicated, before the allegation was made, that she did not like you. Further, let's say this allegation came only after your company had had a staff meeting where they all talked about you getting the promotion, and the accuser did not make the accusation at that time, but only talked privately with a few of her friends about making the accusation.
In this situation, with this uncorroborated allegation, if your bosses decided to pass you over for the promotion, it would be a monumental injustice. Further, if they passed you over for the promotion because they believed the accuser, it would also mean they have to take disciplinary action against you, most likely firing you. And worst of all, it could set a precedent in your company that anyone can get anyone else fired as long as they make up a story that is impossible to disprove.
Of course my analogy is not a perfect match to Kavanagh's situation, no analogy is perfect. But it's pretty close to the same level of lack of supporting evidence, and the consequences to the accused and the institution, if it was believed.