They could have gone after him for trying to personally profit from the presidency on day one. He quite flagrantly did not properly divest himself from his business interests. And, he could very plausibly been impeached for obstruction of justice during the Mueller investigation (wasn't it John Bolton who said he could "personally attest" that attempts to obstruct justice were routine in the Trump White House?).
As for the impeachments, that they took as long as they did, and did not lead to conviction in the Senate, in spite of having been based on extremely strong evidence, says as much about wariness around setting risky precedents wrt to holding presidents to legal account as it does about partisanship.
At every juncture of the Trump drama, there were Democrats who expressed extreme wariness about going after any President, even one who was all but daring them to, lest it break with the precedent of almost carte blanche political impunity for senior US leaders, and thus endanger national stability. This is exactly what the leadership on both sides are referring to when they say that only the electorate should be allowed to punish a rogue president.
What enrages Trump about the predicament he's in is not at all any belief that he's been falsely accused. I don't think he operates in terms of truth or falsity, but rather winning and losing. What drives him crazy is the double standard he believes is being applied to him as President (e.g. recall his response to Bill O'Reilly's comment about V. Putin being a "killer": "We have killers too, you know".) You see some of this same outrage in the comments of people who support him even in this thread. When he's being a sober realist about the US and its politics (rare, but not unheard of), Trump is often not wrong.
But what Trump and his supporters have lost sight of is that, cynical as they are, there eventually ARE limits to what the American political establishment-- and probably electorate, too, but we'll see-- will allow its leadership to do. And he, as has been his wont all his life, has blown well past them.