This is precisely the reason it's hard to draw conclusions about the effectiveness, or not, of altitude training.
There have been "experts" who'll tell you that you have to re-acclimate to low altitude running after a stretch at altitude. One reason given for Ron Hill's "failure" to win the marathon at Munich was that he stayed too long at altitude prior to the race. But no one knows what would have happened if he'd come down sooner, or even later.
In your case I'm not sure how much training effect you'd get from three weeks at altitude. Maybe you got quite a lot and just need to have more time to re-acclimate, though after only three weeks I wouldn't think it's an issue. And there are so many other possible variables in play here that I don't see how you'd begin to isolate them. Here are just a few examples:
Did you train at altitude pretty much like you did at sea level? If so, the added stress of altitude running may have worn you down and that's manifesting itself now. Did you change your life routine at altitude? Maybe that wore you down. Maybe you've gone into your races expecting to have acquired some miraculous level of fitness at altitude and went out harder than you can sustain. Maybe you got slightly anemic or dehydrated at altitude. Maybe you're reacting poorly to the business of travelling to altitude and back.
At this point, if altitude really did make you worse, the only solution I see is to wait it out. I doubt that three weeks there did any permanent damage. But here's another tricky question. Let's say that in two months you're busting PRs like mad. Will you attribute that success to having spent time at altitude or will you wonder how much faster you'd be if you hadn't gone to altitude and had it make you worse?