What do you actually do? That's a huge factor and would help a lot in helping to gauge your situation.
ie. You sound like you probably do bush work or mining of some sort but there is a HUGE difference.
For example; If you are a geologist or accountant required to grind it out at a mine site I would say stick with it! Even if the job and colleagues are horrible you can likely network in the offseason and find a great transitional opportunity down the line.
BUT
if you're doing tree planting or pipefitting etc. I would maybe urge you to start looking for something else ASAP as the longer you stay the more you screw your chances to transition later.
Please let us know what you do. You're an anon poster!