They did bring in the Borg, but like Alan said, there was no connection to those Borg and Earth or the rest of the Federation. Compare Voyager's Borg episodes with the two part episode on Next Generation where the Borg are headed for Earth, assimilate Picard, and wipe out forty starships. To me there was just no comparison in terms of urgency or consequences.
I also never found any of the Voyager characters as interesting as the characters on other Star Trek incarnations. I thought they were trying too hard to make the characters interesting rather than letting them develop into someone interesting or likable.
And I think you're absolutely right about the time travel thing. Used occasionally, I think it happened 2-3 times each on both the original series and The Next Generation, it's kind of interesting and novel. But using it as a story line regularly just didn't work well on either Voyager or Enterprise. It does seem like a cheap way of coming up with a plot when you can't come up with anything else.