One downside, as I pretty much constantly run hills also, is that you will be really good at running hills (both up and down). That may sound like a joke, but it isn't, at least not completely. The potential problem is that when you are in a race and run up a hill, you may actually pick up your pace naturally and without realizing it. I do this all the time. On my training runs, I almost always start increasing my pace on uphills.
Anyway, you will have to watch yourself in races to make sure you don't mess up your pacing on the hills. I've done it more than once. I have two very hilly marathons I do a year. I've blown up several times by not taking it easy on the uphill portions on the first half.
Another potential problem is when your races have substantial flat portions. The last 14k or so of one of the marathons I run is mostly pancake flat. If you are tired and trying to run flat areas without training on flat areas, you'll really, really struggle. Last year I included some of my long runs on the track (mind-numbingly boring), just for this purpose, and it made a huge difference in that last 14k.