It can get hard on your when you are trying to get lighter than your body wants to be.
I'm curious, what is your training like? If you could possibly up your miles over the winter, you might find you naturally drop a few pounds. Also, cutting down on alcohol (if that is a thing for you) can easily lead to another couple pounds dropped.
If you want to go for it, don't be drastic with the change. Being healthy and a tad heavier always trumps being skeletal and breaking down.