The reason you can't get your belly fat to go away probably has to do with two hormones: Insulin and Cortisol. Insulin is the homrone responsible for taking glucose out of your blood and into your cells (sometimes your fat cells) and cortisol is a stress hormone which promotes central obesity if its levels are too high for too long. When you go on extreme diets (eating next to nothing) your body reacts in two key ways to the extreme stress of starvation: It pumps out a bunch of stress hormones (cortisol being one) and it increases your cells' sensitivity to insulin. The overall effect is that once you DO eat, your cells are so glucose hungry that they hold on to every little calorie you give them as fat. The role of cortisol in this equation is that once insulin gives the message to store fat, cortisol makes sure that the fat is directed to your core. This equates to being able to gain weight off of VERY few calories. Good news if you are a starving cave man, bad news if you want to lose body fat.
Essentially, this is a game of blood sugar balance. You want to let your body know that it is okay to lose fat since you are not a starving cave man. To do this, you have to eat a little more. Trust me. I went from eating 1,500 calories a day and gaining weight to eating 2,500 calories a day and losing weight so fast that I couldn't believe it, especially in my stomach region.
This being said, if you want to get rid of your belly fat, try eating a little more protein and fat (yes, fat... especially olive oil which actually promotes FAT LOSS from your midsection) and less sugar, because sugar is the surest way to gain fat on your belly. A good rule of thumb is to try to eat things that are as close to their natural states as possible: non-processed meat, eggs, vegetables, fruit, oatmeal, rice, most dairy products, etc. You don't have to go completely natural, but aiming for a majority of the food you consume to be unprocessed is a good start.
Next, make sure that your blood sugar never gets too low during the day. Eat a decent breakfast (doesn't have to be huge or anything, just something, even a banana.) If you feel hungry, EAT! Just make sure that you are eating good, whole foods. A good rule of thumb here is to eat something every three hours. Not necessarily a meal, but just something. Walnuts, string cheese, yogurt, fruit, etc. are all good snacks. You want to keep you blood sugar steady so that your body uses it for fuel instead of storing it as fat.
Conversely, if you don't feel hungry, don't eat. Your body will learn to regulate itself and you won't have to starve yourself to stay trim.
Lastly, don't worry too much! Eat enough to stay comfortable most of the time and you will be surprised at how much you don't think about food. You will also be very pleased with your fat loss.
For further reading, I would suggest a book called "The Schwarzbein Principle." I learned a lot when I read it and it is not a diet book like others in that it is not some silly quick-fix weight loss tool. It tells you how to modify your nutrition and exercise in a sensible and manageable way. No cutting out entire food groups or anything.
Anyway, I hope this helps!