You sound like you are pretty responsible. You just need to understand that in many ways the Credit Score is a game. To play the game well, you need to understand the rules so you can maximize your score.
Check out the link posted by NoFatty (http://en.wikipedia.org/wiki/VantageScore). It breaks down the components of your score.
There are some differences between the VantageScore and the FICO score, but they're more similar than different.
The biggest component is "Payment History". 32% of your score. With 3 late payments, you're likely getting killed here. 99% on-time doesn't cut it. Being timely from here on, and letting those late payments age and eventually fall off is the only option.
23% of Vantage score is "Credit Utilization". This is overall utilization (of which you are fine, at 5%...$2k of $40k available). Also though, high utilization of individual cards can hurt you. So, if that $2000 balance is more than 30%, 50%,70%, or 90% of that card's limit you are going to get hurt here. Paying the balance down, or increasing the limit on that card so you are using less than 30% of the limit should help your score.
13% is "Depth of Credit", or length of credit history. The older your active accounts, the better. If you closed older accounts and kept newer ones, you screwed yourself here. Also, it is often the average age of accounts, so if you have one old account, and 4 newer ones you will score poorly.
** FICO also gives you a higher score if you have a variety of credit in your history (credit card, car loan, mortgage, etc..). The more variety, the better for your score in this category.
10% is "Recent Credit". If you have any inquiries, or new accounts in the last 2 years, you'll hurt this part. The more inquiries, the more you get hit. For example, I think each new inquiry hit my FICO score by 2 pts each. 5 new inquiries dropped it 10 points.
7% is "Available Credit". This is unique to the Vantage Score, and not a factor in FICO. It looks like by having access to $40k of credit that you could spend tomorrow you are hurting yourself here too.