It's all about the number of beds!
The IOC along with the host city agree on the number of beds that would be required to house the athletes, coaches, etc.
Then working with the different international sports federations, the IOC determines the number of entries each sport will be allowed. The sports federations then go back and formulate the standards to hopefully not exceed that limit. This is easy for team sports but very difficult for a sport like athletics.
There has been much talk over the years to add other sports to the Olympic program that would attract a younger audience. This would make it difficult for athletics to add any additional events to the program. The only realistic thing we could do would be to add new events with athletes who are already there.