A permanent location does not necessarily solve all the issues. The main problem with the modern games is that it is just so huge that it takes billions to get all the venues ready. If you repeat the games every four years, the permanent host has to be able to maintain the venues in the intervening years. That is a very tall task given that most of the Olympic sports are not regular spectator sports.
The solution may be to have the games take place over a longer period of time to allow venues to be reused over and over. Basketball, volleyball, gymnastics, handball, and other indoor stadium sports end up in multiple venues at the Olympics. They should all be able to be held in maybe two or three existing venues (pro basketball arena, civic center and college sports arena). Likewise, soccer, field hockey, and track and field should all be held in the main stadium and possibly one or two smaller stadiums. But instead, hosts have to build a dedicated volleyball arena and extra fields for outdoor sports. These venues almost always end up mothballed following the games.
Also, stringing out the games over a longer period would reduce the need for athlete housing, security and take pressure of local hotels and restaurants.