There are lots of variable involved here.
One of those variables is what the purpose of the workout is.
As a guy who rarely races anything under 5k, the purpose of my doing 400s is to improve running economy by running relatively fast. Since my goal is to improve running economy, feeling recovered enough such that I maintain good form for each rep is not only important, but it is the entire point of the workout for me. So I take enough time such that I feel comfortable that my form for the next rep will not be impacted by fatigue. That usually means somewhere between 2 and 3 minutes. For other runners who race other distances, they may be looking to get something else out of a 400m reps workout, so it may be important for them to take less rest.
As to mile repeats, I tend to take 3:30 to 4:00 of rest. My goal for mile repeats is to get in as much work at my vo2max pace as is possible without beating my body up too much. Miles at 5k pace are hard, and they get your hear rate up pretty quickly (probably takes about 2-3 minutes for your heart rate to get to the max range at 5k pace early in the interval session) and keep it up after you finish your intervals pretty good too. Also, being in my mid 40s, my heart rate tends to jump back up pretty quickly on subsequent intervals, so while the rest may seem long to you, there is very little downside for me since I am getting back into the right zone pretty quickly as the workout progresses.
With vo2max intervals, getting back into the right range quickly can be facilitated by using active rest during the recovery periods. This helps maintain a relatively elevated heart rate, meaning that you will get back up to the desired range more quickly than if you took passive rest.
On cruise intervals at tempo pace, however, I would take short periods of passive rest. Again, the purpose of the workout is the driver here. The purpose of this workout for me is to allow my body to adapt to the buildup of the byproducts associated with lactic acid. Since lactic acid and related byproducts clear relatively quickly, and since even light jogging facilitates their prompt clearance, I avoid jogging and keep the rest short so as to maintain as high a level of byproducts as possible.
So, basically, duration and nature of rest really depends on what you are trying to get out of a workout and what your personal circumstances are.