Here's my Hobbyjogger take on OP's question. Take with a grain of salt.
In HS XC our coach never ran us more than 30 mpw, and each year I gradually improved throughout the season. Then the season ended while I was still improving and I wasnt consistent enough to reach a peak or plateau. I wasnt the top guy on the team so nobody cared, and I just thought that I sucked.
Couple years later (I didn't run in college), I started running on my own to test my potential. After a few months of 40 mpw I caught up to my HS times, and a few months after that, started PRing. Granted, these were crappy PRs, brought on by lack of prior consistency. After a full year of 50-60 mpw, continued to make improvements and finally saw some plateau. I never did speedwork, but I did run an all-out mile every few weeks to assess fitness gains (going from 5:20 to 4:40 in a year, and 18:00 to 16:20 in the 5k).
Based on personal experience, I would argue that your fitness level continues improving off consistent mileage until your body adapts to the level of fitness needed for that workload (aka you plateau), then up the mileage, adapt, etc, up to a certain limit that's dictated by your talent.
Most posters on this board are probably already at a high level of fitness so they need the speed workouts as means for marginal improvement.