The reality is they had just as much right to be there as you did.
If you know how to read the staggered start lines, you can easily do accurate 800s in any lane. Many tracks have staggered start lines marked for 4x200: run entirely in lanes.
I would have explained what I was doing and asked them to leave a lane open.
You are correct that many people who use tracks have little clue about running fast. But that is ok. Maybe they value a track for the same reasons you do or might do(even at 12 min pace):
smooth surface
faster then roads, grass, sidewalks
easy on legs
safe: no traffic or dogs or bikes
accurate distances
repetitive loop allows you do bail out at any time in longer run without planning how to return to start
aid every 400 meters
no route finding
spectators or crew every 400 meters
This said, I understand your frustration.