I sometimes take my athletes to an all comers meet toward the end of our summer base and they rarely run well at those. Granted we don't do a lot of mile specific stuff during the summer, but we do get some fast 200s at the end of tempo runs at least once a week.
I'm always surprised at how much slower they run after a summer of good training than they did during track season. I was completely different. I could go jump in one of those all comers meets and run within 10-12 seconds of my pr on just about any day of the week. I'd often run a meet in the evening after a solid run in the morning and still do ok.
I don't know what that says about my training. I did run lots of hilly runs back in the day, so perhaps I was getting some sneaky VO2 max type training without realizing it. Maybe it means that something was wrong with my training once the season started....idk.
for your case, don't be disappointed with a slow mile TT. Someone above said your body probably isn't used to the higher workload. I believe this is likely the case. BTW, my athletes who ran poorly at the summer all comers meet would usually go and have a very good season in the fall.