In high school, I would usually do shorts in anything down to about 20. Now, I don't really like to wear shorts in anything below about 50 unless it is for a race (I always race in shorts, which has been as cold as 15 degrees).
Note how the Kenyans often wear long pants in anything below 60. Sure, they are from a warmer climate, but at the very least, it does not seem to hurt anything to imitate. I would rather be a little too warm than a little too cold.
20-50 degrees (and sometimes even warmer) is sweat pants weather for me. Below 20 and I will usually put on tights under the sweat pants. Below -10 and I will probably have tights, and two layers of sweat pants.
Pants might add 10-20 seconds per mile, but is that even necessarily a bad thing? It is not as if you are not doing the work; and you still gain the training benefit.
Just a side note - I will probably wear shorts in colder weather in the spring (anywhere between 30 and 50 F), since I am used to the harsher winter conditions instead of coming off of a hot summer.