I was just about done with a fairly long post saying pretty much what you are saying. Arthur thought you should take your pulse when you woke up and if it was 10% or more above normal for at least three consecutive days you probably needed to rest. But for him rest usually meant a very, very, easy run ("so slow that grandmothers pushing their grandchildren in prams are passing you.")
There are some people who have a day with no running every week. There are some who take complete breaks from running, maybe two weeks to a month at the end of a season. You can make a case for doing or not doing something like this but in such cases the off days are done as part of a plan. But taking a random day off at irregular time intervals, which is what I think the OP's asking about,probably doesn't matter much either way; it's not going to get you the kind of rejuvenation a couple weeks off likely will but it's also not going to make you any less unfit.
Most of us have a day every now and again when fitting in a run is very hard to do, e.g. you're flying to the other end of the world and are looking at twenty plus hours in airplanes and airports, your wife goes into labor, people around you have gotten the flu and you're feeling fairly shaky yourself. Sure, on days like that not running makes sense.