A simplified model: The world is 3-D. We only have the capacity to experience one place at a time, but the rest of it still exists as a unified whole even when we are not there. When you walk in a straight line things appear, pass by, and recede into the distance behind you. If you are doing it at night with a candle you only see the things that are immediately around you, then retain the memory.
Time is the same way. When it comes to our perception of time we are walking through the dark with a candle and only perceive things in terms of whether we can see them or not. We don't have the physical capacity to experience that fourth dimension directly, just as stingrays, for example, have a two-dimensional perception of the world because their primary sense, the electrosensory system, is built to model a flat disc, but we can extrapolate what that extra dimension means from our perception of three dimensions.