Actually there is no way to compare the size of the Earth with the "size" of the "universe" because the two are fundamentally different kinds of things.
The size of the Earth is a standardized, local measurement. It can be made in Earth's frame of reference and isn't affected by relative motion or distance.
The universe is mostly not here, and even though relativity provides a way to correlate distances far away with distances here, there is no way to do that without establishing simultaneity of events. Because of the snail-paced speed of light, that is impossible to do. All the astrophysicist talk about something "is" 4 billion ly away, or appears as it "was" 4 billion years ago, ignores this inconvenient fact. There is no correlation between time here and time there.
So what is all the "age of the universe" stuff? Well, it doesn't come from direct measurement of time at all, it's estimates based on the Hubble model. The basis of these estimates is the speed and behavior of light. Light, however, is not a clock unless it makes a round trip. So the concept of "time" on the large scale is simply not the same thing as "time" on the local scale. Because time is essential to estimating distance on the large scale, distance is not the same kind of thing as on the local scale either. On the small scale, distance starts with standardized measurement and works its way up. On the universal scale, distance starts with a model of the whole universe and works its way down. Metaphysically, two entirely different concepts. Apples and oranges.