They can determine your location whenever your wifi is on. Even you aren't connected to a wireless router all nearby routers and your phone are communicating. On a phone wifi location is much more accurate than GPS because the GPS has to triangulate with satellites. Your phone uses this more often than GPS. GPS doesn't work well inside a building or even in a car.
Wireless router physical locations are determined, probably via the ISP and eventually mapped to the MAC address of the router. When a phone pings a router the phone's general location is established. As the phone's wifi pings more local routers the physical location becomes more well defined. They may be using signal strength measurements to help pinpoint the physical location.
A few years ago when I moved my router's physical location it took about a week before it figured out the new location. This was evident because my phone kept showing that it was at my old address when I opened up the map program.
If you don't want people to know where you are then buy a burner and leave your phone on at home.