Depending on the doctor, you should wait. PCPs are not as experienced as orthopedics at diagnosing that type of injury, so if you are only seeing a PCP and not a specialist, I would definitely pay attention to what the radiologist says.
If it says "stress fracture" it's most likely a stress fracture. The manual diagnostic tests that an orthopedic uses to determine a "stress" fracture are not 100% reliable; more often than not, they are very UNRELIABLE.
It's very typical for pain with a stress fracture to decrease dramatically just by ceasing the activity or assault that caused it.
I'm treating a young woman right now who is recovering from a femoral neck stress fracture and she is a nightmare to treat for the same reason. Her pain decreases very quickly when she rests. She has ignored her doctor and attempted multiple times to get back out on the roads and it never ends well. Not saying you are ignoring your doctor or doing the same thing, but just referring to the fact stress fractures can be frustrating and everyone's pain perception is different. Take it easy until you see a specialist who can help you understand just how affected the bone is/isn't.