In the OP’s case I just move away.
Couple stories.
One time I was walking home at 1am in a bad area and someone put a gun to my head and demanded I give him a quarter. I sighed because I knew it was an initiation to a gang and there was probably a car watching down the street. Anyways, I had a casual conversation with a gun to my head and gave him a dollar because that’s all I had. I was pretty sure he wouldn’t shoot but my life is worth more than $1 to risk it.
Another time I was the subway in NY and some thugly looking guy entered and told me to move. I said, “no.” So he said, “move out of my seat.” So I said, “no” and directed him to sit in the sit in the seat next to me. He walked away. Afterwards I thought I could have been killed over a subway seat. Not worth it if you ask me.
Long story short, it’s not worth serious injury or death over things as small as a quarter, a subway seat, or standing too close to someone. Just do what they say and move on.