Ok, first off a picture would be very helpful, but i think this is just a glorified algebra problem with some trig.
Also you mention both reflection and refaction so i just assumed that you meant only one.
So here is the info given:
Msinx = 2Nsiny
y = 1/2 x (the (non-zero) angle of incidence whose angle of refraction is half the angle of incidence)
From here its just simplify for x given the 2 equations, 2 unknowns.
Plugin for y:
Msinx = 2Nsin(1/2 x)
Substitute u = 1/2 x x = 2u in order to use the trig identity below:
Msin(2u) = 2Nsin(u)
Now use this trig identity sin(2u) = 2sin(u)cos(u) to deal with the sins with different arguments:
2Msin(u)cos(u) = 2Nsin(u)
What do you know, stuff cancels:
Mcos(u) = N
Solve for u:
u = arccos(N/M)
And finally substitute x back for u:
x = 2u = 2arccos(N/M)
Hopefully that is right and makes sense to you. Keep in mind there are probably other ways to solve this, particularly with how you deal with simplifying the two sins with different arguments. And don't worry, I had to look up a chart of trig subs to know how to do that. Thanks for the little brain teaser.