What a lot of people do not know is that MJ had a huge ego and was also very conscious of how his race affected his fame. Off the Wall was MJs big breakout solo album. It was a huge hit for a solo R&B album, but did not have a very big impact on the Billboard 100. Rock with You was no. 4 in 1980, but lost out to songs by Blondie, Pink Floyd and Olivia Newton John. MJ also could not crack into any of the non-R&B categories on the awards shows.
Thriller was clearly an attempt to take MJ out of the R&B idiom and make him the top pop star of his day. This is when the plastic surgery started. MJ was also under a lot of stress after constantly being accused of fathering children with groupies. Billie Jean is about that experience. Of course, the change in his artistic approach in Thirller worked. It was the most successful album in history by a lot.
If you look at MJs performances touring for Off the Wall and his later performances for Thriller, you can really see a huge difference. He is free and easy doing his hits like Rock with You. His performances are joyful. But with Thriller, it is very different. He is very aggressive and almost angry as if he is offended that Off the Wall was only considered a R&B album and wants to prove to the world that he is the biggest star in the music world. I think that the conflict between the music MJ wanted to make that was more R&B influenced and the music he had to make to have success that crossed racial lines really caused him a lot of pain and forced him into his self mutilation.