Ex-lifeguard/competitive swimmer here... while I'm no means an authority on how easy it is to forcibly drown someone, I would say that it is much easier than most think. People are not water animals, and our intuitive response to danger in the water is usually counter-productive.
In lifeguarding, one of the most dangerous situations is rescuing a DNS (drowning non-swimmer). This is because someone who is actively drowning is in a panicked state, and will latch on to anything they can as a last ditch attempt to save themselves. This includes lifeguards, or any other person who attempt to help them. They are not thinking clearly and will not wait for you to shove a flotation device at them. The classic story is that the DNS lunges and grabs the approaching rescuer, and tries to climb them like a ladder, pushing them underwater. The result is that both people may drown if there is no other intervention, which is why lifeguards and rescuers follow a very specific approach procedure. Sometimes, if a victim is too risky, it is advised to wait until they pass out/go under to grab them.
If lifeguards (usually young, fit, strong swimmers) have to be vigilant about being held under by someone who is not actively trying to kill them, then being held under by someone (who is not drowning and intending to kill you) is absolutely within the realm of possibility. Of course someone in this situation would try to fight off their attacker, but I wouldn't be surprised if this was not successful. Drowning victims typically panic in the struggle and inhale lots of water out of desperation, thus accelerating their demise. This would likely be especially true if you weren't expecting it and didn't have a chance to take in a big breath before going down.
FWIW, when I was a kid (10 or so), one of my friends who was a weaker swimmer grabbed me because she was in trouble. I was a competitive swimmer, and I was much stronger and larger physically. It was like a death grip and I got pushed underwater for a second. I tried to escape, but I couldn't, and knew that I wouldn't be able to. Luckily a parent noticed and intervened from the pool side.