I think you are spot on about women not having an interest in golf, most of us don't have the time or patience to chase after a ball.
I did take lessons for about 6 months a few years ago and got really into it but realized after a while it took too much time. The sport can be addictive because you're constantly trying to improve on it.
If you notice the women who are out on the golf courses, they are usually retirement age and most due it for the 'social' aspect. Now if you're talking about professional women golfers then different story.
Like someone mentioned before, I think strength and drive skills play a big part too. Again, you can't compare men and women to these skills--we just don't have the testosterone like men do, if we did then we would compete in the same league ie; just have one tee-off box.
I think too, it would be interesting to see if women are better at the short game then at the drive. The short game takes more 'mental' skills. This is just my own observation.