His logic is basically "it's immoral for some people to work from home because not everyone can work from home."
I'm a physician. Most of my work has to be done in the hospital or clinic where my patients are. If your job is conducive to working from home and you choose to do so, I have absolutely NO animosity toward you. It would be moronic for me to expect a graphic designer to chain themselves to a desk in an office just because I can't place a central line from my living room.
It's a matter of practicality, not morality. Some tasks are conducive to a work from home model and others aren't. Do what works for you.