It's about how you value yourself compared to others. It's not that you have to shame yourself for being attracted to other people. But it is important to recognize your potential for infidelity. Even if you have not cheated, you are qualitatively no better than a person who has cheated. "There but for the grace of God go I." This is a good attitude to have. It can keep you humble and also can help prevent you from cheating.
If you have never cheated on your spouse, but you spend all day fantasizing about the 25 year old in your office, would this make your spouse happy? The attitude that you can think about lust all you want, but you are fine as long as you never act on it, will most likely lead to you cheating. As with many things in life, the "analogy of the 2 wolves" applies. You have one wolf that lusts, and another wolf that wants to stay faithful. These wolves are inside you, and are constantly fighting. The wolf that wins is the one you feed.