Chlorine is in tap water. Sometimes where I live, I can taste it. I can taste it at most restaurants. If I can taste it, I'm sure a cat can taste it. Get a good water filter with a carbon filter that removes chlorine.
We have an indoor water fountain that doubles as a water dish for our cat and dog. After 24 hours of circulation, most of the chlorine evaporates out of the fountain. They prefer the fountain to their water dishes, which only get tap water. Still, they're stuck with fluorine, which is a much smaller particle.
We used to double-filter our water. The first stage would remove the bigger particles and the final stage would slowly remove everything, leaving what was tantamount to distilled water, but it was a slow, expensive process.
Now to avoid fluorine, we only drink reverse osmosis filtered water at our house. I fill up 2 3-gallon containers every couple of weeks at the supermarket.
If you want to try an inexpensive approach for your cat, try filling the water bowl from your water dispenser on your refrigerator. Assuming you change the filter regularly, it should remove some of the chlorine. You can also try filling a large pitcher with refrigerator filtered water and pouring it back and forth a few times into another pitcher. This will remove a tiny amount of chlorine. You can then let it sit until the next day. They will allow another tiny bit of chlorine to evaporate.
Chlorine has done its job by getting water to your house bacteria-free. It is a poison and is no longer needed. Get rid of it before drinking water. I've been filtering our drinking water for 35 years. Just another step in staying as young as possible for as long as possible.