I do not think that it is silly to be upset about the cats. You literally smell them, are physically exposed to them, every second that you are there. If that is not an agreeable sensation, then every second of your time there has that disagreeable tinge to it.

I don't know whether or not it would be a dealbreaker for me, but I can certainly understand that it could be a dealbreaker.