There is nothing wrong with casual sex. It just doesn’t seem to make you happy in a long run. These encounters with married men were about sex and it left you unhappy and devastated. It’s nust not satisfying emotionally for you but if that’s what you want it’s ok. Just have to be careful. It could be dangerous. Especially if these are strangers not FWB whom you already know.