I agree with everyone who says it's fine as long as you are safe and remember that it is "casual" (as much as sex can be casual, as seeker said)

I guess I was in a casual relationship once like that, but the communication was horrible and while I thought that he liked me, he just wanted the casual thing, so I ended up extremely hurt. Just be careful when it comes to that and make sure everyone involved knows what's going on 100%