You are not obligated to have sex with anyone, no matter what, even if you said yes earlier. Everyone is entitled to change our minds. I suggest you go out for some coffee so you're not home for a while, but leave the person a phone/text/email explaining that you apologize for the inconvenience, but have decided it's not best for you to do this and you hope they will please not contact you and that it's nothing personal at all.