It doesn't seem to me like you're being honest with Ben. You told him you want a casual friendship. That seems like you do still want him in your life. And if he asks you about the party, you plan to lie to him and say you didn't get the message. He might believe you and think you still want to be friends with him. But you don't. I think it's important to make a clean break rather than keeping him hanging on. I know that's how I would prefer to be treated. Having false hope about someone who is not interested is not a fun way to live.