You are setting boundaries for yourself and there is nothing wrong with that. He should understand this, as your friend.