It's important that he respect your boundaries. If he's willing to just be friends and that's what you want then great! You can never have enough friends. But don't make the mistake of doing things you are not ready for in hopes that he changes his mind about the relationship, that rarely happens.

What you had sounded very much like a booty call or friends with benefits. If that's what he's looking for, of course he's unhappy it's not working out for him. If this is to be a healthy relationship, both of you must get something positive from it.