Have you voiced this concern to him and had a thorough discussion about it? If not, he may just not understand that you feel like you're being left out.