Usually people who have faced rejection (and/or been badly hurt) before tend to function by always expecting/fearing they will be abandonned... again. Also usually accompanied by low self-esteem/confidence.

It is a difficult situation. If you have been in such a situation before (rejection etc.) maybe you could talk to him and explain why you are so worried. If he knows where you're coming from he could support you and calm your fears.

On the other hand, and this is the difficult part, you do need to let go of your fears - or at least don't vocalise it as much as he gets frustrated. Is there anyone who could help you work through these issues?