I've done this too. Sadly, I've even do it with distant family if they come for the holidays. Just hide out in my bedroom with all the bare necessities to avoid any and all contact. I can't explain it but it can make things tough. Things have gotten better with time, seems to come and go in cycles, but would definitely try not to beat yourself up about it.

I imagine exposure is the key to building on this "weakness", but exposure is the terrifying part. Good luck!