Wow. He sounds very controlling, like he wanted to 'punish' you for daring to take a break against his wishes.

You are well rid of him. Well done for having the strength to stand up to him. Hopefully you won't be referred back to him now or he'll probably be worse. For your sake I just hope you end up working with someone ethical and responsible and patient-led, not someone on a power trip.