I was wondering if it has something to do with work relations, ie colleagues and / or line manager. This has largely been the case for me.

This would explain the fact that you like your work but you want to go home when you are there.