This sounds like an emotional connection, at least on your side. And I'm concerned that he may not be a trustworthy, healthy friend. Does your partner have a gut feeling about this relationship and does it in any way complicate your your relationship? I'd say good goodbye to your old boss because as you said, "he's not really great for me."