Some with BP might be far more like you and some less so. Some have difficulty withstanding a borderline personality attack, so to speak, some are great at it (either because they have at least some aspects of a borderline personality or because they have some more SZ abilities).

It's important to establish what is the reason behind his passivity, if that's what it is. If he's in great pain, you might think you pick up on emotions, but you don't. It's not uncommon with a borderline personality. When reacting, it's likely you distance/detach yourself from your emotions, which means you cannot pick up on his emotions. You should stop. You have no idea how much you hurt him.

If it's more SZ-like, emotional detachment, distancing, very similar to borderline personality but also unlike it, far more under control, you can't hurt him. So don't try to (which you do, if a borderline personality causes your reaction). It's useless. You might be the (only) one with a problem.

Do know which it is. The first possibility can traumatise someone further. Depression is something you can make worse. Not just a current depression, but any future ones.

You may not think so, but (I believe and experience) that it may not appear like it, but a borderline personality is very much unemotional, as well as emotional. BP can be very much the opposite: it looks like there's emotional detachment, but there are extreme emotions instead.

I don't believe anyone who says they know all there is to know about borderline personalities. If that were true, you wouldn't have one.

Be careful with others and with yourself.