You could try saying 'yeah my boyfriend and I would LOVE to have a coffee with you' or take your boyfriend with you next time you go to the libary ?