Is he flirting with you or does he just enjoy your company?

I don't see anything wrong with telling him that you have to work and can't chit-chat anymore, especially now that the semester is starting up again in full swing. I think you could pull that off without letting him know the real reason.