I'd be very uncomfortable if my colleagues showed up to work on
Their day off to see me or stay late until I am off. In fact id feel stalked and harassed and likely would complain to hr if they didn't stop. It's very likely they liked you but not after you started a bit of stalkerush behavior. Just be nice to them but do not see them on your day off.