Your first love will never leave you, whether their impact on your life was good or bad. It is harmless to still think about that, unless it interferes with your current relationship with your husband, or if you are spending substantial amounts of time delving into the past. It might be wise to keep those thoughts to your self, as it seems your husband is worried at this point. You need to reassure him that you have no desire or intention to get back together with him, you just sometimes remember the relationship, for whatever reason. I'm sure he thinks about his Xs too. Ask your self if you feel alright with that, and then decide whether it's a good thing to talk to your husband about.

I sometimes think about old friends or Xs, but that doesn't mean I want to talk to them or get back together with them. It was a completely different life. But just because things have changed it doesn't mean they won't always be in my memory. And as long as that memory isn't interfering with my current life, there is no harm to it.