You gotta just talk to him about it, how you're thinking about what he said. Find out if he thinks your moods are the cause for his moods or problems? My guess is he didn't mean it negatively and did not want to hurt your feelings.

Do you feel like he minimalized your problems by saying that? Maybe he just cares for you so much that when he sees you hurting, he hurts.