Nah, don't feel horrible about it. Not everyone can afford to buy gifts or have the time to get gifts.

Not to sound cheesy, but I think people value other people's presence more than they value presents. Memories last longer than gifts do, anyway.