I think it's disrespectful not to cancel if you're not planning on going. If you have a reason you want to be disrespectful, I'd say go ahead. But be aware of what you're doing.