I agree with sometimes giving an excuse. There's a difference between rationalizing your decision out of fear and doing so in the interest of being charitable and preserving goodwill. If this gal has been a dear friend, you do want to avoid being hurtful. Just keep it simple, and know you are well within your rights to decline attending. An invitation is not a summons.