You did the right thing. You never have to feel bad about saying no to anything that seems off or that you aren't comfortable with.