You have the option to say "I enjoy your company but that isn't a good time for me."
You can have "other plans"--the other plans are not having company. You don't have to say why it isn't a good time for you and if your friend persists in asking why, you repeat it just isn't a good time for me. You could also suggest another time that would be better. Then you would be setting your boundary, defending it with a 'no', yet adding a 'yes' because you enjoy your friend, but on your terms.