You could temper that statement a bit by saying...."It was a kind gesture last time when you held my hand but it triggers me and I prefer not to do that today"