"I'll do my best" would have caused me to panic too. Not to add to your panic, but my very good male friend just went through this. The ex said she needed to move, and so she did, and then wanted a visit and then never brought their son back. No formal arrangement and now being out of province... not good.
NOT to make you feel worse, but just so you know i think that your feelings are quite appropriate and that your panic is completely normal.