Yes it’s important to put yourself first. It’s not very considerate of others to invite you last minute and then get upset if you deny an invite. Because that day may have been the case where you had no other plans but they can’t always have that expectation.