You can call your Pdoc office and just be matter of fact. Tell the office person that your dr recommended that you go in to the hospital when you saw him last weds, you said no, but now things have gotten worse and you've reconsidered his recommendation. ask them what you need to do to follow his advice.

It might seem like eating humble pie but it is better than going though the ER. Most doctors have a after hours number and you can call them if you don't want to wait for the 9-5 shift. They can deal with it, without making it into a big deal. It would just take a phone call on you Pdocs part, something pdocs do as part of their job.