I'm a 100% P&T SC disabled veteran and use the VA Medical Center (VA Benefits) as my primary health care. If you have SSDI and Medicare, you can explain to Medicare why you may not want to use their services, but you have to make sure to only quit Medicare AFTER you establish VA Benefits for the ENTIRE YEAR. Otherwise, Medicare will penalize you for the months or years that you don't use their services UNLESS you show proof that you had VA coverage for all of your healthcare needs BEFORE you quit Medicare. I had to do this recently when I decided to add back Medicare's Part D, just in case I wanted to see an outside doctor for prescriptions. I had to prove to Medicare that I had coverage from the VA BEFORE I left Medicare's Part D plan. I've kept Medicare Part A and B, just as a backup, because many times the VA will NOT pay for paramedics. It's frustrating. And I'm covered 100% (I don't pay a dime for VA Benefits, but they will still periodically deny paramedics care, including the time I was short of breath in March 2020, when I didn't know if it was Covid or not; the VA still denied the ambulance, so I had them bill Medicare, and they covered it).
As far as primary care for all other things, the VA is a lot better than it was decades ago. You do get really good care, and you can get referrals from your primary care to see a specialist. Now when they prescribe you anything, sometimes they do NOT ASK YOU if you want the meds or not, so you will get charged for each and every prescription that they send out to you. You'd have to tell them to stop sending the prescriptions and to request a refund if they send you the wrong one or one that you didn't okay with your doctor first. Before I had my 100% rating, I was charged $8 per prescription, and then I owed money. I was eventually reimbursed for some of that when I received my 100% rating, but it was a pain to deal with.
Anyway, do what you can to navigate your VA's system. Every VA is different. The VA I used to go to covered both contact lens and eyeglass exams, whereas the one I go to now only covers eyeglasses (not contacts). Also, the one I used to go to allowed me to get referrals to specialists, whereas the one I attend now are sticklers about having the primary care do everything, even if they aren't as qualified. I'm a woman with reproductive problems, for instance, and I used to be referred to an OBGYN specialist because I have many issues going on. But when I moved out of state to a different VA, they failed to hire a female OBGYN at their Women's Clinic (they only have a male doctor, which doesn't help female MST survivors), so I would either need to see a PA, a regular primary care, or use my Medicare to seek care outside of the VA. Sadly other female veterans have complained about similar things at the particular VA I'm now enrolled in. So it really differs with each VA Medical Center you are enrolled in. Still, I am satisfied with most of the care I receive at the current VA, even though I got better care at the previous VA.
You also have a patient advocate, should you run into any problems with any of the VA staff. You can also ask for an escort if you are struggling with PTSD or TBI symptoms and need that extra assistance to walk you to and from your vehicle or the entrance to your appointment and back. They also sometimes have transportation services, such as DAV volunteers who drive a government vehicle with the VA logo on it to pick veterans up for their appointments (from their homes to the the VA and back, or at least from the train/bus station - sometimes in a VA bus - and back). You can ask about all of their transportation services as well, which is something you wouldn't typically find in civilian centers.
When you make your first appointment, also ask about getting assistance with getting a premiere account on my Health e Vet. There, you can download your medical reports (Blue Button), including your vaccine status, should you need it. You can also make appointments online and message your primary care doctor and other treatment staff you engage with.
You can PM me if you have any questions you'd rather ask in private.
I hope this helps.