I don’t know if this will help at all but my friend’s son who is autistic says he feels compelled with his special interest, I don’t think it’s a frustration free experience for him all of the time, although broadly speaking he is still enjoys it. I think feeling compelled is not always comfortable for him though.
Lastly I think a special interest can be anything and it can be physical, for example a sport.
Are you happy to talk about your special interest and the problems you’ve encountered? You don’t have to if you’re not comfortable, of course.