I like to first say that what most think of as dyslexia is development dyslexia and I use dyslexia for the same, in what follows.

I would say dyslexia is to dyscalculia what manic-depression is to schizophrenia (or the other way around: it is besides the point, though may be a better comparison than I thought, since it could be that the problems causing dyslexia arise earlier in life than for dyscalculia). It could be argued that there are varieties of BP and SZ distinguishable by a different ratio of exogenous to endogenous influence in the onset of the disorder. I suppose there is pretty much a continuum from one variety to the next, but for the sake of argument in this text, I refer to the hypothetical purest forms of endogenous disorders, meaning the most developmental, the most hereditary and unprovoked types of BP and SZ, the classic or typical psychotic disorders.

The point is: it is highly probable that they are different expressions of, and reactions to, the same underlying problem. Low overall expression/reaction (arguably) leads to SZ and higher overall expression/reaction to BP. Simplistic, but essentially it is (again arguably) like this. It is possible that some preference for mathematics over language may mean that, given problems severe enough to make learning either very difficult, one person may focus on mathematics/arithmetic, becoming dyslexic, while another might focus on language, becoming dyscalculic.

The underlying problem is basically one of perception. More specifically, appraisal. If one has difficulty (rote) learning anything with enough precision (as is requirement for learning arbitrary sequences, like times tables, complex movements and reading), dyslexia or dyscalculia may arise.

As I've already mentioned, it could even be that problems with learning of language, before learning any mathematics/arithmetic, is indicative of an earlier onset, which shifts the (compensatory) focus to mathematics, while no apparent problems early on may cause an eventual focus on language (because, says the linguist, language is just more fun, right?).

I would focus on the underlying problem and call both (a specifier of either dyslexia or dyscalculia would be clinically helpful) something like dysreductia, given my assumption that people experiencing these problems have a difficulty preserving detail in memory: they reduce too much. They abstract too much away.

I would go one step further: I think dysreductia (or however you might call it) is a necessary, but not sufficient, prodrome of the purest endogenous, "developmental"/early-onset and most severe psychotic disorders. How much of the psychotic spectrum is difficult to ascertain since dyslexia (and certainly dyscalculia) often goes unnoticed (i.e. the problems are there, but circumstances prevent proper intervention), underestimated and there's quite a lot of controversy about what is actually means.

I am dyslexic and I have difficulty with all things mentioned (including times tables, motor skills, clockwise and counterclockwise, counting, directions; any arbitrary sequence, in space or time).

I am very interested to know: do you also have difficulty with the alphabet (as in knowing the relative order of two, arbitrary, letters)?

Edit:
The hemispheres of the brain are not switched round, but less lateralised: less specialised.

Many psychiatrists have professional self-esteem problems and like to think they know everything about any medical specialty. Psychiatry is not seen as real medicine by many other specialists, as you may know.

Another edit:
I also have difficulties with names and faces. As I look at it, we might be better at gauging emotions on faces, but we abstract away the (persistent) particulars. The same for routine or well-known environments: we stop noticing the things that remain the same and especially details. We experience changes, but might not know what changed. All dysreductia or, in my case, dyslexia. It can escalate from a normal, moderate, form and state to a more severe, manic, state where everything is malabstracted or malreduced and everything blends (far more) together. Everything is transcended, including the self.

The over-reduction in a normal state of being might even lead to mild depression, due to under-stimulation. Maybe even when, severely, manic and unable to find new excitement. Together with any fear, of course.