This offhand makes me think of
Sherlock Holmes
but I'm not sure how to apply that to all the lines so here are the ones I can think of at the moment:
I am a genius.
He is, simple fact.
You all miss me
Perhaps because he faked his death?
I specialise in Mathematics and crime. A deadly combination, isn't it?
I'm not sure about this one.
People consult me for both.
He's a consultant.
I have only one nemesis of my level of intelligence.
Moriarty
He destroyed my reputation.
In an effort to outsmart Holmes he destroyed his reputation. (I forget the details but I believe he did this by having people believe that Holmes had wrongly accused him of a crime which there by put all of his cases under question.)
But, Don't worry, I will be back soon.
Sherlock faked his death in order to beat Moriarty with the intention of coming back eventually.
In the mean time, Can you guess Who am I?