Is it
Cards?
I saw kings but no kingdom ,
Four kings no kingdom
I saw a half dozen lovers but one was left over ,
Based on OP's comment, there are 13 hearts (love) out of which 6 have a pair, leaving one that is "alone"
I saw racism but no one discriminated on color ,
4 "races": Spades, Clubs,(black) Hearts and Diamonds(red) - in most games you would group them by "race" and not actual color
I saw mathematicians but all of them didn't knew why they started everything from two,
The decks start from 2
All of them lived happily but there was a serious villain among them ,
Based on OP's comment, the Joker
More I saw them , more poorer I became in my life ,
Gambling problem? The more you see the cards the more money you lose