You're a
Neutrino
Invisible to the naked eye.
Very little are as small as I.
Of unknown size, but certainly small. Smaller than an atom. Smaller than an electron, if size means anything at that scale
Matters rarely matter to me.
I go through them unaffectedly.
Known for interacting with next to nothing (and thus passing straight through most objects). Speculated for some time to have no mass due to their lack of gravitational interactions (now known to have nearly infinitesimally low mass).
For me there's no charge, but I do have an anti.
and flavours known to amount to three.
Chargeless particle, anti-neutrinos exist with opposite spin, and there are three flavours (referred to as such) - electron, muon and tau neutrinos