Another answer that fits the clues is:
I am a common word that refers to a person.
Commonly used to refer to men, and often people in general due to gender-biased language (e.g. "early man", "man in the street", etc.).
I rarely refer to very young people – in fact, the older a person is, the more likely they are to be called me.
One usually only calls an adult a man, although male children can also be referred to using the term man, e.g. "little man", "be a man" etc.
However, if you pick two random siblings, the younger of them is more likely than the older one to be called me.
It's not a large effect, but the fact that sex-selective abortion in favour of male foetuses is more prevalent when a couple's first-born child is female [citation] would suggest that on average a second born child is statistically more likely to be male, and therefore be referred to at some point as a man.