You may be
A pimple (or acne).
When you were young, you didn't fear me
Young people don't get pimples.
But when you became older you found me, and you really had a scare
Adolescents start getting them, and hate them, often afraid they'll be made fun of for their appearance and/or that they'll seemingly never go away.
Black or white you didn't care, as long as I wasn't there
Blackheads or whiteheads.
If you try to be rid of me I'll serve you a scar
Trying to pop pimples can leave scarring
It'll make perfect even more far
Scarring, or acne, makes perfection even farther away. Either fits. I think scarring was intended?
(Thanks @Daedric for helping me parse this line)
And I dare you to, smoke a cigar
Smoking has been linked to acne.
If you do, we might both get worse
Smoking can make acne worse, and lead to scarring on its own. It can also harm your health.
And I might stick around, a hole to be your curse
The hole-like scar left behind.