I think the answer could be:
Death
People gather because of me, but they're never happy eventually, because:
A death is a loss and no one want to lost someone, especially if it is a friend or relative
My name has five letters.
D E A T H
My favourite number is four.
In Chinese culture, number 4 means Death
If you are reading this, you haven't got the honor to meet me yet.
In this moment, I'm not dead yet, it's true
The most feared and hated side of two opposites.
It can be scary like it can cause anger to lose someone
The title might also help you
It happens only once