If anyone leaves me, I don't leave him.
But if I leave once, I will not return.
What am I?
Could it be
a secret?
If anyone leaves me, I don't leave him.
If someone forgets a secret, it remains still.
But if I leave one, I will not return.
Once a secret is revealed, it is no longer a secret.
I think that you might be
Hair
If anyone leaves me, I don't leave him.
After having a haircut, hair normally grows back.
But if I leave once, I will not return.
Once somebody starts going bald (through aging) their hair tends not to return.
Perhaps it is
a legacy
If anyone leaves me, I don't leave him. But if I leave once, I will not return.
After leaving a legacy behind, it will stay attached to you until you are forgotten.