Now that you've added 3 more lines and reading the answer of this riddle made by you, I think that the answer is
As a young child, you live here:
You live in the present
Later you leave me often
But you always return, sooner or later:
You start to think about your future and make plans. But some time or the other, you return to the present moment
I have no beginning or end
I am infinite
The present moment hasn't got a beginning or an end. It is boundless.
Therefore, if God exists,
God must live here, too.
God lives in the present moment.
Worry, doubt and fear are rarely
found with me.
You worry,doubt and fear about something that is going to happen in the future ,not the present.
I only have two siblings.
The two siblings of the present moment is the past and the future
If you are not with me, you are
with one of them.
If one is not thinking about the present, then he/she is definetely thinking about the past or the future.