Could it be
a fire engine?
As a secondary answer, the "you" that the riddle is addressed to is
a match, or fire in general.
I know you wanna see each other more often, but...
I'm letting you go!
The fire engine is used to put out fires, which could be seen as "letting the fire go" or even "letting the flame die", which would fit with the love story in the title.
I've always put away the fire in your eyes,
And made your world spin,
Always anxious on me,
The one to set you free.
Obviously, the fire engine "puts away the fire". The rotating light in a siren could be the spinning. The last two lines refer yet again to extinguishing the fire.
Was that a lifetime ago?
Or mere seconds, or both?
A match can start a huge fire, but the match itself only lasts a few seconds.
Don't look back,
You won't see me anymore!
After the fire is out, the engine is returned to the station, so it's not where the fire was any more.
Finally, the title:
Fires can be a major hazard in densely populated urban areas, and the continued "visits" that the fire engines make to the fires could be imagined as a romance that the fire wants to continue, but the fire engine wants to end.