Are you...
Space?
I am an invisible key but I don't open a lock.
(Thanks, @WhatsUp!) Space is a key on the keyboard, but you can't see it when you type it. Shift or capslock have no effect when used with space.
I am infinite and I am growing faster than anything.
"Space" or "outer space" is constantly expanding, and there is no known limit to our universe.
You are in me but you cannot see me.
Space is all around us (we are in it), but space is invisible.
You need me when there is too much going on around you.
People say they "need space" or "give me space" when they get stressed or crowded.
I am a physical quantity.
Not sure about this one, possibly space-time (any mathematical model which fuses the three dimensions of space and the one dimension of time into a single four-dimensional manifold)