You are
The knight chess piece
I don't like my neighbors.
When you move, you won't be next to either.
One of them wishes I would move so that he can visit the others on our street.
The rook needs you out of the way to move along the back row.
The other is stuck up and only associates with one color.
The bishop moves diagonally so always stays on the same colour.
I can't stand to be in either of their homes. ...at least, not while they're in them.
You can't take your own pieces or have two pieces in one square.
Alternatively,
You are
The queen chess piece
Most of the answers are the same:
I don't like my neighbors.
They get in the way of you moving.
One of them wishes I would move so that he can visit the others on our street.
The king needs you out of the way to castle safely.
The other is stuck up and only associates with one color.
The bishop moves diagonally so always stays on the same colour.
I can't stand to be in either of their homes. ...at least, not while they're in them.
You can't take your own pieces or have two pieces in one square.