You describe
a game of four player ludo
We are very unlucky in my court,
because after beating a pawn (maybe wrongly translated, I mean the figurines) it gets reset to point zero and has to start over.
But with rival groups, we play sport.
The groups of pawns race to get into the "home". Without rivals, it wouldn't be sport
There are 3 other groups, just like us,
there are 4 groups of pawns similar to the first group. One group for each player.
Who we all are, out you must suss.
you are the pawns of a player of the game.