Could it be a
punching bag?
I am ready to fight with the duke.
as other answers mention, put up your dukes is a way of talking about fists.
If you stop me, Of course I will lose.
Hit the bag, stop it swinging, control its motion, and you've won - beaten the bag into submission.
But if you celebrate your victory,
I will definitely win,
and
it will be your loss.
Raise your arms in victory, or fail to stop the last swing, and it can swing forward and strike you even if you've declared victory. It will keep swinging until it slows or something stops it, so sayeth momentum.
What am I?
a punching bag
Normally I have two opponents, but in rare cases I have to fight four of them.
Possibly referring to hitting the bag with two fists, and then rarely kicks as well (two feet)? Or some method of sharing a punching bag between two people - I am imagining one on each side, so the bag swings back and forth between them. This would not be common, so therefore, rare.