Wild guess, but I’ll put it on here. The word is
My prefix is ironic.
FE is the symbol for iron on the periodic table, as Omega Krypton indicated
My infix is well-oiled.
Not sure about this one - maybe a reference to how a collective (WE) can act like a machine that’s well-oiled, or perhaps a wordplay clue?
My suffix briefly handled a worm.
This one’s also stretch, but ERs - “emergency rooms” - have the ability to remove or at least mitigate the damage from worms that burrow inside the body
There aren't many of me - what is my name?
If there aren’t many of something, there are FEWER of that thing
As for the title,
having fewer of something would make it rarer to find