Pardon my language, but I'm going to guess
Shit
So often forgotten, unless accidentally used,
A piece of shit is something that's usually worthless and overlooked. You realize that something is a piece of shit when you try and use it.
How am I so rotten; why am I so often abused?
Shit is another word for poop -- it stinks. People can kick the shit outta you.
I'm loud and obnoxious, it's obvious when I'm there,
When people are cursing, they normally do it loudly. You could say that some people who like to sing loud and out of tune are shit singers. You can instantly hear when someone isn't a good one. Another common phrase would be, "They can't sing for shit."
For I'm not for the cautious, you might use me when you swear.
If you don't care what others hear (like children) then you're likely to use it in your swear vocabulary
Take away all restraint, and I'm what you're left with.
The first letter of each line spells out SHIFT, people sometimes use this in place of shit, for example: Shift happens. The F line talks about being cautious, so if you remove it (take away the restraint), you're left with SHIT.
Oh, and the title:
There are people out there who truly annoy the shit outta you