I am fairly sure your word is
Don't say one synonym's a type of fish:
It's the same word, not the same meaning.
To cavil is to CARP, same word as CARP, a kind of fish.
Don't say one synonym's a type of bird:
That's a different form of the word.
To cavil is to GROUSE. So far as I know, grouse=complain and grouse=bird are no more related to one another than carp=complain and carp=fish, and while the former is a verb and the latter a noun the same is equally true for CARP, so I fear I'm missing something here.
Don't say one synonym's a type of drink:
In fact the sound is the only real link.
To cavil is to WHINE, which sounds like WINE, a type of drink.
"Replace an approximation by a constant to find badness"?
Not a good clue at all! This riddle is madness!
Take CAVIL and replace CA (circa = approximately) with E (e, mathematical constant) to get EVIL = badness. Though this seems like a perfectly reasonable clue to me.
You should travel to find a rhyme:
Even here you've failed one time.
CAVIL and TRAVEL are almost rhymes, though to my ear imperfect ones because that last vowel isn't quite right. (Perhaps this is why "you've failed one time".)
And of course (I didn't originally say this explicitly because I thought it was obvious enough not to be needed)
each stanza is itself something of a CAVIL, about the riddle itself. (And even-more-of-course the title describes this.)
I wondered whether
there might be one further level of subtlety that explains why these are all slightly wrong in ways not already covered above
but if I have correctly understood Rand's comments, that isn't the case. As Rand also suggests in comments
you may consider this a meta-cavil from me :-).