I tried to join two different groups from very different towns
I was sure that one would let me in but both have turned me down
The groups are "Senses" and "Directions", two very different groups in many aspects. (Also, "down" is one of the directions)
The first said they already have all personnel they need
they didn't seem to care that I adhere to their creed
"The five of us can manage without another in the mix
Although humans definitely have some kind of a sense of time (and many others senses as well), the only senses "in the elite club" are the classical five: sight, hearing, taste, smell and touch.
In any case you belong elsewhere in a different gang of six!"
But when I tried to join the other group of which they spoke
their members viewed my presence as if it were a joke
The other group are the six directions, forward, backward, up, down, left, and right.
At first glance, it seems ridiculous to think time should belong to the same group.
The first group were convinced that this was where I would belong
Given Einstein's theory of special relativity, it makes sense that time should be treated similarly to the spatial dimensions, only that it doesn't really go in the negative direction like the other ones, which is why it makes some sense to think about it as "half a dimension".
but the second group insisted that they must have heard me wrong
Hearing (wrong) is one of the five senses.
(In the correct solution, the wordplay most likely goes here.)
I don't belong in the sextet or the previous quintet
I must belong to something, have you figured it out yet?
Time belongs to the Tower. --Stephen King
At this point I'm pretty much convinced this isn't IT. (Trying to stay on the King theme here..) Posting anyway, since a couple of the clues fit quite nicely.