INTENDED SOLUTION (SELF-ANSWER)
Tommy Jollyboat pretty much has it! The answer is: Viola
My family and I love to be united in four or five
As TJ guessed, this refers to string quartets and string quintets.
My name can be hard to pronounce, and I won't apologize
When people see the word viola, they often attempt to pronounce it similar to violin (it should rhyme with bee-oh-lah, not bye-oh-la)
My brother's always crying, really pulling on the heart
TJ got this one too; violin is often stereotyped as "sad" and "mournful"
And not many people can tell us two apart
Very few people can tell violas and violins apart, as TJ notes. Even I can't unless one is held next to the other!
Upon hearing my name, others shudder
Viola is often mocked as sounding ugly and nasty compared to violin (this is a major clue as to which string instrument it is)
I don't neck in the halls, unlike my brother
This refers to the chinrest on a violin, which can make it seem as though the violin is kissing the player's neck, and the "halls" are concert halls. I was under the mistaken impression that violas didn't use chinrests in the orchestra, so this clue is pretty much invalid.
I've always read differently than the rest of my family
Violas, of course, read alto clef, unlike violins, cellos, and double basses (this was the other big hint about which string instrument it was)
And here's a hint: I've not got an E.
No E-string on the viola.