An invading force was once named my prefix
What they left after, colored my suffix
Forgetting my infix is just not allowed
When applied to people, I'm more than a crowd
What am I?
Hint:
The 3rd line is a cultural reference to mid '90s American music, but you should be able to solve this riddle with only the other 3 lines.