>An invading force was once named my prefix  
>What they left after, colored my suffix  
>Forgetting my infix is just not allowed  
>When applied to people, I'm more than a crowd


What am I?

Hint:
>! The 3rd line is a cultural reference to mid '90s American music, but you should be able to solve this riddle with only the other 3 lines.