This must surely be
Hogwarts School of Witchcraft and Wizardry, from J K Rowling's "Harry Potter" books.
It all started with a group of four.
The founders: Godric Gryffindor, Helga Hufflepuff, Rowena Ravenclaw, and Salazar Slytherin.
Their legacy inheritors compete round the clock for the annual prize.
The four houses that bear their names.
People say the lovely head has got style.
Not sure exactly why "lovely", but at least one character says in so many words that Albus Dumbledore "has style".
Fluffy the pet can be found in the belly.
Hagrid's three-headed dog, guarding the entrance to the place where the Philosopher's Stone is hidden in the first book.
Even the Riddle that shan't be spoken of, had been here.
Tom Marvolo Riddle, also known as Lord Voldemort, also known as "He who must not be named".
What did the four start?
Not sure right now what the first one is about. [EDITED to add: ffao points out in comments that this must be about Quid-ditch.] The second one is just a slightly cryptic way of telling us the answer: swine=hog, blemishes=warts.