Ok, again I think I got this.
Fleet Street, in London.
I was confused by the fact that...
The two last lines didn't rhyme! But what does rhyme with harbour? Hmm, I don't know... Barber?
A murderous barber? Sweeney Todd? From Fleet Street? You can definitely find a fleet in a harbour.
So, let's go backwards to fit in the four clues:
East of Fleet Street is St. Paul's Cathedral. St. Paul was born in Turkey. Maybe not exactly a priest, but it's too much of a coincidence at this point.
The Royal Courts of Justice are in the West End of the street.
There's no bigger book of words than a dictionary. Such as the Dictionary of English Language penned by Samuel Johnson, whose house is a couple streets north of Fleet Street.
I actually have no clue about this, but it might have something to do with the Temple Church right below Fleet Street. Maybe because a king of France decided to kill the Templars. Although I guess that would have left a bad taste in the Templars... I dunno, but I'm so sure of the other three that I will edit this when I find out what I'm missing.
Or, as M Oehm so cleverly indicates, it might have to do with the fact that the Waterloo district is right across the Thames south of Fleet Street. And of course it was in the Battle of Waterloo where Napoleon was finally defeated by the Seventh Coalition. Since France and England are mortal enemies, it probably leaves a bad taste in French people's mouths when they're reminded of any occasion where England has prevailed over them.