I'm going to have to disagree with BianB BB's answer, which was formerly the accepted answer:
If there are 98 blue-eyed people, then each of them sees 97 blue eyes. They know the number is not prime, so they leave immediately on day 1.
99 blue-eyed people wait 1 day, see there are not only 98 blue eyes, and leave on day 2.
100 blue-eyed people need therefore to wait only 2 days, see there must be more than 99 blue-eyed islanders, and leave on day 3.
For other values of n, the number of blue-eyed islanders:
If $n$ is smaller than 25, then it must be prime. This means that for every n different from 3, the islanders will see $n-1$, which is not prime, and conclude that they should leave on day 1. The exception is 3, for which $n-1$ is also a prime, so the islanders will need to wait one extra day to rule the possibility that $n=2$ out, and leave on day 2.
If $n$ is 25, then the islanders realize it can't be 24 because it is not a prime, and leave immediately. If $n$ is greater than 25 and one more than a prime, the islanders know it can't be a prime and also leave immediately. For all other $n$ greater than 25, the blue-eyed islanders leave on day $k+1$, where $k$ is the difference between $n$ and the greatest number $d$ such that the islanders leave immediately (either 25 or one more than a prime) and $d<n$.