My sniper longshot
6 doesn't represent the number 6. That's why Grandpa says "No. I never said number (6)".
We know it's not a letter and not a number, so we are left with symbols.
Unicode U+212X is the third row of Unicode characters, if X is 6 we get Omega. Omega, in maths, is used to denote how many prime divisors a number has. So Grandpa is not saying 6, but Unicode 6 is prime.
Previous answers (none is the actual one)
Six has 3 letters. 3 is prime.
6 = G => G = 7 and 7 is prime.
TV Primetime. But it is between 8-11 PM.
6 is between Twin Primes (5-7).