Rules
With integer programming, I managed to place
8 pieces, proved to be optimal
like this.
$$\begin{array}{cccccccc} 3&3&3&3& &5&5&5\\ 3& & & & &5& &5\\ &4&4&4&4& &A&\\ 6& & &4& &A&A&A\\ 6&6&6& &8& &A&\\ & &6& &8&8& &B\\ 2&2& &8&8& &B&B\\2&2&2& & &B&B&\\ \end{array}$$
Here is my formulation. I happened to solve a similar model to solve a puzzle called One puzzle a day. Let $B$ be the set of cells in the 8x8 board, and $P$ be the set of all kinds of pieces (rotating and flipping count difference here), so I have $|P|=63$, then we need to choose a subset of $P$ and place them only by translation. Let binary variable $x_{pb}$ indicate whether the piece $p$ is placed on the cell $b$ (some predefined reference point on the piece being on $b$). Let the set $P_i\subset P$ represent all flipped and rotated version of a same pentomino, and binary variable $y_i$ indicates whether the pentomino $i$ is placed on the board, so we have $$ y_i = \sum_{p\in P_i} \sum_{b\in B} x_{pb} $$ and the objective is to maximize $\sum_i y_i$.
Putting the reference point on some cells make the tile out of the board, let that set be $F_p$, we have $$ x_{pb} = 0 \quad\forall b \in F_p. $$
Now the main part. Actually in every cell $b$, we could find all possible $(p,b')$ that will cover the cell $b$, say the set is $Covered(b)$. By constraining $\sum_{Covered(b)} x_{pb'} \leq 1$, we could prevent overlap. To prevent neighboring, I build sets $Edge(b) = \{(p, b')\}$ similarly, which means if putting $p$ in $b'$, the cell $b$ will be on the edge of that piece. Then we need some constraints like $$ \alpha \sum_{(p,b')\in Covered(b)} x_{pb'} + \beta \sum_{(p,b') \in Edge(b)} x_{pb'} \leq \delta $$
The parameters $\alpha, \beta, \delta >0$should satisfy that $\alpha \leq \delta$, $4\beta \leq \delta$, $\alpha + \beta > \delta$, $2\alpha > \delta$, meaning that "one cover is allowed", "four edges on the same cell is allowed", "one cover and one edge is not allowed" (that is when two pieces are next to each other) and "no overlap". I chose $\alpha=\delta=1, \beta=0.25$ (actually any $\beta > 0$ works...).
That is all constraints we need.
I solved this completely by hand.
Here is a clean proof of its optimality. No computer is needed. Mere pencil and paper suffice.
Expand each pentomino by adding little right-angled isosceles triangles with area 1/4 as follows:
Here is an example (with added triangles yellow in step 1 and green in step 2):
Observe that the number of little triangles added is at least the perimeter, regardless of the pentomino. The perimeter is at least 12 except for the P-pentomino, but the P-pentomino has 4 little triangles added in step 2. Except the X-pentomino, every other pentomino has at least 2 little triangles added in step 2. Thus the expanded X-pentomino has area $5+12/4 = 8$, and the expanded P,F,W-pentominos have area $5+14/4 = 8.5$, and every other pentomino has area at least $5+16/4 = 9$.
It is easy to see that, for any solution to the puzzle, the expanded pentominos will still not overlap. So if there are 9 distinct pentominos, then their total area is at least $8 + 8.5×3 + 9×5 = 78.5$.
Now expand the 8×8 board as well in the same manner, which adds 15 little triangles to each side. The expanded board has area $8×8 + 15 = 79$, but at least one of the added triangles next to each side of the board will not be covered, so the actual covered area is at most 78. Hence 9 distinct pentominos cannot fit.
In fact, we can show that 9 pentominos cannot fit even if each pentomino can be used 4 times! To see why, observe that each pentomino can only cover 6 units of the board perimeter, after which there is a gap of at least 1 unit before the next pentomino along the board perimeter. Let T be the expanded board area. Due to the gaps, not all of T can be covered by the pentominos. First, each isolated gap 'reduces' T by 1 unit. Two adjacent gaps 'reduces' T by at least 1.5 units (attained at the board corner). In general, $(k+1)$ adjacent gaps 'reduces' T by at least $(k+1)/2$ units. Thus the average reduction is at least 1/7 per board perimeter. Thus the gaps 'reduces' T by at least $\lceil32×1/7\rceil = 5$, so the covered expanded board area is at most $8×8 + 15 − 5 = 74$. But 9 pentominos (even with quadrupled pentominos) would have total area at least $8×4 + 8.5×5 = 74.5$. I presume that these bounds can be improved to show that 9 pentominos regardless of how many repeats cannot fit, but I did not bother to squeeze this last bit.
An equivalent counting method is to count the cell edges instead of area. Each pentomino already has 15 or 16 cell edges, and between each pair of adjacent cells along its perimeter we can add 1/2 edge sticking out, because that goes at most halfway to any other pentomino. In total we would get at least 17 edges per pentomino except 16 for the X-pentomino, which corresponds to 8.5 area for each pentomino except 8 for the X-pentomino.
The slight advantage of this counting is that it is easier to see where we are making losses. For instance, every untouched corner incurs a loss of at least 1 edge (i.e. 1/2×2), and this applies at the boundary as well, so we automatically know that we want to minimize the number of untouched corners. Also, every empty 2×2 square in the board incurs a loss of at least 2 edges, twice as bad as an untouched corner.
I initially did not mention this approach because the area equivalent is a bit easier to understand. However, I have decided to add it in, because it is much easier to write a program that uses this for a branch-and-bound search that counts the total edges plus the (necessary) losses so far. This makes it easy to prune most branches of the search. We even get a reasonable heuristic for branch-ordering, namely to first try placing the next pentomino to minimize the total edges+loss so far.
I can manage
seven pentominoes, in a few different ways:
There is an obvious upper bound of
twelve (the total number of distinct pentominoes, and $12\times5$ is still less than the number of squares on the board),
but this is nowhere near achievable given the restriction that pentominoes are not allowed to touch each other orthogonally. This means that
every placed pentomino creates at least 4 squares around it that no other pentomino can occupy. At least 4, in the best possible scenario where the pentomino is placed in a corner - usually more. (I think 4 is only achievable with the P-pentomino in the corner and 5 is only achievable with the symmetric L-pentomino in the corner, otherwise it's more.)
Because of this, I'm pretty sure the number I managed above is optimal and it's not possible to place more.
