As described in many answers, five colors is the minimum. Here we bring in the theory of pandiagonal Latin squares to show some hidden features of the solution and allow a generalization to $n×n$ arrays.
A pandiagonal Latin square is one where all the labellings are different:
Pandiagonal Latin squares exist if and only if the order is one more or one less than a multiple of $6$, so we can arrange our five colors into a square of order five:
$\begin{array}
&\color{blue}{■}&\color{brown}{■}&\color{gold}{■}&\color{tan}{■}&\color{black}{■}\\
\color{gold}{■}&\color{tan}{■}&\color{black}{■}&\color{blue}{■}&\color{brown}{■}\\
\color{black}{■}&\color{blue}{■}&\color{brown}{■}&\color{gold}{■}&\color{tan}{■}\\
\color{brown}{■}&\color{gold}{■}&\color{tan}{■}&\color{black}{■}&\color{blue}{■}\\
\color{tan}{■}&\color{black}{■}&\color{blue}{■}&\color{brown}{■}&\color{gold}{■}\\
\end{array}$
Since all rows, columns, and diagonals of the larger square have different colors by this construction, any $3×3$ bloc will provide the requited $3×3$ array. All such blocks are identical except for the way the colors are labeled. But also any $4×4$ block or the entire $5×5$ Latin square will also meet the problem conditions. The five colors needed for the $3×3$ square are also sufficient for $4×4$ and for $5×5$!
In general, given any $N$ an $N×N$ array can be differently colored along rows, columns and diagonals using no more than $N+3$ different colors because we can always construct a pandiaginal Latin square of order $N$, $N+1$, $N+2$ or $N+3$.
Note also that with order five there is just one pandiaginal square, as above, plus its mirror image; but for higher orders the number of possible pandiagonal Latin squares grows expinentially thus enabling many solutions.