I think this works:
11 53 37 41 23 19 17 13 29 7 71 67 31 5 59 43 47 2 3 61
And thus the pattern of sums and differences is:
S64, D16, D4, S64, D4, S36, D4, D16, S36, D64, D4, D36, S36, S64, D16, D4, S49, D1, S64
Basically it just
Took a lot of brute forcing on my end — seeing which numbers were able to be connected to which, and noting to the best of my knowledge that 61 had to go on the end and had to be attached to 3. With the exception of working with 2, all squares had to be even squares. And I started by trying to fit as many D4s together as possible and then adjusting/flipping where needed.
The other N for which this occurs, of course, is
N = 2 (ie. 2 3)