Partial answer:
We have a 4x4 grid, so we first want to find all possible ways to place 5 vertices in 16 slots, while skipping rotations and reflections. Then for every unique vertex configuration, count the number of pentagons that can be formed.
Finding the non-congruent configurations by hand would be very tedious. We're looking at $C(16,5)$ ways to arrange the vertices, then eliminating all reflections or rotations that have already been counted. Most configurations have 8 rotations and reflections, so expect somewhere around $C(16,5)/8=571$ total.
Also, we can skip any configuration with 4 collinear vertices, which cannot form a pentagon.
Iterating by hand would be extremely tedious, so instead I wrote a program to do the work for me. It calculated 549 valid non-congruent configurations.
(For the curious: The program is a brute-force approach in Python. Keep a list to track various 4x4 grids. Iterate through all possible grids with 5 spaces filled in. For each grid, if it has 4 collinear filled spaces, or if any reflections and rotations have already been included in the tracker list, then continue to the next iteration; otherwise add it to the list. Then print and number the grids in the tracker list. It's not elegant but it only takes a few seconds to complete.)
Next, we would check each configuration and count the pentagons that can be formed. There are 12 different ways to connect any 5 vertices (thanks JaapScherpius for the correction), many of which won't result in valid pentagons. So our workload looks like 6588 different arrangements to check!