The side length $s$ of the square is:
$$
s = \sqrt{500} = 10\sqrt{5}
$$
The radius of the circle $r$ is half the diagonal of this square:
$$
r = \frac{s}{\sqrt{2}} = 5\sqrt{10}
$$
The diagonal of the rectangle (of width $w$ and height $h$) is also equal to twice the radius, and its area is equal to the product of its dimensions:
$$
w^2 + h^2 = (2r)^2 = 1000\\
wh = 400
$$
Solving for $h$ and $w$, we have:
$$
\begin{align}
w &= 20\sqrt{2} \\
h &= 10\sqrt{2} \\
\end{align}
$$
The angle $\phi$ between the long side ($w$) of the rectangle and its diagonal is:
$$
\phi = \arctan\frac{h}{w} = \arctan\frac{1}{2}
$$
Consider a vertex of the square adjacent to the endpoints of the short side of the rectangle. The angle between the two corners of the rectangle is $2\phi$, so if the rectangle is oriented so that there is an angle $\theta$ between one of its corners and the vertex of the square, there will be an angle $2\phi-\theta$ between that vertex and the other corner.
Since each of the sides of the square occupies an angle of $\pi/2$ measured from the circle's center, the angles between the corners of the rectangle and the next two corners of the squares are $\pi/2-\theta$ and $\pi/2-(2\phi-\theta)$.
The final four angles are repeats of these, since the arrangement has twofold rotational symmetry.
Thus the octagon is made of eight sectors. Each has two side lengths of $r$ and the angles between those sides are (two each of):
$$
\theta, \\
2\phi-\theta, \\
\pi/2 - \theta, \\
\pi/2 - 2\phi + \theta
$$
The total area of the octagon is then just:
$$
r^2\left(\sin\theta + \sin(2\phi-\theta) + \sin(\pi/2 - \theta) + \sin(\pi/2 - 2\phi + \theta)\right) \\
=300(\sin\theta + 2\cos\theta)
$$
Plotting this quantity, we can see that it is maximized when $\theta=\phi$, and its maximum value is:
$$
300\sqrt{5}
$$