The answer that @jarnbjo found is demonstrably the best possible.
I wrote in a comment on that answer,
"Because of the continuous motion, and the way [OP has] defined 'best', I'm pretty sure the exact (infinite decimal places) best will be when the hour and second hand meet at exactly 120°. Any slight deviation from this will move the second hand away more than the minute hand moves closer..."
and I stand by this, to elaborate:
OP defines "best" by averaging the absolute values of the differences of the three angles from 120°. If we put the hour and second hand at exactly 120° and the minute hand at 120° plus x, then the differences are (0, +x, -x) which averages to 2x/3. If we try to move it a tiny bit to bring the minute hand deviation closer to zero, the second hand will move away from zero 60 times as fast.
Now OP's "continuity" thing was a bit confusing, but I interpret it as follows: either they are moving at a constant speed; or there are one-second ticks, but even so, for a mechanical clock, the motion from one tick to the next takes a finite time and is continuous, and the three hands are moving in a 1:12:720 ratio. So you do still get a 120° conjunctions of hour and second hands somewhere, and the three hands will be in the same positions as they would be in the constant speed case. (As for time, it would be within one second of when it would be in the constant speed case, though exactly what fraction of a second, that will depend on the nature of the ticking.)
In what follows, let's just assume constant speed.
So now I am going to
divide the circle into 2157 equal slices (about 0.16690°)
which sounds like a pretty random thing to do, but
the minute hand moves 12 times faster than the hour hand, and the second hand moves 60 times faster than the minute hand. Now imagine we start from noon when all hands are aligned.
- When the hour hand has moved ahead one whole slice, the second hand has moved 720 slices, so it is 719 slices ahead of the hour hand, which is exactly one third of a circle, or 120° exactly.
- When the hour hand has moved ahead two whole slices, the second hand has moved 1440 slices, and it is exactly 120° behind.
- When the hour hand has moved ahead three whole slices, the second hand has moved 2160 = 2157+3 slices, so it is exactly aligned with the hour hand.
And the pattern repeats every three whole slices the hour hand moves.
Therefore we know that the solution involves
whole numbers of slices
Now a tiny bit of number-theoretic voodoo,
$$1961 = \frac{1}{11} \mod 2157$$
I mean that 1961 is the only integer (between 1 and 2157) such that, when you multiply by 11, divide by 2157, and take the remainder, you get 1. It exists and is unique because 11 and 2157 do not have a common divisor.
The result is that, when doing arithmetic 'modulo 2157', multiplying by 1961 is like dividing by 11.
This matters because
When the hour hand moves ahead one slice, the minute hand moves ahead twelve, so the difference increases by eleven.
So if you want to know where the hour hand is when the minute hand is exactly X slices ahead of the hour hand, take X, multiply by 1961, divide by 2157 and take the remainder. This will be the only answer that involves a whole number of slices.
Now OP told us that the perfect 120° conjunction doesn't exist. But what does OP know! Let's try to find it
Using the above formula, for X = 719 (120°) we get the hour hand at 1438 slices ahead of noon and for X = 1438 (240°) we get 719. Well that's no good, this is 4 o'clock and 8 o'clock exactly, there will indeed be 120° between the hour and minute hands, and 120° between the hour and second hands, but 0° between the minute and second hands. We want the minute and second hands on opposite sides of the hour hand!
Well, I guess OP was right. Ok, next best thing.
Since we know whole numbers of slices are involved, one slice off is the best we can hope for. So X = 719±1 or 1438±1. But 720 and 1437 are divisible by 3, that will give us the second and hour hands aligned.
You can compute X=718 gives us the hour hand at 1634 slices ahead of noon. You can then compute the minute hand is at 195, and the second hand at 915, and see that indeed the minute hand is 718 slices ahead of the hour hand, and the second hand 719 slices behind.
And the mirror image solution
X=1439 gives 523, 1962, and 1242 slices for hour, minute, and second hands.
Converting this into hours, minutes, and seconds
- 9 hours, 5 minutes, 25 325/719 seconds (325/719 = 0.4520...)
- 2 hours, 54 minutes, 34 394/719 seconds (394/719 = 0.5480...)
Which is what @jarnbjo found. Yay!