My own solution
The following is the solution I had when I posted this puzzle. It loses to Albert.Lang's answer, but beats others.
First observation:
Once we reinvest the interest, we end up with an similar problem as we started with, but with a larger principal, a shorter remaining duration, and the same transaction fee.
Thus, if we can find out when to make the first reinvestment, we can re-use the same method for all subsequent reinvestments.
Second observation:
As the transaction size get smaller relative to the principal, we will reinvest sooner, regardless of remaining duration. Since the transaction size stays the same but the principal increases, this means that if we let $t_0$ be the time before first reinvestment and $t_1$ be the time before the second, then $t_0 > t_1$.
Third observation:
For any duration $t$
Let $i_0$ be the interest we would earn on the principal during $t$.
Let $i_1$ be the interest we would earn on $i_0$ during $t$ - that is, if we reinvested after $t_0 + t$, $i_1$ represents the compound interest we would have earned at $t_0 + 2t$.
If $i_1$ is less than the transaction fee, then it is definitely too soon to reinvest.
That gives us a lower bound on how often to reinvest. Furthermore:
If we knew how much shorter $t_1$ was going to be than $t_0$, we'd be able to use that method to find an exact answer.
Unfortunately, I can't find a way to compute that ratio. However, we can
Pretend it's a constant and plug in different numbers until we find the one that gives the best results.
I was rather surprised to find that I got best results was to assume that $t_1 / t_0 = 0.5$, even though the actual ratio approaches $1.0$ as the size of the principal increases. I believe this is because the earlier reinvestments (when the ratio is large) have much more significance than the later ones.
With a bit of algebra:
Let $p$ be the principal, $c$ be the transaction cost, and $i$ be the interest rate.
Find the time $t$ such that $ti(pti - c) = 2c$. (Which is equivalent to saying "The excess interest from this deposit will make back the transaction fee in at half $t$")
We can computer our money at that to be is $\sqrt{2pc} + c$
From there, we have a method:
Reinvest whenever we have $\sqrt{2pc} + c$ available.
We can do slightly better by knowing the duration - we can avoid making the final reinvestment if it won't pay itself back before the scores are calculated. Long term (if we don't know the duration), it seems to always best to invest on this schedule.
With that method, I get the following results:
Result for 3 = 1334.4774464102625 in 6 transactions
Result for 10 = 2626.26209457277 in 28 transactions
Result for 30 = 18736.580608089855 in 150 transactions
Result for 40 = 50518.34501317744 in 275 transactions
Result for 100 = 20137653.64033103 in 6308 transactions
Result for 300 = 9763932019226500 in 139742111 transactions