Haven't heard this one before, though I believe there is a class of similar puzzles. I don't know if they have a name.
Naturally, the puzzle can be solved.
The simplest way, without requiring any special memorization prowess, is as follows:
Consider the 13 positions as numbered from 0-12. Note that this solution doesn't actually require a line with a start and end, as long as there is an order (a circle works as well if we consider clockwise).
With two numbers among 13 values, we know they can be at most 6 apart (modulo 13). For example, 1 is 5 away from 6 (2,3,4,5,6), 6 away from 7 (2,3,4,5,6,7) and then back to 5 away from 8 (12,11,10,9,8).
The assistant notes
The ball that is 6 or less away from the other (modulo 13). That is, they would choose the smaller position if the distance is <=6, otherwise the larger one. For example, if the balls are in 7 and 10, they choose 7. If they were in 1 and 10, they would choose 10.
The assistant then considers
The balls to be relabeled as follows:
_ _ _ X ? ? ? ? ? ? _ _ _. Where X is the noted ball. By construction, the other ball must be in one of the positions marked with question marks.
Finally, the assistant chooses
One specific ball in one of the three positions before the noted ball, as follows:
_ _ A X a a a ~ ~ ~ _ _ _. Selecting A if the second ball is the 1st, 2nd, or 3rd after the noted
_ B _ X ~ ~ ~ b b ~ _ _ _. Selecting B if the second ball is the 4th or 5th after the noted
C _ _ X ~ ~ ~ ~ ~ c _ _ _. Selecting C if the second ball is the 6th after the noted
The magician then enters, and reverses as follows
Starting with the position after the chosen, they lift up until they find the first ball.
If they find it after one lift, they are in scenario A, and can continue lifting with their remaining 3 guesses.
If they find it after two lifts, they are in scenario B, and skip ahead 3 before continuing to lift.
If they find it after three lifts, they are in scenario C, and skip ahead 5 to find the last ball.
I choose positions 3 and 10.
The assistant notes 10, since 3 is 6 ahead of 10 (
3-10 mod 13 ~= -7 ~= 6).
The assistant chooses 7 (3 before 10) and is done.
The magician chooses 8, 9, finding nothing.
The magician chooses 10, and then skips 5 (11, 12, 0, 1, 2), choosing 3.