First, number the coins using Balanced Ternary. This is a ternary numeral system using the digits -, 0, and + to represent -1, 0, and 1.
For the sake of brevity I'm going to illustrate this using 12 coins, but this can be easily extended to larger numbers. Just add more trits to each number. (For example, 120 coins would require 5 trits, corresponding to 5 weighings.)
- 1 = 00+
- 2 = 0+-
- 3 = 0+0
- 11 = ++-
- 12 = ++0
Each of these can also be inverted by swapping each symbol.
- -1 = 00-
- -2 = 0-+
- -3 = 0-0
This numbering now tells us where to place the coins.
For the first weighing, put all the coins with a + in their first trit on the right pan, and all the coins with a - in their first trit in the left pan. Leave all the coins with a 0 in their first trit off. Then record the result: if it tips to the left, that's a -. If it tips to the right, that's a +. If it is balanced, that's a 0.
Repeat as many times as necessary, then look at the number you've recorded. With 12 coins there will be three weighings, so the result might be e.g. +-+ (tipped to the right, tipped to the left, tipped to the right). This is the number of the false coin. If it's positive, that coin is heavier. If it's negative, that coin is lighter. If the number is zero, no coin was fake.
HOWEVER, this doesn't actually work precisely as I have written it. No positive number has a - in its first trit, so the first weighing will have one pan empty. Unless you have a stock of real coins to compare against (a variant of the problem), it won't be a useful measurement.
So we need to invert some of the coins' numbers (so coin 5 might become coin -5), then interpret the result differently if one of these is chosen: a negative number will now mean that the fake coin is heavier, and a positive number that it is lighter.
This is where I got stuck in my own experimentation, but some web research indicates that John Conway came up with a very clever way to determine which ones to flip: go through the coins, and look at the first "change" in the sequence. If it is - to 0, or 0 to +, or + to -, keep it positive. Otherwise, invert that number.
This isn't as elegant as I would like—it feels like there should be some way to decide mathematically which numbers to invert. But for now, that solution is the best I have.