This is a "canonical" answer meant to move the hints and partial results out of the question, as well as present a unified description of the solution and explain some of my thoughts about the puzzle.
Warning, unmarked spoilers ahead!
Squeamish ossifrage Found a method that demodulates the signal into images in his answer. Using his technique we can extract a series of 12 'pages,' which are described below. Click on any of the pages to see an "annotated" version with the numbers converted into familiar units.
Page 1 — Counting
This page introduces us to the number system that the message uses, a form of balanced ternary. "Ternary" means it's base-3; instead of ones, tens, and hundreds places, we have ones, threes, and nines. "Ordinary" base-3 uses the digits $0$, $1$, and $2$, but balanced ternary instead uses the digits $-1$, $0$ and $+1$. $\def\T{\mathtt T}\def\Z{\mathtt 0}\def\E{\mathtt 1}$
In the message these digits are represented by "×," "−," and "+," respectively, but in this explanation I'll use the more standard $\T$, $\Z$, and $\E$ instead, to avoid confusion with mathematical operators. The examples given:
$$
\begin{array}{rcrcr}
\E &=& {}+1 &=&1 \\
\E\T &=& {}+3-1 &=& 2 \\
\E\Z &=& {}+3\phantom{{}+0} &=& 3 \\
\E\E &=& {}+3+1 &=& 4 \\
\E\T\T &=& {}+9-3-1 &=& 5 \\
\E\T\Z &=& {}+9-3\phantom{{}+0} &=& 6 \\
\E\T\E &=& {}+9-3+1 &=& 7 \\
\E\Z\T &=& {}+9\phantom{{}+0}-1 &=& 8 \\
\E\Z\Z &=& {}+9\phantom{{}+0+0} &=& 9 \\
\end{array}
$$
Notice that counting is similar to ordinary base-3; each digit cycles from $\Z$ to $\E$ to $\T$ and back to $\Z$, and when we "roll over" from $\E$ to $\T$, the next digit increments as well.
I chose balanced ternary not just because it's a little unusual, but because it also has a number of cool properties (no pun intended). For example:
- You can represent negative numbers without a minus sign; take the positive form of the number and replace all $\E$s with $\T$s and vice versa.
- Addition and subtraction are easy. When adding, a $\E$ and $\T$ cancel, and a pair of the same digit turn into the opposite, with a same-signed carry: $$\begin{array}{r|rrr}+&\T&\Z&\E\\\hline \T&\T\E&\T&\Z\\\Z&\T&\Z&\E\\\E&\Z&\E&\E\T\end{array}$$
- Multiplication is similarly easy. multiplication by $\E$ leaves the multiplicand unchanged, by $\Z$ yields $0$ of course, and by $\T$ flips the sign of the multiplicand: $$\begin{array}{r|rrr}\times&\T&\Z&\E\\\hline \T&\E&\Z&\T\\\Z&\Z&\Z&\Z\\\E&\T&\Z&\E\end{array}$$ This means that when multiplying two numbers, you can skip writing down the multiplicands and go straight to writing the partial products. For example to multiply $\E\T\Z$ ($6$) by $\E\T\E$ ($7$) we would write: $$\begin{array}{r}\E\phantom{\Z\Z\Z}\\\E\T\Z\\\T\E\Z\phantom\Z\\{}+\E\T\Z\phantom{\Z\Z}\\\hline\E\T\T\T\Z\end{array}$$ Which is, of course, $81-27-9-3=42$.
Page 2 — Arithmetic
This page demonstrates the basic arithmetic operators (equality, addition/subtraction, and multiplication/division) and their notation.
Expressions are represented in prefix notation, instead of the typical infix notation. In infix notation the operator is placed in between its operands, for example $1+1$. In prefix notation, the operator comes before its operands. In prefix notation, $1+1$ might be written ${}+1\ 1$.
The reason I chose prefix notation is that it doesn't require parenthesis, making the number of symbols I had to invent and explain smaller; similar to how I eliminated the need for a minus sign by choosing balanced ternary. However, I do use parenthesis in the annotated drawings to make things a little easier to read.
Expressions are read in the direction of the triangles. They serve as a graphic indicator of which end of the numbers are most and least significant. This direction indicator allows expressions to be written in any direction (or in multiple directions, as we will see later). I'll call the most significant end (flat end of the triangle) the upstream end, and the least significant end (point of the triangle) the downstream end. Note that the prefix operators are upstream of their operands.
Operands with no operator between them are separated by a small vertical bar, which acts a sort of comma. If the vertical bar comes next to the direction marker, the two are combined together into a single symbol that looks sort of like a "fast-forward" or "skip" button.
The operators themselves are based on diagrams of the operations they represent: comparing two things side-by-side for equality, placing two things end-to-end for addition, and forming an area from two perpendicular lengths for multiplication.
The block-like depiction of the example operations are probably inspired by the manipulatives we used in elementary school math.
Addition and multiplication both have three parts: the two addends (or augend and addend) and the sum, and the multiplier, multiplicand, and product. However, their operators only have two arguments or inputs. Which roles the arguments play are determined by the two parts of the operator that are shown; the missing third part is the value of the whole expression. For example:
- Addition is represented by the two addends: two short bars placed end-to-end.
- Subtraction is represented by a sum and addend (minuend and subtrahend): one long bar next two one short bar.
- Multiplication is represented by the multiplier and multiplicand: two bars at right angles.
- Division is represented by the product and multiplicand (dividend and divisor): a rectangle with a bar on its side.
- Equality has no "output" value, so it is always represented by both its left-hand and right-hand sides: two bars of equal length side-by-side.
The example for equality has the first mistake of this puzzle: the example equation reads $16=5$ instead of $5=5$ due to an extra $\E$.
One final detail: the order of the parts in the operator corresponds to their order in the expression. $a-b$ is represented by 'long bar, short bar, $a$, $b$.' But if we want to switch the order around and do $b$-$a$, we could represent it without changing the order of the operands as 'short bar, long bar, $a$, $b$.'
Page 3 — Decimals
Here we see that the triangle functions not only as a direction indicator, but also as a decimal point. We also introduce the circle as a symbol for repetition. A small circle at the end of a number it functions like ellipsis, while a circle or oval around a set of digits denotes a repeating decimal (like a vinculum or overbar).
Probably the most interesting detail on this page is that one number is shown to have two repeating decimal expansions! This is the balanced-ternary equivalent to the expression $0.999\ldots=1$.
Page 4 — $\pi$
This page shows the geometric meaning of $\pi$, its approximate value, and one continued fraction expansion for it. The expression is two-dimensional, with the dividends branching off at the division operators. In (linear) prefix notation the expression is:
= π + 3 ÷ × 1 1 + 6 ÷ × 3 3 + 6 ÷ × 5 5 + 6 ÷ × 7 7 + 6 ...
This is equivalent in standard notation to:
$$
\pi = 3 + \cfrac{1\times 1}{6+\cfrac{3\times 3}{6+\cfrac{5\times 5}{6+\cfrac{7\times 7}{6+\cdots}}}}
$$
Note the explicit multiplication, since we haven't introduced a notation for exponentiation (squaring).
Looking back, I wish I had used $\tau$ (tau) instead of $\pi$ here.
Page 5 — $e$ and Scientific Notation
Like the previous page, this one shows a geometric definition of $e$, its approximate value, and one continued fraction expansion for it. In prefix notation the expression is:
= e + 2 ÷ 1 + 1 ÷ 1 + 2 ÷ 2 + 3 ÷ 3 + 4 ÷ 4 ...
This is equivalent in standard notation to:
$$
e = 2 + \cfrac{1}{1+\cfrac{1}{2+\cfrac{2}{3+\cfrac{3}{4+\cfrac{4}{\cdots}}}}}
$$
We also see a form of scientific notation. Repeated zeros are "pulled" into the decimal point; to denote this, the decimal point is circled and labeled with the number of decimal placed moved (the exponent). Remember these are base-three decimal places, so increasing the exponent by one increases the value by three times, not ten!
Page 6 — Atoms
Here four atoms are depicted, along with one lone electron. The electron is labeled $1$. The atom with one nucleon and one orbiting electron, hydrogen, is labeled $1837$. This is the ratio in mass between a proton and electron, so the labels are the masses of the atoms in electron mass units. Note that the mass of the carbon atom is slightly different from the value you find in the periodic table: that value is a weighted average of the different isotopes of carbon that occur on Earth, while the value given on this page is the weight for pure carbon-12 (the isotope depicted). The values for hydrogen, helium, and oxygen will be the same as in the periodic table because they are all dominated by one isotope on Earth.
I chose these specific elements because they represent the main stages in the stellar evolution of a Sun-like star: hydrogen burning on the main sequence produces helium, helium burning forms carbon as the star transitions into its red giant phase, and the carbon fuses into oxygen as the star contracts into a white dwarf.
There is also a line next to the hydrogen atom equal in length to the radius of the electron's circle; this line is also labeled $1$. This indicates that the distance unit is the Bohr radius $a_0$, in some sense the "radius" of that electron's orbital.
If you look carefully you will notice that the electrons are labeled with $\E$ and the protons with $\T$, a sign convention opposite to ours. This probably makes electronics a little easier for these aliens!
Page 7 — Water
This page shows a water molecule. The atoms, hydrogen and oxygen, are labeled by their number of electrons. The atomic weight is given in electron mass units; the bond length is given in Bohr radii; and the bond angle is given in radians.
Labeling by the number of protons probably would have made more sense, since the electrons move around when atoms are bonded; but since the electrons are represented as positive, the aliens probably think of them as the more important part of the atom.
I initially considered putting an ammonia molecule here, but decided to make it a little more obvious for the puzzle-solvers.
Page 8 — Transmitter
A parabolic reflector is depicted on this page. The diameter of the dish and the wavelength are labeled in Bohr radii. The wavelength, 21.16 cm, is the wavelength of the hydrogen line, equivalent to a frequency of 1420 MHz, theorized to be an important frequency for SETI.
I thought about making the frequency "pi times hydrogen," or 4462 MHz as a reference to Contact, but eventually decided against it.
The number to the right along the transmitted beam is the speed of light, given in atomic units. In atomic units, $c$ is dimensionless and equal to my favorite number: $1/\alpha$, the inverse of the fine-structure constant.
Finally, the number at the focus of the transmitter is the transmitting power (again, in atomic units).
Page 9 — The Solar System
This page gives an overview of the solar system the aliens are transmitting from. From left to right, we have:
- The star;
- A small Mercury-like planet;
- Two hot, Neptune-like gas giants:
- One with one moon,
- and another with a ring system and five moons;
- A cool, rocky super-Earth with two moons;
- And a cold, rocky planet with one moon.
The distances from the star are given in some as-yet unspecified planetary distance units. The large fourth moon of the second gas giant is shown transmitting; it's (presumably) the home of the aliens who sent the signal.
Designing the solar system was probably the most fun part of this puzzle. Other than the star and the two gas giants, everything is made up (including the rings).
Page 10 — Uninhabited Planets
The characteristics (mass, radius, and orbital distance) of the first two and last two planets and their moons are described on this page. They are given in the same unspecified distance and mass units.
The second mistake in this puzzle is an extra $\T$ in the orbital radius of the gas giant's moon. The correct value is given in the annotated image.
Page 11 — Inhabited Planetary System
The characteristics of the ringed gas giant and its system of moons are described on this page. Note that the first two distances near the planet are supposed to be the inner and outer radii of its rings. One of the moons (the large one shown transmitting on page 9) has its characteristics described by two equations:
$$
1 = 1.45\times 10^{53} \\
1 = 4.88\times 10^{16}
$$
These give the values of the planetary mass and radius units in terms of the electron mass units and Bohr radius units used in the earlier part of the message.
Page 12 — Stars
The final page describes two stars, one of which has two planets depicted. The stars' and planets' characteristics are given in the same planetary units as before. Converting to our units, we can see that the second system (mass and radius of the star, and the mass and orbital radii of the planets) exactly match the Sun, Jupiter and Saturn.
The third error in this puzzle is that the masses for Jupiter and Saturn are swapped. The corrected values are in the annotated image.
The distance between the stars is also given; but it is not in the planetary distance units: it is in units of the orbital radius of the aliens' moon!
This distance was originally supposed to be in terms of the radius of the aliens' star. I thought that this had a nice symmetry: distances between objects are measured in terms of a "standard" object of that type. However, I mistakenly used the wrong unit in my calculations and ended up with what you see in the puzzle.
Conclusion
Using the information from the message, the only star that matches is Gliese 785.
math... $\endgroup$ – Bobson Jun 30 '15 at 17:34