The problem with the Rube Goldberg machine (which I think are awesome, btw) has to do with Newton's laws of motion.
His first law is that of inertia - an object will not change its velocity unless acted upon by an outside force. This definition of inertia is slightly a misnomer, but the point we need from this is that mass and movement are two separate things, which also sets up the second law of motion.
His second law of motion describes force. Force is given as $F=ma$, where $m$ is mass in kilograms and $a$ is acceleration in meters per second, and it is calculated in Newtons. It is obvious (that is, most of us can tell from when we throw our laptops at the wall in frustration over not being able to solve a riddle) that if an object with $F_1$ Newtons of force hits another object with $F_2$ Newtons of force, the first object's new force is simply $F_1 - F_2$. If the first object has less force than the second object, its new force will, obviously, be negative, meaning that the second object pushes the first one backwards.
Finally, the third law of motion says that every action has an equal and opposite reaction.
Say that the first domino has a force of 10 and a mass of 1. The second domino, based on the number given in the riddle, will have a mass of 1.65. You'll need enough force to overcome the second domino's inertia. As we said above, inertia is essentially the same thing as mass, and thus we need at least 1.65N to push the domino over.
Now, based on the third law of motion, the second domino will exert 1.65N of force back on the first domino. Based on the second law of motion, there are 8.35N of force remaining in the domino.
Since each domino still has 1.65 times the mass of the previous domino, the third domino will have a mass of about 2.72. Repeating the steps from above means that 5.63N continue onto the fourth domino, 1.13N to the fifth domino, and -6.28N to the sixth domino. Because of the negative amount of force, not only does the sixth domino not fall, but it applies 6.28N back on the fifth domino, thus causing it to slip, as you described.
Since each domino has a progressively lower amount of force combined with a higher amount of mass, the acceleration must be decreasing significantly, which explains why the dominoes take slower and slower to tip over.
The reason that each time the machine is run a different domino is left standing is because the above numbers were based on an initial force of 10. If you increase or decrease that, the number of dominoes could increase or decrease as well, depending on how much you changed the amount of force.
The above model is an extremely simplified version of how the real world works. I didn't take into account things like the increase in acceleration that occurs due to gravity after the domino begins falling, or the decrease in acceleration that occurs due to air resistance/friction. However, the general idea is the same.
