Since every mating produces two offspring and results in the immediate death of both parents, and since there is no other source of death on the planet, the total population of creatures is constant over time; only the relative population levels of the genders can change. (We'll leave the question of how these creatures reached their current population level for worldbuilding.SE)
The population naturally self-balances. That is, if all but one of the population is of gender 'alpha', and if the last other individual is of gender 'beta', then an alpha will mate with the beta, producing two gammas. And each of those will then mate with another of the alphas (since they can't mate with each other), producing four more betas. And so on, with the overbalance of alphas slowly decreasing with each cycle.
Let's do some eugenics, boys and girls!
So in order to prove what would have to happen to make the population collapse to a single gender, let's try to engage in selective breeding to make that happen intentionally. For the following discussion, assume that we want only alphas to remain.
Now, it's easy to reduce the population to only alphas if we have precisely the same number of betas as of gammas; we'd just pair the betas with the gammas and produce entirely alpha offspring, while the betas and gamma parents die, and we're done. But to do this, we need to have precisely the same number of betas as gammas, because if there is even a single surviving, unmated beta or gamma, then the population diversity will eventually recover given enough mating operations (as per the second observation, above).
So let's assume that we don't have the same number of betas as of gammas. If the number of betas doesn't match the number of gammas, can we perform selective breeding in order to make those numbers match?
Well, there are three moves we can make:
- Mate alpha with beta. We wind up with one fewer beta, and two more gammas.
- Mate alpha with gamma. We wind up with one fewer gamma, and two more betas.
- Mate beta with gamma. We wind up with one fewer beta and one fewer gamma.
We can eliminate move #3 as irrelevant; it doesn't bring the number of betas closer to the number of gammas. Moves #1 and #2, on the other hand, can be used to bring their numbers closer together by 3.
Regardless of the total population of the planet, if the difference in the number of beta individuals from the number of gamma individuals is a multiple of 3, then we can perform selective breeding in order to make the total number of betas equal to the total number of gammas. And once we've done that, then we can make the betas and gammas mate and die off, resulting in a population consisting only of alphas.
And on the other hand, if the difference between the number of betas and the number of gammas is not a multiple of 3, then we cannot ever make them equal via the available mating operations, and it's impossible for there to ever be a population consisting only of alphas.
Of course, this is also true for each of the other pairs of genders; if the difference between alphas and betas is a multiple of three, we can equalise their numbers and produce a population entirely of gammas, and if the difference between alphas and gammas is a multiple of three, we can equalise their numbers and produce a population entirely of betas.
And finally, the answer:
The puzzle specifies that we do eventually determine that it's impossible for the population to become entirely composed of any one gender.
To reach that conclusion, we must have taken a census of how many individuals there are of each gender, and determined that no pair of genders have a difference that is a multiple of three.
As a result of this, by the logic above, no mating operations can possibly occur which would result in any pair of genders having the same number of individuals. And since no two genders can ever have the same number of members, they cannot ever be bred together to remove all members of both genders at the same time.