I love this question! Here is what I managed to think of. The puzzle statement uses minimal math as required. I'm less sure whether it counts as a "good" puzzle in its current state, I think it's a bit too long and convoluted. But hopefully someone more experienced in puzzle-making can take it as a starting point, or at least find some interesting idea they can use. (I also apologize in advance for my English, I'm a non-native speaker).
Puzzle:
A group of evil aliens have at their disposition a powerful spaceship fleet, which they use to destroy planets whenever they feel like.
The fleet consists of 100 ships in total, numbered from $1$ to $100$. It is known that the aliens use a set of connectors, filled with a very powerful but very unstable chemical substance, to power up these ships. This chemical flows through a number of cables, which join the inputs of the connector to its outputs in some specific way. The cables can't be seen from the outside, but each connector comes with a corresponding label depicting what's inside.
The circuit board of Ship $N$ uses a fixed sequence of chained $N$-cable connectors, with the $N$th overall input of the sequence connected to the $N$th respective overall output as in the picture below.
The circuit is always designed in such a way that the chemical flows in a single closed loop. This is extremely important, otherwise the resulting magnetic instability would make the ship explode.
The aliens collectively enter a deep hibernation state every night. There is a group of saboteurs whose home planet is next in line to be destroyed, and so would like to disable the alien fleet before that can happen. They plan to do this by swapping the labels of the connectors in the right way while the aliens are sleeping.
But the aliens are aware of the possibility of a sabotage: after all, they have garnered many enemies during their planet-destroying quest. As mentioned, the connectors contain an extremely unstable chemical, so they cannot be opened or tampered with at all. They can't even check the inputs one by one: the chemical must always flow through all the cables at the same time so as not to disturb the delicate equilibrium of forces. Despite these limitations, the aliens have in their possession the Connector Chain Comparing Contraption (abbreviated C4), a miracle of engineering which is able to compare any two arbitrary chains of connectors and tell whether they act the same for all possible input signals or not. This allows the aliens to make sure that the labels of the connectors are internally consistent. For example, the C4 can be used to check whether two copies of the three-cable connector labelled XI chained together act the same as a single three-cable connector labelled III; if the C4 returns a "No" answer, it is certain that at least one of these three connectors is mislabeled. To prevent the ships from being harmed, the aliens always use the C4 to meticulously test every possible pair, triplet, etc. of $N$-cable connectors (not only those who will be loaded onto the ship, but the whole supply) for inconsistencies in the labels before loading the connector chain onto Ship $N$'s circuit board.
However, the aliens know that the C4 is not powerful enough to ensure that the labels match the true content of the connectors. Long ago, there was an incident where the manufacturer sent all the connectors of Ship $57$ horizontally flipped by mistake (that manufacturer is now dead). Clearly, the C4 is unable to detect that kind of mistake. Fortunately for the aliens, such "C4-invisible" mislabelings have always turned out to be inoffensive in practice: in every known case, the mislabeled sequence of connectors still produces a circuit with a single loop when loaded onto the ship. The following image shows another example, where a former group of saboteurs managed to bypass C4 by following the strategy depicted in the upper part of the image: assign a given $4$-cable label to the connector obtained by moving the third input and output to the leftmost position (keeping the connections the same), and do this for all possible $4$-cable labels. This sabotage failed; as shown in the lower part of the image, the mislabeled chain of connectors still produced a single loop of cable, allowing Ship $4$ to function normally.
After checking lots of examples by hand, the aliens have concluded that all "C4-invisible" mislabelings are most likely harmless, and consequently that their routine checks with the C4 device will suffice to prevent any possible sabotage. But are they right...?
Can the saboteurs succeed? If so, what strategy must they follow?
Solution:
An $N$-cable connector can be thought of as a map from the set of $N$ inputs (taken in order) to the set of $N$ outputs (taken in order). This evidently corresponds to a permutation, i.e., an element of the group $S_N$. The existence of the C4 machine forces the saboteurs to relabel these elements in such a way that composition of permutations is preserved, so a "C4-invisible" label swapping defines an automorphism of $S_N$.
If we load an arbitrary chain of connectors to Ship $N$, generically we will have $n$ loops, with respective lengths $l_1, l_2, \ldots, l_n$, such that $\sum_{i=1}^n l_i=N$. The unordered list $(l_1, l_2, \ldots, l_n)$ defines the cycle type of the permutation associated to the chain of connectors. The statement of the puzzle implies that Ship $N$ will work if and only if the circuit forms a single loop, in which case the cycle type is necessarily $(N)$. It is known that inner automorphisms of $S_N$ preserve the cycle type of its elements, so since $S_N$ has no outer automorphisms for $N\neq 6$, it is mathematically impossible for the label swapping to make Ship N malfunction if $N\neq 6$.
However, if $N=6$, it can be easily checked that the outer automorphism of $S_6$ sends permutations of cycle type $(6)$ into permutations of cycle type $(1,2,3)$, i.e., it turns the single loop into three loops of different lengths. So this is the solution of the problem: swapping the labels of all the $6$-cable connectors according to this automorphism (i.e., according to any representative in the equivalence class corresponding to the nontrivial element of $\mathrm{Out}(S_6)$) will make Ship $6$ explode and save the saboteurs' home planet.
I don't know how plausible it is that someone who isn't already familiar with group theory could stumble upon the solution, but at least the emphasis on loops of cables naturally leads one to think about cycle types, which is a crucial element of most proofs that $S_6$ is the only symmetric group with a nontrivial outer automorphism (see e.g. here).