Partial Answer (almost done; can't figure out the last part)
Warning: The design of this puzzle pretty much requires some mild profanity, so consider yourself warned.
Riddle
Note: I'd already figured this part out before @bobble posted her answer.
Looking at the code, I recognized it as
based on the programming language called "Brainfuck."
Looking at the riddle,
My prefix is something that controls you.
The prefix is "brain," which is also the name of what controls us.
My suffix is something that students mustn't do.
The suffix is "fuck," which, of course, students shouldn't do.
My infix is one over nothing.
The infix is "inf." $\frac{1}{0}$ (one over nothing) is sometimes said to represent infinity, which is often abbreviated "inf." More formally, $$\lim_{n\to 0^+}{\frac{1}{n}}\to\infty$$
My whole is the key to the lines below.
The whole, "Brainfuck," is the name of the programming language for the code.
Code
Brainfuck only has the following characters: ><+-.,[]. You can read more about these commands on Wikipedia. To summarize them, > and < move the data pointer one cell right or left, respectively. + and - respectively increment and decrement the data at the pointer. It does not have numbers, so I had to figure out what the numbers were for.
Numbers‽
My first guess was that a number meant to add that number. That was wrong. My next guess was that a digit meant to add that number. Also wrong. Then, I noticed that it alternated between numbers and commands, so I decided that the numbers probably meant to run the following command that many times. This turned out to make sense.
Decoding
I tried using Hint 6 as a "practice run" to see if I was right. I noticed that each . was preceded by a 1 most of the time, with the 2. corresponding to what I hoped was merely a double letter. This gave me more confidence. Using the value at the pointer for each output command as the ASCII value for a character, Hint 6 says "FIVE CROSSES" makes 0. At this point I didn't know what that meant, but I knew "FIVE CROSSES" was definitely not gibberish. (I'll explain this later at the point in the process that I figured it out.)
Using the principles I used to decode Hint 6, the 6 ASCII-encoded messages are:
(1) 82 69 68 32 87 72 73 84 69 32 83 80 76 73 84
(2) 70 76 65 71 32 85 83 69 68 32 73 78 32 83 69 77 65 80 72 79 82 69
(3) 70 79 85 82 32 67 79 76 79 82 32 83 81 85 65 82 69
(4) 70 76 73 80 32 84 79 80 32 79 70 32 49
(5) 83 67 79 84 76 65 78 68
(6) 65 32 82 79 85 78 68 32 80 69 71 32 73 78 32 65 32 83 81 85 65 82 69 32 72 79 76 69
Converting these numbers to text, the messages/clues read:
(1) RED WHITE SPLIT
(2) FLAG USED IN SEMAPHORE
(3) FOUR COLOR SQUARE
(4) FLIP TOP OF 1
(5) SCOTLAND
(6) A ROUND PEG IN A SQUARE HOLE
These clues each correspond to a signal flag as follows:
(1) ? (RED WHITE SPLIT)
(2) O (FLAG USED IN SEMAPHORE)
(3) ? (FOUR COLOR SQUARE)
(4) ? (FLIP TOP OF 1)
(5) M (SCOTLAND)
(6) ? (A ROUND PEG IN A SQUARE HOLE)
To figure out which system to use for the numeral 1, I noticed that five crosses makes 0 in the U.S. Navy only. (This is when I finally understood Hint 6.)
This is the farthest I got. I've gotten past all the hints, but I'm stuck.
