How do I go about solving this cipher?

Question

A friend and I signed up for a pay-to-play D&D campaign some time ago. Near the start of the campaign, our characters were given a puzzle to figure out. We're approaching the end of the game in a couple weeks and we still haven't figured it out. We asked the GM for clues and context, but he declined saying we had everything we should need already to solve it. This is what we were given:

---

As you move into the next room of the Quizmaster, the letters on the wall are painted in red 'ink'. Something feels extremely discomforting about them. There is a musty smell in the room. This room has the letters in rows, but the size of them seems to increase as each 'phrase' progresses.

On the door opposite, you see the following:

There is a small tablet on which to write your answer.

---

We've tried most of the brute force options on Dcode and attempted to guess the keyword to no avail.

I will accept any answer that successfully decrypts the message above and includes a brief, step-by-step approach to how you solved it. I'm hoping to learn from this experience for the future.

Answer 1

Each letter in the plaintext is shifted forward by a number determined by

The Fibonacci sequence: +1, +1, +2, +3, +5... ...+610, +987, +1597.

My process:

Since it's a question, the first word might be WHAT or WHEN—meaning the first two letters are shifted by +1, but (if it's WHEN) the third letter is +2 and the fourth is +3. At first glance this seems so arbitrary that I must be barking up the wrong tree! But the clue "the size of [the letters] seems to increase as each 'phrase' progresses" supports the idea that the "key" is a number that keeps going up. If we hypothesize that the "key" is defined mathematically, then the sequence 1, 1, 2, 3... looks more familiar.

The answer:

Take the first dozen or so terms of the sequence, modulo 26, and reverse-shift each letter to get: WHEN WILL YOU GIVE UP?

Answer 2

[Ryan Veeder posted his solution whan I was about to write mine, so the Green Tick should go to him. I shouldn't post a solution that was already given, but I wrote a longer section on how I go about solving such things that also includes dead ends, so I'll let it stand.]

The message reads:

When will you give up?

How is it encoded?

The room description talks about "letters in rows, but the size of them seems to increase as each 'phrase' progresses". This seems to indicate an ever increasing shift, but it's not a linear increase.

Instead, the shift of the n-th letter is by the n-th number of the Fibonacci series, modulo 26:

1, 1, 2, 3, 5, 8, 13, 21, 34 (8), 55 (3), 89 (11), ...

I'm not quite sure how you are supposed to find out that the Fibonacci series is used here. There are many series that are increasing. The Fibonacci numbers are related to the golden ratio, but I can't see any hint to that, either. (Sure, the letters look kinda "golden", but the other letters are supposed to be written in blood.)

How did I find it?

The first things to check when seeing such a message is to do a frequency analysis and to check for simple substitutions.The frequency analysis isn't really useful for such short texts, but it usually allows you to tell whether the cipher is a transposition cipher, where the original letters are used, but "scrambled". With three Q's and only one U, that's not likely. (But if you have a good consonant-to vowel ratio, you can try to rearrange the letters in rectangles and see whether the columns spell something useful or try a Railfence cipher.)

Simple substitutions can be found with Quipqiup. This doesn't yield anything useful here. Use both settings to rule out that spaces are just decoration, that is the real spaces are somewhere else. Also check the message reversed. I usually also check all 26 possible Caesar shifts. A popular trick seems to be that each word is shifted by the number of its letters, but since there are so many four-lterre-words, that doesn't seem likely.

The steps that lead to actually finding the code were the same as Ryan's: The question is likely to begin with an interrogative word and "when" and "what" seem likely. (The word could of course be any verb, but that would lead to either yes or no as answer, which seems a bit lame. We're probably supposed to write "never" or perhaps "ofxhw" on the tablet.)

My first thought was that the first letter was misencoded, but then I saw that 1, 1, 2, 3 is the beginning of the Fibonacci seriesand tried that. (But just the first four items are not enough to identify a series.)