Yes! I think I got it, the solution is:
Steps to solution:
Walrus from Beynac
Wikipedia says Beynac is 'a commune in western France', Walrus translates to french Morse, that suggests Morse code was used
Oddly placed divs in
<div>tags amid binary numbers (in the puzzle's edit history)
After splitting binary by divs
<div>and substituting ones for dots and zeros for dashes and decoding message using Morse code I got WBGGSPUNZAHJRLEJOHUNL (also there is a new line at end of that string but I just ignored it)
Number hidden in first stanza
Dozen less half = 12 - 12/2 = 6
Baker's dozen = 13
Together it's 19
I guess last past of the riddle was supposed to guide me to Caesar cipher but I just blind guessed to use it :)
Now we just need to apply 19 to shift string from step 2. Et voilà!
OP Reason: Actually the first stanza's solution is this:
When all is going = rot
a dozen less half, be sure to bring the Baker back = 6+1 aka 7
So together they make rot7.
But this is how the cipher was made, so in actuality, rot19 is required to decipher it.