Some things that appear not to work
This is in some sense "less than a partial answer". But since this simple-looking puzzle has been here unsolved for so long, even with a bunch of hints, it seems worth doing anything that might help. So, here are some things I've considered that don't appear to lead anywhere useful. (Obviously I'd prefer to post a list of things that do lead somewhere useful. If only there were any.) I might have screwed some of them up, of course. Maybe something here will give someone an idea that leads somewhere. Or maybe something here actually does work and OP will take pity on me and indicate somehow how I goofed.
First of all, some obvious observations.
OP's hints make it very clear that the authors are relevant. Exactly one of the "annoying books" has an author. What's notable about that one? Well, the title is John Buchan and the author is William Buchan. So (1) the title is shorter than the full author name and (2) the author's surname is a substring of the title. (I guess the author is actually related to the subject of the biography, but we've already been told in hints that this sort of external information isn't required.)
So, what might we try? Well, first of all,
OP's hints kinda suggest that we might want to give very close visual scrutiny to the authors' names on the spines of the books. Some sort of steganographic wizardry, perhaps? I've looked closely and haven't found anything obviously interesting, and (though I haven't attempted any sort of serious analysis) it doesn't look as if there's anything magical in e.g. the low bits of the pixel values. In any case, it seems like that wouldn't be fair; this is meant to be an ordering of the books, not randomized books with some sort of ordering then encoded into the picture.
More to the point:
Take the author's name (or just the surname?) as given on the spine, and remove from it all letters found in the title. Hope that we end up with something that yields an obvious ordering. Doesn't seem to work because for the second book on the shelf (Stringer) no letters are left after doing this.
Look for letters shared between author's name and title. Doesn't seem to work because it's hard to see why the Buchan biography would be Annoying in this case, and e.g. the first book has T,I,N (in order of appearance in either title or name) and there's no way that's coming out alphabetically ahead of the second, no matter how we order it. (Well ... sort into order and use reverse lexicographic order; T matches T, N is earlier than S. But then other books later on the shelf are going to come out earlier than that.)
Take the author's name (whole name or surname), index by its length (with or without spaces) into the title (with or without spaces, respectively), either from the start or from the end, and see if we get anything that yields an obvious ordering. Doesn't seem to work because many variants are impossible because either Tavris & Aronson or Emanuel & Mannheim have too-long author names (depending on whether we include spaces and punctuation), and I think I've tried most of the possibilities without finding anything useful. (There are a lot of different possibilities and I certainly haven't been completely exhaustive. Trying out dozens of essentially-identical things in the hope that one of them will turn out to work is Not Fun.)
Something to do with words embedded in the authors' names. ("Buchan" doesn't have much, though "ch", "ha", "a", and "an" are all Scrabble-legal and the last two are hard to quibble with.) Doesn't seem to work because "Buchan" does have some embedded words, and no ordering is apparent in the ones found in the authors' names on the shelf; e.g., "smith" is later in the alphabet than anything you can get from "Caplin" if we use only surnames, and if not then "a"/"all"/"ally" are all earlier in the alphabet than anything you can get from "Stringer".
Look for some physical feature of the book spines that somehow picks out particular bits of the authors' names. Doesn't seem to work because for some (Smith, Caplin) there really truly seem to be no such features we can use -- unless, again, there's some steganographic wizardry going on.