There is a nice general theorem governing this sort of thing (which I'll give at the end), but here's how to do it from first principles.
I'll assume, for reasons set out in my comment on the original question, that A and B appear independently with probability 1/2 each in each position. The probability we're after will be different if the frequencies of A and B are different, or if their probabilities in different places are not independent.
OK. Now draw a graph (in the dots-and-arrows sense, not the how-y-depends-on-x sense) whose vertices are the prefixes of the words ABA and BABA. That is: [empty], A, B, AB, BA, ABA, BAB, BABA. We want to find, for each prefix p, the probability that if we have just read p (and not any longer prefix) we will find ABA before BABA when we continue.
The cases p=ABA and p=BABA are easy (the probabilities are 1 and 0 respectively) and the case p=[empty] is the one whose answer we actually want, which I'll call q. (With hindsight, using "p" to stand for "prefix" may have been a bad choice.) Let's call the others pA, pB, pAB, pBA, pBAB. We can write down a bunch of simultaneous equations for them, the idea being that e.g. if we've just seen BA then after the next letter we'll have seen BAB or BAA, with probability 1/2 for each -- and the latter is exactly equivalent to having just seen A.
So:
$\eqalign{q &= (pA+pB)/2 \\
pA &= (pA+pAB)/2 \\
pB &= (pBA+pB)/2 \\
pAB &= (1+pB)/2 \\
pBA &= (pA+pBAB)/2 \\
pBAB &= (0+pB)/2
}$
The first equation doesn't tell us much yet. The next two tell us that $pAB=pA$ and $pBA=pB$ and the last tells us $pBAB=pB/2$, so now we have everything in terms of pA and pB.
$\eqalign{q &= (pA+pB)/2 \\
2pA &= 1+pB \\
pB &= (pA+pB/2)/2
}$
So, finally,
the simultaneous equations yield $pA=3/4,pB=1/2$ and then the first equation gives $q=5/8$.
I promised to tell you the general theorem, so here it is. You can find it in the "Discrete probability" chapter of "Concrete Mathematics" by Graham, Knuth and Patashnik. It's due to John Horton Conway (as if he didn't have enough other neat mathematical things to his credit!).
Given a string s of As and Bs, define prefix(s,k) to be its first k letters and suffix(s,k) to be its last k letters. For two strings s,t, say s ~k~ t if suffix(s,k) = prefix(t,k): the end of s matches the beginning of t. And now define a number s:t which has bit k-1 set iff s ~k~ t.
Theorem: the odds in favour of seeing s before we see t are t:t-t:s to s:s-s:t.
So in the present case s=ABA and t=BABA; so
in binary we have s:s=101, s:t=010, t:t=1010, t:s=101, so the odds are (10-5) : (5-2) = 5:3, which is the same thing as the probability being 5/8.