OK, I see what's gone wrong. I had blithely assumed your initial ansatz was correct, but it wasn't. I should have realized this earlier since you mixed annihilation and creation operators, which usually means the B-transformed Fock space is unitarily inequivalent to the original. But it's not.
The diagonalization should be dead easy for this case. Suppressing your scalar coefficients, one can write the Hamiltonian as:
$$
H ~=~ a^\dagger b + b^\dagger a
~=~ \pmatrix{a^\dagger &b^\dagger} \pmatrix{0 & 1 \\ 1 & 0} \pmatrix{a \\ b}
$$Then
find a diagonalizing matrix P for the matrix in the middle of the above (by finding its eigenvectors). The diagonalizing matrix P in this case turns out to be
$$
P ~=~ \frac{1}{\sqrt{2}} \pmatrix{1 & 1 \\ 1 & -1}
$$
Apply ##P## to the column vector of ##a,b## to get ##c,d##. Or rather, apply its inverse to get ##a,b## in terms of ##c,d##. I get
$$
\pmatrix{a \\ b} ~=~ \frac{1}{\sqrt{2}} \, \pmatrix{c+d \\ c-d} ~.
$$
When you substitute for ##a,b,## in the original ##H## it becomes diagonal -- without the pesky constant term. (The latter was happening because you'd mixed annihilation and creation operators, and unwittingly transformed to a new vacuum, with different ground state energy from the old.)
Sorry for not recognizing all this sooner. Hopefuly you can generalize the above to your specific case.