Consider the product:
[tex]
\left\langle p_{1}, \ldots, p_{m} | q_{1}, \ldots q_{n} \right\rangle \equiv\left\langle 0 | a_{p_{1}} \, \ldots a_{p_{m}} \, a^{\dagger}_{q_{n}} \ldots a^{\dagger}_{q_{1}} | 0 \right\rangle[/tex]
Let us start moving [itex]a_{p_{m}}[/itex] to the right. With each creation operator it gives a contraction:
[tex]
a_{p_{m}} \, a^{\dagger}_{q_{k}} = \delta_{p_{m}, q_{k}} + \zeta \, a^{\dagger}_{q_{k}} \, a_{p_{m}}[/tex]
and one factor of [itex]\zeta[/itex] when the flipping occurs. When it reaches the furthest right, it annihilates the vacuum giving zero [itex]a_{p_{m}} \, | 0 \rangle = 0[/itex]. Finally, we are left with the following sum:
[tex]
\sum_{k_{m} = 1}^{n}{\zeta^{k_{m}} \, \delta_{p_{m}, q_{k}} \, \langle 0 | a_{p_{1}} \, \ldots a_{p_{m - 1}} \,\ldots a^{\dagger}_{q_{k_{m} + 1}} \, a^{\dagger}_{q_{k_{m} - 1}} \ldots | 0 \rangle}[/tex]
The brackets in the sum differ from the original one by a [itex]a_{p_{m}}[/itex] and a [itex]a_{q_{k_{m}}}[/itex] and each one has a factor of [itex]\zeta^{k_{m}}[/itex], where [itex]\zeta = +1[/itex] for bosons and [itex]\zeta = -1[/itex] for fermions.
We may repeat this procedure for [itex]a_{p_{m - 1}}[/itex] for each of the terms in the sum. Then, we will get a double sum with one pair of creation-annihilation operators less and some power of [itex]\zeta[/itex]. Continuing this procedure, we will be left with either a bracket of the form:
[tex]
\langle 0 | a_{p_{1}} | 0 \rangle = 0, \; m > n[/tex]
or:
[tex]
\langle 0 | a^{\dagger}_{q_{j_{n - m}}} \, a^{\dagger}_{q_{j_{n - m - 1}}} \, a^{\dagger}_{q_{j_{1}}} | 0 \rangle = 0, \; n > m[/tex]
So, we can only get a non-zero matrix if [itex]m = n[/itex], i.e. we have the same number of annihilation operators as creation operators. Furthermore, due to the Kronecker deltas, we must have the single-particle states [itex]p_{1}, \ldots , p_{n}[/itex] be some permutation of the states [itex]q_{1}, \ldots q_{n}[/itex]. Finally, the power of [itex]\zeta[/itex] is equal to the number of transopsitions necessary to bring the sequence [itex]\{q_{1}, \ldots , q_{n}\}[/itex] to be identical to the sequence [itex]\{p_{1}, \ldots, p_{n}[/itex] (notice that [itex]\zeta^{2} = 1[/itex] for any kind of particle).
You could have used this rule if you wrote the one-body operator as:
[tex]
O = \sum_{m, m'}{(m | O | m') a^{\dagger}_{m} \, a_{m'}} = \zeta \, \sum_{m, m'}{(m' | O | m) a^{\dagger}_{m} \, a_{m'}} - \zeta \, \sum_{m}{(m | O |m)}[/tex]
EDIT:
I noticed that you wrote the one-body operator with two creation operators. If this is so, then the matrix element is zero.