my take:
selecting ##A=0## and ##B=0## obviously satisfies the equation which knocks out choice (iii)
now consider a few basic things:
##AB = -(A+B)##
##(AB)^2 = (-1)^2(A+B)^2 = A^2 + AB +BA +B^2##
simplify for case (i) and case (ii)
if you go back to the original problem and square it you get
##\big(AB +A+B\big)^2 = \mathbf 0##
so
##(AB)^2 +A^2+B^2 +ABA + AB^2 +A^2B+ BAB + AB +BA= \mathbf 0##
##\big\{(AB)^2 +(A^2+B^2)\big\} +(ABA + A^2B)+ (AB^2 + BAB) + (AB +BA)= \mathbf 0##
simplify that.
in case (i) it should imply nilpotence of ##(AB)## and in case (ii) it should imply idempotence of ##(-AB)## -- which in combination with commuting matrices means simulataneously
diagonalizable triangularizable (and nice trace properties with rank).
the intersection of (i) and (ii) then is when ##AB## is the zero matrix because otherwise nilpotent matrices are not diagonalizable.
I still don't have a particularly clean finish, but you might try solving this by assuming ##A## and ##B## are diagonal matrices and seeing if any complex numbers fit this bill. Scalar field was not stated nor were any relevant equations given so I'm guessing at what is useable, though ideas related to commuting and working in ##\mathbb C## seems reasonable to get going.
- - - -
edit:
if you work through case (ii) of commuting, carefully, what you get is a nice blocked structure, after similarity transform
where ##n=3 ##
##m \in\{0,1,..n\}##
##(AB) = \begin{bmatrix}
-I_m & \mathbf 0 & \\
\mathbf 0& \mathbf {00}^T_{n-m}
\end{bmatrix} = A \cdot B
= \begin{bmatrix}
J_m & \mathbf 0 & \\
\mathbf 0& N^{(A)}_{n-m}
\end{bmatrix}
\begin{bmatrix}
-J_m^{-1} & \mathbf 0 & \\
\mathbf 0& N^{(B)}_{n-m}
\end{bmatrix}## |
with ##N^{(B)}_{n-m}, N^{(A)}_{n-m}## being nilpotent and commuting and obeying
##N^{(B)}_{n-m}N^{(A)}_{n-m} = \mathbf {00}^T_{n-m}##
and the matrices ##J_m## having non-zero eigenvalues.
This is reducible in blocked structure and if we ignore the nilpotent piece, we see that that ##B## is in effect the negative inverse of ##A## (sometimes called a pseudo inverse)
.. I of course agree with Fresh that this problem seems to be designed to look up the definition of a commutator and applying it, though OP seems quite against doing so -- I dropped in some other ideas accordingly.
also there's a link with Fibonnaci numbers in here...