Edit 2: Indeed, the first labda is NOT supposed to be squared, that is a typo! In that case I know of a method on how to solve it, it basically just follows the website I posted below.
Dear SammyS,
Thank you for your response. Indeed, I also thought that it would be most natural to think of λ to be the changing factor, while keeping A and B constant. Interpreting what it meant has proven to be a bit problematic for me, so your input is very useful.
The 'book' I use in my course - complex analysis - is Schaum's Outline of Complex Variables, which is basically just a summary type of book. It does have 1 worked out exercise dealing with orthogonal families, and that is the approach I followed. It uses implicit differentiation to find dy/dx, for both of the equations, and then you can see that the product is -1 without too much hassle.
This problem is clearly not in the same ballpark, as it is very nontrivial (if it can even be seen?) that the product is -1 in this case. I'm going to work out a few examples with values of a and b, and see if that gets me anywhere..
Alright, so what this shows me is that the approach does not work, as far as I can tell. I don't get that the product is -1, for specific values of a, b and the lamda's. Guess I'll have to think of something else..
Making a few contour plots in mathematica, for different values of lamda, the first family is indeed a family of hyperbola's and the second one a family of ellipses.
Now, I don't understand the approach, but
http://planetmath.org/hyperbolasorthogonaltoellipses treats how to show that a certain family of ellipses, not too much unlike the one I have here, is perpendicular to a family of hyperbola's.
Looking at it, it does show some promise, but I can't reproduce the line after (3). Applying the same method to my system seems to be working, up to that point, where I extract a quantity that is not equal to zero.
Edit: Alright, so I used the method posted on that website. It ALMOST works! The only issue is that, in the second denominator, lamda is not squared, while the first one is. So if the first one is also not supposed to be squared, then it works.
Edit 2:
Indeed, the first labda is NOT supposed to be squared, that is a typo!