For a more detailed explanation please see http://math.stackexchange.com/questions/52767/integral-of-a-function-with-a-variable-along-a-axis-in-a- rotated-positionf
The first integration isn't correct. You can see that immediately by differentiating the primitive. Use a substitution if you don't see it right away, [itex]u=1-xy,du=-ydx[/itex]. The hard part comes after this however. Hint: Use the power series of the logarithm.
Ok let's say z=xy, then [tex] \frac{1}{1-xy}= \frac{1}{1-z}=\sum_{n=0}^\infty z^n=\sum_{n=0}^\infty (xy)^n=\sum_{n=0}^\infty x^n y^n[/tex]. Then you take the integral over the sum.
[tex]\int_0^1 \int_0^1 \sum_{n=0}^\infty x^n y^n dxdy=\sum_{n=0}^\infty \left(\int_0^1 \int_0^1 x^n y^n dxdy\right)[/tex]. Can you see how to continue from here?
Edit: I think you're expected to know that the series converges to pi^2/6 by heart. However http://en.wikipedia.org/wiki/Basel_problem shows you how to get to that value in case you're interested.