Fun problem that leads to a surprisingly simple result. There are many ways to do this. Avoid the cumbersome trigonometric formulas and divide triangle APB with a vertical line parallel to the y-axis through P. Call the point where this vertical line intersects the line y=mx+b, C. Now the area of triangle APB is the sum of the areas of triangles ACP and BCP. The difference being that now you may use double integrals to calculate the area of each separately. The cumbersome part comes in finding the equation for the line AP and the line BP. But once this is done the problem is straight foward. Find the double integral expression for the area of ACP and for BCP. You will get an expression in terms of A, B, and P. Before all this you should have calculate A and B which is trivial, set mx+b=x^2. P may be regarded as P=(p1,p2)=(p1,p1^2), since P lies on the parabola y=x^2. Once you have the expression for the area in terms of A, B, C, differentiate with respect to p1 and set the result equal to zero and solve for p1. The result is surprisingly concise.