Why don't you just do the pieces first? Derivative of cos(x^2)? You aren't going to get anywhere without that. Or did you actually mean to write cos(x)^2 in the denominator instead of cos(x^2)?
(cos(x^2))^2 is different from cos((x^2)^2). One is the square of cos(x^2). The other is cos(x^4). They are different. If you don't believe me pick a number and put it into your calculator. Which do you want?
As Dick asked at least a couple of times, use parentheses. Your last answer would be correctly interpreted as
[tex]
5cos(5x)cos(x^2) + \frac{2x sin(5x)Sin(x^2)}{cos^2(x^2)} [/tex]
but I don't think that's what you had in mind.