For (a), write out the power series for cos(x), then replace x with x^2 to get the power series for cos(x^2). Now Let [itex]y= \sum a_nx^n= a_0+ a_1x+ a_2x^2+ a_3x^3+ /cdot/cdot/cdot[/itex] and differentiate. [itex]y'= \sum na_nx^{n-1}= a_1+ 2a_2x+ 3a_3x^3+ \cdot\cdot\cdot[/itex]. Set the two power series equal so that corresponding coefficients (i.e. same power of x) are equal. That gives you an infinite number of equations to solve for the infinite unknowns, an! Hopefully, after doing a few you will recognize a pattern. Notice that you have lost a0. That's what you need y(0)= 0 for.
For (b), much the same. If [itex]y= \sum a_nx^n[/itex] then [itex]y''= \sum n(n-1)a_n x^{n-2}[/itex] while [itex]xy= x\sum a_n x^n= \sum a_n x^{n+1}[/itex]. You will want to "change indices" on the two sums in order to be able to match up the same powers of x.