"So, we take the domain on ##x## to be:"
Wait, what? The domain of ##x## is already defined, it's (0,1]. Why would you change it?
Your integral will is calculating the cumulative distribution of ##x##, the probability that ##x## is less than or equal to ##\sqrt y##. That isn't what was asked for. However ##P(X \leq \sqrt y)## = ##P(X^2 \leq y)## = ##P(Y \leq y)##. So you've calculated the cumulative distribution of ##y##. IOnce you have the CDF for ##y## you can get the ##PDF##.
Now there's a lot wrong with your concluding statement ##f_x(y) = y^2; y \leq x^2##.
- It's not ##f_x## as this is supposed to be the density of ##y##.
- As explained already, you didn't calculate a density, you calculated a CDF.
- Since ##y = x^2## and the domain of ##x## is (0, 1], the domain of ##y## is (0, 1]. The density of ##y## is not related to a particular value of ##x##.
Working with the CDF is a useful thing to do, you're just misinterpreting what every single thing in this calculation is for.
Let's say we want the CDF of ##y##, the probability ##P(Y \leq y)## for any value of ##y##, which as we said could be anything in (0, 1] based on the domain of ##x##. ##Y \leq y \iff X^2 \leq y \iff X \leq \sqrt y##. With a different definition of ##X## that last event might be (##0 \leq X \leq \sqrt y## or ##-\sqrt y \leq X \leq 0##) but X can't be negative here.
OK, so we've established that ##P(Y \leq y) = P(X \leq \sqrt y) = \int_0^{\sqrt y} 4x^3 dx##, and that should address your questions as to whether you're integrating with respect to ##x## or ##y##, and also what you've calculated here and how it relates to what you were asked for.
To summarize: You decide to calculate the cumulative distribution of Y. You want to know how many Y's fall into a given range ##(0,y]##, for any choice of ##y##. You know what X values will give you Y's in the desired range, so you know how to use the density of X to calculate that probability. And that gives you an expression for the CDF of Y.