Ok, um, a lot of questions.
If you can't even draw the shape, determining boundaries for the region become even harder. To my knowledge, there is no transformation rule that will make them fall into place. You do, however, have a set of equations that determine some boundaries for you. Those are the equations for the boundaries on the original integral. In this case:
y = \sqrt{3} x
and
y = \sqrt{1 - x^2}
If you did your polar substitutions, you would get:
r Sin(\theta) = \sqrt{3} Cos(\theta)
Sin(\theta) = \sqrt{3} Cos(\theta)
\theta = \frac{\pi}{3}
and (skipping some algebra)
r = 1
It still takes some vision to see how everything falls into place, but these can usually give you some boundary lines in your new coordinate system, if you couldn't draw the original.
This isn't like a single variable substitution, because, although x is a function of r and \theta, r and \theta are both functions of x and y. My best advice is to just practice at drawing regions. Textbooks tend to use the same variety of boundaries, especially when you are expected to change coordinate systems.
the limits of y is actually a circle of radius 0 to 1 right?
The limits of r are from 0 to 1. In this case, one of the bounds of y was a semicircle of radius 1. It is an actual curve that serves as an upper bound. The lower bound for y was just a line with constant slope of root(3).
but the limits of x is from 0 to 1/2. so can i assume that the angle is always from smaller to bigger? so it becomes pi/3 to pi/2? are there cases where i have to take a bigger angle to smaller angle?
There are cases where textbooks will give you reversed boundaries, but it is rare. In general, use the orientation given in the problem to determine if you are going from smaller to bigger angle. Don't make the assumption that it will always be positive orientation. In this case, we used the lower bound of y to determine that the integral needed to start at \frac{\pi}{3}. Always use the information given. I would say, if you are in doubt, stick with positive orientation, but don't assume it is impossible to see otherwise.
