
Let R and H represent the radius and height of the circular cylinder that is inscribed in a cone with height
[FONT=MathJax_Math]h and base radius
[FONT=MathJax_Math]r.
We know that if two triangles are similar, their corresponding side lengths are proportional. We see that the two triangles ACD and BCE are similar, hence,
$$\frac{r}{h}=\frac{R}{h-H}$$
Rearrange the equation to make H the subject (because in this problem we want to find the maximize volume of the cylinder where its formula is $$V_{\text {cylinder}}=\pi R^2h$$, it would be easier for us to make the equation of V in terms of only one variable, i.e. to get the formula of H in terms of R.):
$$r(h-H)=rR$$
$$H=\frac{h(r-R)}{r}$$
Hence,
$$V_{\text {cylinder}}=\pi R^2h=\pi R^2\left(\frac{h(r-R)}{r}\right)=\frac{\pi R^2h(r-R)}{r}$$
To find $$V_{\text{max}}$$, we find its first derivative w.r.t. R and then use the second derivative test to determine if it's a maximum value:
$$\frac{dV_{\text {cylinder}}}{dR}=\frac{d\left(\frac{\pi R^2h(r-R)}{r}\right)}{dR}=\frac{\pi Rh(2r-3R)}{r}$$; $$\frac{d^2V_{\text {cylinder}}}{dR^2}=\frac{2\pi h-6 \pi R}{r}$$
$$\frac{dV_{\text {cylinder}}}{dR}=0$$ iff $$2r-3R=0\;\rightarrow R=\frac{2r}{3}$$ $$\rightarrow \frac{d^2V_{\text {cylinder}}}{dR^2}=\frac{2\pi h-6 \pi (\frac{2r}{3})}{r}=-2\pi h (<0)$$
Hence we can conclude now that $$V_{\text{max}}=\pi R^2 h=\pi(\frac{2r}{3})^2(\frac{h(r-R)}{r})=\pi(\frac{4r^2}{9})(\frac{h(r-(\frac{2r}{3}))}{r})=\frac{4 \pi r^2h}{27}$$.