Consider the figure below.
View attachment 2560
Choose $Q$ as the origin and the positions of $P$ and $R$ are denoted by vectors $\vec{p}$ and $\vec{r}$.
Also, let $m/n=\lambda$.
Clearly,
$$\vec{QC}=\frac{1}{\lambda+1}\vec{p}$$
$$\vec{QA}=\frac{\vec{r}}{2}$$
$$\vec{QB}=\frac{\vec{r}+3\vec{p}}{4}$$
Next, I find the areas in terms of $\vec{r}$ and $\vec{p}$,
$$x=\frac{1}{2}\left| \vec{QA}\times \vec{QC}\right|=\frac{1}{4(\lambda+1)}\left|\vec{r}\times \vec{p}\right|$$
$$y=\frac{1}{2}\left|\vec{AR}\times \vec{AB}\right|=\frac{1}{2}\left|\vec{AR}\times \left(\vec{QB}-\vec{QA}\right)\right|=\frac{1}{2}\left|\frac{\vec{r}}{2}\times \left(\frac{\vec{r}+3\vec{p}}{4}-\frac{\vec{r}}{2}\right)\right|=\frac{3}{16}\left|\vec{r}\times \vec{p}\right|$$
$$z=\frac{1}{2}\left|\vec{CP}\times \vec{CB}\right|=\frac{1}{2}\left| \frac{\lambda}{\lambda+1}\vec{p}\times \left(\vec{QB}-\vec{QC}\right)\right|=\frac{\lambda}{8(\lambda+1)}\left|\vec{r}\times \vec{p}\right|$$
As per the question,
$$x^2=yz \Rightarrow \frac{1}{16(\lambda+1)^2}=\frac{3}{16}\cdot \frac{\lambda}{8(\lambda+1)}\Rightarrow 3\lambda^2+3\lambda-8=0$$
$$\Rightarrow \boxed{\lambda=\dfrac{1}{6}\left(\sqrt{105}-3\right)}$$
(neglecting the negative root)