I would orient my coordinate axes such that the $x$-axis coincides with the plane, and the origin coincides with the initial position of the particle and so we have:
$$a_x=-g\sin(a)$$
$$a_y=-g\cos(a)$$
And so:
$$v_x=-g\sin(a)t+u\cos(b)$$
$$v_y=-g\cos(a)t+u\sin(b)$$
And:
$$x=-\frac{g\sin(a)}{2}t^2+u\cos(b)t$$
$$y=-\frac{g\cos(a)}{2}t^2+u\sin(b)t$$
When the particle has reached its range up the plane, we have $y=0$ and $t>0$ giving us:
$$t=\frac{2u\sin(b)}{g\cos(a)}$$
And so the range $R$ of the particle is:
$$R=\left(\frac{2u^2\sin(b)}{g\cos(a)}\right)\left(\cos(b)-\tan(a)\sin(b)\right)$$
When the particle has reached its maximum height, we find from $v_y=0$:
$$t=\frac{u\sin(b)}{g\cos(a)}$$
We are told that we must have:
$$x\left(\frac{u\sin(b)}{g\cos(a)}\right)=\frac{3}{5}R$$
$$-\frac{g\sin(a)}{2}\left(\frac{u\sin(b)}{g\cos(a)}\right)^2+u\cos(b)\left(\frac{u\sin(b)}{g\cos(a)}\right)=\frac{3}{5}\left(\frac{2u^2\sin(b)}{g\cos(a)}\right)\left(\cos(b)-\tan(a)\sin(b)\right)$$
This simplifies to:
$$2\cos(b)-\tan(a)\sin(b)=\frac{12}{5}\left(\cos(b)-\tan(a)\sin(b)\right)$$
Divide through by $\cos(b)$:
$$2-\tan(a)\tan(b)=\frac{12}{5}\left(1-\tan(a)\tan(b)\right)$$
And this reduces to:
$$\tan(a)\tan(b)=\frac{2}{7}$$