I'm going to assume the 3 forces all act within the same plane. And so, I would choose one of the forces to act along the $x$-axis in the positive direction (I'll choose the one with the largest magnitude so our unknowns involve the vectors in question), and I would make the following definitions:
$$F_1=\langle x_1,y_1 \rangle$$ where $$\left|F_1\right|=5$$
$$F_2=\langle x_2,y_2 \rangle$$ where $$\left|F_2\right|=6$$
$$F_3=\langle 7,0 \rangle$$
Next, let the resultant force be:
$$F_R=F_1+F_2+F_3=\langle 0,0 \rangle$$
And so we obtain:
$$x_1+x_2+7=0$$
$$y_1+y_2=0$$
$$x_1^2+y_1^2=5^2$$
$$x_2^2+y_2^2=6^2$$
We have 4 equations in 4 unknowns...can you proceed?