Your answer 0.4 m is correct.
Richie Smash said:
The first being, Why does mass by length work? I know moments to be Force by Distance.
That's because if you put the forces (in this case weights) in, you get
##m_{mass}~g~x-m_{plank}~g~(L/2-x)=0##
Since ##g## appears in
both terms on the left and the right side is zero, we can divide by it to get
##m_{mass}~x-m_{plank}~(L/2-x)=0##
Richie Smash said:
The second was your point, the centre of gravity of the mass relative to the right end of the top plank would be x/2 = 0.2m
That's not how it works. The center of gravity from the right end is given by
$$X_{cg}=\frac{W_{mass}~x_{mass}+W_{plank}~x_{plank}}{W_{mass}+W_{plank}}$$
where ##x_{mass}## and ##x_{plank}## are distances measured from the right end. Here ##x_{mass}=0## and ##x_{plank}=L/2##
so
$$X_{cg}=\frac{0+W_{plank}~L/2}{W_{mass}+W_{plank}}=\frac{W_{plank}~L}{2(W_{mass}+W_{plank})}=x=0.4 m$$
So the cg is at the same position as the edge of the lower plank. That's no coincidence and the point to be made is this.
For the net torque to be zero, the cg must be vertically above or below the pivot point. If you stack the planks with their edges flush, there will be no tipping when you put some mass on the right end because the cg of the assembly {top plank + mass} is to the left of the edge so a point can be found on the bottom plank to provide zero torque. Now if you pull the top plank with the mass to the right, the top assembly will tip if its cg goes past the pivot point at the edge because you run out of points on the bottom plank that can provide zero torque. Therefore to find ##x## all you have to do is find where the cg of the top assembly is with respect to the right edge and that's your answer. That's where I was leading you at first and that's the easier way to do this. Now you know.
A simple convincing experiment about the cg and tipping thresholds is this. Go to the middle of the room and stand on your toes. Keep the position long enough to convince yourself that you can do it. Then stand facing a bare wall with your toes right up against the baseboard. Try standing on your toes. You will fail. Why? Answer: When you stand on your feet, your cg is somewhere above your heels, not above your toes and the torque due to gravity is zero. When you stand on your toes, you have to lean a bit forward to bring your cg above the pivot point at your toes which again makes the torque due to gravity zero. Now when you try to stand on your toes facing the wall, you are prevented from leaning forward as the torque due to gravity keeps you standing on your heels.