Hi Himanshu
Yes 2g/23 is the correct acceleration. I think the main reason you are getting a wrong answer is because you are using Newton's laws which become very complicated when you have systems which have moving parts (moving relative to each other) because then, in addition to reaction forces (i.e. contact forces such as normal reaction) you also need to consider the so called "pseudo/inertial forces" since relative accelerations are computed in non-inertial frames e.g. forces on the smaller blocks w.r.t 5M block. But instead if you write down the lagrangian, the correct expression pops out in a two lines of algebra/calculus. This is how I solved this problem:
Let X(t) is the position of the left end of the 5M block from some fixed inertial frame origin of coordinates, say O. Let x(t) be the position of the block of mass M (sitting atop of 5M) wrt to the left pully/left edge of 5M block. And let L be distance of the 2M mass block from the left edge of the 5M block (L remains unchanged as 2m block always remains in contact with 5M). This means with respect to the inertial frame origin, O
a) the x-axis coordinate of top mass (M) block is XM = x(t)+X(t) while
b) the horizontal coordinate of 2M mass block is "X2M =X(t)+L", and
c) the vertical/y coordinate of 2M mass block is "Y2M = x(t)" (because top block and 2M block are connected by an inextensible string, so horizontal displacement of top block on 5M mass by x means 2M block gets pulled up by equal amount x).Now we can write down the lagrangian, which is difference of kinetic and potential energies,
L= 1/2 (5M) (V2)+ 1/2 (M) (dxM/dt)2 + 1/2 (2M) (dX2M/dt+dY2M/dt 2)
= 1/2 (5M) (V2) + 1/2 (M) (V+v)^2+ 1/2 (2M) (V2+v2) - 2M g x
where V= dX/dt, v= dx/dt and we have used, Y2M = x(t). So the lagrangian is a function of x,v,V.
The lagrange's equations are then,
d(∂L/∂v)/dt = ∂L/∂x,
d(∂L/∂V)/dt = ∂L/∂X.
The first equation simply gives, acceleration of the top box is, a = -2g/3 -A/3 while the second equation gives, 8 A + a =0. Substituting expression for a from the first into the second, I get, A = 2g/23!
Lagrangian method is very straightforward compared to Newton's laws where you have to first make free body diagrams WITH all contact reaction forces and pseudoforces. Students inevitably make mistakes when doing that second step, like you did. All you have to do is just write down the lagrangian and then take derivatives and wallah! the result stares at you.
Of course I am assuming that you know what derivatives/ differential calculus is and that the velocity is first derivative of position/dispalcement while acceleration is the second derivative of displacement/position. V =dX/dt, A = dV/dt