Here is what I think. I know that the center of mass of the rope starts to accelerate. The acceleration has a downward component, call it a_y. Then I know that there is an imbalance between the normal force and the force due to gravity. Let m be the mass of the rope that is hanging over the edge of the table (this changes with t, time or with x, the distance the rope (on top) has moved across the table). So the net force on the center of mass in the y direction is M*a_y or m*x'' where x'' is the acceleration of the rope (on top) across the table. To get the normal force N, N = M*g - m*x'' . So, the normal force is proportional to the product of the distance x, and the acceleration x'' .
Keep in mind the x and x'' correspond to only the acceleration of the rope on top of the table moving horizontally; these are the same as y' and y'' of the rope hanging over the table if you just consider the rope to only move in one path, that is, it doesn't sling off the table realistically.
The reason I need the correct normal force is to know how the friction changes so I can find x as a function of t; to solve the correct differential equation (If I can). When I did the problem on the test, I said the normal force was constant, Mg, but after the test, I ask my professor what it should really be and she told me it should just be the mass of the rope on top times g. But, I really don't know what to think so... please help.