Hi Gavroy. Maybe a simple example will help you so consider the case of a block sliding down a fixed frictionless incline of angle ##\theta##. Obviously, the easiest approaches to this would be to either just write down Newton's 2nd law in a convenient coordinate system or to use the generalized coordinate ##q## representing the distance traveled along the incline by the block and just writing down Lagrange's equations. For the second method, the choice of generalized coordinate ##q## will implicitly take into account the constraint which is that the block must stay on the incline during the trajectory before reaching the ground. However, let's do this using the method of Lagrange multipliers.
Using ##x## for the horizontal coordinate and ##y## for the vertical coordinate, we can write out Lagrangian as ##L = \frac{1}{2}m\dot{x}^{2} + \frac{1}{2}m\dot{y}^{2} + mgy## but note that ##x,y## are not independent. They are related by the holonomic constraint ##g(x,y) = y - x\tan\theta = 0##. Letting ##L' = L - \lambda g## we simply write down Lagrange's equations as ##\frac{\partial L'}{\partial x^{\alpha}} - \frac{\mathrm{d} }{\mathrm{d} t}\frac{\partial L'}{\partial \dot{x}^{\alpha}} = 0##. Doing this and combining with the constraint (this is important!) gives us ##m\ddot{x} = \lambda \tan\theta## and ##\lambda\cot\theta = mg\cot\theta - m\ddot{x}## so adding these two gives us ##\lambda\sec^{2}\theta = mg\Rightarrow \lambda = (mg\cos\theta)\cos\theta## . As we know, ##N = mg\cos\theta## is the magnitude of the normal force on the block from the incline so ##\lambda## is the magnitude of the vertical component of the normal force and ##\lambda\tan\theta = (mg\cos\theta)\sin\theta = m\ddot{x}## is the magnitude of the horizontal component of the normal force, as expected.
As you can see this is more work than simply choosing the generalized coordinate ##q## as the distance traveled along the incline, which implicitly takes the constraint into account and deals with a much nicer coordinate system for this problem.
As far as the mathematical nature of lagrange multipliers goes, for holonomic constraints it is very easy to understand in terms of constraint submanifolds of the configuration space and the implicit function theorem. For non-holonomic systems, there are subtleties involved that go into Frobenius' theorem and the tangent bundle of the configuration space but if you want I can direct you to books / resources for these matters.