The math for working with the squared error is simpler than the math for working with the absolute error. You can do things like take the derivative of the squared error or express it with matrices.

I really don't know if this is correct at all, but I've always thought that one reason at least is that by squaring the errors you're working with the magnitudes of the errors without regard to the signs of the raw errors.

I'm very weak in statistics, however, so I could well be wrong about that ...

You can show that for a joint Gaussian distribution that the minimization of mean squared error is equivalent to maximum likelyhood. I suppose then you could ask, "Why maximum likelyhood".

So that measure is rather useless. Some condsider [itex]\sum |x_i - \overline{x}|/n[/itex] instead, and this leads to some useful information.

However the second order measure (typically called the root mean square or some such) is analogous to the "moment of inertia" of the distribution about the mean and has been mentioned is useful in analysing minimization of error.

It was decided (I am not sure when, early 20th century?) that this measure would be the "standard" one (hence "standard deviation"), but the other higher order ones are also valid measures.

This may not fully answer your question, but statisticians have put some thought into which measure is best.

The squared error was historically used because that is the natural method to use when you assume the errors behind your data are normally distributed. The fact that using the squares made the following mathematics easier to work with was a bonus.

The reason of using MSE is that you will have a quadratic curve, similar to [tex]y=x^2[/tex], which has a unique minimal value, which is its derivative equal to zero.

A normal distribution has only two parameters, its mean and its standard deviation. The average of your data is an unbiased estimator of the mean, and the mean square error is an unbiased estimator of the standard deviation.

this is why these two statistics are used. they are unbiased estimators for the defining parameters of the unknown normal distribution.

In reality, a lot of data is close to normal. For instance stock price returns are nearly normal. In these cases one can estimate the distributions directly with the sample mean and standard deviation. If data is not normal then one can take averages before estimating parameters. Before computers, this is what statisticians did because they needed to know something about the mathematical form of their sample distribution. if you average your data to create a new sampling distribution then this new distribution of averages is approximately normal. this reduces the problem of data analysis to estimating which normal distribution your sample data comes form.

Interestingly enough the use of the divisor (n - 1) is a rather recent development. I've read texts as recent as the 40's where the standard deviation has a devisor of n. As such the expected value of s^{2} is [itex]n^2\cdot \sigma^2 / (n-1)^2[/itex] if I'm not mistaken. Since is was an underestimator of the true variance, statisticians and probabilists switched to the current divisor of (n - 1).

Whether the divisor is n or (n - 1), these expressions are all measures of dispersion; whether they're good measures is another matter.